When deploying Exchange hybrid, one of the aspects that can prove to be the most challenging is Exchange Online mailboxes trying to view calendar information for on-premises mailboxes.
This is because there are multiple potential issues which can be caused due to management of Exchange on-premises, firewall rules and reverse proxy configuration issues. What was slightly interesting with this customer was that the cross-premises calendar functionality had been working when they initially setup the hybrid. They documented the testing performed and saved the results in the project's folder. However, that was more that 18 months ago.
Like many customers, they started down the path to Exchange hybrid but for one reason or another they ceased work on the project. The upcoming end of support for Office 2010 and Exchange 2010 resurrected the project.
The steps to renew the federation trust were previously followed as the initial certificate to the Federation Gateway had expired. The steps are listed here: Renew the federation certificate
To simulate the issue, an older Exchange 2010 hybrid lab was started up.
Starting Configuration
At the customer's location, the issue was discovered when trying to view the free/busy information of an on-premises mailbox. Note that no information was returned, and all that we see are the dreaded backslashes.
One of the first things to check is the request from Exchange Online actually getting to the on-premises server? Maybe a firewall rule change now blocked it. The initial location to check is the IIS logs on the server(s) which are published to the Internet.
In the below, we can see that the expected cross-forest requests are indeed hitting the Exchange server. However a HTTP status code of 500 indicates a server error.
To increase readability and search engine access, the log entries from the image are included below for reference.
2020-07-17 17:34:48 10.0.0.12 POST /autodiscover/autodiscover.svc/WSSecurity - 443 - 52.96.32.133 ASAutoDiscover/CrossForest/EmailDomain//15.20.3195.017/MailTips - 500 0 0 3140
2020-07-17 17:34:48 10.0.0.12 POST /autodiscover/autodiscover.svc/WSSecurity - 443 - 52.96.32.133 ASAutoDiscover/CrossForest/EmailDomain//15.20.3195.017/Freebusy - 500 0 0 3218
2020-07-17 17:34:48 10.0.0.12 POST /autodiscover/autodiscover.svc/WSSecurity - 443 - 52.96.32.133 ASAutoDiscover/CrossForest/EmailDomain//15.20.3195.017/Freebusy - 500 0 0 0
So now that we know the traffic is getting to the Exchange server, what is Exchange doing with it? Any errors in the application event log? Well, there was the below:
The text from the above event log entry is shown below:
Log Name: Application
Source: MSExchange Common
Date: 7/17/2020 5:34:47 PM
Event ID: 403
Task Category: Configuration
Level: Error
Keywords: Classic
User: N/A
Computer: Exch-2010.Tailspintoys.ca
Description:
The certificate named '791BC6AD9893AA570DF03452B4F8069C8A743C29' in the Federation Trust 'Microsoft Federation Gateway' is expired. Please review the Federation Trust properties and the certificates installed in the certificate store of the server.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
<Provider Name="MSExchange Common" />
<EventID Qualifiers="49152">403</EventID>
<Level>2</Level>
<Task>2</Task>
<Keywords>0x80000000000000</Keywords>
<TimeCreated SystemTime="2020-07-17T17:34:47.000000000Z" />
<EventRecordID>190831</EventRecordID>
<Channel>Application</Channel>
<Computer>Exch-2010.Tailspintoys.ca</Computer>
<Security />
</System>
<EventData>
<Data>791BC6AD9893AA570DF03452B4F8069C8A743C29</Data>
<Data>Microsoft Federation Gateway</Data>
</EventData>
</Event>
The above message was "interesting". The word interesting is probably not what we want to hear whilst troubleshooting.....
Why was that interesting? Well the certificate which is referenced 791BC6AD9893AA570DF03452B4F8069C8A743C29 does not actually exist on the server. Why is the federation trust trying to use the previous certificate?
Certificate '791BC6AD9893AA570DF03452B4F8069C8A743C29' is not present:
EC0793D4A0F2C4F7AF0104FB46DDB81439640FE1
C8BCAC52CF86C3515C569E63E8928C32D4B246EB
86C94FFB5529C84CC2F8E3B317C87FEE4C8BD7CE
7E24D43625AF86DB9996898156C53471F3D14C68
50CDE53C6572A2496AB35AAA74CDD6CBBC212713
Something is not quite right with the certificate used by the Federation Trust. There are some other issues associated with an error of "Federation certificate with the thumbprint "<thumbprint of the current certificate>" cannot be found" and they have a separate resolution - for example:
The next step is to have a look at the Federation Trust - does it still exist?
let's confirm that the trust exists with the MFG - Microsoft Federation Gateway
Get-FederationTrust -Identity "Microsoft Federation Gateway"
The Federation Trust is listed in there, so let's look at the details of the trust.
Piping the output to a Formal-List shows the Thumbprints of the current and previous certificate. We do see that the certificate thumbprint listed above in the TokenissuerPrevCertificate 791BC6AD9893AA570DF03452B4F8069C8A743C29 - this is indicated with the yellow arrow below.
What results do we get when testing the Federation Trust?
The error which is highlighted is:
The federation trust doesn't contain the same certificates published by the security token service in its federation metadata.
Ah ha! This is the classic issue with outdated Federation Metadata.
Updating Federation Metadata
This is an issue which has come up multiple times over the years and is caused when the certificate infrastructure is updated in the Federation Trust infrastructure. Microsoft is responsible for maintaining and managing the the infrastructure which is used as the foundation of trust for the Organization Relationships. The certificates typically have a 5 year validity, and so when we roll to a new certificate the on-premises Exchange servers need to know about the change.
There have been multiple messages inside the Office 365 Message center.
2014 - MC9420
2019 - MC184636
KB 2928514 also discusses this issue and resolution:
Free/busy lookups stop working in a cross-premises environment or in an Exchange hybrid deployment
Get-FederationTrust | Set-FederationTrust –RefreshMetadata
In the lab, we will update the federation metadata and then re-test.
The federation trust contains the same certificates published by the security token service in its federation metadata.
Now that the metadata has been refreshed, the Test-FederationTrust cmdlet now succeeds.
On-premises free/busy is now visible from the tenant once again.
And the entries in the IIS logs now indicate success with a 200 HTTP status code.
2020-07-17 19:38:01 10.0.0.12 POST /autodiscover/autodiscover.svc/WSSecurity - 443 - 52.96.32.133 ASAutoDiscover/CrossForest/EmailDomain//15.20.3195.017/Freebusy - 200 0 0 5267
2020-07-17 19:38:01 10.0.0.12 POST /autodiscover/autodiscover.svc/WSSecurity - 443 - 52.96.32.133 ASAutoDiscover/CrossForest/EmailDomain//15.20.3195.017/Freebusy - 200 0 0 5517
Also of note, now that the metadata is updated the Federation Trust is able to use the first certificate and does not fall back to the second. No more Event ID 403 errors.
Keep Your Federation Trust Up-To-Date
As noted in the EHLO post back in 2014, certificate changes are expected. Microsoft periodically refreshes certificates in Office 365 as part of our effort to maintain a highly available and secure environment. On September 23, 2014, we are making a certificate change on our Microsoft Federation Gateway that could affect some customers as detailed in knowledge base article 2928514.
This certificate change can affect any customer that is using the Microsoft Federation Gateway. If you are in a hybrid configuration or if you are sharing free/busy information between two different on-premises organizations using the Microsoft Federation Gateway as a trust broker, you need to take action.
If you’re using Exchange Server 2013 SP1 + on Windows Server 2012 or later no action is required. The federation metadata will be automatically updated.
If you are running Windows Server 2008 with Exchange 2013, the automatic update feature will not work (it will only work with Windows Server 2012 and newer). Therefore, you should instead follow the instructions in KB 2928514 to update your metadata.
Since Exchange 2010 now exits out of support on the 13th of October 2020, and most Exchange 2013 installs are on a newer version of Windows, hopefully this will become a non-issue.
Cheers,
Rhoderick
