Previously I managed to break one of my labs when replicating a customer situation and then had to fix it as noted in this post from 2017.
This time around though I really raised my game, and instead of one certificate being expired, all of them were. Yup every cert was toast. Trying to install the Exchange CU to update to the latest build did not go well at all. As you see below, all of the certificates that were bound to Exchange had expired. This was the:
- Third party certificate for IIS
- Default SMTP self-signed certificate
- Exchange Auth cert for OAUTH
- Federation would also be here, but since this was never setup that cert did not exist.
All the certificates were all dead to me. How did I manage this? By not powering this lab on for 4 and a half years….
When I went to update it to work on a customer issue, it went kaboom.
Mailbox role: Transport service FAILED
The following error was generated when "$error.Clear();
Install-ExchangeCertificate -services IIS -DomainController $RoleDomainController
if ($RoleIsDatacenter -ne $true -And $RoleIsPartnerHosted -ne $true)
{
Install-AuthCertificate -DomainController $RoleDomainController
}
" was run: "System.Security.Cryptography.CryptographicException: The certificate is expired.
This was the 3rd party certificate which was bound to IIS. At the time of writing it is late 2020.
Simply Renew Certificate? No….
So you may be tempted to say that we can just request and update the certificates. Well not so fast. Both EAC and Exchange Management Shell were not operative due to the failed install.
The Exchange Management Shell was a spectacular wall of colour.
No other Exchange 2013 servers were available in the lab, so we had to fix this locally without relying on additional servers.
Fixing Locally
Since the Exchange Management Shell was inoperative, we needed to load up the snap in directly.
Note that this should not be done without engaging Microsoft support as this is not supported for general use. This is also just a lab.
Add-PSSnapin Microsoft.Exchange.Management.PowerShell.SnapIn
First step was to clean up the expired certificates. I removed the expired:
- Exchange Server auth certificate
- 3rd party IIS cert
- Exchange self signed cert
Requested new 3rd party cert for IIS. Steps are in this article to to this outside of Exchange.
The request process was completed and the newly issues certificate was installed. You can see new certificate 34B18F6F914747FE8D2FC930CEDD03D40D190DA3 below.
All the other Exchange certificates are gone, only OS specific ones remain. Note that Exchange has no bindings to any of these certs.
Fix default SMTP cert
Let's start with this one as we want to overwrite the default SMTP cert as it is expired.
It is also the simplest operation.
New-ExchangeCertificate -Services SMTP
We check to see that it is bound to SMTP server, and it is. IMAP and POP also decided to hitch a lift but I do not run those services.
Fix IIS cert
If you recall we requested and installed a new certificate from a 3rd party CA with a thumbprint of 34B18F6F914747FE8D2FC930CEDD03D40D190DA3.
Let's enable this certificate for IIS
Enable-ExchangeCertificate -Services IIS -Thumbprint 34B18F6F914747FE8D2FC930CEDD03D40D190DA3
This will create the binding for TCP 443 on the Default Web Site. We need to manually create the TCP 444 binding on the Exchange Back End Web Site.
Using IIS manager assign the new self signed cert to the Exchange Back End site.
For the Exchange Back End web site, the HTTPS binding should be TCP 444. Initially, the SSL certificate is listed as "Not Selected".
We need to correct that.
Chose the recently created Exchange Self Signed certificate. Typically this will have a friendly name of "Microsoft Exchange".
By default the certificate will be shown as below, i.e. not trusted.
Despite this, Exchange will allow the proxy traffic to this sever as the certificate is used for encryption purposes.
While Exchange does not verify the certificate for the purpose above, you are not done. Please see this article to resolve Managed Availability issues.
Fix Auth Cert
We need to generate another self signed certificate which can be used as the new Auth certificate. The below article outlines how to perform this task.
Since the article relates to Exchange hybrid, we will perform only the required steps.
First up, let's review the current layout. The installed certificates are shown. Then we check to the thumbprint for the current Auth certificate.
Note that the current Auth certificate thumbprint is not listed, as it was deleted since the certificate had expired.
Get-AuthConfig | FL CurrentCertificateThumbprint
We need to generate a new self signed certificate that can be used as the Auth certificate. The below sample command generates the certificate with required options.
New-ExchangeCertificate -KeySize 2048 -SubjectName "cn= Microsoft Exchange ACS Certificate" -FriendlyName "Microsoft Exchange Server ACS Certificate" -PrivateKeyExportable $true -Services SMTP -DomainName Tailspintoys.org
Note that we do NOT overwrite the default SMTP certificate. This is the Auth certificate and its prime use is NOT for SMTP. SMTP will still bind to it, but it will not be the default SMTP certificate.
Note that this is certificate 099B3C877881077A083FB4084260799891AE2B95
Now that we have the new Auth certificate, we need to instruct Exchange to use it for this purpose. Normally we should be proactively managing this certificate and gracefully transitioning to a new certificate whilst the current one remains valid. This is why you see the note below stating that the certificate effective date is not in the future. We are setting the effective date to be right now so the newly generated certificate is used. Once that is done we then publish the certificate and the Exchange Service Host processs will replicate it across the organisation.
$Date=Get-Date Set-AuthConfig -NewCertificateThumbprint 099B3C877881077A083FB4084260799891AE2B95 -NewCertificateEffectiveDate $Date Set-AuthConfig –PublishCertificate
Certificate Endgame
Now we have new certificates for IIS, SMTP and Exchange server Auth. These new certificates can be seen below.
Now that was corrected, let's re-launch setup and get the latest CU installed.
Done!
Cheers,
Rhoderick
Happy Halloween! ...and thank your for this post. Needed it today when I was going to play in my lab, after a few months of total neglect, followed by an "OK, let's see what happens" to Exchange 2019 CU 10 installing via Windows Update (which ended in the described symptoms here:)).
It is hard to say enough how welcomed a post like this one is, from a reliable source such as yourself. Please keep'em coming.
Hey!
That's typically where I run into this issue as well. The new health check script checks for this now, so will be a good idea to run that as an extra step to see what state a lab is in prior to installing the CU etc.
I was thinking about you the other day. Lowered my lawn mower from the normal summer setting to mulch up the leaves, it is normally at the elevated Jeremy level 😉
Cheers,
Rhoderick