You have prepped for months, and tonight is the night! It is the night of CAS namespace cutover to move the HTTPS namespaces from Exchange 2010 so that the point to Exchange 2013 or 2016. DNS TTL was decremented. Servers are all fully updated as are the Outlook clients and we are good to go.
The DNS change kicks in, and Outlook HTTPS traffic hits Exchange 2016 and we enter a spiral of never ending authentication prompts. Oh...
Users are greeted with Outlook clients in the below state:
In desperation you try to create a new Outlook profile.
There is no relief. It is the same issue, in fact you can not even get past the initial screen to the main Outlook window without getting the same issue.
OWA is perfectly fine, and users are able to open their mailbox that way with no issues.
Exchange 2016 IIS Logs
What do we see in the front end IIS log files? Does that match with what we see on the client - yes!
Note the series of 401 errors - access denied.
2020-10-09 13:30:59 10.0.0.16 POST /Autodiscover/Autodiscover.xml &CorrelationID=<empty>;&cafeReqId=388a5fa2-6684-4fed-8475-a40678816ef8; 443 - 10.0.0.14 Microsoft+Office/16.0+(Windows+NT+10.0;+MAPICPL+16.0.12527;+Pro) - 401 0 0 16
2020-10-09 13:30:59 10.0.0.16 POST /Autodiscover/Autodiscover.xml &CorrelationID=<empty>;&cafeReqId=4d7bb567-27f0-429d-8798-ec57baaed8fd; 443 - 10.0.0.14 Microsoft+Office/16.0+(Windows+NT+10.0;+MAPICPL+16.0.12527;+Pro) - 401 1 2148074254 8
2020-10-09 13:30:59 10.0.0.16 POST /Autodiscover/Autodiscover.xml &CorrelationID=<empty>;&cafeReqId=fbbd30b5-5693-477b-94b8-2b86c7977ff0; 443 - 10.0.0.14 Microsoft+Office/16.0+(Windows+NT+10.0;+MAPICPL+16.0.12527;+Pro) - 401 1 2148074254 47
Test Autodiscover
Running a test Autodiscover on a PC with an existing Outlook profile shows the exact same symptom. Autodiscover does not complete and it receives a series of 401 errors.
Note though that the account is able to communicate to AD and retrieve the SCP information. So the user is able to authenticate to AD, and the issue is specific to Exchange.
Doing the same thing multiple times does not change the outcome.
The results are the same and Outlook is unable to even communicate to the Autodiscover web service.
Brave Brave Sir Robin
<gratuitous link to Monty Python>
Maybe you are thinking about backing out the change? If so then I really hope you reduced the DNS TTL as a preparatory step. Else you will be waiting a while for the default TTL to expire...
Battling On - Check The IIS logs On Exchange 2016
If you selected to continue to troubleshoot the issue, or are looking back afterwards; is IIS able to perform Windows authentication and is that why we saw the 401 errors?
The below are some sample lines with the 401 error highlighted.
2020-10-09 01:09:59 10.0.0.16 POST /autodiscover/autodiscover.xml &CorrelationID=<empty>;&cafeReqId=b5139f68-4a91-4131-83ab-8b992242c529; 443 - 10.0.0.14 Microsoft+Office/16.0+(Windows+NT+10.0;+Microsoft+Outlook+16.0.12527;+Pro) - 401 1 2148074254 6
2020-10-09 01:09:59 10.0.0.16 POST /autodiscover/autodiscover.xml &CorrelationID=<empty>;&cafeReqId=55841d89-72bf-43c0-88e0-d600a37413c2; 443 - 10.0.0.14 Microsoft+Office/16.0+(Windows+NT+10.0;+Microsoft+Outlook+16.0.12527;+Pro) - 401 1 2148074254 44
In IIS manager note that the Autodiscover site is setup for Windows integrated auth, and that both of the expected providers are present.
Hmm. There are a few things to mull over:
- Outlook does not work for existing profiles.
- Unable to create a new Outlook profile
- Outlook receives 401 error from Autodiscover
- OWA works with no issues
- Users can log onto Windows clients with no issues
- All other applications are fine, only Exchange has a problem
- AD is healthy with no issues
Why is Outlook unable to authenticate? The user account is fine. AD is fine.
Why is Outlook unable to authenticate to the DCs using NTLM or Kerberos?
Then it hits you.
Epiphany
After some deep soul searching, you recall that previously Kerberos was enabled for Exchange 2010. Oh right, that was the Alternate Service Account (ASA) credential that we created many years ago and never really documented as it was not done as part of the initial project. Kerberos has been the recommended authentication to help mitigate NTLM bottleneck issues for any years.
Let's check to see if we setup the ASA on either Exchange 2010 or 2016. In Exchange Management Shell on an Exchange 2016 server:
Get-ClientAccessService -IncludeAlternateServiceAccountCredentialStatus | Select Name, *Alternate*
Oh bugger, that's it. We forgot to deploy Kerberos for Exchange 2016. Note that we really should run the command on Exchange 2016 so it is able to see all versions.
Note that the same command on Exchange 2010 ignores Exchange 2016.
Time to regroup and plan to fix this.
This may mean having to back out the CAS namespace cutover change if you do not have the ability or permission to perform the additional required steps. Also, does this now mean that you have to perform additional testing and validation with Kerberos and Exchange 2016? Ultimately that's your call and will be dictated partly by your organisation's change control process.
Back To The Drawingboard
In the post Exchange 2016 Coexistence with Kerberos Authentication Ross lists out the requirements for the ASA when different versions of Exchange are present. The key point is that if Exchange 2010 is still present, then you require two ASA accounts. This is because the ExchangeMDB, ExchangeRFR and Exchange AB SPNs must continue to point to Exchange 2010. The HTTP based SPNs need to be transitioned to Exchange 2013/2016. They need to be associated with a separate ASA so that we can split out the traffic.
The RollAlternateserviceAccountCredential.ps1 script cannot deserialize objects and pass them between servers that are different versions. This means the script cannot be used to copy the credentials from an Exchange 2010 server or push the credentials to an Exchange 2010 server. As a result, Exchange 2013/2016 and Exchange 2010 cannot share the same Alternate Service Account (ASA) credential.
There are multiple configuration steps required and issues to address as part of the transition to Kerberos. The relevant articles are linked below.
Enable Kerberos for Exchange 2013
Enable Kerberos for Exchange 2016
Enable Kerberos for Exchange 2019
To summarise, we need to:
- Create new ASA
- Deploy new ASA to Exchange 2016 servers
- Move SPNs to new ASA
In the below example, a new ASA is created for Exchange 2016 called E15ASA. A computer account is used.
New-ADComputer -Name E15ASA -AccountPassword (Read-Host 'Enter password' -AsSecureString) -Description 'Alternate Service Account credentials for Exchange' -Enabled:$True -SamAccountName E15ASA
Set-ADComputer E15ASA -add @{"msDS-SupportedEncryptionTypes"="28"}
Note – move this to an OU where it will not be deleted by any automatic stale object purge scripts.
Wait for AD to replicate, then in the Exchange Management Shell we need to generate a new ASA password. There is a script provided for this purpose which is in the scripts directory (unsurprisingly!). You can use the $EXScripts variable to easily jump to the correct location.
CD $ExScripts
By default this is: C:\Program Files\Microsoft\Exchange Server\V15\scripts
.\RollAlternateServiceAccountPassword.ps1 -ToSpecificServers exch-2016.tailspintoys.ca -GenerateNewPasswordFor Tailspintoys\E15ASA$
Now that the E15ASA is available, we need to deploy it to the other Exchange 2016 servers as required. To do this, the –CopyFrom parameter is used.
For example:
.\RollAlternateServiceAccountPassword.ps1 -ToSpecificServer Exch-2016-2.tailspintoys.ca -CopyFrom Exch-2016-1.tailspintoys.ca
Then we should confirm that the E15ASA credential was updated on the relevant servers:
Get-ClientAccessService -IncludeAlternateServiceAccountCredentialStatus | Select Name, *Alternate*
Note that there is a single Exchange 2010 and 2016 server in this lab, so your list will be larger.
Update SPN
At this point we will make changes to the existing E14ASA as some of the SPNs (the HTTP based namespaces) need to move to Exchange 2016. Prior to making any changes, we should document and verify the starting configuration. Use the SetSpn.exe utility with the -L option to list out the SPNs associated to an object.
setspn -L E14ASA
setspn -L E15ASA
Note that the initial E14ASA has all of the names in the Exchange 2010 CAS namespace design. As stated above the RPC namespaces must remain on Exchange 2010, and the HTTP namespaces must be moved to Exchange 2016.
To do this we delete the SPNs we want to move from the E14ASA. Then they will be added to the E15ASA. Note that -D is Delete and -S is Set.
setspn -D http/owa.tailspintoys.ca E14ASA
setspn -D http/mail.tailspintoys.ca E14ASA
setspn -D http/autodiscover.tailspintoys.ca E14ASA
setspn -S http/owa.tailspintoys.ca E15ASA
setspn -S http/mail.tailspintoys.ca E15ASA
setspn -S http/autodiscover.tailspintoys.ca E15ASA
The below shows these commands being run in the lab.
Now that we have made the changes, again we verify that no mistakes were make and that each ASA has the correct SPNs associated to it.
We should also verify that Exchange 2016 is configured for client Kerberos authentication. We need to check the settings for Outlook Anywhere and MAPI/HTTP.
First up - Outlook Anywhere. Note that Exchange 2016 is set for NTLM, so that needs to be changed.
To do this in PowerShell requires specifing additional parameters, and the options have changed in different versions of Exchange so please check the latest documentation for your version.
The below example uses Negotiate internally and externally, which may not be suitable for your deployment. Please review your CAS namespace design. The focus here is on internal clients.
Get-OutlookAnywhere -Server Exch-2016 | Set-OutlookAnywhere -SSLOffloading $False -ExternalClientsRequireSsl $True -InternalClientsRequireSsl $True -ExternalClientAuthenticationMethod Negotiate -InternalClientAuthenticationMethod Negotiate
Once have made the change, we again verify, and note that Negotiate is used now internally for Exchange 2016.
Next we want to review the MAPI/HTTP settings.
Get-MapiVirtualDirectory | Select Name, *URL,*auth*
Note that in production ADPropertiesOnly should not be used to verify authentication flags.
Note that the authentication options and parameters available with the Set-OutlookAnywhere cmdlet are different in Exchange 2010 compared to Exchange 2016. The options can be reviewed here. Also worth remembering is that Outlook Anywhere was typically not used internally with Exchange 2010. That is why there is only a single configuration line for it on Exchange 2010, the assumption there is that the client is external so that the is the Internet endpoint it will connect to.
Verification Time
If you have to change the authentication options in Exchange, then it is always prudent to run an IISRESET on those servers so that the change is processed. While that is another step, it is preferable to having to wait hours for the cache to expire and reload. No one got time for that.
Once the changes are effective, restart Outlook and you should see that the client is able to connect. In the example below, the same client is now able to authenticate to Exchange and retrieve the settings.
Correspondingly, in the IIS logs the requests are now successful and we can weep with tears of joy!
2020-10-09 15:58:49 10.0.0.16 POST /Autodiscover/Autodiscover.xml &CorrelationID=<empty>;&cafeReqId=50e4ea36-3751-4187-a7e1-d9c9c53eb499; 443 TAILSPINTOYS\Local-1 10.0.0.14 Microsoft+Office/16.0+(Windows+NT+10.0;+Microsoft+Outlook+16.0.12527;+Pro) - 200 0 0 66
Cheers,
Rhoderick
