The amazing Exchange Healthcheck script really has gotten to the point where Exchange Best Practices Analyser (ExBPA) used to be. Exchange admins should heavily use the Healthcheck script. This should be done before and after maintenance tasks such as installing CUs and security updates.
There is a handy dandy shortcut URL to download the script: https://aka.ms/ExchangeHealthChecker
Note that this is the new name for the tool, the initial incarnation was the Exchange Performance Health Check Script.
By default the script will check the local server where it is being run, displaying results on screen and also writing them to disk next to the script’s location.
In this case there was an unexpected finding on a single server. Note that the red error says:
EWS Internal Bypass URL Set: https://exch-2019-2.tailspintoys.ca/ews/exchange.asmx - Can cause issues after KB 5001779
This is actually an error that bubbled up in one of my lab environments. A logical question will be, is that the only misconfiguration?
Let’s have a look…
Checking All InternalNLBBypass URLs
If the terms used need a bit explanation please check out https://aka.ms/CASBible - that is a quick redirect to the Exchange 2010 Proxy and Redirection article.
We can run the below command to check for the presence of this URL on the Front End virtual directories.
Get-WebServicesVirtualDirectory| Select-Object Server, *URL* | Format-Table –Autosize
There is only a single server that has this issue, Exch-2019-2. Interestingly, the names are not in order even though they were installed in that numerical sequence. We will come back to that.
Do we even need this? Why is Healthcheck flagging an issue?
In short, we do not need the InternalNLBBypassURL to be set on the Front End virtual directory. This is something that David Paulson mentioned in a comment to this issue.
As David correctly states, this is not needed on the Front End virtual directory due to the architectural changes that were introduced with Exchange 2013 and newer. The phrase “every server is an island” was how the new proxy architecture was explained and you can see that in the documentation. In Exchange 2007 and 2010 there were a couple of EWS calls that required targeting a single server as they could be be moved to another server by a load balancer in the remote site. Hence the name of the URL, NLB Bypass. Nowadays this is a non-issue as Exchange 2013+ addresses all of that murky version specificness . Those calls only every happen on one server – the server where the mailbox is mounted. Between servers are simpler proxy commands. This is illustrated using the arrows in the diagram below. The black arrows indicate the Exchange 2013+ behaviour of proxy commands being sent between servers. The red “banned” arrows illustrate the murky Exchange 2010 and older Store Driver behaviour which created version specific requirements for those RPC calls to succeed.
If you want to delve back into the murky past of Exchange 2007/2010 cross-site CAS-CAS proxy feel free to take a look at https://aka.ms/CASBible.
Note that the comment above specifically applies to the Front End virtual directories. The InternalNLBBypassURL *IS* required on the Backend. Normally you do not see those virtual directories. If you do want to see the Backend, then add the ShowMailboxVirtualDirectories parameter.
An example of that command is shown below:
Get-WebServicesVirtualDirectory -ShowMailboxVirtualDirectories | Select-Object Server, *URL*
Again you will note that only server Exch-2019-2 has the InternalNLBBypassURL set.
How To Remediate
The resolution harkens back to Setting Backend Exchange 2013 / 2016 Virtual Directory - 250 Hello (rmilne.ca as we need to set the virtual directory to a null value.
An example command is shown below:
Get-WebServicesVirtualDirectory -Server Exch-2019-2 | Set-WebServicesVirtualDirectory -InternalNLBBypassUrl $Null
That’s great, but why are the servers not in order?
Unexpected ordering?
Active Directory has a habit of returning things in a date sorted order, the Autodiscover SCP are a prime example. The same would be expected here for the servers as they were originally installed in order.
This is because the server in question had to be recovered using the /RecoverServer setup option. RecoverServer deletes and re-creates each of the virtual directories to ensure that the settings in AD DS, the registry and IIS are all consistent. The Exchange server object must *NOT* be deleted, else all of those descendent settings are lost. This is why you see the behaviour below. The WhenCreated timestamp of the Exchange server object reflects the original installation date.
However the WhenCreated timestamp of the OWA vDIR shows that Exch-2019-2 was installed on the 16th December 2021 – a day later than the other servers.
ExBPA Flashback
As a gratuitous flashback to a distant time, here is the Exchange Best Practices Analyser. Props for this tool, and MS award, go to Paul Bowden.
Anyway, back to the future!
Cheers,
Rhoderick
