After going through the steps to disable TLS 1.0 and TLS 1.1, it was noted that Managed Availability was not happy with one particular component in Exchange 2013. This was the OnPremisesSmtpClientSubmission probe and the monitor which was associated to it. The below is a reproduction of the customer environment.
For reference, you can review Protocols in TLS/SSL (Schannel SSP) for a listing of which versions of Windows have particular SSL & TLS protocols enabled by default. Managing SSL/TLS Protocols and Cipher Suites for AD FS also has some good background.
Though Transport Layer Security (TLS) registry settings is probably my favourite reference.
I have seen some deployments where the assumption that SSL 3 has been disabled was incorrect, so please also ensure that you are reviewing the overall Schannel configuration and not just fixating on TLS 1.0 and 1.1. Exchange 2013 onwards inherit from the underlying OS. This change is described below.
Change in TLS Settings Behavior in Exchange Server 2013 and 2016
The cumulative updates for Exchange Server 2013 and 2016 released today include a change in behavior as it relates to configuring TLS and cryptography settings. Previous cumulative updates would overwrite a customer’s existing configuration. Due to customer feedback, we have changed product behavior to configure TLS and cryptography settings only when a new Exchange server is installed. Applying a cumulative update will no longer overwrite the customer’s existing configuration. In the future, the Exchange team will publish guidance on what we believe customers should use to optimally configure a server. It will be up to customers to ensure their servers are configured to meet their security needs. Exchange SETUP will ensure that our current recommendations are applied automatically when a new Exchange server is installed using current and future cumulative updates.
This post will focus on the Exchange 2013 probe as described in the title. However, the steps and troubleshooting methodology are detailed as we go down this road so you can take the same process and apply it to other Managed Availability issues.
Starting Configuration
This is an Exchange 2013 CU23 server with the latest security updates at the time of writing. Exchange 2013 CU23 was installed directly onto Windows Server 2012 R2, and all the Windows updates had been installed so the platform was fully patched.
In other words, this is a fresh brand new Exchange 2013 server with no history. There are multiple Exchange servers in the lab, but we will focus on this one – a server called Exch-3.Tailspintoys.org.
Initially all is well, and there are no issues reported by Managed Availability.
We can see this by running the Get-ServerHealth using the QuickTip to help filter on unhealthy elements.
The same picture was obtained when running Get-HealthReport. No issues are present.
Repro Destructions Instructions
The steps in the blog posts to enable TLS 1.2 and subsequently disable TLS 1.0 and 1.1 were followed. Please read the aforementioned posts for all of the details, but the changes made to the server can be summarised as:
-
Disabled TLS 1.0 client and server
-
Disabled TLS 1.1 client and server
-
Set .NET to use SystemDefaults for both x86 and x64
Restarted server to apply the changes. Please note that SSL 3 was already disabled.
Symptoms of The issue
After restarting the server, we need to wait for Managed Availability to re-assess the health and status of the components. This can take 30 minutes or so to shake out all of the changes. Give it time.
All of the required services were running, so that would not cause any issues. Once Managed Availability had time to fire the necessary probes, we saw errors in the crimson channel.
In the Microsoft-Exchange-ActiveMonitoring/ProbeResult log there was:
Looking at the details tab showed the following error:
<Exception>System.IO.IOException: Authentication failed because the remote party has closed the transport stream.
at System.Net.Security.SslState.InternalEndProcessAuthentication(LazyAsyncResult lazyResult) at System.Net.Security.SslState.EndProcessAuthentication(IAsyncResult result)
at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
Looking for the monitor which consumed the output, we noted the below in the Microsoft-Exchange-ManagedAvailability/Monitoring event log.
The details reported:
Subject>FrontendTransport health set unhealthy (OnPremisesSmtpClientSubmissionMonitor) - The client submission probe failed 3 times over 15 minutes.</Subject>
<Message>The client submission probe failed 3 times over 15 minutes. Authentication failed because the remote party has closed the transport stream.
Probe Exception: 'System.IO.IOException: Authentication failed because the remote party has closed the transport stream.
at System.Net.Security.SslState.InternalEndProcessAuthentication(LazyAsyncResult lazyResult) at System.Net.Security.SslState.EndProcessAuthentication(IAsyncResult result)
at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)'
Failure Context: 'One or more errors occurred.' Execution Context: '' Probe Result Name: 'OnPremisesSmtpClientSubmission'
Probe Result Type: 'Failed' Monitor Total Value: '3' Monitor Total Sample Count: '3' Monitor Total Failed Count: '0' Monitor Poisoned Count: '0' Monitor First Alert Observed Time: '11/16/2020 8:32:03 PM'</Message>
Hunt Down the Probe
Managed Availability does not have a UI, and since most of the logic is stored in code, there are only a few places to look when troubleshooting issues. In addition to the crimson channel storing the errors from probes, when Managed Availability starts up it writes the probe definitions there. We can use those event log entries to help ascertain what the probe is attempting.
Since all of the details are stored in these event logs as XML entries, we can search the event log looking for matches to our failing probe. In this case we wanted to find the OnpremisesSmtpClientSubmission probe.
(Get-WinEvent -LogName Microsoft-Exchange-ActiveMonitoring/ProbeDefinition | % {[XML]$_.toXml()}).event.userData.eventXml | ?{$_.Name -like "*SmtpClient*"}
Good. We confirmed that the probe definition actually exits. Additionally we can see some details on how the probe operates.
Let us confirm that there are no other issues with the other probes and monitors which are related to this area of Managed Availability. We need to confirm the name of the HealthSet to which the probe belongs. So we can filter on the HealthSetName attribute. In the example below, we see that the name to focus on is FrontEndTransport.
Get-ServerHealth (Get-PSSession).ComputerName | Where-Object {$_.Alertvalue -eq "UnHealthy"} | select Name, HealthSetName
Now that we have the HealthSetName, let us review all of the other Managed Availability elements which it contains. To do this, we can use the Get-Monitoringidentity cmdlet. An example is show:
Get-MonitoringItemIdentity -Server <ServerName> -Identity <HealthSetName> | Format-Table Identity,ItemType,Name -Auto
Note the arrow to indicate that OnPremisesSmtpClientSubmission is a probe.
Ok so far. We confirmed that Managed Availability has loaded the probe. We can attempt to manually run the probe. The below PowerShell command can be used to manually invoke the probe:
Invoke-MonitoringProbe -Server Exch-3 -Identity FrontendTransport\OnPremisesSmtpClientSubmission | FL
Error : Authentication failed because the remote party has closed the transport stream.
Exception : System.IO.IOException: Authentication failed because the remote party has closed the transport
stream.
at System.Net.Security.SslState.InternalEndProcessAuthentication(LazyAsyncResult lazyResult)
at System.Net.Security.SslState.EndProcessAuthentication(IAsyncResult result)
at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2
endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
We confirm that the same error is logged when manually running it, as to when Exchange runs it automatically. This will be useful when testing as we can manually run the probe rather than having to wait…
Server Verification
Before we go further, let's verify that all of the expected configuration is present. This is also to illustrate the build for reference purposes. As noted in the Transport Layer Security (TLS) registry settings article the DisabledByDefault is a separate key.
To disable TLS 1.0 for client or server, change the DWORD value to 0. If an SSPI app requests to use TLS 1.0, it will be denied.
To disable TLS 1.0 by default, create a DisabledByDefault entry and change the DWORD value to 1. If an SSPI app explicitly requests to use TLS 1.0, it may be negotiated.
TLS Registry Entries
Do we have the expected TLS registry entries, and are they correct?
Note that the below illustrates that:
-
SSL2 is disabled
-
SSL3 is disabled
-
TLS 1.0 is disabled
-
TLS 1.1 is disabled
-
TLS 1.2 is enabled
The configuration to .NET Framework 4.8 is shown below for the x64 path in the registry
And also for x86.
PowerShell & .NET
Quick check to ensure that .NET, and by extension PowerShell processed the TLS change to use SystemDefaults.
The below image shows both x86 and x64.
Now that we have done the initial verifications to show that the change was made correctly, let's move on to running some analysis of the issue.
Troubleshooting Issue
Now that we have verified the base configuration, we can move on to some troubleshooting details. We will step through some of the tools and techniques used.
Schannel Event Logging
This was basically useless in helping to get to the bottom of the issue. The event messages were too generic.
This KB has the steps should you want to review them.
SMTP Protocol Logging
I always recommend that SMTP logging is enabled for all send and receive connects.
See the Exchange 2013 Tweaks post for details. For the front end connector we can look at the logs inside this folder:
C:\Program Files\Microsoft\Exchange Server\V15\TransportRoles\Logs\FrontEnd\ProtocolLog\SmtpReceive
Note that there is a difference in Client Submission on TCP 587 entries versus regular SMTP TCP 25. TCP 587 is the client submission port, and TCP 25 is regular server to server SMTP.
The full SMTP session is shown, this is identified by the Session ID. For example this is session 08D88A6C5AFA85C5 in the first example shown below.
Client Submission Receive Connector – TCP 587
#Fields: date-time,connector-id,session-id,sequence-number,local-endpoint,remote-endpoint,event,data,context
2020-11-17T00:01:08.020Z,Exch-3\Client Frontend EXCH-3,08D88A6C5AFA85C5,0,127.0.0.1:587,127.0.0.1:27647,+,,
2020-11-17T00:01:08.020Z,Exch-3\Client Frontend EXCH-3,08D88A6C5AFA85C5,1,127.0.0.1:587,127.0.0.1:27647,*,None,Set Session Permissions
2020-11-17T00:01:08.020Z,Exch-3\Client Frontend EXCH-3,08D88A6C5AFA85C5,2,127.0.0.1:587,127.0.0.1:27647,>,"220 Exch-3.tailspintoys.org Microsoft ESMTP MAIL Service ready at Tue, 17 Nov 2020 00:01:07 +0000",
2020-11-17T00:01:08.020Z,Exch-3\Client Frontend EXCH-3,08D88A6C5AFA85C5,3,127.0.0.1:587,127.0.0.1:27647,<,EHLO SmtpClientSubmissionProbe,
2020-11-17T00:01:08.020Z,Exch-3\Client Frontend EXCH-3,08D88A6C5AFA85C5,4,127.0.0.1:587,127.0.0.1:27647,*,None,Set Session Permissions
2020-11-17T00:01:08.020Z,Exch-3\Client Frontend EXCH-3,08D88A6C5AFA85C5,5,127.0.0.1:587,127.0.0.1:27647,>,250-Exch-3.tailspintoys.org Hello [127.0.0.1],
2020-11-17T00:01:08.020Z,Exch-3\Client Frontend EXCH-3,08D88A6C5AFA85C5,6,127.0.0.1:587,127.0.0.1:27647,>,250-SIZE 36700160,
2020-11-17T00:01:08.020Z,Exch-3\Client Frontend EXCH-3,08D88A6C5AFA85C5,7,127.0.0.1:587,127.0.0.1:27647,>,250-PIPELINING,
2020-11-17T00:01:08.020Z,Exch-3\Client Frontend EXCH-3,08D88A6C5AFA85C5,8,127.0.0.1:587,127.0.0.1:27647,>,250-DSN,
2020-11-17T00:01:08.020Z,Exch-3\Client Frontend EXCH-3,08D88A6C5AFA85C5,9,127.0.0.1:587,127.0.0.1:27647,>,250-ENHANCEDSTATUSCODES,
2020-11-17T00:01:08.020Z,Exch-3\Client Frontend EXCH-3,08D88A6C5AFA85C5,10,127.0.0.1:587,127.0.0.1:27647,>,250-STARTTLS,
2020-11-17T00:01:08.020Z,Exch-3\Client Frontend EXCH-3,08D88A6C5AFA85C5,11,127.0.0.1:587,127.0.0.1:27647,>,250-AUTH GSSAPI NTLM,
2020-11-17T00:01:08.020Z,Exch-3\Client Frontend EXCH-3,08D88A6C5AFA85C5,12,127.0.0.1:587,127.0.0.1:27647,>,250-8BITMIME,
2020-11-17T00:01:08.020Z,Exch-3\Client Frontend EXCH-3,08D88A6C5AFA85C5,13,127.0.0.1:587,127.0.0.1:27647,>,250-BINARYMIME,
2020-11-17T00:01:08.020Z,Exch-3\Client Frontend EXCH-3,08D88A6C5AFA85C5,14,127.0.0.1:587,127.0.0.1:27647,>,250 CHUNKING,
2020-11-17T00:01:08.020Z,Exch-3\Client Frontend EXCH-3,08D88A6C5AFA85C5,15,127.0.0.1:587,127.0.0.1:27647,<,STARTTLS,
2020-11-17T00:01:08.020Z,Exch-3\Client Frontend EXCH-3,08D88A6C5AFA85C5,16,127.0.0.1:587,127.0.0.1:27647,>,220 2.0.0 SMTP server ready,
2020-11-17T00:01:08.020Z,Exch-3\Client Frontend EXCH-3,08D88A6C5AFA85C5,17,127.0.0.1:587,127.0.0.1:27647,*,,Sending certificate
2020-11-17T00:01:08.020Z,Exch-3\Client Frontend EXCH-3,08D88A6C5AFA85C5,18,127.0.0.1:587,127.0.0.1:27647,*,CN=Exch-3,Certificate subject
2020-11-17T00:01:08.020Z,Exch-3\Client Frontend EXCH-3,08D88A6C5AFA85C5,19,127.0.0.1:587,127.0.0.1:27647,*,CN=Exch-3,Certificate issuer name
2020-11-17T00:01:08.020Z,Exch-3\Client Frontend EXCH-3,08D88A6C5AFA85C5,20,127.0.0.1:587,127.0.0.1:27647,*,150DB71FAD847FB44AEFCA6192E2B6F9,Certificate serial number
2020-11-17T00:01:08.020Z,Exch-3\Client Frontend EXCH-3,08D88A6C5AFA85C5,21,127.0.0.1:587,127.0.0.1:27647,*,4069974D790C81EC43DBC54F944300BF98BBEA0F,Certificate thumbprint
2020-11-17T00:01:08.020Z,Exch-3\Client Frontend EXCH-3,08D88A6C5AFA85C5,22,127.0.0.1:587,127.0.0.1:27647,*,Exch-3;Exch-3.tailspintoys.org,Certificate alternate names
2020-11-17T00:01:08.020Z,Exch-3\Client Frontend EXCH-3,08D88A6C5AFA85C5,23,127.0.0.1:587,127.0.0.1:27647,*,,TLS negotiation failed with error AlgorithmMismatch
2020-11-17T00:01:08.020Z,Exch-3\Client Frontend EXCH-3,08D88A6C5AFA85C5,24,127.0.0.1:587,127.0.0.1:27647,-,,Local
Default Front End Receive Connector – TCP 25
2020-11-17T00:01:36.098Z,Exch-3\Default Frontend EXCH-3,08D88A6C5AFA85C6,0,127.0.0.1:25,127.0.0.1:27675,+,,
2020-11-17T00:01:36.098Z,Exch-3\Default Frontend EXCH-3,08D88A6C5AFA85C6,1,127.0.0.1:25,127.0.0.1:27675,*,SMTPSubmit SMTPAcceptAnySender SMTPAcceptAuthoritativeDomainSender AcceptRoutingHeaders,Set Session Permissions
2020-11-17T00:01:36.098Z,Exch-3\Default Frontend EXCH-3,08D88A6C5AFA85C6,2,127.0.0.1:25,127.0.0.1:27675,>,"220 Exch-3.tailspintoys.org Microsoft ESMTP MAIL Service ready at Tue, 17 Nov 2020 00:01:35 +0000",
2020-11-17T00:01:36.098Z,Exch-3\Default Frontend EXCH-3,08D88A6C5AFA85C6,3,127.0.0.1:25,127.0.0.1:27675,<,EHLO,
2020-11-17T00:01:36.098Z,Exch-3\Default Frontend EXCH-3,08D88A6C5AFA85C6,4,127.0.0.1:25,127.0.0.1:27675,*,SMTPSubmit SMTPAcceptAnySender SMTPAcceptAuthoritativeDomainSender AcceptRoutingHeaders,Set Session Permissions
2020-11-17T00:01:36.098Z,Exch-3\Default Frontend EXCH-3,08D88A6C5AFA85C6,5,127.0.0.1:25,127.0.0.1:27675,>,250-Exch-3.tailspintoys.org Hello [127.0.0.1],
2020-11-17T00:01:36.098Z,Exch-3\Default Frontend EXCH-3,08D88A6C5AFA85C6,6,127.0.0.1:25,127.0.0.1:27675,>,250-SIZE 37748736,
2020-11-17T00:01:36.098Z,Exch-3\Default Frontend EXCH-3,08D88A6C5AFA85C6,7,127.0.0.1:25,127.0.0.1:27675,>,250-PIPELINING,
2020-11-17T00:01:36.098Z,Exch-3\Default Frontend EXCH-3,08D88A6C5AFA85C6,8,127.0.0.1:25,127.0.0.1:27675,>,250-DSN,
2020-11-17T00:01:36.098Z,Exch-3\Default Frontend EXCH-3,08D88A6C5AFA85C6,9,127.0.0.1:25,127.0.0.1:27675,>,250-ENHANCEDSTATUSCODES,
2020-11-17T00:01:36.098Z,Exch-3\Default Frontend EXCH-3,08D88A6C5AFA85C6,10,127.0.0.1:25,127.0.0.1:27675,>,250-STARTTLS,
2020-11-17T00:01:36.098Z,Exch-3\Default Frontend EXCH-3,08D88A6C5AFA85C6,11,127.0.0.1:25,127.0.0.1:27675,>,250-X-ANONYMOUSTLS,
2020-11-17T00:01:36.098Z,Exch-3\Default Frontend EXCH-3,08D88A6C5AFA85C6,12,127.0.0.1:25,127.0.0.1:27675,>,250-AUTH NTLM,
2020-11-17T00:01:36.098Z,Exch-3\Default Frontend EXCH-3,08D88A6C5AFA85C6,13,127.0.0.1:25,127.0.0.1:27675,>,250-X-EXPS GSSAPI NTLM,
2020-11-17T00:01:36.098Z,Exch-3\Default Frontend EXCH-3,08D88A6C5AFA85C6,14,127.0.0.1:25,127.0.0.1:27675,>,250-8BITMIME,
2020-11-17T00:01:36.098Z,Exch-3\Default Frontend EXCH-3,08D88A6C5AFA85C6,15,127.0.0.1:25,127.0.0.1:27675,>,250-BINARYMIME,
2020-11-17T00:01:36.098Z,Exch-3\Default Frontend EXCH-3,08D88A6C5AFA85C6,16,127.0.0.1:25,127.0.0.1:27675,>,250-CHUNKING,
2020-11-17T00:01:36.098Z,Exch-3\Default Frontend EXCH-3,08D88A6C5AFA85C6,17,127.0.0.1:25,127.0.0.1:27675,>,250 XRDST,
2020-11-17T00:01:36.098Z,Exch-3\Default Frontend EXCH-3,08D88A6C5AFA85C6,18,127.0.0.1:25,127.0.0.1:27675,<,QUIT,
2020-11-17T00:01:36.098Z,Exch-3\Default Frontend EXCH-3,08D88A6C5AFA85C6,19,127.0.0.1:25,127.0.0.1:27675,>,221 2.0.0 Service closing transmission channel,
2020-11-17T00:01:36.098Z,Exch-3\Default Frontend EXCH-3,08D88A6C5AFA85C6,20,127.0.0.1:25,127.0.0.1:27675,-,,Local
Note that the probe to TCP 25 does not issue the STARTTLS verb. Is that why it does not report an error?
Hmmmm. Let's do a network trace to look at the TLS handshake.
Wireshark Network Trace
Wireshark was installed and we set a capture filter to only capture a certain port – TCP 587. The capture was started and included both the loopback and ethernet interface.
Promiscuous mode was disabled as this is not needed. The below shows the capture filter which was created.
After manually invoking the OnPremisesSmtpClientSubmission probe, the below capture was obtained. Immediately we see red at the bottom of the capture. That is generally not good, but don't get distracted.
There is something much worse logged before that…
Take a close look…
Look at the Client Hello frame #14.
It is trying to use SSLv3. Oh dear.
That is disabled as we noted earlier, and is simply never going to work. That is a problem.
Review Default Install
On a different server where the OnPremisesSmtpClientSubmission is working, let's look at the SMTP protocol log and perform the same network capture to review the TLS handshake details.
This other server is called TO-Exch-2 and is the same build Exchange 2013 CU23 on Windows Server 2012 R2. All updates are installed for both Exchange and Windows, just the same as the failing server.
However the Schannel registry keys have not been modified, so this server can still perform SSL3, TLS 1.0 and 1.1 and 1.2 – the default values are present.
The entire SMTP session is included for completeness.
2020-11-17T00:03:23.391Z,TO-EXCH-2\Client Frontend TO-EXCH-2,08D88A3BBD31567D,0,127.0.0.1:587,127.0.0.1:57852,+,,
2020-11-17T00:03:23.391Z,TO-EXCH-2\Client Frontend TO-EXCH-2,08D88A3BBD31567D,1,127.0.0.1:587,127.0.0.1:57852,*,None,Set Session Permissions
2020-11-17T00:03:23.391Z,TO-EXCH-2\Client Frontend TO-EXCH-2,08D88A3BBD31567D,2,127.0.0.1:587,127.0.0.1:57852,>,"220 TO-Exch-2.tailspintoys.org Microsoft ESMTP MAIL Service ready at Tue, 17 Nov 2020 00:03:22 +0000",
2020-11-17T00:03:23.407Z,TO-EXCH-2\Client Frontend TO-EXCH-2,08D88A3BBD31567D,3,127.0.0.1:587,127.0.0.1:57852,<,EHLO SmtpClientSubmissionProbe,
2020-11-17T00:03:23.407Z,TO-EXCH-2\Client Frontend TO-EXCH-2,08D88A3BBD31567D,4,127.0.0.1:587,127.0.0.1:57852,*,None,Set Session Permissions
2020-11-17T00:03:23.407Z,TO-EXCH-2\Client Frontend TO-EXCH-2,08D88A3BBD31567D,5,127.0.0.1:587,127.0.0.1:57852,>,250-TO-Exch-2.tailspintoys.org Hello [127.0.0.1],
2020-11-17T00:03:23.407Z,TO-EXCH-2\Client Frontend TO-EXCH-2,08D88A3BBD31567D,6,127.0.0.1:587,127.0.0.1:57852,>,250-SIZE 36700160,
2020-11-17T00:03:23.407Z,TO-EXCH-2\Client Frontend TO-EXCH-2,08D88A3BBD31567D,7,127.0.0.1:587,127.0.0.1:57852,>,250-PIPELINING,
2020-11-17T00:03:23.407Z,TO-EXCH-2\Client Frontend TO-EXCH-2,08D88A3BBD31567D,8,127.0.0.1:587,127.0.0.1:57852,>,250-DSN,
2020-11-17T00:03:23.407Z,TO-EXCH-2\Client Frontend TO-EXCH-2,08D88A3BBD31567D,9,127.0.0.1:587,127.0.0.1:57852,>,250-ENHANCEDSTATUSCODES,
2020-11-17T00:03:23.407Z,TO-EXCH-2\Client Frontend TO-EXCH-2,08D88A3BBD31567D,10,127.0.0.1:587,127.0.0.1:57852,>,250-STARTTLS,
2020-11-17T00:03:23.407Z,TO-EXCH-2\Client Frontend TO-EXCH-2,08D88A3BBD31567D,11,127.0.0.1:587,127.0.0.1:57852,>,250-AUTH GSSAPI NTLM,
2020-11-17T00:03:23.407Z,TO-EXCH-2\Client Frontend TO-EXCH-2,08D88A3BBD31567D,12,127.0.0.1:587,127.0.0.1:57852,>,250-8BITMIME,
2020-11-17T00:03:23.407Z,TO-EXCH-2\Client Frontend TO-EXCH-2,08D88A3BBD31567D,13,127.0.0.1:587,127.0.0.1:57852,>,250-BINARYMIME,
2020-11-17T00:03:23.407Z,TO-EXCH-2\Client Frontend TO-EXCH-2,08D88A3BBD31567D,14,127.0.0.1:587,127.0.0.1:57852,>,250 CHUNKING,
2020-11-17T00:03:23.407Z,TO-EXCH-2\Client Frontend TO-EXCH-2,08D88A3BBD31567D,15,127.0.0.1:587,127.0.0.1:57852,<,STARTTLS,
2020-11-17T00:03:23.407Z,TO-EXCH-2\Client Frontend TO-EXCH-2,08D88A3BBD31567D,16,127.0.0.1:587,127.0.0.1:57852,>,220 2.0.0 SMTP server ready,
2020-11-17T00:03:23.407Z,TO-EXCH-2\Client Frontend TO-EXCH-2,08D88A3BBD31567D,17,127.0.0.1:587,127.0.0.1:57852,*,,Sending certificate
2020-11-17T00:03:23.407Z,TO-EXCH-2\Client Frontend TO-EXCH-2,08D88A3BBD31567D,18,127.0.0.1:587,127.0.0.1:57852,*,CN=TO-Exch-2,Certificate subject
2020-11-17T00:03:23.407Z,TO-EXCH-2\Client Frontend TO-EXCH-2,08D88A3BBD31567D,19,127.0.0.1:587,127.0.0.1:57852,*,CN=TO-Exch-2,Certificate issuer name
2020-11-17T00:03:23.407Z,TO-EXCH-2\Client Frontend TO-EXCH-2,08D88A3BBD31567D,20,127.0.0.1:587,127.0.0.1:57852,*,43DA91D2D9D5EDB74F2320CE76783064,Certificate serial number
2020-11-17T00:03:23.407Z,TO-EXCH-2\Client Frontend TO-EXCH-2,08D88A3BBD31567D,21,127.0.0.1:587,127.0.0.1:57852,*,B8A812D891F10046C7B212F30DF61A04EC044269,Certificate thumbprint
2020-11-17T00:03:23.407Z,TO-EXCH-2\Client Frontend TO-EXCH-2,08D88A3BBD31567D,22,127.0.0.1:587,127.0.0.1:57852,*,TO-Exch-2;TO-Exch-2.tailspintoys.org,Certificate alternate names
2020-11-17T00:03:23.407Z,TO-EXCH-2\Client Frontend TO-EXCH-2,08D88A3BBD31567D,23,127.0.0.1:587,127.0.0.1:57852,*,,"TLS protocol SP_PROT_TLS1_0_SERVER negotiation succeeded using bulk encryption algorithm CALG_AES_256 with strength 256 bits, MAC hash algorithm CALG_SHA1 with strength 160 bits and key exchange algorithm CALG_ECDHE with strength 384 bits"
2020-11-17T00:03:23.407Z,TO-EXCH-2\Client Frontend TO-EXCH-2,08D88A3BBD31567D,24,127.0.0.1:587,127.0.0.1:57852,<,EHLO SmtpClientSubmissionProbe,
2020-11-17T00:03:23.407Z,TO-EXCH-2\Client Frontend TO-EXCH-2,08D88A3BBD31567D,25,127.0.0.1:587,127.0.0.1:57852,*,,Client certificate chain validation status: 'EmptyCertificate'
2020-11-17T00:03:23.407Z,TO-EXCH-2\Client Frontend TO-EXCH-2,08D88A3BBD31567D,26,127.0.0.1:587,127.0.0.1:57852,*,,TlsDomainCapabilities='None'; Status='NoRemoteCertificate'
2020-11-17T00:03:23.407Z,TO-EXCH-2\Client Frontend TO-EXCH-2,08D88A3BBD31567D,27,127.0.0.1:587,127.0.0.1:57852,*,,TlsDomainCapabilities='None'; Status='NoRemoteCertificate'
2020-11-17T00:03:23.407Z,TO-EXCH-2\Client Frontend TO-EXCH-2,08D88A3BBD31567D,28,127.0.0.1:587,127.0.0.1:57852,*,None,Set Session Permissions
2020-11-17T00:03:23.407Z,TO-EXCH-2\Client Frontend TO-EXCH-2,08D88A3BBD31567D,29,127.0.0.1:587,127.0.0.1:57852,>,250-TO-Exch-2.tailspintoys.org Hello [127.0.0.1],
2020-11-17T00:03:23.407Z,TO-EXCH-2\Client Frontend TO-EXCH-2,08D88A3BBD31567D,30,127.0.0.1:587,127.0.0.1:57852,>,250-SIZE 36700160,
2020-11-17T00:03:23.407Z,TO-EXCH-2\Client Frontend TO-EXCH-2,08D88A3BBD31567D,31,127.0.0.1:587,127.0.0.1:57852,>,250-PIPELINING,
2020-11-17T00:03:23.407Z,TO-EXCH-2\Client Frontend TO-EXCH-2,08D88A3BBD31567D,32,127.0.0.1:587,127.0.0.1:57852,>,250-DSN,
2020-11-17T00:03:23.407Z,TO-EXCH-2\Client Frontend TO-EXCH-2,08D88A3BBD31567D,33,127.0.0.1:587,127.0.0.1:57852,>,250-ENHANCEDSTATUSCODES,
2020-11-17T00:03:23.407Z,TO-EXCH-2\Client Frontend TO-EXCH-2,08D88A3BBD31567D,34,127.0.0.1:587,127.0.0.1:57852,>,250-AUTH GSSAPI NTLM LOGIN,
2020-11-17T00:03:23.407Z,TO-EXCH-2\Client Frontend TO-EXCH-2,08D88A3BBD31567D,35,127.0.0.1:587,127.0.0.1:57852,>,250-8BITMIME,
2020-11-17T00:03:23.407Z,TO-EXCH-2\Client Frontend TO-EXCH-2,08D88A3BBD31567D,36,127.0.0.1:587,127.0.0.1:57852,>,250-BINARYMIME,
2020-11-17T00:03:23.407Z,TO-EXCH-2\Client Frontend TO-EXCH-2,08D88A3BBD31567D,37,127.0.0.1:587,127.0.0.1:57852,>,250 CHUNKING,
2020-11-17T00:03:23.407Z,TO-EXCH-2\Client Frontend TO-EXCH-2,08D88A3BBD31567D,38,127.0.0.1:587,127.0.0.1:57852,<,AUTH LOGIN,
2020-11-17T00:03:23.407Z,TO-EXCH-2\Client Frontend TO-EXCH-2,08D88A3BBD31567D,39,127.0.0.1:587,127.0.0.1:57852,>,334 <authentication response>,
2020-11-17T00:03:23.407Z,TO-EXCH-2\Client Frontend TO-EXCH-2,08D88A3BBD31567D,40,127.0.0.1:587,127.0.0.1:57852,>,334 <authentication response>,
2020-11-17T00:03:23.454Z,TO-EXCH-2\Client Frontend TO-EXCH-2,08D88A3BBD31567D,41,127.0.0.1:587,127.0.0.1:57852,*,SMTPSubmit SMTPAcceptAnyRecipient BypassAntiSpam AcceptRoutingHeaders,Set Session Permissions
2020-11-17T00:03:23.454Z,TO-EXCH-2\Client Frontend TO-EXCH-2,08D88A3BBD31567D,42,127.0.0.1:587,127.0.0.1:57852,*,HealthMailbox8ca3d86576df4ffd9f919b94136056e6@tailspintoys.org,authenticated
2020-11-17T00:03:23.500Z,TO-EXCH-2\Client Frontend TO-EXCH-2,08D88A3BBD31567D,43,127.0.0.1:587,127.0.0.1:57852,*,,Proxy session was successfully set up. Session forHealthMailbox8ca3d86576bf4ffd9f919b94136056e6@tailspintoys.org will now be proxied
2020-11-17T00:03:23.500Z,TO-EXCH-2\Client Frontend TO-EXCH-2,08D88A3BBD31567D,44,127.0.0.1:587,127.0.0.1:57852,>,235 2.7.0 Authentication successful,
2020-11-17T00:03:23.766Z,TO-EXCH-2\Client Frontend TO-EXCH-2,08D88A3BBD31567D,45,127.0.0.1:587,127.0.0.1:57852,-,,Local
Note that the STARTTLS verb is issued, and we can complete the TLS handshake. We will confirm the highlighted section by using Wireshark as before.
The SMTP protocol log entries for this session are much more complete, and we can see front end transport has then gone on to proxy to a back end server as expected.
What about the network trace?
Note that the Client Hello in frame #14 states that TLS 1.0 is used. The TLS handshake completes, and we are able to then see the Server Hello.
Re-Enabling TLS 1.0 Client and Server
On the broken server if we re-enable TLS 1.0 client and server and restart the server to apply the change, does that now allow the probe to function?
The below changes were made, and the server restarted.
Yes it does. Note that there are no longer any issues reported by Get-ServerHealth.
Additionally, the probe is able to be executed and we have a successful return code.
Looking at the crimson channel we can also see success, and this can be compared to the previous issues.
In the details:
220 Exch-3.tailspintoys.org Microsoft ESMTP MAIL Service ready at Wed, 18 Nov 2020 00:58:42 +0000EHLO SmtpClientSubmissionProbe 250-Exch-3.tailspintoys.org Hello [127.0.0.1]STARTTLS 220 2.0.0 SMTP server readyEHLO SmtpClientSubmissionProbe 250-Exch-3.tailspintoys.org Hello [127.0.0.1]AUTH LOGIN 334 VXNlcm5hbWU6 SGVhbHRoTWFpbGJveDlhYzc3NzBjYWUxMTQyYTA4ZDA0YmExMmZlZWQ4NjYxQHRhaWxzcGludG95cy5vcmc= 334 UGFzc3dvcmQ6 OzI3KDYpbExiUnFCS3hmUUVbNk1pSjZrWXtNQGdFTF0qKz9odkx5bkM1eExzdEBuJHFNKl9qZ0VjIXYwRG99WDE6MiMta18zPngrPk93TXU1T0dyVGF9eks+dkF6OXNXMigpV0wvcTMqfWU1ND9zMCp9TUkkQGhTYU8mb3MhTk4= 235 2.7.0 Authentication successfulMAIL FROM: HealthMailbox9ac7770cae1142a08d04ba12feed8661@tailspintoys.org 250 2.1.0 Sender OKRCPT TO: HealthMailbox9ac7770cae1142a08d04ba12feed8661@tailspintoys.org 250 2.1.5 Recipient OKDATA 354 Start mail input; end with <CRLF>.<CRLF> X-MS-Exchange-ActiveMonitoringProbeName:c511c6fbf635465f9c72e6d972b8fb53-OnPremisesSmtpClientSubmission Subject:Client submission probe X-LAMNotificationId:00000000
As a test, changes were made to the TLS 1.0 configuration to see if that would make it work
Note the same behaviour was noted if the TLS 1.0 Disabled and DisabledByDefault were set to zero. Only SSL3 was attempted. This did not allow the probe to connect.
This aligns with the documentation.
This leads me to think that this Managed Availability probe is hard coded to use only specific SSL and TLS versions.
"Fixing" The OnPremisesSmtpClientSubmission With TLS 1.2
Getting the probe to use TLS 1.2 will most likely require a code change. Give the support lifecycle position for Exchange 2013 I doubt that will happen.
So we have a couple of options:
- Leave TLS 1.0 enabled
- Configure an override to disable this probe
Consider that many customers do not allow end users to connect directly to Exchange to send SMTP messages, and that this issue relates to only the SMTP client submission probe then option #2 is probably the best one for most people. However, if you do have a business requirement for SMTP client submission then you can consider probing the server via the load balancer or a separate service/appliance.
In this case we deployed a global override so that and targeted all of the Exchange servers. Once this replicated, the Health Manager service was restarted on each server.
Add-GlobalMonitoringOverride -Identity "FrontendTransport\OnPremisesSmtpClientSubmission" -PropertyName Enabled -PropertyValue 0 -ItemType Probe
If we want to target this to only Exchange 2013 CU23 servers, then we can add the ApplyVersion and specify the relevant build number. Note that duration can not be combined with the ApplyVersion. You may need to come back and re-issue the command if this expires in 12 months and you still have Exchange 2013 installed.
Add-GlobalMonitoringOverride -Identity "FrontendTransport\OnPremisesSmtpClientSubmission" -PropertyName Enabled -PropertyValue 0 -ItemType Probe –ApplyVersion "15.0.1497.2"
Should you want to remove this global override, then you can run:
Remove-GlobalMonitoringOverride -Identity FrontendTransport\OnPremisesSmtpClientSubmission –ItemType Probe -PropertyName Enabled
Cheers,
Rhoderick
Hi Rhoderick,
Super brilliant article, many thanks!