Just like the Exchange 2010 version of this post, the intent of this blog post is to be a throwdown of common issues that can arise when deploying Exchange 2013. Issues are not listed in order of significance/priority. Think of this more as a smorgasbord…
Exchange 2013 Servicing Changes
Be aware that Exchange 2013 is not serviced in the same way as Exchange 2010 and 2007. Those older versions used Rollup Updates or RUs, whereas Exchange 2013 uses Cumulative Updates (CU) instead.
Be sure to avoid these 7 common CU installation issues! (6 tips plus a bonus one!)
Ensure that Exchange is updated with new CUs as they become available. This is critical to ensure that the product receives the necessary servicing and support. Also if you are in a Hybrid configuration with Office 365, only the most current and N-1 CUs are supported.
Co-Existence Issues
One great way to minimise issues is to maintain Exchange and Outlook patch levels. As issues are identified and resolved, the fixes are made available through product updates. If you are not installing updates, then you don’t get the fixes….
Some specific examples:
KB 2834139 - Users of Exchange Server 2013 or Exchange Online can't open public folders or shared mailboxes on an Exchange 2010 or Exchange 2007 server
KB 2839517 - Outlook is unable to connect to Exchange 2013 public folder or auto-mapped mailbox
Updated Legacy Public Folder Coexistence in Exchange 2013 CU7
Prior to CU7, Exchange 2013 mailboxes using the Outlook client were proxied to the legacy mailbox server hosting the Public Folder being accessed either via RPC/TCP or RPC/HTTP depending on the client’s location, the connectivity model being used, and the configuration on the legacy Exchange servers.
Exchange administrators must follow the documented steps to enable the new Public Folder discover method. With the settings configured Exchange 2013 will begin returning a new section in Autodiscover responses to Exchange 2013 mailbox users similar to the following and using the new coexistence code paths:
<PublicFolderInformation>
<SmtpAddress>PFDiscovery-001@contoso.com</SmtpAddress>
</PublicFolderInformation>
Move Mailbox Issues
Mailbox Size Growth
Note that when moving mailboxes from Exchange 2007/2010 to Exchange 2013, the size of the mailbox will appear to grow. It was taking the space previously, we just did not have the ability to fully track all of the consumed space in the older versions. This is a serious planning issue as if your users are sailing close to their quota on Exchange 2010, then the simple act of moving their mailbox to 2013 will most likely take them over quota and block access to their mail. Quota limits may need to be increased *PRIOR* to moving those mailboxes. Rough rule is to expect an average of ~20% growth. BUT – you need to analyse moves in your environment, as that is the figure that will be applicable. Individual mailboxes may experience growth significantly above 20%….
Unable to Logon To Exchange 2013 Mailbox After Move Completes
Move mailbox issue. Unable to open mailbox when moved to Exchange 2013.
When mailboxes are moved to Exchange Server 2013 or Exchange Server 2016, users can no longer access those mailboxes.
This issue occurs in the following scenario:
-
A user typically uses Outlook Anywhere to connect to his or her Exchange Server 2010 mailbox.
-
The user's mailbox is moved to Exchange Server 2013 or Exchange Server 2016.
-
After the mailbox is moved and the user tries to log on, he or she is prompted that “The Microsoft Exchange administrator has made a change that requires you quit and restart Outlook.”
-
After Outlook is restarted, the client remains disconnected for up to 12 hours.
This issue occurs as Exchange 2013’s Autodiscover cache still believes the mailbox is on Exchange 2010, so it hands out the Exchange 2010 configuration not Exchange 2013 configuration. To workaround this the Autodiscover cache needs to be flushed on the Exchange 2013 servers. You can restart the app pool using IIS Manager or if you want to script it to run every X minutes the below command can be automated:
Restart-WebAppPool MSExchangeAutodiscoverAppPool
There are other issues that can and will cause the same symptom. One is an out-dated Outlook build. As discussed here, do not underestimate the importance of properly maintaining Outlook. If Outlook is not updated, then it may unable to connect to Exchange 2013 once the move request completes. This is discussed in: KB2934750 Mailbox move to Exchange Server causes Outlook connectivity issue.
Update 14-7-2016 Exchange 2013 CU13 reduced the cache of the app pool to one hour to mitigate this issue.
Move Arbitration Mailboxes To Exchange 2013 At The Start of The Project
This is often overlooked as admins typically want to move the carbon based life unit mailboxes first, and system mailboxes at the end. However TechNet has the below guidance, and outlines what will fail to function if you ignore this:
If you do not move this system mailbox to Exchange 2013, the following issues will occur when Exchange 2010 and Exchange 2013 coexist in your Exchange organization:
-
Exchange 2013 tasks aren’t saved to the administrator audit log. When you run the Search-AdminAuditLog cmdlet or try to export the administrator audit log in the EAC, you’ll receive an error that says you can’t create an administrator audit log search because the system
mailbox, SystemMailbox{e0dc1c29-89c3-4034-b678-e6c29d823ed9}, is located on a server that isn’t running Exchange 2013. A Microsoft Exchange error with an Event ID of 5000 is also logged in the Windows Application log each time a command is run. -
You can’t run eDiscovery searches using the EAC or the Shell in Exchange 2013. Mailbox searches can be created and queued, but they can’t be started. An error with an Event ID of 6 is logged in the MsExchange Management log, stating that the Start-MailboxSearch cmdlet failed. However, you can search mailboxes using the Shell and the Exchange Control Panel (ECP) in Exchange 2010.
You can do this using the EAC or through PowerShell – for example:
Get-Mailbox -Arbitration "SystemMailbox{e0dc1c29-89c3-4034-b678-e6c29d823ed9}" | New-MoveRequest -TargetDatabase "Exchange 2013 Database Name"
Or
Get-Mailbox –Arbitration | New-MoveRequest -TargetDatabase "Exchange 2013 Database Name"
Note that the database where the arbitration mailboxes are being moved to must be protected, as we do not want to lose them. We can recover from that, but why have the hassle and stress if all we need to do is to make sure we move the arbitration mailboxes to a database that has multiple copies?
For example to Re-create the Discovery system mailbox
Maintain .NET Framework
Exchange 2007 and 2010 started the move to managed code. With the Exchange 2013 information store being re-written in managed code, keeping .NET maintained is even more important.
For example when MAPI/HTTP was first released, there was a strong recommendation to install .NET 4.51 framework.
As always, ensure that you are following the Exchange 2013 system requirements. Do not install unsupported versions of .NET and Windows Management Framework onto Exchange servers.
As of March 2016, .NET 4.6 and 4.6.1 are not supported on Exchange servers.
This is discussed further on the Exchange team blog. Symptoms of installing unsupported .NET framework are mailboxes becoming quarantined, unexpected database failover and databases dismounting.
EventID 1014 and EventID 1002.
ID: 1014 Level: Warning Source: MSExchangeIS Machine: exch.tailspintoys.ca Message: The mailbox with mailboxguid "3d488bd9-cfe0-493e-907a-2aa299885f45" has been quarantined. Access to this mailbox will be restricted to administrative logons for 1.00:00:00 since the last report.
ID: 1002 Level: Error Source: MSExchangeIS Machine: exch.tailspintoys.ca Message: Unhandled exception (System.Threading.LockRecursionException: Recursive read lock acquisitions not allowed in this mode.
You may also wish to review: How to temporarily block the installation of the .NET Framework 4.6.1 to set:
LOCAL_MACHINE\Software\Microsoft\NET Framework Setup\NDP\
BlockNetFramework\461 DWORD to "1".
PowerShell Support
In the same vein as the above note on .NET framework, the version of PowerShell that is installed must also listed on the Exchange 2013 system requirements.
Sleepy NIC
Be aware on an issue where the NIC can have siesta in the middle of the day.
Server Sizing
Do not use Exchange 2010 tools to size Exchange 2013. It will not end well. Use the right tool for the right job.
Do not install Exchange 2013 to the C: drive without sufficient planning. If you fail to provision sufficient capacity then you will be running out of space very quickly.
I’m a little tired of seeing servers with a 40 GB or 50 GB C: drive. That is not even enough space for Windows, how is Exchange going to function? Also bear in mind that SafetyNet maintains copies of all messages for 2 days by default. That will be in the mail.que file, which will grow….
Server Sizing Changes
Exchange 2013 SP1+ has changed the sizing for pagefile. It is now possible to set pagefile to be 32GB + 10MB on Exchange 2013 SP1+ servers.
Server Sizing – Hyperthreading Stance Not Changed
Please see the following post and the links contained therein.
Ask The Perf Guy: What’s The Story With Hyperthreading and Virtualization
CPU sizing is a critical issue, You need to have enough CPU resources to deal with the worst case situation in your design, but not so much that it causes an issue. See the recommended sizing for Exchange 2013.
Event log Sizing
Exchange 2013 is chatty. Very!
Ensure that the application event log is correctly sized. On most production Exchange servers that I’ve seen the default 20MB Application Event Log does not even hold a days worth of data. Not good for troubleshooting. This is easy to configure via a GPO. While you are in there, also disable that pesky Server Manager from automatically starting up!
Client Connectivity
Exchange 2013 architecture is different from Exchange 2010. The following post illustrates how different versions of Exchange will coexist for client connectivity
Client Connectivity in an Exchange 2013 Coexistence Environment
Modern Public Folder Limits
Exchange 2013 introduced Modern Public Folders. The scalability of this solution has increased greatly during the lifecycle of Exchange 2013.
Please ensure that you Exchange 2013 implementation is within the supported limits for public folders
Transport Architecture
Exchange 2013 has a different transport architecture due to the every server is an island design methodology. The intent was to move away from the strict version coupling store driver exhibited in Exchange 2007/2010. Richard’s post below is good visual resource”":
Exchange 2013 Mail Flow Demystified…Hopefully!
And the way transport now achieves HA has changed. Details below:
Enable SMTP Protocol Logging
Enable protocol logging for all SMTP send connectors, receive connectors and the intra-organization send connector.
Get-SendConnector | Set-SendConnector -ProtocolLoggingLevel Verbose
Get-ReceiveConnector | Set-ReceiveConnector -ProtocolLoggingLevel Verbose
Get-FrontEndTransportService | Set-FrontEndTransportService -IntraOrgConnectorProtocolLoggingLevel Verbose
Get-MailboxTransportService | Set-MailboxTransportService -MailboxDeliveryConnectorProtocolLoggingLevel Verbose
Failed Content Indexes
In some environments an issue with the search framework may cause all or most of the database copies are in a failed state. This can be viewed with Get-MailboxDatabaseCopyStatus
In this case you will find EventID 1009 in the Application event log with a source of MSExchangeFastSearch
The indexing of mailbox database DB1 encountered an unexpected exception. Error details: Microsoft.Exchange.Search.Core.Abstraction.OperationFailedException: The component operation has failed. ---> Microsoft.Exchange.Search.Fast.FastConnectionException: Connection to the Content Submission Service has failed. ---> Microsoft.Ceres.External.ContentApi.ConnectionException: Given CSS specification failed:
Could not connect to CSS node at net.tcp://localhost:17028/ContentSubmissionServices/content with flow: Microsoft.Exchange.Search.Writer.12.14. Error: Object reference not set to an instance of an object
And also EventID 1010
An operation attempted against a FAST endpoint exprienced an exception. This operation may be retried. Error details: Microsoft.Exchange.Search.Fast.PerformingFastOperationException: An Exception was received during a FAST operation. ---> System.ServiceModel.FaultException: The server was unable to process the request due to an internal error. For more information about the error, either turn on IncludeExceptionDetailInFaults (either from ServiceBehaviorAttribute or from the <serviceDebug> configuration behavior) on the server in order to send the exception information back to the client, or turn on tracing as per the Microsoft .NET Framework SDK documentation and inspect the server trace logs
This issue may occur if the search platform tries to check its membership in a security group that is named "ContentSubmitters." This group is not created by the search platform or by Exchange Server 2013 and is therefore not usually present. Although the check usually fails silently, without any consequences, an exception sometimes occurs. This causes the search component to fail
To resolve the issue, do the following:
- Create a new Active Directory group that is named "ContentSubmitters" and then grant Admistrators and NetworkService full access to the group. This is a dummy group and should be used as a placeholder only. You might want to add a description so that the group is not removed.
- Force or wait for Active Directory replication.
- Restart the following services:
- Microsoft Exchange Search
- Microsoft Exchange Search Host Controller
The services must be restarted on all Exchange servers. To assist with this, you can modify the script to restart the Exchange Health Manager service.
DAG Changes
Exchange 2013 SP1 was the first version of Exchange which was supported on Windows Server 2012 R2. One of the new clustering features in Windows Server 2012 R2 is the ability to create a cluster with an administrative IP address. This is sometimes called the IP less DAG.
While Exchange will function quite happily without an administrative IP address this may not be the case for your third party software, especially backup software. This is because there is no DAG IP or network name resource in such a configuration. Currently a lot of backup software is configured to point to the DAG network name to initiate backups. If there is no network name then the backups cannot be configured.
Check with the respective vendors to see if they support the new DAG. This must be done before creating the DAG and adding servers to it. This is because the type cannot be changed when the DAG is populated. If you want to change the configuration, then it would involved undoing the current configuration and building a new one.
Managed Availability
Managed Availability is the focus moving forward for Exchange monitoring and health detection.
Ensuring that users have a good email experience has always been the primary objective for messaging system administrators. To help ensure the availability and reliability of your Microsoft Exchange Server 2013 organization, all aspects of the system must be actively monitored, and any detected issues must be resolved quickly. In previous versions of Exchange, monitoring critical system components typically involved using an external application such as Microsoft System Center 2012 Operations Manager to collect data, and to provide recovery action for problems detected as a result of analyzing the collected data. Exchange 2010 and previous versions included health manifests and correlation engines in the form of management packs. These components enabled Operations Manager to make a determination as to whether a particular component was healthy or unhealthy. In addition, Operations Manager also used the diagnostic cmdlet infrastructure built into Exchange 2010 to run synthetic transactions against various aspects of the system.
Exchange 2013 takes a new approach to monitoring and preserving the end user experience natively using a feature called Managed Availability that provides built-in monitoring and recovery actions.
Manage health sets and server health
Configure managed availability overrides
Load Balancer Health URLs
Exchange 2010 required load balancers to have significant business logic, and also made them responsible for determining Exchange server health. With the addition of Managed Availability, Exchange is now able to state its health. The load balancer does not have to poke at the server to try and determine health.
Exchange 2013 has specific URLs for health checks, there is one per protocol and load balancers should configured to use these ghosted pages.
For example:
https://exch-2013.wingtiptoys.ca/owa/healthcheck.htm
https://exch-2013.wingtiptoys.ca/ecp/healthcheck.htm
https://exch-2013.wingtiptoys.ca/oab/healthcheck.htm
https://exch-2013.wingtiptoys.ca/rpc/healthcheck.htm
The below example is for the /OWA virtual directory.
The load balancer needs to be configured to parse the output from the page. If OWA is considered healthy then the response is 200 OK.
Exchange 2013 Throttling Is Not The Same As In Exchange 2010
Do not assume that the throttling framework is the same in Exchange 2010 and 2013. This is not the case. There have been multiple improvement areas in Exchange 2013.
By default, there is one default throttling policy named GlobalThrottlingPolicy_<GUID> with a throttling scope of GLOBAL. Microsoft Exchange Setup creates a default client throttling policy as part of the Client Access server role. You should not replace, re-create, or remove the existing default throttling policy. However, you can create additional throttling policies with the scope of Organization or Regular to change your user throttling settings. You can also edit policies with the scope of Organization and Regular that you've created using the Set-ThrottlingPolicy cmdlet.
For more information about how to control how resources are consumed by individual users, see Exchange workload management.
Exchange 2013 throttling functionality and deployment considerations
Whether you perform a clean installation of Exchange 2013 or install Exchange 2013 into a coexistence environment that includes Exchange 2010 computers, all users with mailboxes on computers running Exchange 2013 are throttled using the new Exchange 2013 throttling functionality. However, Exchange 2010 mailboxes will remain throttled by Exchange 2010 throttling functionality when they access their mailboxes through Exchange 2010 Client Access servers.
When Exchange 2013 is installed into a coexistence environment, the Exchange 2013 installation process may try to carry forward some of the throttling settings that you had set in your Exchange 2010 configuration. However, the Exchange 2013 throttling functionality is so different that the impact of any legacy throttling settings will generally not impact how throttling works in Exchange 2013.
Managing throttling policies by using scopes
Similar to Exchange 2010, there’s a single default throttling policy in Exchange 2013. In Exchange 2013, the default throttling policy is named GlobalThrottlingPolicy. This policy has the Global scope. The other available user throttling scopes are Organization and Regular. Due to the introduction of scope assignment for Exchange 2013 user throttling policies, you manage throttling policies differently than in Exchange 2010. The GlobalThrottlingPolicy defines the baseline default throttling settings for every new and existing user in your organization unless you have customized throttling policies for your organization. In many typical Exchange deployment scenarios, the GlobalThrottlingPolicy will be adequate to manage your users.
Replication Service Memory Use
MSExchangeRepl.exe process terminates when you try to replay more than 1,000 transaction logs in Exchange Server 2013.
See KB 2892330 for details.
Exchange 2013 Search Issues
When you search for email messages that contain a specific keyword in Microsoft Outlook in a Microsoft Exchange Server 2013 environment, Outlook returns 250 items even though you know that there are more results available. This occurs even after you click the More option at the end of the results list to load additional search items.
Exchange TLS and SSL Best Practices
Do NOT look at standard IIS documentation, instead refer to the Exchange specific guidance:
Exchange TLS & SSL Best Practices
Exchange Using Out of Site Domain Controllers
Starting with Microsoft Exchange Server 2013 Cumulative Update 6 (CU6), Exchange Server may unexpectedly use out-of-site domain controllers and global catalog servers. This is discussed in KB 3088777.
In pre-CU6 installations, when the Exchange process requests domain controllers, as long as there is a single suitable domain controller in the In-Site list, the topology service returns it and does not search the Out-of-Site list any further. This behaviour occurs regardless of how many domain controllers are requested by the client. This may cause an unbalanced load issue, especially during site failover. Domain controllers that remain in the failed-out site bear a greater load than domain controllers from outside the site.
To fix this issue, the MinSuitableServer configurable setting has been introduced. The topology service first checks whether there's a sufficient number of suitable servers in the In-Site list. If not, it adds servers from the Out-of-Site list. A similar change has also been made in topology discovery.
MinPercentageOfHealthyDC = "50"
EnableWholeForestDiscovery = "true"
ForestWideAffinityRequested = "true"/>
High Exchange Server CPU
Adding the below, as there have been a few cases around this. If you see this issue, and the below updates resolves it – please leave a comment indicating if you are doing session affinity or not on Exchange 2013.
This issue occurs because Internet Information Services (IIS) changed the method of closing the HTTP connection in Windows 8.1 or Windows Server 2012 R2
Exchange Server Authentication Prompt When Down-level Proxying
Windows 2008 R2 has an issue which may cause users to be prompted for credentials when connecting to their down-level mailbox through Exchange 2013. This may also happen when the download the OAB
Exchange Server Authentication Prompt When Down-level Proxying - Group Bloat
There is another prevalent issue which I frequently encounter. This is due to excessive security group memberships and is similar to classic Kerberos token bloat issues except this is happening to IIS. Note that the below fix is made on the down-level Exchange 2010 server and requires a server reboot to apply.
Cheers,
Rhoderick
