Exchange 2010 added multiple features to improve the resiliency of messaging services. Notable additions included client throttling to ensure that a single mailbox would not consume excessive resources and mailbox quarantine.
Mailbox quarantine is enabled by default, and the first time a lot of admins discover the feature is when a mailbox gets quarantined and the user loses access to their mailbox. In Exchange 2010 the default quarantine value is 6 hours. Think about that for a minute, if a mailbox gets quarantined at 09:00, then it will exit quarantine at 15:00. Whilst a mailbox is quarantined there is no access to the mailbox. Only by passing the open as admin flag can it be opened. The mailbox cannot be moved, indexed, opened in OWA/EAS/Outlook or anything whilst it is quarantined. Quarantined really does mean quarantined…..
Some customers may be OK with the Exchange 2010 6 hour default quarantine duration, others not so much. TechNet also states that Exchange 2013 has a 24 hour default quarantine duration.
Lets take a look at the feature to investigate what we can configure. Some other questions that we want to answer are:
-
How to configure quarantine
-
How to change the quarantine duration
-
I changed the quarantine duration, but it made no difference. Mailboxes are still quarantined for 6 hours.
-
How do I test quarantine
Defend The Flag
Exchange 2010 has a single store.exe process where all the databases are loaded, so it is imperative that this critical process is as well defended as possible. If the store were to crash or get hung up on a single thread then all mailboxes would be affected. Exchange 2013 implements multiple store.exe processes to mitigate impact. By analysing the status of mailbox threads, Exchange can determine if a single mailbox is impacting the store. It is possible that a single mailbox with corrupted data could cause store to crash or become unresponsive. If this happens repeatedly, then that would be considered a poison mailbox. As described on TechNet there are a couple of items that store considers naughty:
-
Mailbox Threads crashing
-
Stuck threads that have not progressed for an excessively long time
A mailbox that exhibits these behaviours is tagged, and a count is kept. So that this data is non-volatile and made available to multiple servers in a DAG, it is persisted in the registry. In a DAG the cluster service replicates this information via the cluster database. If a mailbox does get tagged with one of these issues you will see the entry here:
HKLMSYSTEMCurrentControlSetServicesMSExchangeIS<ServerName>Private-{dbguid}QuarantinedMailboxes{mailbox guid}
With CrashCount or LastCrashTime holding the necessary data.
The key is not created until the store has crashed at least one time by a mailbox.
The default behaviour is to quarantine a mailbox if identified as causing a failure or deadlock three times in a two hour timespan. Store tags the mailbox as quarantined in the registry and the user cannot get access to the mailbox. The only access allowed is if the Open_As_Admin flag is passed, you can do this with MFCMapi for example and take a look at the mailbox contents.
The QuarantineState and QuarantineTime registry keys are used to keep track of the quarantine status.
Mailboxes are automatically released from quarantine if quarantined for longer that the quarantine duration (MailboxQuarantineDurationInSeconds) since it’s last LastCrashTime.
If the mailbox does not cause further issues, then the registry will be cleaned up. So if there are no failures in the previous two hours and the mailbox is not currently quarantined the registry will be cleaned up.
How to Configure Mailbox Quarantine
This is where it gets a little bit interesting! There are a couple of registry keys that we want to examine:
MailboxQuarantineCrashThreshold - number of failures which cause mailbox to be quarantined. By default this is three (3).
MailboxQuarantineDurationInSeconds – amount of time a mailbox will stay quarantined. This is specified in seconds. By default the Exchange 2010 value is 21,600 (6 hours).
TechNet documents the time period for resetting quarantined mailboxes is controlled by the registry key:
HKLMSYSTEMCurrentControlSetServicesMSExchangeIS<ServerName>Private-{dbguid}MailboxQuarantineDurationInSeconds.
Unfortunately there is a lot of content out on them interwebs which state that the registry value must be created in other locations. For example a quick search suggested these locations:
HKLMSYSTEMCurrentControlSetServicesMSexchangeISParameterSystemServernamePrivate-dbguidQuarantined MailboxesMailboxQuarantineDurationInSecond
or
HKLMSYSTEMCurrentControlSetServicesMSExchangeIS<ServerName>Private-{dbguid}QuarantinedMailboxesMailboxQuarantineDurationInSecond
Note that the MailboxQuarantineDurationInSeconds value is NOT below the QuarantinedMailboxes key, it is actually above it.
How To Test Quarantine
Exchange 2013 has the Enable-MailboxQuarantine and Disable-MailboxQuarantine cmdlets which allows the admin to easily control when a mailbox is placed in and out of quarantine.
Exchange 2010 however today does not have a simple way of really testing the quarantine values. As mentioned, Exchange 2013 has the Enable-MailboxQuarantine and Disable-MailboxQuarantine cmdlets. These cmdlets are not available in Exchange 2010. This means there is no easy way to validate that the change was applied apart from waiting for the next reoccurrence.
How To Remove Mailbox From Quarantine
In Exchange 2013 the Disable-MailboxQuarantine cmdlet can be used.
For Exchange 2010, which does not have the Disable-MailboxQuarantine cmdlet we have to take a different approach. KB 2603736 states that to take a mailbox out of quarantine immediately all that we need to do is to delete the mailbox’s GUID entry from under the QuarrantinedMailboxes registry key. Store should process the registry key deletion and since the registry is the authoritative source of quarantine state the mailbox should be released. If no action is taken, then the mailbox will exit quarantine after the MailboxQuarantineDurationInSeconds period has expired.
How To Check If A Mailbox Is Quarantined
There are a few ways to look at the status of a given mailbox:
-
Event log entries
-
Get-MailboxStatistics
-
Registry
-
ExBPA
-
PerfMon
When a mailbox is quarantined, EventID 10018 will be logged into the application event log and this can be easily picked up by monitoring tools.
We can take a peek at the registry to see if there are any mailbox GUIDs listed there under:
HKLMSYSTEMCurrentControlSetServicesMSExchangeIS<Server Name>Private-{db guid}
Then run Get-Mailbox <GUID> to see which mailbox it is.
Get-MailboxStatistics for a mailbox also has a property to indicate if a mailbox is quarantined.
Get-MailboxStatistics Administrator | Select DisplayName, IsQuarantined | Format-Table -AutoSize
ExBPA will also check to see if a mailbox is quarantined.
Finally Exchange also exposes a performance monitor counter to indicate the number of quarantined mailboxes. SCOM will pick this up with the Exchange Management Pack. You can manually look at the counter -
MSExchangeIS MailboxQuarantined Mailbox Count
Cheers,
Rhoderick
