When To Move Arbitration Mailboxes

The concept of arbitration mailboxes is not really new, they were first introduced in Exchange 2010.  Arbitration mailboxes are used for several special tasks within Exchange and for the most part will tick away quite happily with minimal administrator intervention.  System mailboxes have existed for much longer in Exchange, and again perform special tasks which are required by Exchange.

A lot of admins first discovered arbitration mailboxes when they ran into an issue trying to removed the default Exchange 2010 mailbox database on the first Exchange 2010 Mailbox server.  This is because the arbitration mailboxes were created in that database, and the UI does not display arbitration mailboxes, you need to use PowerShell.

The other issue which happens is that they are very often overlooked in subsequent migrations.  For various reasons arbitration mailboxes are not moved when they should.  The below is an example of what I often discover.  Note there are two versions of Exchange in this lab: 2010 SP3 RU15 and 2013 CU13.

The CAS namespaces have all been cut over and mailboxes moved to Exchange 2013.

PowerShell was used to see the arbitration mailboxes.  Specifically we need the –Arbitration parameter in Get-Mailbox.

If we look to see which version of Exchange hosts the arbitration mailboxes, note that there are two versions .  Only the bottom two ones, which are highlighted are on an Exchange 2013 server.  All of the others are still on Exchange 2010.  We can tell by looking at the mailbox’s AdminDisplayVersion, which is the version of Exchange they are located on.

Get-Mailbox –Arbitration | FL Name, Database, AdminDisplayVersion
Get-Mailbox -Filter {RecipientTypeDetails -eq "DiscoveryMailbox"} | FL name,ProhibitSendQuota, AdminDisplayVersion

Note that only the arbitration mailboxes in the highlighted yellow rectangle are on Exchange 2013.  All of the others are on Exchange 2010.  The DiscoverySearchMailbox is also on Exchange 2010,

Why are arbitration and system mailboxes on two different Exchange Versions?

Who Made Who

<courtesy link to AC/DC>

Good question!  When Exchange 2010 was first installed, the initial arbitration mailboxes were created on that first Exchange 2010 mailbox server.  Specifically they were created with the /PrepareAD command.  Exchange 2013 has more arbitration mailboxes compared to Exchange 2010, and when the first Exchange 2013 server was introduced the net-new Exchange 2013 arbitration mailboxes were created on Exchange 2013.  This is why we see the split.

For a more detailed post on where the arbitration mailboxes were first introduced by a particular version of Exchange please see Arbitration Mailboxes – Lay of The Land.

In Exchange 2010 there was no great way to determine what each arbitration mailbox was used for apart from looking up the GUID of the mailbox name.  These IDs are documented in the article Re-create Discovery and Other System Mailboxes in Exchange 2010.  Note that there are three arbitration mailboxes listed and the DiscoverySearchMailbox.

• SystemMailbox{e0dc1c29-89c3-4034-b678-e6c29d823ed9}    – This account is used by the Multi-Mailbox Search process to store discovery search metadata.
• FederatedEmail.4c1f4d8b-8179-4148-93bf-00a95fa1e042       – This account is used for federated mail.
• SystemMailbox{1f05a927-af78-475a-aba4-fc281398eb54}      – This account is used for moderated transport.

• DiscoverySearchMailbox{D919BA05-46A6-415f-80AD-7E09334BB852} – This account is used as the target mailbox for discovery searches in the exchange control panel. Discovery search is used when you search for emails across all the mailboxes.

Custom greetings, announcements, menus, and prompts are used by Unified Messaging dial plans and auto attendants. The system mailbox named {e0dc1c29-89c3-4034-b678-e6c29d823ed9} is created when you install Exchange 2010, 2013 or 2016 and is used to support features such as Message Approval and Multi-Mailbox Search. This system mailbox is also used to store dial plan and auto attendant custom greetings, announcements, menus, and prompts.  This is discussed in Upgrade Exchange 2010 UM to Exchange 2013 UM.

This screenshot is from an Exchange 2010 only environment, note the three arbitration mailboxes and the one DiscoverySearchMailbox.  The names match with the four mailboxes listed above.

Having to remember/lookup what each arbitration mailbox does is a pain.  In Exchange 2013 onwards a new attribute was added – PersistedCapabilities.  This provides insight into what purposes the arbitration mailbox serves.  In the below we can see this attributed added to the command being executed in the Exchange 2013 Management Shell:

Get-Mailbox -Arbitration | FL Name, Database, AdminDisplayVersion, PersistedCapabilities

For make benefit, most glorious search engines the pertinent persisted capabilities data is added below:

Name                  : SystemMailbox{e0dc1c29-89c3-4034-b678-e6c29d823ed9}
Database              : Mailbox Database 1880456259
AdminDisplayVersion   : Version 14.3 (Build 123.4)
PersistedCapabilities : {OrganizationCapabilityUMDataStorage}

Name                  : SystemMailbox{bb558c35-97f1-4cb9-8ff7-d53741dc928c}
Database              : Mailbox Database 1674805942
AdminDisplayVersion   : Version 15.0 (Build 1210.3)
PersistedCapabilities : {OrganizationCapabilityUMGrammarReady, OrganizationCapabilityPstProvider,
OrganizationCapabilityMessageTracking, OrganizationCapabilityMailRouting,
OrganizationCapabilityClientExtensions, OrganizationCapabilityGMGen,
OrganizationCapabilityOABGen, OrganizationCapabilityUMGrammar}

Name                  : Migration.8f3e7716-2011-43e4-96b1-aba62d229136
Database              : Mailbox Database 1674805942
AdminDisplayVersion   : Version 15.0 (Build 1210.3)
PersistedCapabilities : {OrganizationCapabilityManagement}

When To Move

The above discussion unpacks the history and reason why arbitration mailboxes are often found in the above situation.  However this should not be the case.

Arbitration mailboxes must be moved at the beginning of the project, before moving user mailboxes onto the new version of Exchange.  It is expected and required that the arbitration mailboxes are moved to the latest version of exchange unless a specific article discusses an issue.

If you fail to do this simple step, then big issues are heading your way.  They include:

• Administrator actions are not saved to the administrator audit log
• Cannot run E-Discovery searches
• Unified Messaging prompts may not function as expected

In Exchange 2010, the Microsoft Exchange system mailbox is an arbitration mailbox used to store organization-wide data such as administrator audit logs, metadata for eDiscovery searches, and Unified Messaging data, such as menus, dial plans, and custom greetings. The Microsoft Exchange system mailbox is named SystemMailbox{e0dc1c29-89c3-4034-b678-e6c29d823ed9}; the display name is Microsoft Exchange.

When you upgrade your existing Exchange 2010 organisation to Exchange 2013, you have to move the Microsoft Exchange system mailbox to a mailbox database on an Exchange 2013 Mailbox server. You should move this mailbox after you’ve installed and verified Exchange 2013. If you don’t move this system mailbox to Exchange 2013, the following issues will occur when Exchange 2010 and Exchange 2013 coexist in your Exchange organization:

• Exchange 2013 tasks aren’t saved to the administrator audit log. When you run the Search-AdminAuditLog cmdlet or try to export the administrator audit log in the EAC, you’ll receive an error that says you can’t create an administrator audit log search because the system mailbox, SystemMailbox{e0dc1c29-89c3-4034-b678-e6c29d823ed9}, is located on a server that isn’t running Exchange 2013. A Microsoft Exchange error with an Event ID of 5000 is also logged in the Windows Application log each time a command is run.

• You can’t run eDiscovery searches using the EAC or the Shell in Exchange 2013. Mailbox searches can be created and queued, but they can’t be started. An error with an Event ID of 6 is logged in the MsExchange Management log, stating that the Start-MailboxSearch cmdlet failed. However, you can search mailboxes using the Shell and the Exchange Control Panel (ECP) in Exchange 2010.

This is from Move the Exchange 2010 system mailbox to Exchange 2013

In-Place eDiscovery uses a system mailbox to store In-Place eDiscovery search metadata. This Discovery system mailbox has the display name SystemMailbox{e0dc1c29-89c3-4034-b678-e6c29d823ed9}. Because system mailboxes aren't visible in the Exchange Administration Center (EAC) or in Exchange address lists, they are rarely deleted inadvertently.

However, if the Discovery system mailbox is deleted accidentally, discovery managers will be unable to perform In-Place eDiscovery searches or manage existing searches. In this case, to enable eDiscovery functionality, you must re-create the Discovery system mailbox.

This is from Re-create the Discovery system mailbox

Separate Issue

In case additional arbitration mailboxes were created, you may need to then also discover who is configured to leverage the non-default ones.

An arbitration mailbox can be used to handle the approval workflow for moderated recipients and distribution group membership approvals. You use PowerShell to find all the recipients that are configured to use the arbitration mailbox. After you identify the recipients, you can either configure them to use a different arbitration mailbox, or you can disable moderation for them.

 
$AM = Get-Mailbox "<arbitration mailbox>" -Arbitration
$AMDN = $AM.DistinguishedName
Get-Recipient -RecipientPreviewFilter "ArbitrationMailbox -eq '$AMDN'"

This is from Manage and troubleshoot message approval

Cheers,

Rhoderick