Exchange 2007 Outlook Anywhere White Paper

This is one of those older documents, that for many reasons I seem to keep coming back to over the years.  Even as we transition away from classic Outlook Anywhere to MAPI/HTTP, knowing some of the features/limitations of Outlook Anywhere and why MAPI/HTTP is a better replacement is something that will be around for a long time to come.

Since TechNet is dead, even though I know exactly what I'm looking for it has become harder and harder to find.  This comment applies to all of the search engines.  So to make life a little easier, well at least for me,  let's park a copy of the article here for future reference.

The original link to the Outlook Anywhere White Paper is:

https://docs.microsoft.com/en-us/previous-versions/office/exchange-server-2007-technical-articles/cc540453(v=exchg.80)

 

Single PDF

A print to PDF was taken, and the file is located on this blog.

Exchange 2007 Outlook Anywhere White Paper

If you wish to simply ready the original page content, it is included below in the next section.

 

 

Original Webpage Content

 

White Paper: Outlook Anywhere Scalability with Outlook 2007, Outlook 2003, and Exchange 2007

• 07/23/2014
• 20 minutes to read

Tom Di Nardo, Senior Technical Writer, Microsoft Exchange Server

May 2008

Summary

This white paper provides an analysis of the scalability of the Outlook Anywhere feature for Microsoft Exchange Server 2007, Microsoft Office Outlook 2007, and Microsoft Office Outlook 2003, and an analysis of expected client network traffic between enterprise e-mail clients and Exchange Server 2007 SP1 in non-Outlook Anywhere scenarios.

 

Applies To

Microsoft Exchange Server 2007

Table of Contents

• Introduction
• Outlook Connections
• TCP Protocol Connection Limit
• Outlook Anywhere Path
• Port Exhaustion Causes Scalability Limitations
• Mitigating Port Exhaustion Scalability Limitations
• Windows Server 2008 and Multiple IP Addresses
• Refer Outlook Directly to Global Catalog Servers
• Worker Process Recycling May Cause Performance Issues
• Mitigating Process Recycling Issues
• LoadGen Does Not Simulate DSProxy Connections
• Client Network Traffic
• Conclusion
• Additional Information

Introduction

Outlook Anywhere scalability and client network traffic are two areas that generate many questions about the number of connections Outlook makes and sustains to an Exchange server. This area is frequently the subject of discussion when site consolidation is being discussed which also raises the issues of network costs and Transmission Control Protocol (TCP) connection limits. The TCP connection limitations are largely hit by hosting companies and large enterprise customers who force all MAPI connectivity through RPC over HTTP (RPC/HTTP). In the following sections we will cover each of these areas in detail to help show the behavior you can expect to see when using Outlook Anywhere in your Exchange 2007 deployment.

Outlook Connections

Because of many variables that can exist in an Exchange 2007 environment, it is difficult to provide a solid number of client connections for all possible variables. The actual number of connections that will be seen in a non-default Exchange 2007 environment can vary based on using ISA Server, public folders, Outlook Add-ins, and so on. Outlook connections can also vary based on the features or usage patterns of the client, including accessing shared calendars, public folder usage, or offline address books. Because of these variables, it is most useful to provide information about the connection values that will be seen in a default Exchange 2007 installation. Keep in mind that a larger number of connections will be seen during initial logon. After a minute you will see the number of connections reach a steady state. It is important to be aware that these startup connections are not accounted for in the illustrations below. This behavior is seen in both Outlook 2003 and Outlook 2007. It is impossible to predict exactly how many connections will be used because of the previously mentioned variables. However, an increase of between 25 and 50 percent at startup has been regularly observed.

The numbers provided in this topic were collected by running TCPView on a default installation of Exchange 2007. This includes a server that has the Mailbox server role installed, a server that has the Client Access server role installed, Windows Server 2003 Active Directory, and default Outlook 2003 and Outlook 2007 clients. For more information about TCPView, see TCPView for Windows.

The illustration below details how these connections look from inside the firewall on the corporate network (inside firewall):

The illustration below details how these connections look from an Outlook Anywhere perspective when coming into the corporate network via RPC over HTTP (outside firewall):

TCP Protocol Connection Limit

The TCP protocol has a requirement that each connection have a unique ordered list, also known as an n-tuple, which consists of source address, source port, destination address, and destination port. All incoming connections use the same destination address or port, so the number of incoming connections is limited by the non-paged pool size. Each outgoing connection consumes a port on an address. The TCP port is a 16-bit number, so there are at most 65,535 ports.

The change to 64-bit hardware in Exchange 2007 exposed this scalability limit. In Exchange 2003, the memory constraints in 32-bit hardware hide this limit and because of those memory constraints, memory availability would be exhausted before the TCP connection limit could be reached. Now, with 64-bit hardware and an almost endless amount of memory, Exchange is no longer limited in this area and can therefore hit the TCP connection limitation. Usually, this affects enterprise customers who are running at very high scale and who are trying to maximize as much scale-up from their hardware as possible.

 

Outlook Anywhere Path

RPC/HTTP is a tunneling protocol where Exchange uses a pair of virtual channels to create a virtual connection from Outlook to Exchange. Each virtual channel is a single directional data stream that is transported over various real channels. The client to RPC Proxy channel is HTTP/HTTPS and the RPC Proxy to Exchange channel is TCP. The client then establishes four channels. Data flows on these as follows:

1. Client to Proxy
2. Proxy to Exchange
3. Exchange to Proxy
4. Proxy to Client

Once all four channels are established, RPC then treats this as a single full duplex tunneled connection from Outlook to Exchange. Each real channel can be replaced without interrupting the data flow over the virtual connection.

Exchange has two kinds of connections, mail and directory. Each of these connections will appear as a pair of virtual channels. Mail connections flow from Outlook to the RPC Proxy component on the Client Access server to the Mailbox server. In deployments were Internet Security and Acceleration (ISA) Server is used, ISA Server will proxy these connections to the Client Access server (RPC/HTTP Proxy). Because ISA Server is still a 32-bit application, it will be unable to scale the TCP connections to the physical connections limit before it runs out of available non-paged pool memory. Non-paged pool memory is used for managing the high number of connections established. This limit will be reached before any Exchange limits are reached. The testing documented here does not deal with this issue. However, it is an important consideration for any real-world deployment. Exchange then uses its data store to service the requests and replies to the client. The directory connections flow from Outlook to the RPC Proxy component on the Client Access server to the DS Proxy component on the Mailbox server to an Active Directory global catalog server. The RPC connection is processed on the DC (not on the Exchange server), with the DS Proxy component merely copying bytes from one TCP connection to the other. The large number of outbound connections from Exchange to the DC is a function of the DS Proxy component that tunnels connections.

The TCP connections limits discussed earlier in this topic exist in both Windows Server 2003 and Windows Server 2008 as consumers of the TCP protocol. One IP address is used as the source IP address when it opens a connection to a remote computer. Each Client Access server is bound by the Windows port limit of 65,535 available ports. The Client Access server depletes the available pool of ports as each client uses anywhere between 2 and 8 connections. The information store process has a hard limit of 60,000 RPC context handles which are associated with each RPC/HTTP virtual connection between Outlook and Exchange. Therefore, the store process is limited to 60,000 of these mail connections.

The following performance counters are helpful in determining whether a server is reaching this limit:

RPC/HTTP Proxy (Windows Server 2008 Only)

• Current Number of Incoming RPC over HTTP Connections
• Current number of unique users
• RPC/HTTP requests per second
• Number of Failed Back-End Connection attempts per Second

MSExchangeIS

• RPC Averaged Latency
• RPC Requests
• RPC Operations/Sec

Web Service (Windows Server 2003 Only)

• Current Connections

Note

These counters are only useful on servers where no other Web service is used.

Memory

• Pool Nonpaged Bytes
• Pool Paged Bytes

Process

• Private bytes / LSASS, W3WP and any Exchange-specific processes running

The current number of incoming RPC/HTTP connections and current number of unique users counters that are available with Windows Server 2008 will determine how many user connections there are and how many different NT user accounts are connected. The other counters will help determine potential causes of the denial of new user connections to a server and how the server is failing.

 

Port Exhaustion with Outlook Anywhere Causes Scalability Limitations

The key discovery and conclusion is that a Mailbox server will deplete outbound source IP ports far more quickly than it will reach any inbound limit. This occurs because of the way that DSProxy operates. DSProxy opens a single outbound connection for every inbound connection it receives. For every inbound connection to DSProxy, the Mailbox server opens an equivalent number of outbound connections to a global catalog.

Other related observations are:

• Clients do not share connections. New connections are established for every new client connecting.
• Connections are removed as soon as a client logs off.
• If a client opens a mailbox on a Mailbox server other than the one hosting their mailbox, or views a calendar or folder on another Mailbox server, additional TCP connections are established from the Client Access server to that Mailbox server.
• If multiple mailboxes or calendars are viewed on a Mailbox server other than the one hosting their mailbox, no additional connections are created beyond those established for the first mailbox or calendar viewed.
• Because ISA Server is still a 32-bit application, it will be unable to scale the TCP connections to the physical connections limit before it runs out of available non-paged pool memory. Non-paged pool memory is used for managing the high number of connections established. This limit will be reached before any Exchange limits are reached. The testing documented here does not deal with this issue, but it is an important consideration for any real-world deployment.

 

Mitigating Port Exhaustion Scalability Limitations

There are two possible ways to mitigate the issue of port exhaustion: by using Windows Server 2008 with multiple IP addresses, reverting to Exchange 2003 RTM behavior and scaling out the Client Access server deployment by adding additional Client Access servers to service client connections.

 

Windows Server 2008 and Multiple IP Addresses

A registry key was created in Windows Server 2008 which allows the server to use 65,535 outbound connections for each IP assigned to the system, even for multiple IPs assigned to the same network adapter. This feature will allow some additional headroom but does not address the other limits, such as the store connection limit. It should also be noted that each RPC/HTTP virtual connection consumes 61 KB of RAM on the Client Access server so that a server that will be using many TCP connections must be configured with sufficient RAM to manage the connections and also the load Exchange puts on it. If you do not plan for this, you will be encountering issues related to memory pressure which can cause the server to thrash (continuously page). For information about implementing the registry change detailed here, see Microsoft Knowledge Base article, How to enable the port scalability feature for RPC proxies and for applications in Windows Server 2008.

 

Refer Outlook Directly to Global Catalog Servers

Another method that is available for mitigating the issue of port exhaustion is to refer Outlook directly to global catalog servers and scale out your Client Access server deployment by adding additional Client Access servers to service the client connection load.

Important

This change should only be considered in exceptional circumstances. This change presents the problem of requiring the manual configuration of all possible global catalogs into the registry of every Client Access server. The supportability issues of this change should be fully understood before you implement it in a production environment.

The following change must be made to enable this configuration for the Mailbox server:

The Do Not Refer HTTP to DSProxy key must be set as detailed in How to control the DSProxy process for RPC over HTTP connections in Exchange Server 2003 SP1.

The following changes must be made to enable this configuration for the Client Access server:

Use the following procedure to modify the ValidPorts setting in registry key HKLM\Software\Microsoft\RPC\RPCProxy so that the entries referring to 6004 point to every available global catalog in addition to a Mailbox server. These entries must exist for both the fully qualified domain name (FQDN) and the short NETBIOS name of every available global catalog.

Procedures

Warning: Incorrectly editing the registry can cause serious problems that may require you to reinstall your operating system. Problems resulting from editing the registry incorrectly may not be able to be resolved. Before editing the registry, back up any valuable data.

Use the following procedure to modify the PeriodicPollingMinutes value in the following registry subkey:

HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\MSExchangeServiceHost\RpcHttpConfigurator\

To modify this value, set it to 0 to prevent the Microsoft Exchange Service Host service from updating the ValidPorts subkey automatically.

Use Registry Editor to modify the PeriodicPollingMinutes value

1. On the Exchange server that has the Client Access server role installed, open Registry Editor.
2. Browse to: HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\MSExchangeServiceHost\RpcHttpConfigurator\
3. Right-click PeriodicPollingMinutes and then click Modify.
4. In the Value data field, enter a value of "0" without quotation marks.
5. Close Registry Editor.
6. Restart the Microsoft Exchange Service Host service for changes to take effect.

Use the following procedure to modify the ValidPorts value in the registry.

Use Registry Editor to modify the ValidPorts value

1. On the Exchange server that has the Client Access server role installed, open a Registry Editor.
2. Browse to: HKLM\Software\Microsoft\RPC\RPCProxy.
3. Right-click ValidPorts and then click Modify.
4. In the Value data field, enter the FQDN and NETBIOS name for every available global catalog in the following format: GC01:6004;gc01.contoso.com:6004.
5. Close Registry Editor.
6. The new setting will take effect in approximately five minutes.

Conclusion

Note: IIS reads the Enabled and ValidPorts registry entries on startup. In addition, RPC over HTTP rereads the contents of the ValidPorts key approximately every five minutes. If the ValidPorts entry is changed, the changes are implemented within five minutes.

The following changes must be made to enable this configuration for the global catalog server:

Use the following procedure to create a Multi-String Value entry named NSPI protocol interface sequences in the registry key HKLM\System\CCS\Services\NTDS\Parameters\ on each global catalog server and set its value to ncacn_http:6004.

Use Registry Editor to create the NSPI interface protocol sequences Multi-String Value entry

1. On the Exchange server that has the Client Access server role installed, open Registry Editor.
2. Browse to: HKLM\System\CCS\Services\NTDS\Parameters\.
3. Right-click in the action pane, select New\Multi-StringValue, and enter the name NSPI interface protocol sequences.
4. Right-click NSPI interface protocol sequences and then click Modify.
5. In the Value data field, enter a value of "ncacn_http:6004" without quotation marks.
6. Close Registry Editor.
7. Restart the server for changes to take effect.

 

 

 

Worker Process Recycling May Cause Performance Issues

RPC over HTTP runs in the Default Application Pool (DefaultAppPool) in Internet Information Services (IIS). By default, this application pool is configured to recycle worker processes every 29 hours. During the recycling process, IIS allows active worker threads an additional 90 seconds to finish servicing requests before IIS terminates the active threads.

Because RPC over HTTP uses long-running connections, the connections may not finish within the additional 90 seconds that are given to the worker threads. In this scenario, the connections are terminated, which causes Outlook to lose connectivity with IIS. When this action occurs, Outlook immediately tries to reconnect. If many Outlook clients are disconnected at the same time, the large number of simultaneous reconnections may overwhelm the server.

Mitigating Process Recycling Issues

To mitigate any performance issue that may occur because of worker process recycling, configure the following items in IIS:

• If practical, move the RPC over HTTP component into its own application pool.
• Turn off worker process recycling on application pools in which RPC over HTTP is configured.
• Increase the HTTP.sys queue limit from the default value of 1,000 to 10,000.

 

Procedures

Use Internet Information Services Manager to move the RPC over HTTP component to a new application pool in IIS 6.0

1. Start Internet Information Services Manager.
2. Expand the local computer, right-click Application Pools, point to New, and then click Application Pool.
3. In the Add New Application Pool dialog box, type a descriptive name such as MSExchangeOutlookAnywhere, click Use existing application pool as template, click DefaultAppPool in the Application pool name list, and then click OK.
4. Expand Web Sites, expand the Web site in which the Rpc Web application is located. For example, expand Default Web Site. Right-click Rpc, and then click Properties.
5. On the Virtual Directory tab, click the new application pool in the Application pool list. For example, click MSExchangeOutlookAnywhere.
6. Click OK.

Use Internet Information Services Manager to turn off worker process recycling in IIS 6.0

1. Start Internet Information Services (IIS) Manager.
2. Expand the local computer, expand Application Pools, right-click the appropriate application pool, such as DefaultAppPool or the new application pool that you created, and then click Properties.
3. Click to clear the Recycle worker processes (in minutes) check box, and then click OK.

Use Internet Information Services Manager to increase the queue length in IIS 6.0

1. Start Internet Information Services (IIS) Manager.
2. Expand the local computer, expand Application Pools, right-click the appropriate application pool, such as DefaultAppPool or the new application pool that you created, and then click Properties.
3. Click the Performance tab, and then modify the value in the Request queue limit box. Replace the default value of 1000 with 10000.
4. Click OK.

Use Internet Information Services Manager to move the RPC over HTTP component to a new application pool in IIS 7.0

1. Start Internet Information Services Manager.
2. Expand the local computer, click Application Pools, and then click Add Application Pool.
3. In the Name box, type a descriptive name, such as MSExchangeOutlookAnywhere, and then click OK.
4. In the Connections pane, expand Sites, expand the Web site in which the Rpc Web application is located. For example, expand Default Web Site Click Rpc, and then click Advanced Settings.
5. Note any settings that appear in the Advanced Settings dialog box.
6. Under General, click the ellipsis (…) button that appears next to DefaultAppPool.
7. In the Application pool list, click the new application pool that you created, and then click OK two times.

Use Internet Information Services Manager to turn off worker process recycling in IIS 7.0

1. Start Internet Information Services Manager.
2. Expand the local computer, and then click Application Pools.
3. In the Application Pools pane, click the appropriate application pool, such as DefaultAppPool or the new application pool that you created, and then click Advanced Settings.
4. In the Recycling section, modify the Regular Time Interval (minutes) value. Replace the default value of 1740 with 0 (zero). A value of zero turns off worker process recycling.
5. Click OK.

Use Internet Information Services Manager to increase the queue length in IIS 7.0

1. Start Internet Information Services Manager.
2. Expand the local computer, and then click Application Pools.
3. In the Application Pools pane, click the appropriate application pool, such as DefaultAppPool or the new application pool that you created, and then click Advanced Settings.
4. In the General section, modify the Queue Length value. Replace the default value of 1000 with 10000.
5. Click OK.

 

 

LoadGen Does Not Simulate DSProxy Connections

The Microsoft Exchange Load Generator (LoadGen) tool does not simulate any DSProxy connections. The affect of this is not significant from a performance perspective, but it is significant from a scale testing perspective. Customers who use LoadGen to simulate Outlook Anywhere users will not hit the outbound ports scalability issue described earlier in this topic. This will result in a test where a much larger number of users will be able to connect to Exchange by using Outlook Anywhere than would be able to do this in a production environment. The load on the server is believed to be minimal, but the missing DSProxy connections will allow the server to support far more clients during LoadGen testing than it would allow in a production environment.

The LoadGen Tool team is investigating adding support for directory connections in a future release of the LoadGen tool. Until LoadGen is updated to reflect these connections, it is critical that scalability testing of Outlook Anywhere with LoadGen not be used exclusively to determine the maximum number of concurrent users that a server can support.

 

 

 

Client Network Traffic

As part of the Outlook Anywhere scalability testing, analysis was done to determine the network costs between enterprise e-mail clients and Exchange 2007 SP1. The values presented here may help an organization determine an estimated value for the network use requirements that are part of connecting end-users to the Exchange 2007 infrastructure. The testing performed in this analysis included the following scenarios: Outlook 2007 online mode; Outlook 2007 cached mode; Outlook 2007 cached mode through RPC/HTTP (Outlook Anywhere) and Outlook Web Access. No reporting on the network bytes passed between Exchange roles was performed. This analysis is limited to the bytes entering and leaving the datacenter. Outlook Anywhere and Outlook Web Access connect to the Exchange servers with the Client Access server role installed, while Outlook 2007 (in both online and cached mode) connects directly to the Exchange servers that have the Mailbox server role installed. The network traffic from previous Outlook versions can be estimated from the Exchange 2003 results that are published in the Client Network Traffic with Exchange Server 2003 white paper because there have not been fundamental changes in Exchange-Outlook communications in the 2007 releases.

The user profile started with the message send and delivery rates from the "light", "medium", "heavy" and "very heavy" knowledge worker profiles. The following assumptions were made for the purposes of these tests:

• An average message size of 50 KB
• Every message delivered was read
• Half of all incoming mail was deleted
• Web clients logged on and logged off two times per day
• Logon and logoff costs from the other client types were not evaluated because enterprise e-mail users generally stay logged on for days at a time.

 

 

 

The network bytes transferred for each action is independent of mailbox size, so separate measurements for each profile were not performed, but measurement of the costs of the actions were made and totaled for each profile.

Note

For Outlook 2007 in cached mode and Outlook Anywhere, which work from a local copy of the user mailbox, there is insignificant traffic associated with reading or deleting mail because these actions work against the local copy. However, every e-mail received is downloaded to the client.

In the following table, all values are in kilobytes per day per user. The sending portion has been separated from the other actions, which are labeled as 'aggregate'.

 

 

 

 

To better understand how these values can be used, consider the following example:

Suppose a datacenter has 10,000 "Medium" Outlook cached mode users. Further assume these users are in the same time zone and they perform most of their work during the same 8-hours of the day. The graphic here predicts what the average network bytes per second would be.

or

Assuming a daily peak of twice the average value, the network coming into the datacenter would have to support approximately 15 megabits per second from these users alone.

If these users were running in online mode, per-user bandwidth consumption value would be replaced as shown in the following formula:

or

Assuming a daily peak of twice the average value, the network coming into the datacenter would have to support approximately 30 megabits per second from these users.

 

 

Conclusion

By using the information that is provided here, you can start to evaluate how to properly size your Outlook Anywhere deployment and the network utilization requirements for your Exchange 2007 environment.

Additional Information

For the complete Exchange 2007 documentation set, see the following resources:

• Exchange Server 2007 Help
• Exchange Server TechCenter