One of the issues we seem to encounter more often than not with Exchange 2010 is NTLM authentication issues. Due to the architectural changes in Exchange 2010 (with all connectivity running through the Client Access servers), a high volume of NTLM traffic is increasingly likely to create an authentication bottleneck on the Client Access Servers and Domain Controllers.
This bottleneck is due to there only being 2 MaxConcurrentAPI channels available (by default) in Windows Server 2003 or 2008 to service authentication requests. The default value of 2 (that even Windows Server 2008 R2 uses by default) is a legacy setting that has been used since Windows 2000. Naturally the MaxConcurrentAPI value of 2 was used back in the Windows 2000 days to avoid creating performance bottlenecks on the server, since the hardware capabilities back then were nowhere near what they are today. Unfortunately today this antiquated setting is not aggressive enough for all the applications (i.e. Exchange, Lync, SharePoint etc.) that may be creating additional NTLM authentication traffic to your Domain Controllers.
The most common symptom of an authentication bottleneck in Exchange 2010 is the Outlook client will receive an authentication prompt. More often than not if you enter your credentials you will be prompted again and again, until the authentication bottleneck has subsided. I typically explain it to customers as a busy 2-lane highway. The bottlenecks occur during peak times as more cars try to get on the highway from ramps and by other means. Naturally you may be waiting a while to get on the highway when it is full. For Exchange, an authentication prompt will only wait so long to get "on the highway" before it times out. Since that authentication attempt timed out, you must retry and thus you will be prompted for your authentication credentials again.
So what is the best way to fix this problem in Exchange 2010? Naturally if you have more lanes available to you then more cars can get on the highway. Imagine adding another 8 lanes to this two-lane highway. You have just significantly increased the ability to get more and more cars onto the highway and avoiding bottlenecks (and the corresponding problems that come from those bottlenecks). Windows Server 2012 now utilizes a default value of 10 for MaxConcurrentAPI. However if you don't have Windows Server 2012, we can add "more lanes" to Windows Server 2003 & 2008 by adjusting the MaxConcurrentAPI value in the registry (see KB: http://support.microsoft.com/kb/975363).
Windows Server 2003 supports a maximum MaxConcurrentAPI value of 10.
Windows Server 2008 & 2008 R2 can be set up to 10without a hotfix, and can go all the way to 150with a hotfix.
Thus we typically recommend changing the MaxConcurrentAPI value on Windows Server 2003 & 2008 to 10 on your Domain Controllers and Exchange 2010 Client Access Servers. This will address most issues with NTLM authentication bottlenecks.
If authentication issues still exist, as long as you have Windows 2008 (or 2008 R2) DCs, you can load the hotfix and set the MaxConcurrentAPI value higher as needed. Another recommendation we typically have for Exchange 2010 is enabling Kerberos authentication to help reduce the high volume of NTLM authentication in Exchange. I will discuss enabling Kerberos Authentication in another blog post.
For more information on MaxConcurrentAPI, see the following links:
You are intermittently prompted for credentials or experience time-outs when you connect to Authenticated Services: http://support.microsoft.com/kb/975363
How to do performance tuning for NTLM authentication by using the MaxConcurrentApi setting: http://support.microsoft.com/kb/2688798
Optimizing NTLM authentication flow in multi-domainenvironments: http://blogs.technet.com/b/isrpfeplat/archive/2010/11/05/optimizing-ntlm-authentication-flow-in-multi-domain-environments.aspx
NTLM’s time has passed: http://blogs.technet.com/b/authentication/archive/2006/04/07/ntlm-s-time-has-passed.aspx
NTLM Bottlenecks and the RPC runtime: http://blogs.technet.com/b/askds/archive/2011/09/15/is-this-horse-dead-yet-ntlm-bottlenecks-and-the-rpc-runtime.aspx