How to Enable Logging for RPC Client Access Throttling in Exchange 2010

Thursday, July 12, 2012
Throttling is a resource protection feature introduced with Exchange 2010.  It is designed to prevent a single user or groups of users from consuming all the Exchange resources and causing a denial of service (DoS) attack.

Users will see various warnings and errors in Outlook when RPC throttling occurs.  Two of the most common warnings and errors in Outlook are shown below:

Unable to open your default e-mail folders. The Microsoft Exchange Server computer is not available. Either there are network problems or the Microsoft Exchange Server computer is down for maintenance.

Unable to expand the folder. The set of folders could not be opened.

By default RPC throttling is not logged anywhere, which makes it very difficult to troubleshoot.  Without logging you normally need to load up all the Perfmon counters and watching them increment.  This does nothing to tell you who is being throttled, though.

You can enable logging for RPC throttling by configuring the Microsoft.Exchange.RpcClientAccess.Service.exe.config file.  This file is located in the \Program Files\Microsoft\Exchange Server\V14\Bin folder on the Client Access Servers.

Open the config file in Notepad and edit the LoggingTag tag key to add the Throttling value as follows:

<add key="LoggingTag" value="ConnectDisconnect, Logon, Failures, ApplicationData, Warnings, Throttling" />


Save the Microsoft.Exchange.RpcClientAccess.Service.exe.config file and restart the Microsoft Exchange RPC Client Access service on the CAS.  This needs to be done on all CAS servers.

Meaningful RPC throttling events are then logged in the \Program Files\Microsoft\Exchange Server\V14\Logging\RPC Client Access folder.  Open the latest log file to search for RPC throttling events.  They usually include the term "exceeded":
2012-06-26T17:32:23.301Z,19,0,/o=theguillets/ou=Exchange Administrative Group (FYDIBOHF23SPDLT)/cn=Recipients/cn=Jeff Guillet,,OUTLOOK.EXE,14.0.6117.5001,Cached,192.168.1.5,192.168.1.30,ncacn_http,,Connect,2614 (rpc::MaxConnectionsExceeded),00:00:00,"SID=S-1-5-21-117020884-2285600563-2343042490-1113, Flags=None; Connection Limit Exceeded",RpcDispatch:

You can adjust throttling using client throttling policies.  Throttling policies are groups of settings that are used to control how much resources that a user or connection can use in an Exchange organization. Throttling polices can only be used against users that are using Exchange 2010 servers. They do not apply to previous versions of Exchange.  See the TechNet article Understanding Client Throttling Policies (http://technet.microsoft.com/en-us/library/dd297964) for more information.

3 comments:

  1. Hi Jeff, I have in the RCA log file the following message:

    2013-01-23T07:53:14.080Z,100,2801,/o=XXX/ou=Exchange Administrative Group (FYDIBOHF23SPDLT)/cn=Recipients/cn=Angelika dd1,,OUTLOOK.EXE,14.0.6126.5000,Classic,,,ncacn_ip_tcp,,,0,00:00:00,"BS=Conn:0,HangingConn:0,AD:3000/3000/0%,CAS:123000/123000/0%,AB:3000/3000/0%,RPC:120000/120000/0%,FC:1000/0,Policy:DefaultThrottlingPolicy_7ed58c68-57f5-4e40-92a0-08adfe168c0b,Norm[Resources:(Mdb)Mailbox Database (Health:-1%,HistLoad:0),(Mdb)Public Folder Database (Health:-1%,HistLoad:0),(DC)SERVER.****(Health:-1%,HistLoad:0),(DC)SERVER****(Health:-1%,HistLoad:0),];GC:137/43/2;;BudgetExpired;",

    Have you ever seen the message "BudgetExperied"?

    Regards Henning

    ReplyDelete
    Replies
    1. Hi Henning,

      According to the Exchange product group:

      "It’s part of the built-in internal throttling mechanism. BudgetExpired means that the budget wrapper outlived the underlying budget so that at some point in the call, access to the wrapper caused us to rehydrate the budget from the cache. It really isn’t something to be concerned about, though typically it suggests a really long call for the wrapper to outlive the budget. It’s there for diagnostics purposes so that if there was something funky going on, we would be aware that maybe the budget instance swap had something to do with it."

      Thanks to Scott Schnoll for the information above.

      Delete
  2. Hi Jeff,

    I am ingesting messages into exchange 2010 personal archives (via MAPI connection) and found your article around logging the activities. I haven't found any maximum connections are MAPI, but I too am seeing the 'BudgetExpired' in the logs and am trying to understand if my ingestion into Exchange is being throttled (expected ingestion rate is much slower than tested in labs, but production infrastructure is MUCH better).

    I have checked the exchange policies for the environment and service accounts and there is nothing that stands out, most policies that i am interested in is set as $null.

    I am also seeing : "CAS:$null/$null/94%" and i am trying to figure out if this means that the CAS server utilisation level is 95%....

    ReplyDelete

Thank you for your comment! It is my hope that you find the information here useful. Let others know if this post helped you out, or if you have a comment or further information.