I came across an interesting scenario relating to the use of a Hardware Load Balancer publishing Exchange Client Access Servers. This relates specifically to how CAS distributes OAB and how Outlook decides which files it needs to perform an OAB update.
First, let's walk through how it is supposed to work starting from a new Outlook Profile (if you already know the OAB download/update process, feel free to skip to the next section):
- Outlook finds (via AutoDiscover) the url to use to connect to the OAB distribution server. It will look something like https://mail.contoso.com/OAB/<guid of OAB>/
- Outlook attempts to download the OAB.XML file at that URL. This is the "manifest" file that lists out the other files that Outlook will need to download.
- Outlook attempts to download the "template" file for whatever language Outlook is running. For English, it finds the entry in the oab.xml that specifies langid 0409. The line in the OAB.xml looks something like this:
<Template seq='7' ver='7' size='6028' uncompressedsize='26470' SHA='df80ec818f23bb6b23aa55cbc49d45986604af34' langid='0409' type='windows'>
1efdeb0a-56fe-488d-aed7-b3d9179193a2-lng0409-7.lzx - Outlook attempts to download the template file 1efdeb0a-56fe-488d-aed7-b3d9179193a2-lng0409-7.lzx
- Outlook finds the file name for the full OAB download in the manifest
<Full seq='7' ver='32' size='2408' uncompressedsize='11481' SHA='c3874445e4942eac788523db974e60a37463cabb'> 1efdeb0a-56fe-488d-aed7-b3d9179193a2-data-7.lzx </Full> - Outlook attempts to download the full OAB file 1efdeb0a-56fe-488d-aed7-b3d9179193a2-data-7.lzx
Now Outlook has an up-to-date OAB. Outlook and the Server track OAB status using a sequence number. In the above manifest, you can see the current sequence number of the OAB is 7.
Next time (say 24 hours later) Outlook goes to update OAB it goes through the following process.
- Outlook finds (via AutoDiscover) the url to use to connect to the OAB distribution server. It will look something like https://mail.contoso.com/OAB/<guid of OAB>/
- Outlook attempts to download the OAB.XML file at that URL. This is the "manifest" file that lists out the other files that Outlook will need to download.
- Outlook compares its own sequence number to that in the manifest. In our example, Outlook downloaded sequence 7. When looking at the manifest we can see that the server is up to sequence 9
<OAL id='1efdeb0a-56fe-488d-aed7-b3d9179193a2' dn='/' name='\Global Address List'>
<Full seq='9' ver='32' size='2422' uncompressedsize='11496' SHA='6852ea7fc58cfe2788766efbec61bc27cf62a96e'>
1efdeb0a-56fe-488d-aed7-b3d9179193a2-data-9.lzx
</Full> - Outlook looks in the manifest to see if there are differential files available to get from sequence 7 to sequence 9
<Diff seq='9' ver='32' size='122' uncompressedsize='11496' SHA='05eed595e0fdf88245a48d42d80431337dff36cc'>
1efdeb0a-56fe-488d-aed7-b3d9179193a2-binpatch-9.lzx
</Diff>
<Diff seq='8' ver='32' size='132' uncompressedsize='11494' SHA='4e0e62a0c3af3786310f9bd6a9b802879bcde07f'>
1efdeb0a-56fe-488d-aed7-b3d9179193a2-binpatch-8.lzx
</Diff> - They are there! Outlook grabs those 2 files and applies them in order. Outlook is now up-to-date at sequence 9.
However, if either of these items are true:
- The diff files that Outlook needed to get current aren't available (they are kept for 30 days by default)
- New profile
Outlook will fall back to a full download and follow the first procedure.
When good manifests go bad…
Now…on to the meat of the matter.
Consider this scenario:
- Client Access web based services live behind a hardware load balancer.
- Outlook decides it needs to do a full OAB download.
Outlook goes to the OAB URL and grabs the OAB.XML file. As laid out in the Full download procedure above, the first thing Outlook is going to look for is the template file.
And then…failure. If you initiated the download manually, you may see an 0x80200011 error code. You will also see an Event ID 27:
Log Name: Application
Source: Outlook
Date: 12/20/2012 1:20:01 PM
Event ID: 27
Task Category: None
Level: Warning
Keywords: Classic
User: N/A
Computer: workstation1.corp.contoso.com
Description:
OAB Download Failed. (Result code in event data).
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
<Provider Name="Outlook" />
<EventID Qualifiers="16384">27</EventID>
<Level>3</Level>
<Task>0</Task>
<Keywords>0x80000000000000</Keywords>
<TimeCreated SystemTime="2012-12-20T19:20:01.000000000Z" />
<EventRecordID>38872</EventRecordID>
<Channel>Application</Channel>
<Computer>cas.corp.contoso.com</Computer>
<Security />
</System>
<EventData>
<Data>OAB Download Failed. (Result code in event data).</Data>
<Binary>0C00000094011980081700000C17000008170000000000000000000000000000E9FD00000000000000000000784E000090500000344D0000FC2500009028000080BDF7080D5D0100000000005C000000200000000400000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000</Binary>
</EventData>
</Event>
If you review Dave Goldman's excellent blog post on this subject (http://blogs.msdn.com/b/dgoldman/archive/2006/06/19/troubleshooting-offline-address-book-downloads-with-outlook-diagnostic-logging-_2800_event-id-27_2900_.aspx ) you will see that the binary data starts with 0C which indicates that Outlooks is doing a full download. You can also see Outlook's current OAB sequence number 0817 and the Server's current sequence number 0C17. To translate these hex sequence numbers into decimal, flip the bytes. The client translates to 5896 and the server to 5900 in this example.
Having said all of that, there is little indication as to why the download failed. To attempt to see what is really going on, we can use the bitsadmin tool. The Binary Intelligent Transfer agent is the code that performs the OAB download for Outlook and it is a built-in part of Windows.
C:\>bitsadmin /list /verbose
BITSADMIN version 2.0 [ 6.6.2600.2180 ]
BITS administration utility.
(C) Copyright 2000-2004 Microsoft Corp.
Listed 0 job(s).
Well, that is not very helpful…we need to run this tool at the moment of the failure.
Let's try this batch file:
@echo off
:work
bitsadmin /list /verbose >> c:\bitsoutput.txt
goto work
Looking over the output file that was created, we see Outlook grabbing the OAB.xml file successfully and then it moves on to download the template file:
BITSADMIN version 2.0 [ 6.6.2600.2180 ]
BITS administration utility.
(C) Copyright 2000-2004 Microsoft Corp.
GUID: {D5820AE4-24C8-4C43-8B1D-728E3C16CB4E} DISPLAY: Microsoft Outlook Offline Address Book
TYPE: DOWNLOAD STATE: CONNECTING OWNER: CORP\owells
PRIORITY: NORMAL FILES: 0 / 1 BYTES: 0 / UNKNOWN
CREATION TIME: 1/8/2013 11:23:46 AM MODIFICATION TIME: 1/8/2013 11:23:47 AM
COMPLETION TIME: UNKNOWN ACL FLAGS:
NOTIFY INTERFACE: UNREGISTERED NOTIFICATION FLAGS: 3
RETRY DELAY: 600 NO PROGRESS TIMEOUT: 1209600 ERROR COUNT: 0
PROXY USAGE: PRECONFIG PROXY LIST: NULL PROXY BYPASS LIST: NULL
DESCRIPTION: Microsoft Outlook Offline Address Book Template
JOB FILES:
0 / UNKNOWN WORKING https://mail.contoso.com/OAB/9165d096-ff22-4a79-ba2d-b47d8645e220/418771d0-6766-4c27-ab27-b6068ffb56a8-lng0409-5958.lzx -> C:\Documents and Settings\owells\Local Settings\Application Data\Microsoft\Outlook\tmplts.tm_
NOTIFICATION COMMAND LINE: none
Listed 1 job(s).
Great. You can see it connecting to grab the template file for English with the sequence number 5958.
Uh oh!
BITSADMIN version 2.0 [ 6.6.2600.2180 ]
BITS administration utility.
(C) Copyright 2000-2004 Microsoft Corp.
GUID: {D5820AE4-24C8-4C43-8B1D-728E3C16CB4E} DISPLAY: Microsoft Outlook Offline Address Book
TYPE: DOWNLOAD STATE: ERROR OWNER: CORP\owells
PRIORITY: NORMAL FILES: 0 / 1 BYTES: 0 / UNKNOWN
CREATION TIME: 1/8/2013 11:23:46 AM MODIFICATION TIME: 1/8/2013 11:23:50 AM
COMPLETION TIME: UNKNOWN ACL FLAGS:
NOTIFY INTERFACE: UNREGISTERED NOTIFICATION FLAGS: 3
RETRY DELAY: 600 NO PROGRESS TIMEOUT: 1209600 ERROR COUNT: 1
PROXY USAGE: PRECONFIG PROXY LIST: NULL PROXY BYPASS LIST: NULL
ERROR FILE: https://mail.contoso.com/OAB/9165d096-ff22-4a79-ba2d-b47d8645e220/418771d0-6766-4c27-ab27-b6068ffb56a8-lng0409-5958.lzx -> C:\Documents and Settings\owells\Local Settings\Application Data\Microsoft\Outlook\tmplts.tm_
ERROR CODE: 0x80190191 - The requested resource requires user authentication.
ERROR CONTEXT: 0x00000005 - The error occurred while the remote file was being processed.
DESCRIPTION: Microsoft Outlook Offline Address Book Template
JOB FILES:
0 / UNKNOWN WORKING https://mail.contoso.com/OAB/9165d096-ff22-4a79-ba2d-b47d8645e220/418771d0-6766-4c27-ab27-b6068ffb56a8-lng0409-5958.lzx -> C:\Documents and Settings\owells\Local Settings\Application Data\Microsoft\Outlook\tmplts.tm_
NOTIFICATION COMMAND LINE: none
Listed 1 job(s).
Requires authentication??? I bet that is a bit of a misinterpretation of what happened. How about we go see what files are on the server? When we looked at the OAB directory (c:\program files\microsoft\exchange\v14\client access\oab) we see that there is a template file there…but it has a sequence number of 5961! Unlike the binpatch diff files, Exchange only keeps the latest template file. If we go looking for 5958 and the server has 5961, we will not find it. The above error really means "file not found".
Why the heck did we go looking for an old template file? Looking at the OAB.xml file on the server, it has the proper sequence number and filename for the template file.
Were we not looking at the current OAB.xml?
We tried opening OAB.xml in IE on the client machine. What the…why does this OAB.xml seem to indicate the 5958 is the current sequence number?
So, we now know what is not working…we can't go looking for old sequences…that won't work.
But, how did this happen?
The Answer (at long last):
The default configuration on many hardware load balancers is to cache some http content. We were looking at a cached copy of the oab.xml! Thanks for the help, Mr. Load Balancer!
The Solution:
Turn off http caching on the load balancer. You can either disable it for all Exchange Web services, the OAB vdir only, or just the OAB.xml file depending on your preference and your load balancers capabilities.