Remove Lingering Objects that cause AD Replication error 8606 and friends

Introducing the Lingering Object Liquidator

Hi all, Justin Turner here ---it's been a while since my last update. The goal of this post is to discuss what causes lingering objects and show you how to download, and then use the new GUI-based Lingering Object Liquidator (LOL) tool to remove them. This is a beta version of the tool, and it is currently not yet optimized for use in large Active Directory environments.

This is a long article with lots of background and screen shots, so plug-in or connect to a fast connection when viewing the full entry. The bottom of this post contains a link to my AD replication troubleshooting TechNet lab for those that want to get their hands dirty with the joy that comes with finding and fixing AD replication errors.

Overview of Lingering Objects

Lingering objects are objects in AD than have been created, replicated, deleted, and then garbage collected on at least the DC that originated the deletion but still exist as live objects on one or more DCs in the same forest. Lingering object removal has traditionally required lengthy cleanup sessions using tools like LDP or repadmin /removelingeringobjects. The removal story improved significantly with the release of repldiag.exe. We now have another tool for our tool belt: Lingering Object Liquidator. There are related topics such as “lingering links” which will not be covered in this post.

Lingering Objects Drilldown

The dominant causes of lingering objects are

1. Long-term replication failures
While knowledge of creates and modifies are persisted in Active Directory forever, replication partners must inbound replicate knowledge of deleted objects within a rolling Tombstone Lifetime (TSL) # of days (default 60 or 180 days depending on what OS version created your AD forest). For this reason, it is important to keep your DCs online and replicating all partitions between all partners within a rolling TSL # of days. Tools like REPADMIN /SHOWREPL * /CSV, REPADMIN /REPLSUM and AD Replication Status should be used to continually identify and resolve replication errors in your AD forest.

2. Time jumps
System time jump more than TSL # of days in the past or future can cause deleted objects to be prematurely garbage collected before all DCs have inbound replicated knowledge of all deletes. The protection against this is to ensure that :

1. your forest root PDC is continually configured with a reference time source (including following FSMO transfers
2. All other DCs in the forest are configured to use NT5DS hierarchy
3. Time rollback and roll-forward protection has been enabled via the maxnegphasecorrection and maxposphasecorrection registry settings or their policy-based equivalents.

The importance of configuring safeguards can't be stressed enough. Look at this post to see what happens when time gets out of whack.

3. USN Rollbacks

USN rollbacks are caused when the contents of an Active Directory database move back in time via an unsupported restore. Root causes for USN Rollbacks include:

• Manually copying previous version of the database into place when the DC is offline
• P2V conversions in multi-domain forests
• Snapshot restores of physical and especially virtual DCs. For virtual environments, both the virtual host environment AND the underlying guest DCs should be Virtual Machine Generation ID capable. Windows Server 2012 or later. Both Microsoft and VMWARE make VM-Generation ID aware Hyper-V host.

Events, errors and symptoms that indicate you have lingering objects
Active Directory logs an array of events and replication status codes when lingering objects are detected. It is important to note that while errors appear on the destination DC, it is the source DC being replicated from that contains the lingering object that is blocking replication. A summary of events and replication status codes is listed in the table below:

A comparison of Tools to remove Lingering Objects

The table below compares the Lingering Object Liquidator with currently available tools that can remove lingering objects

The Repldiag and Lingering Object Liquidator tools are preferred for lingering object removal because of their ease of use and holistic approach to lingering object removal.

Why you should care about lingering object removal

Widely known as the gift that keeps on giving, it is important to remove lingering objects for the following reasons

• Lingering objects can result in a long term divergence for objects and attributes residing on different DCs in your Active Directory forest
• The presence of lingering objects prevents the replication of newer creates, deletes and modifications to destination DCs configured to use strict replication consistency. These un-replicated changes may apply to objects or attributes on users, computers, groups, group membership or ACLS.
• Objects intentionally deleted by admins or application continue to exist as live objects on DCs that have yet to inbound replicate knowledge of the deletes.

Once present, lingering objects rarely go away until you implement a comprehensive removal solution. Lingering objects are the unwanted houseguests in AD that you just can't get rid of.

Mother in law jokes… a timeless classic.

We commonly find these little buggers to be the root cause of an array of symptom ranging from logon failures to Exchange, Lync and AD DS service outages. Some outages are resolved after some lengthy troubleshooting only to find the issue return weeks later.
The remainder of this post, we will give you everything needed to eradicate lingering objects from your environment using the Lingering Object Liquidator.

Repldiag.exe is another tool that will automate lingering object removal. It is good for most environments, but it does not provide an interface to see the objects, clean up RODCs (yet) or remove abandoned objects.

Introducing Lingering Object Liquidator

How to obtain Lingering Object Liquidator

1. Log on to the Microsoft Connect site (using the Sign in) link with a Microsoft account:

http://connect.microsoft.com

Note: You may have to create a profile on the site if you have never participated in Connect.

2. Open the Non-feedback Product Directory:

https://connect.microsoft.com/directory/non-feedback

3. Join the following program:

AD Health

Product Azure Active Directory Connection Join link

4. Click the Downloads link to see a list of downloads or this link to go directly to the Lingering Objects Liquidator download. (Note: the direct link may become invalid as the tool gets updated.)

5. Download all associated files

6. Double click on the downloaded executable to open the tool.

Tool Requirements

1. Install Lingering Object Liquidator on a DC or member computer in the forest you want to remove lingering objects from.

2. .NET 4.5 must be installed on the computer that is executing the tool.

3. Permissions: The user account running the tool must have Domain Admin credentials for each domain in the forest that the executing computer resides in. Members of the Enterprise Admins group have domain admin credentials in all domains within a forest by default. Domain Admin credentials are sufficient in a single domain or single domain forest.

4. The admin workstation must have connectivity over the same port and protocol required of a domain-joined member computer or domain controller against any DC in the forest. Protocols of interest include DNS, Kerberos, RPC, LDAP and ephemeral port range used by the targeted DC See TechNet for more detail. Of specific concern: Pre-W2K8 DCs communicate over the “low” ephemeral port between 1024 and 5000 while post W2K3 DCs use the “high” ephemeral port range between 49152 to 65535. Environments containing both OS version families will need to enable connectivity over both port ranges.

5. You must enable the Remote Event Log Management (RPC) firewall rule on any DC that needs scanning. Otherwise, the tool displays a window stating, "Exception: The RPC server is unavailable"

6. The liquidation of lingering objects in AD Lightweight Directory Services (AD LDS / ADAM) environments is not supported.

Lingering Object Discovery

To see all lingering objects in the forest:

1. Launch Lingering Objects.exe.

2. Take a quick walk through the UI:

Naming Context:

Reference DC: the DC you will compare to the target DC. The reference DC hosts a writeable copy of the partition.

Note: ChildDC2 should not be listed here since it is an RODC, and RODCs are not valid reference DCs for lingering object removal.

Target DC: the DC that lingering objects are to be removed from

3. In smaller AD environments, you can leave all fields blank to have the entire environment scanned, and then click Detect. The tool does a comparison amongst all DCs for all partitions in a pairwise fashion when all fields are left blank. In a large environment, this comparison will take a great deal of time as the operation targets (n * (n-1)) number of DCs in the forest for all locally held partitions. For shorter, targeted operations, select a naming context, reference DC and target DC. The reference DC must hold a writable copy of the selected naming context.

During the scan, several buttons are disabled. The current count of lingering objects is displayed in the status bar at the bottom of the screen along with the current tool status. During this execution phase, the tool is running in an advisory mode and reading the event log data reported on each target DC.

Note: The Directory Service event log may completely fill up if the environment contains large numbers of lingering objects and the Directory Services event log is using its default maximum log size. The tool leverages the same lingering object discovery method as repadmin and repldiag, logging one event per lingering object found.

When the scan is complete, the status bar updates, buttons are re-enabled and total count of lingering objects is displayed. The log pane at the bottom of the window updates with any errors encountered during the scan.
Error 1396 is logged if the tool incorrectly uses an RODC as a reference DC.
Error 8440 is logged when the targeted reference DC doesn't host a writable copy of the partition.

The tool leverages the Advisory Mode method exposed by DRSReplicaVerifyObjects that both repadmin /removelingeringobjects /Advisory_Mode and repldiag /removelingeringobjects /advisorymode use. In addition to the normal Advisory Mode related events logged on each DC, it displays each of the lingering objects within the main content pane.

Details of the scan operation log in the linger.log.txt file in the same directory as the tool's executable.

The Export button allows you to export a list of all lingering objects listed in the main pane into a CSV file. View the file in Excel, modify if necessary and use the Import button later to view the objects without having to do a new scan. The Import feature is also useful if you discover abandoned objects (not discoverable with DRSReplicaVerifyObjects) that you need to remove. We briefly discuss abandoned objects later in this post.

Removal of individual objects

The tool allows you to remove objects a handful at a time, if desired, using the Remove button:

1. Here I select three objects (hold down the Ctrl key to select multiple objects, or the SHIFT key to select a range of objects) and then select Remove.

The status bar updates with the new count of lingering objects and the status of the removal operation:

Logging for removed objects

The tool dumps a list of attributes for each object before removal, and logs this along with the results of the object removal in the removedLingeringObjects.log.txt log file. This log file is in the same location as the tool's executable.

C:\tools\LingeringObjects\removedLingeringObjects.log.txt

the obj DN: <GUID=0bb376aa1c82a348997e5187ff012f4a>;<SID=010500000000000515000000609701d7b0ce8f6a3e529d669f040000>;CN=Dick Schenk,OU=R&D,DC=root,DC=contoso,DC=com

objectClass:top, person, organizationalPerson, user;
sn:Schenk ;
whenCreated:20121126224220.0Z;
name:Dick Schenk;
objectSid:S-1-5-21-3607205728-1787809456-1721586238-1183;primaryGroupID:513;
sAMAccountType:805306368;
uSNChanged:32958;
objectCategory:<GUID=11ba1167b1b0af429187547c7d089c61>;CN=Person,CN=Schema,CN=Configuration,DC=root,DC=contoso,DC=com;
whenChanged:20121126224322.0Z;
cn:Dick Schenk;
uSNCreated:32958;
l:Boulder;
distinguishedName:<GUID=0bb376aa1c82a348997e5187ff012f4a>;<SID=010500000000000515000000609701d7b0ce8f6a3e529d669f040000>;CN=Dick Schenk,OU=R&D,DC=root,DC=contoso,DC=com;
displayName:Dick Schenk ;
st:Colorado;
dSCorePropagationData:16010101000000.0Z;
userPrincipalName:Dick@root.contoso.com;
givenName:Dick;
instanceType:0;
sAMAccountName:Dick;
userAccountControl:650;
objectGUID:aa76b30b-821c-48a3-997e-5187ff012f4a;
value is :<GUID=70ff33ce-2f41-4bf4-b7ca-7fa71d4ca13e>:<GUID=aa76b30b-821c-48a3-997e-5187ff012f4a>
Lingering Obj CN=Dick Schenk,OU=R&D,DC=root,DC=contoso,DC=com is removed from the directory, mod response result code = Success
----------------------------------------------
RemoveLingeringObject returned Success

Removal of all objects

The Remove All button, removes all lingering objects from all DCs in the environment.

To remove all lingering objects from the environment:

1. Click the Remove All button. The status bar updates with the count of lingering objects removed. (the count may differ to the discovered amount due to a bug in the tool-this is a display issue only and the objects are actually removed)

2. Close the tool and reopen it so that the main content pane clears.

3. Click the Detect button and verify no lingering objects are found.

Abandoned object removal using the new tool

None of the currently available lingering object removal tools will identify a special sub-class of lingering objects referred to internally as, "Abandoned objects".

An abandoned object is an object created on one DC that never got replicated to other DCs hosting a writable copy of the NC but does get replicated to DCs/GCs hosting a read-only copy of the NC. The originating DC goes offline prior to replicating the originating write to other DCs that contain a writable copy of the partition.

The lingering object liquidator tool does not currently discover abandoned objects automatically so a manual method is required.

1. Identify abandoned objects based on Oabvalidate and replication metadata output.

Abandoned objects can be removed with the LDAP RemoveLingeringObject rootDSE modify procedure, and so Lingering Objects Liquidator is able to remove these objects.

2. Build a CSV file for import into the tool. Once, they are visible in the tool, simply click the Remove button to get rid of them.

a. To create a Lingering Objects Liquidator tool importable CSV file:

Collect the data in a comma separated value (CSV) with the following data:

3. Once you have the file, open the Lingering Objects tool and select the Import button, browse to the file and choose Open.

4. Select all objects and then choose Remove.

Review replication metadata to verify the objects were removed.

Resources

For those that want even more detail on lingering object troubleshooting, check out the following:

• TechNet - Fixing Replication Lingering Object Problems (Event IDs 1388, 1988, 2042)
• KB 910205 - Information about lingering objects in a Windows Server Active Directory forest
• KB 2028495 - Troubleshooting AD Replication error 8606: "Insufficient attributes were given to create an object"
• Glenn LeCheminant’s weblog: Clean that Active Directory forest of lingering objects

To prevent lingering objects:

• Actively monitor for AD replication failures using a tool like the AD Replication Status tool.
• Resolve AD replication errors within tombstone lifetime number of days.
• Ensure your DCs are operating in Strict Replication Consistency mode
• Protect against large jumps in system time
• Use only supported methods or procedures to restore DCs. Do not:
  • Restore backups older than TSL
  • Perform snapshot restores on pre Windows Server 2012 virtualized DCs on any virtualization platform
  • Perform snapshot restores on a Windows Server 2012 or later virtualized DC on a virtualization host that doesn't support VMGenerationID

If you want hands-on practice troubleshooting AD replication errors, check out my lab on TechNet Virtual labs. Alternatively, come to an instructor-led lab at TechEd Europe 2014. "EM-IL307 Troubleshooting Active Directory Replication Errors"

For hands-on practice troubleshooting AD lingering objects: I'll be presenting instructor-led lab sessions at TechEd Europe 2014. "EM-IL400 Troubleshooting Active Directory Lingering Objects"

Finally, if you would like access to a hands-on lab for in-depth lingering object troubleshooting; let us know in the comments.

Thank you,

Justin Turner and A. Conner