Hello again fellow Windows 10 users!
I'd like to dedicate this post to discussing a reoccurring problem that I am seeing crop up in more and more customer environments as of late. Before I get too entrenched in discussing the details of the issue, see if this scenario hits home for any of you.
The issue.
It's a beautiful and sunny second Tuesday of the month, and you've just finished up a long day of filling out and submitting one TPS report after another. Your Windows 10 machines informs you that updates are available for install, and that they are scheduled to apply to your system at midnight. You decide to log off and let the wonder of patch automation handle installing the updates for you. You get back to the office in the morning ready for another productive day of work, only to find your Windows 10 device sitting at the Windows Recovery screen. You reboot the device and are greeted with a Blue Screen of Unhappy Death shortly after the OS starts to initialize. This pattern repeats two more times before you find yourself back in Windows Recovery area.
For those of you who are finding this post through a Bing or Google search, I reckon the scenario probably sounds mighty familiar to you.
So what exactly happened to cause this tragedy of Greek proportions? Lets dig into the issue a bit.
The back story.
To really get to the bottom of what's happening, you need to be familiar with the patching methodology that Microsoft is utilizing to ensure that Windows 10 stays as up to date as possible. When Windows 10 became generally available to the public, a new patching paradigm was adopted to keep the code that Windows 10 runs off of as up to date and secure as possible for all of our customers. Monthly Cumulative Updates (referred to as CUs from this point on) became the new patching mechanism responsible for delivering all the necessary code changes to the Operating System. Anyone who has ever worked with CUs knows that there is one predominant problem when it comes to CUs... mainly that they are, well, cumulative! Each months update gets progressively larger because of ever increasing amounts of updated code that get added to the updates.
To combat the growth problem, shortly after the release of Windows 10 version 1607 (Anniversary Update), Microsoft introduced a new Windows Update package type effectively dubbed the Delta update. Like the name implies, it only contains code that has changed since the last full CU that was released, which effectively (and dramatically) reduces the overall size of the updates that get pushed out to managed systems. Cool right?
Unfortunately, these two patch types have introduced some peculiar issues into customer environments... especially environments that rely on two different patch deployment solutions... lets say, System Center Configuration Manager and IBM Bigfix just to pick on two.
The problem.
The main problem that arises from utilizing multiple patch deployment services is that it becomes possible to deploy two different types of patches. When it comes to the issue mentioned above, this is exactly what's happening, which results in the BSOD end state that so many customers Windows 10 machines are finding themselves in. The problem state occurs when both a full CU and Delta CU get deployed during the same reboot cycle.
Picture this scenario if you will. Your organization utilizes IBM Bigfix to catch and deploy the full CU out to all Windows 10 machines that need it, and System Center Configuration Manager to deploy the Delta updates. A Windows 10 machine that is a month or so behind on it's updates gets booted up and analyzed by both services. BigFix sees that the device needs the latest deployed month CU, and pushes the update down to the system. No restart occurs due to policies that are in place that prevent this kind of action from occurring during work hours. The Configuration Manager Agent kicks in directly afterwards, and detects that the machine needs the latest Delta update that has been published. This update gets pushed down to the system and installed. No restart occurs due to the same policies mentioned above. When the machine does finally reboot, we immediately encounter the BSOD condition mentioned above. Why?
If we take a look at the stop error code that gets generated by the system during the BSOD event, we will see that it's a 0x7B INACCESSIBLE_BOOT_DEVICE stop error. This error is a direct result of the oddball boot state that the system gets into when both a full CU and Delta update get pushed to, and installed by a system during the same reboot cycle.
Recovery.
So great. You've found yourself stuck in this state. Now what? Fortunately, it's fairly easy to recover Windows 10 devices from this non-bootable condition. Most of the time. Unfortunately, if you use Bitlocker or a third-party encryption service like McAfee or CheckPoint, you're going to find that you need to decrypt or suspend encryption on the system prior to being able to perform the recommended fix action below.
For Bitlocker users, use the following commands below to suspend encryption so that the repair operation can be performed:
manage-bde.exe -protectors -get c:
manage-bde.exe -unlock c: -recoverypassword "xxxxxx"
manage-bde.exe -protectors -disable c:
In the case of third party encryption solutions, I've seen the decryption process take anywhere from a couple of hours, to three or four days... depending on the encryption service that's being utilized. If a customer has a wealth of data that has not been backed up on their system, it will likely be a very worthwhile action for you to perform. The reason the device needs to be either decrypted or unlocked is due to the fact that we will need to utilize some recovery tools from the nifty Windows Recovery Environment that will need access to the volume that Windows is installed on... namely, the Command Prompt, which can be found under the Advanced Options section of the Recovery Environment:
Once you have gotten into the Command Prompt, you're going to want to run the following command:
- dism.exe -image:C: /get-packages > X:packages.txt
*Note - The -image option should point to the volume where the Windows directory is located.
- Open the packages text file with notepad by typing: notepad x:packages.txt
At this point you will want to scroll all the way to the bottom of the text file, and look for any packages that have a state status of Install Pending. Copy the package names, and use the following command to remove the 'Install Pending' packages:
- dism.exe /image:C: /remove-package /packagename:Package_for_RollupFix~31bf3856ad364e35~amd64~~14393.953.1.2
*Note - The above package name was included only as an example. Please ensure you use the package name from the packages.txt file above.
Once the DISM tool has finished removing the package, your system should be good to go. Once it's reboots, Windows 10 will continue to uninstall any packages previously scheduled for removal, at which point the OS should be able to initialize itself properly.
The follow up.
To prevent this issue from reoccurring in the future, it is highly recommended that customers configure their patching infrastructure so that the Windows 10 full CUs and Delta updates are NOT installed at the same time, during the same reboot cycle.
And there you have it folks! Go forth and perform good updating deeds! Until next time.