PowerShell Get-WinEvent XML Madness: Getting details from event logs

Announcements

Before we jump into today’s script here are some current events:

• This blog post celebrates three years of PowerShell blogging on TechNet as GoateePFE.  It has been a great ride, and I am far from done.  See the most popular posts here.  Thank you for making this blog successful.
• The PowerShell Deep Dives book is out now.  I contributed a chapter on Active Directory token bloat taken from my SID history blog series.  This book has a ton of great chapters by a ton of great people. All the proceeds go to Save The Children.  Buy your copy today.
• If you haven’t had a chance to watch the Microsoft Virtual Academy recordings Getting Started with PowerShell 3.0 Jump Start and Advanced Tools & Scripting with PowerShell 3.0 Jump Start then you need to put them on your list.  Jeffrey Snover and Jason Helmick do a fantastic job of covering everything you need to know to get started with PowerShell.  Make time for this over several lunches or knock it out in a couple training days.  These videos will seriously boost your career.  You could even gather the family around with a bowl of popcorn.
• PowerShell Saturday 005 is coming up October 26th in Atlanta, Georgia.  My session is titled It’s Time To Part With Blankie: Moving from command line tools to PowerShell for Active Directory.  If you’re in the area stop by for a good time with several PowerShell celebrities.  I’m looking forward to Ed Wilson’s session PowerShell Workflows for Mere Mortals.

Now for today’s topic…

XML vs. IT Pro

Maybe I haven’t looked hard enough, but I’ve just not found any clear documentation aimed at IT Pros for what I am posting today.  As an IT Pro type guy (not a .NET type guy) I have avoided XML for years.  CSV and HTML are so much easier.  XML seems to be a labyrinth of complexity in my mind, and it still is, at least from a PowerShell perspective.  The object model is convenient, but trying to navigate it loses me.  Yeah, I know XML makes the world a happy place, but I’m just not there yet.

Despite this disparaging disclaimer I believe I have drafted a script that will help many of us IT Pros as we weed through event logs (or ETL trace files or EVTX files).

The Backstory

A couple years ago a distinguished peer of mine, Matthew Reynolds, invited me to contribute to a project parsing ETL trace data with PowerShell.  You can hear him talk about the fruits of this labor in his recent TechEd talk, How Many Coffees (New 2013 Edition) Can You Drink While Your PC Starts?  Through this project I became very familiar with parsing XML event log data.  Most of this learning happened for me while I was parsing GPO processing events.

Recently I had a request from a customer who wanted to parse logon audit events and filter deep into the event message body.  I had to dust off my code from the former project and dive a little deeper.  I figured if I’ve had this much trouble with the topic, then I should blog it so that I have something to refer back to when I forget all this again in a few months.

Events:  The good, the bad, and the ugly

The good:  PowerShell works with event logs out of the box.  You have two cmdlets:  Get-EventLog and Get-WinEvent.  Get-WinEvent is the one we’re all supposed to use now.

The bad:  All of a sudden reading event logs gets complicated.  The filtering in particular requires some crazy syntax.  We are far removed from the simplicity of DUMPEL.  PowerShell team blog posts from 2009 here and here attempt to make this look routine.  Um… yeah.

The ugly:  All of the juicy nuggets of event data in the message body are stored in XML.  And nearly every combination of event ID and provider has a unique event schema for storing the data we want.  Neo’s MSDN blog post gets us most of the way there.  AskDS and Hey Scripting Guy show how we can use the GUI to help write the XML filter syntax.  Now my head is spinning.  This is the farthest point from intuitive.  Don’t even get me started on XPATH.

In the life of every scripter you will come to challenges like this.  You just have to cowboy up and dive in.  I recommend that you cruise through these other articles linked above for some good details on filtering XML event log data.

The thing I’ve not seen in these blog posts is how to dump out the event message data in a CSV file where I can easily report and manipulate the data I need.  For example, if I’m collecting logon failure event 4625, then I want the guts of the message body in separate columns where I can easily summarize and report on the user and computer accounts involved.  While I can harvest event logs from multiple servers in the GUI, it is just not friendly for mass reporting, sorting and visualization like Excel.  This is the problem I am trying to solve. 

The individual nuggets of interest are represented as XML in the EventData portion of the message:

Finding Event Message Schemas

Use the lines below to discover the XML schemas behind the events you want to parse.  Run each line and substitute the event IDs and providers you want to investigate.  These lines serve as a bit of an event log deep dive when you run them on your own machine.  Note that some of these lines may need to run in an elevated shell (Run As Administrator).

# List all event providersGet-WinEvent-ListProvider*|Format-Table# List all policy-related event providers.Get-WinEvent-ListProvider*policy*|Format-Table# List the logs on the machine where the name is like 'policy'Get-WinEvent-ListLog*policy*# List all possible event IDs and descriptions for the provider(Get-WinEvent-ListProviderMicrosoft-Windows-GroupPolicy).Events|Format-Tableid,description-AutoSize# List all of the event log entries for the providerGet-WinEvent-LogNameMicrosoft-Windows-GroupPolicy/Operational# Each event in each provider has its own message data schema.# Use this line to find the schema of each event ID.# For a specific event(Get-WinEvent-ListProviderMicrosoft-Windows-GroupPolicy).Events|Where-Object{$_.Id-eq5314}# For a keyword in the event data(Get-WinEvent-ListProviderMicrosoft-Windows-GroupPolicy).Events|Where-Object{$_.Template-like"*reason*"}# Find an event ID across all ETW providers:Get-WinEvent-ListProvider*|ForEach-Object{$_.Events|Where-Object{$_.ID-eq4168}}

Notice that the Template property holds the XML definition of the event message body.  This is where our event log goodness hides.  The trick is parsing these individual XML values for our reporting.  I’ve highlighted the data entries and their corresponding placeholders in the GPO event schema example below. 

PS C:\> (Get-WinEvent -ListProvider Microsoft-Windows-GroupPolicy).Events |
 Where-Object {$_.Id -eq 5314}


Id          : 5314
Version     : 0
LogLink     : System.Diagnostics.Eventing.Reader.EventLogLink
Level       : System.Diagnostics.Eventing.Reader.EventLevel
Opcode      : System.Diagnostics.Eventing.Reader.EventOpcode
Task        : System.Diagnostics.Eventing.Reader.EventTask
Keywords    : {}
Template    : <template
              xmlns="http://schemas.microsoft.com/win/2004/08/events"><data name="BandwidthInkbps" inType="win:UInt32"
                  outType="xs:unsignedInt"/><data name="IsSlowLink" inType="win:Boolean"
                  outType="xs:boolean"/><data name="ThresholdInkbps" inType="win:UInt32"
                  outType="xs:unsignedInt"/><data name="PolicyApplicationMode" inType="win:UInt32"
                  outType="xs:unsignedInt"/><data name="ErrorCode" inType="win:UInt32"
                  outType="win:HexInt32"/><data name="LinkDescription" inType="win:UnicodeString"
                  outType="xs:string"/></template>

Description : A %6 link was detected. The Estimated bandwidth is %1 kbps. The
              slow link threshold is %3 kbps.

Cracking the XML Nut

So how do I pull those values out of the event message?  This time we’re going to look at a log from one of our domain controllers (DCs).  Notice the filter syntax.  DO NOT pipe the entire event log to a Where-Object if you want results returned in this century.  Let’s grab one event entry to examine:

# Prompts for creds$cred=Get-CredentialContoso\Administrator# Grab the events from a DC# Target DC needs firewall rule enabled:# Remote Event Log Management (RPC)$Event=Get-WinEvent-ComputerName2012DC-Credential$cred` -FilterHashtable@{Logname='Security';Id=4625}` -MaxEvents1# View the event properties.$Event|Format-List*# View the array of message body values.# But the property names are missing.$Event.Properties# Convert the event to XML$eventXML=[xml]$Event.ToXml()# Drill down through the XML to the message goodness# Ah ha! This is what we want.$eventXML.Event.EventData.Data# You have to index each data element to access it.$eventXML.Event.EventData.Data[0].name$eventXML.Event.EventData.Data[0].'#text'

PS C:\> $eventXML.Event.EventData.Data

Name                      #text                          
----                      -----                          
SubjectUserSid            S-1-5-18                       
SubjectUserName           2012DC$                        
SubjectDomainName         CONTOSO                        
SubjectLogonId            0x3e7                          
TargetUserSid             S-1-0-0                        
TargetUserName            DanPark                        
TargetDomainName          CONTOSO                        
Status                    0xc000015b                     
FailureReason             %%2308                         
SubStatus                 0x0                            
LogonType                 4                              
LogonProcessName          Advapi                         
AuthenticationPackageName Negotiate                      
WorkstationName           2012DC                         
TransmittedServices       -                              
LmPackageName             -                              
KeyLength                 0                              
ProcessId                 0x390                          
ProcessName               C:\Windows\System32\svchost.exe
IpAddress                 -                              
IpPort                    -

My Inefficient Magic XML to CSV Event Reporting Machine

The code in this example pulls event 4625 from the Security log on a domain controller, and then it copies each of the XML message body properties into their own event object property for easier reporting.  Note that the DC must have the firewall rule enabled to allow Remote Event Log Management (RPC).

# Prompt for creds$cred=Get-CredentialContoso\Administrator# Grab the events from a DC$Events=Get-WinEvent-ComputerName2012DC-Credential$cred` -FilterHashtable@{Logname='Security';Id=4625}# Parse out the event message dataForEach($Eventin$Events){# Convert the event to XML$eventXML=[xml]$Event.ToXml()# Iterate through each one of the XML message propertiesFor($i=0;$i-lt$eventXML.Event.EventData.Data.Count;$i++){# Append these as object propertiesAdd-Member-InputObject$Event-MemberTypeNoteProperty-Force` -Name$eventXML.Event.EventData.Data[$i].name` -Value$eventXML.Event.EventData.Data[$i].'#text'}}# View the results with your favorite output method$Events|Export-Csv.\events.csv$Events|Select-Object*|Out-GridView

I call this inefficient, because it must go back through the event log data a second time to process the XML message body properties.  This is OK for smaller data sets, but your performance will be slower with larger data sets.

Note that this approach is only appropriate for events that share the same schema.  Some events within a provider will have the same schema, even though the event IDs are different.  In those cases it would be OK to include multiple event IDs in your query.

This is my solution to the problem called out at the beginning of the article.  Now that I have all of the XML data in native event object properties I can easily group, filter, sort, and report.  Yes, there is fancy XML syntax that will filter deep into the message body, but that does not give me visibility to all of the values.  Now I have the data exposed in a CSV spreadsheet where I can quickly visualize patterns and trends.  Add some pivot tables and charts to make your reporting management-friendly.

The Big Finish

You could really amp this up by running it as a workflow in parallel against all of your target computers.  Or you could use Invoke-Command -AsJob for multi-threading.  Unfortunately remote sessions and workflows return their results as deserialized objects that lack the .ToXML() method.  I’m still working on a way to get this working over these remoting technologies.  In the meantime you can scale this out across multiple servers by using the cmdlet Start-Job to spin up a thread for each server you query.

Am I Wrong?

One of the things I like about working in IT is constantly learning new things.  I’ve been doing PowerShell for three years, but I still have much to learn.  If I’m missing something here please let me know in the comments below.  Have you found an easier way to work with XML event message bodies from Get-WinEvent?  I’d love to hear about it.