Collecting text log files to Azure Operational Insights using System Center Operations Manager

Note – the information presented in this post is something you can do today to collect text log files into Operational Insights using System Center Operations Manager. Anyhow, this does not represent ‘full’ nor ‘final’ support for collecting text log files at all - the idea to support this as a first class citizen (i.e. configuring it from the portal, allowing it to work with Directly connected agents, etc) is tracked in our backlog – please vote it here http://feedback.azure.com/forums/267889-azure-operational-insights/suggestions/7113030-collect-text-log-files .

But until we get to prioritize that piece of work, if you have System Center Operations Manager, it is pretty straightforward to author a management pack like the ones I described in my previous posts here http://blogs.technet.com/b/momteam/archive/2014/08/27/anatomy-of-an-event-collection-rule-for-advisor-preview-advanced-targeting.aspx and here http://blogs.technet.com/b/momteam/archive/2014/07/29/how-to-collect-syslog-events-thru-operations-manager-scom-to-advisor-preview.aspx , and start indexing and searching those events in OpInsights today!

So, let’s see how such rules would look like in a management pack.

First of all, like in the other posts, you need to include a reference in your management pack to Microsoft.IntelligencePacks.Types library to get the write action to send to Operational Insights. You’d also need the AppLog library in SCOM to parse/read the actual log files:

<Reference Alias="AppLog">
  <ID>System.ApplicationLog.Library</ID>
  <Version>7.0.8433.0</Version>
  <PublicKeyToken>31bf3856ad364e35</PublicKeyToken>
</Reference>
<Reference Alias="IPTypes">
 <ID>Microsoft.IntelligencePacks.Types</ID>
  <Version>7.0.9802.0</Version>
  <PublicKeyToken>31bf3856ad364e35</PublicKeyToken>
</Reference>

After that, the actual collection rules are very straightforward: a data source reading the log(I am picking up every single line from it, matching wildcard “*” – but you can use this to filter what in each of your logs is ‘interesting’/worth collecting), a condition detection used to MAP fields to the ‘event’ shape that OpInsights Log Management expects, and the write action to send this data to the cloud:

      <Rule ID="MomUIGeneratedRuledc3f1e40e7d54c5a82e408523d686031" Enabled="true" Target="MicrosoftWindowsLibrary7585010!Microsoft.Windows.Computer" ConfirmDelivery="false" Remotable="true" Priority="Normal" DiscardLevel="100">
        <Category>EventCollection</Category>
        <DataSources>
          <DataSource ID="LogDS" TypeID="AppLog!System.ApplicationLog.GenericLog.FilteredEventProvider">
            <LogFileDirectory>c:\Logs\</LogFileDirectory>
            <LogFilePattern>Test*.log</LogFilePattern>
            <LogIsUTF8>false</LogIsUTF8>
            <Expression>
              <RegExExpression>
                <ValueExpression>
                  <XPathQuery Type="String">Params/Param[1]</XPathQuery>
                </ValueExpression>
                <Operator>MatchesWildcard</Operator>
                <Pattern>*</Pattern>
              </RegExExpression>
            </Expression>
          </DataSource>
        </DataSources>
       <ConditionDetection ID="Mapper" TypeID="SystemLibrary7585010!System.Event.GenericDataMapper">
            <EventOriginId>$Target/Id$</EventOriginId>
            <PublisherId>$MPElement$</PublisherId>
            <PublisherName>GenericLog</PublisherName>
            <Channel>$Data/EventData/DataItem/LogFileName$</Channel>
            <LoggingComputer>$Target/Property[Type='MicrosoftWindowsLibrary7585010!Microsoft.Windows.Computer']/PrincipalName$</LoggingComputer>
            <EventNumber>0</EventNumber>
            <EventCategory>3</EventCategory>
            <EventLevel>0</EventLevel>
            <UserName></UserName>
            <Description>$Data/EventData/DataItem/Params/Param[1]$</Description>
            <Params />
        </ConditionDetection>
        <WriteActions>
         <WriteAction ID="HttpWA" TypeID="IPTypes!Microsoft.SystemCenter.CollectCloudGenericEvent" />
        </WriteActions>
      </Rule>

That’s it. Once the rule starts running you should start receiving those logs among the other records of Type=Event.

Here’s an example of how I am pulling up my DNS debug log – you can see how I carry on the ‘Computer’ name (from the rule’s target), I have hardcoded Category and Level to 3 and 0 respectively, EventID to 0, the source is hardcoded to ‘GenericLog’ and the ‘EventLog’ field contains the full path to the actual text file the rule is looking at. The full message payload of each log line is mapped to ‘RenderedDescription’ (which is fully searchable for keywords/terms):

Enjoy!

Full sample MP XML is on GitHub.

Additional things to highlight:

1. the module used to read the logs in this example is a pre-existing module in SCOM, not specific to Operational Insights – details on the module are on MSDN documentation https://msdn.microsoft.com/en-us/library/ee809315.aspx and caveats when using the module are described in this KB article http://support.microsoft.com/kb/2691973
2. be aware that the log lines collected this way will transit from your agent to your management servers – like with other log collection rules, if the amount of events produced is very large, do keep an eye on the agent and management server’s queues.
3. See/vote on the related idea in our backlog about using regular expressions to perform field extraction (useful for those long text fields) http://feedback.azure.com/forums/267889-azure-operational-insights/suggestions/6519270-support-regular-expressions-regex-or-xpath-to-pe
4. This is not the ‘final’ way to collect text logs. That idea can be voted and tracked here http://feedback.azure.com/forums/267889-azure-operational-insights/suggestions/7113030-collect-text-log-files