Sending ETW Logs to Splunk with NXLog

NXLog supports direct collection of Event Tracing for Windows (ETW) data. DNS Analytical logs, for example, can be forwarded to Splunk or another SIEM for monitoring and analysis.

Collecting ETW Logs

Event Tracing for Windows (ETW) is a kernel-level tracing facility that provides high-performance logging of kernel and application events. ETW events can be written to a log file or collected directly from the system in realtime via the Consumers API. ETW is a valuable source of event data that is under-utilized compared to the well-known Windows EventLog. The Debug and Analytical channels, which are based on ETW, cannot be collected through Windows EventLog directly. For more information, see About Event Tracing on Microsoft Docs.

ETW logging is of particular interest for collecting DNS Server activity logs. Prior to Windows Server 2012 R2, activity logs were only available through DNS Server’s debug logging feature (See DNS Debug Logging in the NXLog User Guide.) Unfortunately, debug logging can significantly impact DNS Server performance and is not intended to be enabled permanently. With the introduction of DNS Server Analytical logs in Server 2012 R2 and 2016, high-performance DNS activity logging is available through ETW.

The logman and tracerpt commands can be used to work with traces:

Creating and controlling traces with logman
> logman query providers
Microsoft-Windows-DNSServer              {EB79061A-A566-4698-9119-3ED2807060E7}
[...]
> logman create trace DNSTrace -p Microsoft-Windows-DNSServer -o c:\trace.etl
> logman start DNSTrace
[Wait for activity to be logged]
> logman stop DNSTrace
> logman delete DNSTrace
Using tracerpt to generate a dump file and summary
> tracerpt c:\trace_000001.etl
[...]
DumpFile:           dumpfile.xml
Summary:            summary.txt

While these tools are suitable for working with temporary traces, they do not offer a good solution for setting up regular collection of ETW logs. A similar multi-stage process would need to be implemented to use the Splunk Universal Forwarder for Windows, as it does not support ETW directly.

NXLog Enterprise Edition, however, implements the ETW Consumers API to collect ETW events directly from the source for robust, efficient ETW logging. NXLog handles tracing setup, so it is not necessary for the user to run logman or tracerpt commands. NXLog does not require events to be written to disk, reducing the overhead required and allowing NXLog to perform well even at high event rates.

Configuring the log data source in Splunk

Before adding the NXLog configuration, a new log source needs to be added in Splunk to receive the logs.

  1. In the Settings menu, click Data inputs.

    splunk 1 data inputs
  2. In the Local Input section, for the TCP input type, click Add new.

    splunk 2 add new
  3. Configure the input source. Enter the TCP Port on which to listen for log data (for example, port 514). This same port will need to be specified in the NXLog configuration also. The remaining settings are optional. Click Next.

    splunk 3 select source
  4. Choose the input settings. Set the Source type to syslog (in the Operating System category). Choose an App context; for example, Search & Reporting (search). Adjust the remaining default values if required, and then click Review.

    Syslog is selected as the source type here, because with IETF Syslog, Splunk will automatically parse fields and use the event timestamp without any custom configuration required outside of the dashboard.

    splunk 4 source type
  5. Check the pending changes, then click Submit.

Now that an appropriate Splunk log source has been configured, you can continue by setting up NXLog Enterprise Edition to collect and forward ETW logs.

Configuring NXLog to collect and forward ETW logs

NXLog can collect logs from ETW with the im_etw module. The Provider directive specifies the ETW provider to collect logs from, while the optional Level directive can be used to select a log level (the default is the "verbose" level). It is also possible to collect kernel trace logs with the KernelFlags directive and a comma-separated list of flags. For full configuration details, see im_etw in the NXLog Reference Manual.

In the following examples, NXLog reads DNS Server Analytical logs directly from ETW and forwards them to Splunk via TCP.

Note
DNS Server Analytical logging is available beginning with Windows Server 2012 R2. For more information, and an alternative method for previous versions of Windows Server, see Windows DNS Server in the NXLog User Guide.
  1. DNS Server Analytical logs are available via the Microsoft-Windows-DNSServer ETW provider. Create an im_etw input instance with this provider.

    nxlog.conf
    1
    2
    3
    4
    
    <Input dns_analytical>
        Module      im_etw
        Provider    Microsoft-Windows-DNSServer
    </Input>
  2. There are several options for forwarding events to Splunk, including TCP, TLS, and HTTP Event Collection (HEC)—see Splunk in the NXLog User Guide. This example shows TCP transport: Host should be set to the IP address or hostname of the Splunk host and Port should match the port configured in the Splunk log source.

    Setting the $Message field to an empty string prevents the to_syslog_ietf() procedure from generating a duplicate list of key-value pairs in the Syslog message (from the im_etw $raw_event field). A $vendor_severity field is added to make the severity information available in Splunk.

    nxlog.conf
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    
    <Extension _syslog>
        Module  xm_syslog
    </Extension>
    
    <Output splunk_out>
        Module  om_tcp
        Host    127.0.0.1
        Port    514
        <Exec>
            if $Message == undef $Message = "";
            $vendor_severity = $Severity;
            to_syslog_ietf();
        </Exec>
    </Output>
  3. Add a route for the ETW logs. If no route is specified in the configuration, NXLog will automatically create a single default route connecting all inputs to all outputs.

    nxlog.conf
    1
    2
    3
    
    <Route splunk>
        Path    dns_analytical => splunk_out
    </Route>
  4. The nxlog service should be restarted to apply the new configuration.

Confirming log collection in Splunk

To confirm data collection, go to the Splunk Search & Reporting dashboard and search for the correct logs, using either host name or source name.

splunk out 1

Expand the event to view the list of parsed fields.

splunk out 2

Once forwarded to Splunk, administrators can make use of Splunk features such as visualizations, alerts, and other Splunk apps.

NXLog Ltd. develops multi-platform log collection tools that support many different log sources, formats, transports, and integrations. The tools help administrators collect, parse, and forward logs so they can more easily respond to security issues, investigate operational problems, and analyze event data. NXLog distributes the free and open source Community Edition and offers additional features and support with the Enterprise Edition.

This document is provided for informational purposes only and is subject to change without notice.

Share this post