Sending logs to Azure Sentinel with NXLog

The pivotal part of sending secure HTTPS requests to Azure is the authentication process. Azure validates the values of two custom HTTP headers, Authorization and x-ms-date along with the length of the data payload to determine if the request is authentic. The value assigned to the Authorization header is dynamically generated using a cryptographic hash. For details, see the Azure Monitor Authorization section in the Microsoft documentation.

To allow easy integration with the NXLog HTTP(s) (om_http) module that sends events to REST API endpoints, NXLog provides a Perl script that regenerates the single-use authorization string for each new batch of events to be sent.

NXLog can natively collect ETW logs without the need to capture the trace into an .etl file. Configuring an NXLog agent to capture Windows DNS Server events using the Event Tracing for Windows (im_etw) input module is fairly straightforward as illustrated here:

Please note that the first (opening) line of the Input block defines the name of this instance as DNS_Logs. The output module for sending events to Azure uses this name for creating the Azure Sentinel table that will collect these events.

The Exec statement on line 4 of the DNS_Logs input instance above invokes the to_json() procedure, which converts the Windows events to JSON records, as required by Azure’s HTTP Data Collector API.

The output module is the part that connects directly to Azure. The first step in configuring the output instance is retrieving the Workspace ID and either the Primary key or the Secondary key (also referred to as the shared key). These keys can be found by navigating in the Azure portal to Log Analytics workspace > Settings > Agents management. The same set of keys can be viewed under either the Windows servers or Linux servers tab.

The next step is to add this information to the nxlog.conf file as constants making them accessible to the output instance.

The SUBDOMAIN, RESOURCE, and APIVER are used to construct the complete URL. The value for SIZELIMIT can be tuned to your needs. It represents the maximum size in bytes of the data payload for each batch of events. 65000 is the upper limit. The higher values mean better network efficiency. Lower values mean events can be received faster because they are not waiting for a large buffer to be full before they can be sent.

When looking at the entire output instance that uses the HTTP(s) (om_http) module, you can see how batches of events are buffered and then flushed:

The values for the three HTTP headers Authorization, Log-Type, and x-ms-date are set using the add_http_header procedure as shown above on lines 41-43. Log-Type is dynamically set to $SourceModuleName, the name of the input instance we chose at the beginning. Since all REST API events are categorized by Azure Monitor as Custom Logs, Azure appends _CL to the value of Log-Type in order to prevent naming conflicts with other Azure tables thus the name we originally chose, DNS_Logs, appears in Azure Sentinel as DNS_Logs_CL.

By leveraging $SourceModuleName for defining Log-Type, we have created a completely generic output instance that can be used with any other log sources.

If analytical event logging is enabled, you can capture and view DNS Sever analytical events having EventIDs ranging from 256 to 286. Technically, no further changes are needed for logging and viewing both audit and analytical events in Azure Sentinel. However, there is one enhancement you might want to implement:

Enrich the schema with a new attribute: DNS_LogType. If you need to frequently differentiate between audit and analytical DNS Server events, querying for a range of values on a regular basis is not only tedious and makes queries less readable, but it can also be slower on large data sets. This is as simple as replacing the original Exec to_json(); with an Exec block that sets the new $DNS_LogType field to either Audit or Analytical depending on the value of EventID before calling the to_json() which will then enrich the schema with this new field.

Download/view the entire nxlog.conf configuration file.

Let’s take a look at the configuration file to see how the input module is configured and how the rules are defined.

Instead of defining a small set of audit rules within a Rules block diectly in the LinuxAudit input instance, we use the LoadRule directive to load a more comprehensive collection of rules in an audit rule file which is based on the ruleset maintained by the Best Practice Auditd Configuration project.

The xm_resolver module is needed for the ResolveValues directive in the audit input instance, where it is used for resolving some of the numeric values to more human-readable string values.

It should be noted that there are some configuration differences between Linux and Windows as the NXLog directory structure is slightly different, thus the PerlCode path is as follows:

Also, the first line of Perl scripts on Linux needs to point to the location of the perl binary.

Since the Linux configuration files exhibit only minor differences when compared to their Windows counterparts displayed in the ETW section, we won’t display them here. Instead, you can download them using these links:

Download/view the entire sentinelauth.pl Perl script and the nxlog.conf configuration file.

Once these changes have been implemented and the NXLog service has been restarted, events should be sent to the LinuxAudit_CL Azure Sentinel table based on the name given to the input module, LinuxAudit. The following JSON event was triggered and captured according to the very last line in the im_linuxaudit.rules file.

Upon successful receipt in the Log Analytics workspace by Azure Monitor, events are further processed and finally ingested by Azure Sentinel where they can be viewed via user-defined queries.

After expanding the following event to reveal its columns and their values, it can be verified against the JSON formatted event above that was sent via the REST API.