Windows Event Log stores events in a proprietary binary format which is not human-readable. A consumer, typically Event Viewer, renders the human-readable message by decoding the event records into, in this case, an XML format with each metadata value stored in a separate element. The <System> element contains a standard set of values common to all events, while any additional key-value pairs logged by the event provider are stored in the <EventData> element, as shown in the example below.

As shown in the example XML, the source event data is already structured as key-value pairs. There is no need to use pattern-based parsing when the data is already structured at the source. Unfortunately, however, some log collectors simply discard most of these fields in the EventLog source, collecting only the rendered message and a limited set of data. As a result, not only is some data lost, but what remains must now be parsed from the human-readable message.

For example, here is the same event in the Snare tab-delimited format. The <EventData> element fields are now only available in the free-form message, and some of the values have been replaced by human-readable strings.

Rather than discarding the structure of the event data at the source and then parsing metadata from the rendered message, the fields should be collected directly from the event source. This is the same event in JSON format (with the rendered message truncated for brevity):

With structured logs, accessible event data is not limited to basic metadata such as timestamp and severity. As a result, events can be classified and correlated using any of the values provided by the event set.

For example, an attack originating from a particular IP address may result in events across many different systems. With unstructured logs, the IP address is not available without pattern-based parsing:

The attack can be analyzed much more easily, even across a diverse set of log sources, if all events are structured and include a SourceIPAddress field:

To correlate and filter events across diverse log sources, it is important to use a common set of fields. These field names can be used when parsing unstructured logs, or can be mapped from equivalent fields in a structured log source.

Structured logging often simplifies conversion between formats, allowing event data to be more easily integrated with logging dashboards, analytics platforms, and other log processors. Structured logs can also help work around limitations in log formats.

For example, to transport structured data using syslog, the key-value pairs can be serialized to JSON and used as the payload. In the case of BSD syslog, which lacks a proper timestamp, the full timestamp with year can be preserved in the serialized JSON data (see EventTime below).

Or the key-value pairs can be used as the structured data portion of the IETF syslog format:

This can be done for many other log formats also, by either serializing fields into a free-form message area or using a key-value area specified by the format.

Because unstructured logs contain data in free-form messages, pattern-based parsing is often implemented to extract this data for analysis. When there are multiple points of analysis, for example by both an intrusion detection system (IDS) and a log monitoring application, there may be multiple pattern databases used and these patterns may be implemented in different formats.

When the message format of an unstructured log source changes, the patterns stop matching and message parsing fails. The failed parsing must be investigated and all applicable pattern databases updated to correspond with the new message format. In particular, if a log collector reads the rendered message from Windows Event Log rather than the structured data, an added field can break parsing when included in the human-readable message.

It is not necessary to maintain pattern databases for log sources such as Windows Event Log that are available as structured data. Instead, the structured data should be collected directly. For unstructured logs, the parsing should be done as soon as possible and the events handled as structured data. In either case, once the log data is structured, fields can be freely added to events at any point in the log processing chain.

Modern enterprise systems generate an overwhelming amount of log data which must be collected, transported, managed, and monitored. Environments with complex logging requirements can benefit greatly from structured logging.

NXLog Platform’s log collection technology is designed around the concept of structured logging.

Besides these benefits, NXLog Platform includes capabilities that further support the structured logging strategy:

NXLog Platform is purpose-built to handle structured logging, and support direct collection, processing, and storage of structured logs, streamlining event classification and correlation.

Structured logging is essential for modern log management, enabling more efficient parsing, conversion, and correlation of log data across diverse sources. By representing each event as a set of key-value pairs, structured logging enhances data accessibility, reduces the need for complex pattern-based parsing, and facilitates seamless integration with various log management systems.

By adopting structured logging, enterprises can significantly boost the value of their log management strategy, leading to better system performance, security monitoring, and overall operational efficiency.