What is structured logging? Benefits, examples, and best practices

The BSD syslog format provides only a very restricted set of metadata fields in the header, while any data that is specific to the event or log source, such as username or IP address, appears as a free-form string.

As a result, important data are not accessible during processing without resorting to extraction using regular expressions or other pattern matching. To support full classification and correlation of events, this additional data should also be available. The answer for this issue is structured logging.

This example shows the same log entry but using structured logging, where each event is represented by an unrestricted set of key-value pairs. Structured logs normally use popular formats such as JSON or XML.

Notice that fields have been extracted from the free-form string as well as from the metadata fields in the syslog header. For messages generated by the source in syslog format, this requires maintaining a collection of patterns. However, as logging systems move toward structured logging, pattern-based parsing can be avoided altogether, simply by taking full advantage of the structured log data available at the source.

Windows Event Log stores events in a proprietary binary format which is not human-readable. A consumer, typically Event Viewer, renders the human-readable message by decoding the event records into, in this case, an XML format with each metadata value stored in a separate element. The <System> element contains a standard set of values common to all events, while any additional key-value pairs logged by the event provider are stored in the <EventData> element, as shown in the example below.

As shown in the example XML, the source event data is already structured as key-value pairs. There is no need to use pattern-based parsing when the data is already structured at the source. Unfortunately, however, some log collectors simply discard most of these fields in the EventLog source, collecting only the rendered message and a limited set of data. As a result, not only is some data lost, but what remains must now be parsed from the human-readable message.

For example, here is the same event in the Snare tab-delimited format. The <EventData> element fields are now only available in the free-form message, and some of the values have been replaced by human-readable strings.

Rather than discarding the structure of the event data at the source and then parsing metadata from the rendered message, the fields should be collected directly from the event source. This is the same event in JSON format (with the rendered message truncated for brevity):

With structured logs, accessible event data is not limited to basic metadata such as timestamp and severity. As a result, events can be classified and correlated using any of the values provided by the event set.

For example, an attack originating from a particular IP address may result in events across many different systems. With unstructured logs, the IP address is not available without pattern-based parsing:

The attack can be analyzed much more easily, even across a diverse set of log sources, if all events are structured and include a SourceIPAddress field:

To correlate and filter events across diverse log sources, it is important to use a common set of fields. These field names can be used when parsing unstructured logs, or can be mapped from equivalent fields in a structured log source.

Structured logging often simplifies conversion between formats, allowing event data to be more easily integrated with logging dashboards, analytics platforms, and other log processors. Structured logs can also help work around limitations in log formats.

For example, to transport structured data using syslog, the key-value pairs can be serialized to JSON and used as the payload. In the case of BSD syslog, which lacks a proper timestamp, the full timestamp with year can be preserved in the serialized JSON data (see EventTime below).

Or the key-value pairs can be used as the structured data portion of the IETF syslog format:

This can be done for syslog, as in the the previous example, or many other log formats by either serializing fields into a free-form message area or using a key-value area specified by the format.

Because unstructured logs contain data in free-form messages, pattern-based parsing is often implemented to extract this data for analysis. When there are multiple points of analysis, for example by both an intrusion detection system (IDS) and a log monitoring application, there may be multiple pattern databases used and these patterns may be implemented in different formats.

When the message format of an unstructured log source changes, the patterns stop matching and message parsing fails. The failed parsing must be investigated and all applicable pattern databases updated to correspond with the new message format. In particular, if a log collector reads the rendered message from Windows Event Log rather than the structured data, an added field can break parsing when included in the human-readable message.

It is not necessary to maintain pattern databases for log sources such as Windows Event Log that are available as structured data. Instead, the structured data should be collected directly. For unstructured logs, the parsing should be done as soon as possible and the events handled as structured data. In either case, once the log data is structured, fields can be freely added to events at any point in the log processing chain.