What is structured logging? Benefits, examples, and best practices

Log data analysis is heavily influenced by how the collected data is initially processed and stored. Two primary approaches dominate the logging landscape: structured and unstructured logging.

Choosing between structured and unstructured logging significantly impacts log processing efficiency, especially as systems scale. Ultimately, selecting the right logging approach is crucial for ensuring that log storage and analysis don’t become a bottleneck as system complexity and data volumes evolve.

This white paper explores structured logging, its advantages, and how it compares to the unstructured logging approach.

What is structured logging?

Structured logging records log data in consistent key-value formats, making it easier to parse, search, and analyze. In structured logging, each event is represented by an unrestricted set of key-value pairs. This offers a variety of advantages, including simpler parsing, easier format conversion, and more flexible event classification and correlation, even across diverse log sources.

On the other hand, unstructured logs are often easy to write (as simple text strings) but difficult to search and analyze at scale.

The following examples show an unstructured log entry and its equivalent in a structured JSON format.

Unstructured logging example

The BSD syslog format provides only a very restricted set of metadata fields in the header, while any data that is specific to the event or log source, such as username or IP address, appears as a free-form string.

An unstructured BSD syslog event

<38>Nov 22 10:30:12 myhost sshd[8459]: Failed password for invalid user linda from 192.168.1.60 port 38176 ssh2

As a result, important data are not accessible during processing without resorting to extraction using regular expressions or other pattern matching. To support full classification and correlation of events, this additional data should also be available. The answer for this issue is structured logging.

Structured logging example

This example shows the same log entry but using structured logging, where each event is represented by an unrestricted set of key-value pairs. Structured logs normally use popular formats such as JSON or XML.

The same syslog event as structured JSON

{
  "SourceName": "sshd",
  "ProcessID": 8459,
  "Message": "Failed password for invalid user linda from 192.168.1.60 port 38176 ssh2",
  "Status": "failed",
  "AuthenticationMethod": "password",
  "Reason": "invalid user",
  "SourceIPAddress": "192.168.1.60",
  "SourcePort": 38176,
  "Protocol": "ssh2"
}

Notice that fields have been extracted from the free-form string as well as from the metadata fields in the syslog header. For messages generated by the source in syslog format, this requires maintaining a collection of patterns. However, as logging systems move toward structured logging, pattern-based parsing can be avoided altogether, simply by taking full advantage of the structured log data available at the source.

Why use structured logging?

When deploying a log analysis pipeline, structured logging might seem an unnecessary extra effort. Why not simply go ahead and use unstructured logs? Unless your range of log sources and business requirements starts small, and remains so, using unstructured logging will soon prove to be a bad option.

In the long run, the main advantages of structured logging are:

Improved analysis: A consistent data schema and structured logging formats, such as JSON, provide logs with clearly defined, easily queryable fields like timestamps and IDs. This enables log management tools to efficiently filter, query, and aggregate data.
Easier integration: Most tools are already prepared, or even require, to receive and parse data in structured formats.
Scalability: As your system evolves and expands, structured logging provides an easier correlating framework to aggregate the data from new sources.
Compliance and auditing: With structured logging, you can easily remove or protect sensitive data. You also guarantee that your data is enriched with all the required information, properly formatted and labeled to build clear, verifiable audit trails.

How to use structured logging

The following sections detail techniques associated with enriching and formatting log data for structured logging.

Retaining structured data at the source

Regular expression parsing is unnecessary when the data is already structured at the source

Windows Event Log stores events in a proprietary binary format which is not human-readable. A consumer, typically Event Viewer, renders the human-readable message by decoding the event records into, in this case, an XML format with each metadata value stored in a separate element. The <System> element contains a standard set of values common to all events, while any additional key-value pairs logged by the event provider are stored in the <EventData> element, as shown in the example below.

Windows Event Log XML (excerpt)

<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Microsoft-Windows-Security-Auditing"
              Guid="{54849625-5478-4994-A5BA-3E3B0328C30D}" />
    <EventID>4625</EventID>
    <Level>0</Level>
    <TimeCreated SystemTime="2024-02-02T14:47:43.520229400Z" />
    <Channel>Security</Channel>
    <Computer>WIN-AQ35E3LLSVB.example.com</Computer>
  </System>
  <EventData>
    <Data Name="SubjectUserName">WIN-AQ35E3LLSVB$</Data>
    <Data Name="SubjectDomainName">EXAMPLE</Data>
    <Data Name="TargetUserName">Administrator</Data>
    <Data Name="TargetDomainName">EXAMPLE</Data>
    <Data Name="Status">0xc000006d</Data>
    <Data Name="FailureReason">%%2313</Data>
    <Data Name="LogonType">2</Data>
    <Data Name="WorkstationName">WIN-AQ35E3LLSVB</Data>
    <Data Name="IpAddress">127.0.0.1</Data>
    <Data Name="IpPort">0</Data>
  </EventData>
</Event>

Some log collectors discard most of the fields in Windows Event Log XML

As shown in the example XML, the source event data is already structured as key-value pairs. There is no need to use pattern-based parsing when the data is already structured at the source. Unfortunately, however, some log collectors simply discard most of these fields in the EventLog source, collecting only the rendered message and a limited set of data. As a result, not only is some data lost, but what remains must now be parsed from the human-readable message.

For example, here is the same event in the Snare tab-delimited format. The <EventData> element fields are now only available in the free-form message, and some of the values have been replaced by human-readable strings.

The Windows event in Snare format

win-aq35e3llsvb.example.com        MSWinEventLog        1        Security        971        Fri Feb 02 06:47:43 2024        4625        Microsoft-Windows-Security-Auditing        EXAMPLE\Administrator        N/A        Failure Audit        WIN-AQ35E3LLSVB.example.com        Logon                An account failed to log on.    Subject:   Security ID:  S-1-5-18   Account Name:  WIN-AQ35E3LLSVB$   Account Domain:  EXAMPLE   Logon ID:  0x3E7    Logon Type:   2    Account For Which Logon Failed:   Security ID:  S-1-0-0   Account Name:  Administrator   Account Domain:  EXAMPLE    Failure Information:   Failure Reason:  Unknown user name or bad password.   Status:   0xC000006D   Sub Status:  0xC000006A    Process Information:   Caller Process ID: 0x4d0   Caller Process Name: C:\Windows\System32\svchost.exe    Network Information:   Workstation Name: WIN-AQ35E3LLSVB   Source Network Address: 127.0.0.1   Source Port:  0    Detailed Authentication Information:   Logon Process:  User32    Authentication Package: Negotiate   Transited Services: -   Package Name (NTLM only): -   Key Length:  0    This event is generated when a logon request fails. It is generated on the computer where access was attempted.    The Subject fields indicate the account on the local system which requested the logon. This is most commonly a service such as the Server service, or a local process such as Winlogon.exe or Services.exe.    The Logon Type field indicates the kind of logon that was requested. The most common types are 2 (interactive) and 3 (network).    The Process Information fields indicate which account and process on the system requested the logon.    The Network Information fields indicate where a remote logon request originated. Workstation name is not always available and may be left blank in some cases.    The authentication information fields provide detailed information about this specific logon request.   - Transited services indicate which intermediate services have participated in this logon request.   - Package name indicates which sub-protocol was used among the NTLM protocols.   - Key length indicates the length of the generated session key. This will be 0 if no session key was requested.        4

Rather than discarding the structure of the event data at the source and then parsing metadata from the rendered message, the fields should be collected directly from the event source. This is the same event in JSON format (with the rendered message truncated for brevity):

The Windows event converted to JSON (excerpt)

{
  "EventTime": "2024-02-02T06:47:43.520229-08:00",
  "Hostname": "WIN-AQ35E3LLSVB.example.com",
  "EventType": "AUDIT_FAILURE",
  "Severity": "ERROR",
  "EventID": 4625,
  "SourceName": "Microsoft-Windows-Security-Auditing",
  "ProviderGuid": "{54849625-5478-4994-A5BA-3E3B0328C30D}",
  "Channel": "Security",
  "Message": "An account failed to log on. [...]",
  "Category": "Logon",
  "SubjectUserName": "WIN-AQ35E3LLSVB$",
  "SubjectDomainName": "EXAMPLE",
  "TargetUserName": "Administrator",
  "TargetDomainName": "EXAMPLE",
  "Status": "0xc000006d",
  "FailureReason": "%%2313",
  "LogonType": "2",
  "AuthenticationPackageName": "Negotiate",
  "WorkstationName": "WIN-AQ35E3LLSVB",
  "IpAddress": "127.0.0.1",
  "IpPort": "0"
}

Classifying and correlating events across log sources

Events can be classified and correlated using any of the available data

With structured logs, accessible event data is not limited to basic metadata such as timestamp and severity. As a result, events can be classified and correlated using any of the values provided by the event set.

For example, an attack originating from a particular IP address may result in events across many different systems. With unstructured logs, the IP address is not available without pattern-based parsing:

A basic unstructured syslog event, difficult to correlate

<38>Nov 22 10:30:12 myhost sshd[8459]: Failed password for invalid user linda from 192.168.1.60 port 38176 ssh2

The attack can be analyzed much more easily, even across a diverse set of log sources, if all events are structured and include a SourceIPAddress field:

An excerpt of the same event as structured data, easier to correlate

{
  "SourceName": "sshd",
  "ProcessID": 8459,
  "Message": "Failed password for invalid user linda from 192.168.1.60 port 38176 ssh2",
  "Status": "failed",
  "AuthenticationMethod": "password",
  "Reason": "invalid user",
  "SourceIPAddress": "192.168.1.60",
  "SourcePort": 38176,
  "Protocol": "ssh2"
}

To correlate and filter events across diverse log sources, it is important to use a common set of fields. These field names can be used when parsing unstructured logs, or can be mapped from equivalent fields in a structured log source.

Converting between formats for integration

Structured data can be serialized to JSON and used as a syslog payload

Structured logging often simplifies conversion between formats, allowing event data to be more easily integrated with logging dashboards, analytics platforms, and other log processors. Structured logs can also help work around limitations in log formats.

For example, to transport structured data using syslog, the key-value pairs can be serialized to JSON and used as the payload. In the case of BSD syslog, which lacks a proper timestamp, the full timestamp with year can be preserved in the serialized JSON data (see EventTime below).

BSD syslog event with JSON payload

<13>Nov 22 10:30:12 myhost sshd[8459]: {"SyslogFacility":"USER","SyslogSeverity":"NOTICE","EventTime":"2024-11-22 10:30:12","Hostname":"myhost","SourceName":"sshd","ProcessID":8459,"Message":"Failed password for invalid user linda from 192.168.1.60 port 38176 ssh2","Status":"failed","AuthenticationMethod":"password","Reason":"invalid user","User":"linda","SourceIPAddress":"192.168.1.60","SourcePort":38176,"Protocol":"ssh2"}

Or the key-value pairs can be used as the structured data portion of the IETF syslog format:

IETF syslog event with structured data

<13>1 2024-11-22T10:30:12.000000-06:00 myhost sshd 8459 - [NXLOG@14506 EventReceivedTime="2024-02-06 21:04:36" SourceModuleName="in" SourceModuleType="im_file" Status="failed" AuthenticationMethod="password" Reason="invalid user" User="linda" SourceIPAddress="192.168.1.60" SourcePort="38176" Protocol="ssh2"] Failed password for invalid user linda from 192.168.1.60 port 38176 ssh2

This can be done for syslog, as in the the previous example, or many other log formats by either serializing fields into a free-form message area or using a key-value area specified by the format.

Increasing the reliability of event parsing

Structured logging reduces the dependence on pattern matching, which must be updated anytime there are changes in generated log messages

Because unstructured logs contain data in free-form messages, pattern-based parsing is often implemented to extract this data for analysis. When there are multiple points of analysis, for example by both an intrusion detection system (IDS) and a log monitoring application, there may be multiple pattern databases used and these patterns may be implemented in different formats.

When the message format of an unstructured log source changes, the patterns stop matching and message parsing fails. The failed parsing must be investigated and all applicable pattern databases updated to correspond with the new message format. In particular, if a log collector reads the rendered message from Windows Event Log rather than the structured data, an added field can break parsing when included in the human-readable message.

It is not necessary to maintain pattern databases for log sources such as Windows Event Log that are available as structured data. Instead, the structured data should be collected directly. For unstructured logs, the parsing should be done as soon as possible and the events handled as structured data. In either case, once the log data is structured, fields can be freely added to events at any point in the log processing chain.

Structured logging best practices

Consider the following best practices when deploying your structured logging system:

Select a consistent data structure schema.
Store the data in an easily parseable format, such as JSON or XML.
Classify your log entries with severity levels for easier analysis.
Ensure your contextual metadata is complete (IDs, timestamp, environment, hostname, etc).
Protect sensitive data (for example, for GDPR compliance) and discard unnecessary data.

NXLog Platform: designed for structured logging

Modern enterprise systems generate an overwhelming amount of log data which must be collected, transported, managed, and monitored. Environments with complex logging requirements can benefit greatly from structured logging.

NXLog Platform’s log collection technology is designed around the concept of structured logging.

Internally structured events: When NXLog Platform’s log collector receives an event, it represents the events by a set of fields. Any changes to an event are performed by reading from and writing to those fields. Extraction of unstructured log data can be done with regular expressions, XML pattern databases, or Grok patterns.
Direct collection and storage of structured logs: NXLog Platform’s log collector integrates with many log sources and destinations and is capable of reading logs directly from, or writing to, structured sources. For example, collecting logs from the Windows Event Log API, different databases, or structured formats such as JSON or XML, and mapping the data directly to event record fields.
Event classification and correlation: NXLog Platform can classify, correlate, and filter events according to fields, as required for log processing and alerting. Further analytics, if necessary, can be performed after log data has been collected and processed by NXLog Platform.
Flexible configuration: NXLog Platform’s flexible log collection configuration is tailored to structured logging, allowing reading, writing, and converting log data between many different formats and transports. Additionally, configuration templates accelerate the integration of log sources with a collection of popular SIEMs by defining the configurations to collect, parse, and analyze the most useful log record fields.

Besides these benefits, NXLog Platform further supports the structured logging strategy through efficient log management and analytics. You can visualize and manage your log data and powerful search facilities allow you to keep track of your security, system health, or performance metrics across diverse infrastructures.

NXLog Platform is purpose-built to handle structured logging, and support direct collection, processing, and storage of structured logs, streamlining event classification and correlation. For a deeper analysis on log management, read our log management best practices white paper. For more information on how NXLog Platform can improve efficiency, read our blog post on optimizing log management and cutting costs.

Frequently asked questions

Consider some of the most common questions asked on this topic:

Q: What are the benefits of structured logging?

A: The main benefits of structured logging are:

Consistent formatting.
More precise and efficient search.
Easier integration with log analysis tools.
Scalability.
Simplifies compliance and auditing.

Q: Should I use structured or unstructured logging for my specific use case?

A: The following table provides a comparison of structured and unstructured logging.

Table 1. Structured vs. unstructured logging
Scope	Structured logging	Unstructured logging
Format	Consistent key-value pairs.	Freeform in plain text.
Storage overheard	Small overhead due to structure fields.	No overhead but more difficult to optimize.
Initial implementation	More complex.	Quicker but harder to adapt later.
Data consistency	Enforced by data structure.	Completely dependent on data source.
Content search	Field-based: fast and precise.	Text-based: slower and potentially imprecise.
Scalability	More adaptable for large systems.	Inefficient as data volume increases.

Q: Can one solution manage both structured and unstructured logs effectively?

A: Yes. It’s often beneficial to use a versatile log management solution, even if during the initial deployment your requirements lean fully towards a single data storage methodology. NXLog Platform, for instance, can handle and parse unstructured and structured logs simultaneously.

Conclusion

Structured logging is essential for modern log management, enabling more efficient parsing, conversion, and correlation of log data across diverse sources. By representing each event as a set of key-value pairs, structured logging enhances data accessibility, reduces the need for complex pattern-based parsing, and facilitates seamless integration with various log management systems.

By adopting structured logging, enterprises can significantly boost the value of their log management strategy, leading to better system performance, security monitoring, and overall operational efficiency.

NXLog Platform is an on-premises solution for centralized log management with
versatile processing forming the backbone of security monitoring.

With our industry-leading expertise in log collection and agent management, we comprehensively
address your security log-related tasks, including collection, parsing, processing, enrichment, storage, management, and analytics.

Start free Contact us