Event records and fields

In NXLog, a log message is an event, and the data relating to that event is collectively an event record. When NXLog processes an event record, it stores the various values in fields. The following sections describe event records and fields in the context of NXLog processing.

Event records

There are many kinds of event records. A few important ones are listed here.

The most common event record is a single line. Thus the default is LineBased for the InputType and OutputType directives.
It is also common for an event record to use a single UDP datagram. NXLog can send and receive UDP events with the im_udp and om_udp modules.
Some event records are generated using multiple lines. These can be joined into a single event record with the xm_multiline module.
Event records may be stored in a database. Each row in the database represents an event. In this case the im_odbc and om_odbc modules can be used.
It is common for structured event records to be formatted in CSV, JSON, or XML formats. The xm_csv, xm_json, and xm_xml modules provide functions and procedures for parsing these.
NXLog provides a Binary InputType and OutputType for use when compatibility with other logging software is not required. This format preserves parsed fields and their types.

In NXLog, each event record consists of the raw event data (in a field named $raw_event) and additional fields generated during processing and parsing.

Event fields

All event log messages contain important data such as user names, IP addresses, and application names. Traditionally, these logs have been generated as free form text messages prefixed with basic metadata like the time of the event and a severity value.

While this format is easy for humans to read, it is difficult to perform log analysis and filtering on thousands of free-form logs. In contrast, structured logging provides means for matching messages based on key-value pairs. With structured logging, an event is represented as a list of key-value pairs. The name of the field is the key and the field data is the value. NXLog’s core design embraces structured logging. Using various features provided by NXLog, a message can be parsed into a list of key-value pairs for processing or as part of the message sent to the destination.

When a message is received by NXLog, it creates an internal representation of the log message using fields. Each field is typed and represents a particular attribute of the message. These fields pass through the log route, and are available in each successive module in the chain, until the log message has been sent to its destination.

The special $raw_event field contains the raw data received by the input module. Most input and output modules only transfer $raw_event by default.
The core adds a few additional fields by default:
1. $EventReceivedTime (type: datetime) The time when the event is received. The value is not modified if the field already exists.
2. $SourceModuleName (type: string) The name of the module instance, for input modules. The value is not modified if the field already exists.
3. $SourceModuleType (type: string) The type of module instance (such as im_file), for input modules. The value is not modified if the field already exists.
The input module may add other fields. For example, the im_udp module adds a $MessageSourceAddress field.
Some input modules, such as im_msvistalog and im_odbc, map fields from the source directly to fields in the NXLog event record.
Parsers such as the parse_syslog() procedure will add more fields.
Custom fields can be added by using the NXLog language and an Exec directive.
The NXLog language or the pm_pattern module can be used to set fields using regular expressions. See Extracting data.

When the configured output module receives the log message, in most cases it will use the contents of the $raw_event field only. If the event’s fields have been modified, it is therefore important to update $raw_event from the other fields. This can be done with the NXLog language, perhaps using a procedure like to_syslog_bsd().

A field is denoted and referenced in the configuration by a preceding dollar sign ($). See the Fields section in the Reference Manual for more information.

Example 1. Processing a syslog message

This example shows a syslog event and its corresponding fields as processed by NXLog. A few fields are omitted for brevity.

NXLog receives an event:

<38>Nov 22 10:30:12 myhost sshd[8459]: Failed password for invalid user linda from 192.168.1.60 port 38176 ssh2

The raw event data is stored in the $raw_event field when NXLog receives a log message. The NXLog core and input module add additional fields.

{
  "raw_event": "<38>Nov 22 10:30:12 myhost sshd[8459]: Failed password for invalid user linda from 192.168.1.60 port 38176 ssh2",
  "EventReceivedTime": "2019-11-22 10:30:13",
  "MessageSourceAddress": "192.168.1.1",

The xm_syslog parse_syslog() procedure parses the basic format of the Syslog message, reading from $raw_event by default. This procedure adds a few more fields:

  "SyslogFacility": "USER",
  "SyslogSeverity": "NOTICE",
  "EventTime": "2019-11-22 10:30:12",
  "Hostname": "myhost",
  "SourceName": "sshd",
  "ProcessID": 8459,
  "Message": "Failed password for invalid user linda from 192.168.1.60 port 38176 ssh2",

Further metadata can be extracted from the free-form $Message field with regular expressions or other methods; see Extracting data.

  "Status": "failed",
  "AuthenticationMethod": "password",
  "Reason": "invalid user",
  "User": "linda",
  "SourceIPAddress": "192.168.1.60",
  "SourcePort": 38176,
  "Protocol": "ssh2"
}

Preservation of forwarded data

By default, when the OutputType directive is set to LineBased, only data of the $raw_event field can be forwarded over the network. Contents of other fields from the event record are discarded if not initially converted to a structured format such as JSON, XML, or CSV. Among these formats, JSON provides human-readable syntax, wide support from various software platforms, and inclusion of nested structures.

For example, below is the log message containing the EventTime and Message fields.

"EventTime":2020-05-13 00:00:00
"Message":"The service has started"

After applying to_json() procedure, the message obtains the additional raw_event field containing the EventTime and Message fields in the JSON format.

"EventTime":2020-05-13 00:00:00
"Message":"The service has started"
"raw_event":"{"EventTime":"2020-05-13 00:00:00", "Message":"The service has started"}"

When forwarding over TCP via the default LineBased type, NXLog sends only the contents of the raw_event field.

{"EventTime":"2020-05-13 00:00:00", "Message":"The service has started"}

After this message has been forwarded, the raw_event name returns to it and and other fields are added automatically to create the event record.

"raw_event":"{"EventTime":"2020-05-13 00:00:00", "Message":"The service has started"}"
...

Before it can be processed by the receiving NXLog instance, the event record should be parsed. Consequently, preservation of all fields in the LineBased mode requires conversion and parsing procedures before and after forwarding.

Application of the Binary function in the OutputType directive preserves all fields for forwarding without conversion and parsing procedures, so the forwarded event record will contain unmodified fields.

"EventTime":2020-05-13 00:00:00
"Message":"The service has started"

The binary format is best suited the following cases:

Logs needing long-term archival for compliance reasons
Logs that need to preserve timestamps
Logs unlikely to be read or processed by other applications.

Application of both approaches to the same set of data, i.e. conversion to JSON and NXLog binary formats will duplicate and significantly increase the size of forwarded data.

"EventTime":2020-05-13 00:00:00
"Message":"The service has started"
"raw_event":"{"EventTime":"2020-05-13 00:00:00", "Message":"The service has started"}

The examples below provide detailed explanations of preservation via JSON and NXLog binary format.

Example 2. Preserving data via JSON

This example demonstrates how to preserve forwarded data through conversion to JSON. The sample of the forwarded message is shown below.

"EventTime":2020-05-13 00:00:00 "Message":"The service has started"

The configuration below defines the EVENT_REGEX regular expression to read messages, and the Exec block contains parsing instructions as per this expression.

nxlog.conf (sender)

define EVENT_REGEX /(?x)^"EventTime":(\d+-\d+-\d+)\s+\
                    (\d+:\d+:\d+)\s+"Message":"(.*)"/

<Extension json>
    Module         xm_json
</Extension>

<Input from_file>
    Module         im_file
    File           '/tmp/input'
    <Exec>
        if $raw_event =~ %EVENT_REGEX%
            {
                $EventTime = strptime($1 + $2,'%Y-%m-%d %T');
                $Message = $3;
                to_json();
            }
        else drop();
    </Exec>
</Input>

<Output to_tcp>
     Module        om_tcp
     Host          192.168.31.122
     Port          10500
</Output>

After the message is parsed, this configuration creates the following fields for the event record:

$EventTime is combined from the date and time values of the source log message using the strptime() function,
$Message contains The service has started text of the message.
Fields that are added by NXLog automatically:
- $EventReceivedTime contains the time when the message was received by NXLog,
- $SourceModuleName contains the name of the module instance which initially received the message,
- $SourceModuleType contains the NXLog module name which initially received the message.

Finally, the event record is converted to JSON using to_json() procedure of the xm_json module and can be sent via TCP using the om_tcp module. The result of conversion is shown in the sample below.

Output sample

{
  "EventReceivedTime": "2020-06-18T09:06:03.891218+00:00",
  "SourceModuleName": "from_file",
  "SourceModuleType": "im_file",
  "EventTime": "2020-05-13T00:00:00.000000+00:00",
  "Message": "The service has started"
}

After forwarding, this message can be parsed into fields using the parse_json() procedure, see the configuration below.

nxlog.conf (receiver)

<Extension json>
    Module      xm_json
</Extension>

<Input from_tcp>
    Module      im_tcp
    Host        192.168.31.122
    Port        10500
    <Exec>
        parse_json();
        $EventTime = parsedate($EventTime);
        $Message = $Message;
    </Exec>      
</Input>

The example below demonstrates conversion to the Binary format.

Example 3. Preserving data via the NXLog binary format

The configuration below reads messages from a file. In the Exec block, the $raw_event length is calculated using the size() function. The message is dropped in case its length is not 5 characters or longer. Otherwise, the message length is stored into the $MessageSize field for forwarding over the network.

nxlog.conf (sender)

<Input from_file>
    Module          im_file
    File            '/tmp/input'
    ReadFromLast    TRUE
    <Exec>
        if size($raw_event) < 5 drop();
        $MessageSize = size($raw_event);
    </Exec>
</Input>

<Output to_tcp>
    Module          om_tcp
    Host            192.168.31.150
    Port            10500
    OutputType      Binary
</Output>

This configuration creates the following fields:

$raw_event contains the log message,
$EventReceivedTime contains the time when the message was received by NXLog,
$SourceModuleName contains the name of the module instance which initially received the message,
$SourceModuleType contains the NXLog module name which initially received the message,
$MessageSize contains the size of the $raw_event field.

Finally, the event record is converted to NXLog binary format and forwarded over the network using the om_tcp module.

To accept a message, the InputType directive of the im_tcp module instance can be configured to the Binary function.

nxlog.conf (receiver)

<Input from_tcp>
    Module          im_tcp
    Host            192.168.31.150
    Port            10500
    InputType       Binary
</Input>

<Output to_file>
    Module          om_file
    File            '/tmp/output'
    Exec            to_json();
</Output>