27. Forwarding and Storing Logs

Table of Contents

This chapter discusses the configuration of NXLog outputs, including:

converting log messages to various formats,
forwarding logs over the network,
writing logs to files and sockets,
storing logs in databases,
sending logs to an executable, and
forwarding raw data over TCP, UDP, and TLS/SSL protocols.

27.1. Generating Various Formats

The data format used in an outgoing log message must be considered in addition to the transport protocol. If the message cannot be parsed by the receiver, it may be discarded or improperly processed. See also Parsing various log formats.

Syslog

There are two Syslog formats, the older BSD Syslog (RFC 3164) and the newer IETF Syslog (RFC 5424). The transport protocol in Syslog can be UDP, TCP, or SSL. The xm_syslog module provides procedures for generating Syslog messages. For more information, see Generating Syslog.

Example 142. Generating Syslog and Sending Over TCP

This configuration uses the to_syslog_ietf() procedure to convert the corresponding fields in the event record to a Syslog message in IETF format. The result is forwarded over TCP by the om_tcp module.

nxlog.conf


  1
2
3
4
5
6
7
8
9
10

  <Extension _syslog>
    Module  xm_syslog
</Extension>

<Output out>
    Module  om_tcp
    Host    192.168.1.1
    Port    1514
    Exec    to_syslog_ietf();
</Output>

Syslog Snare

The Snare agent format is a special format on top of BSD Syslog which is used and understood by several tools and log analyzer frontends. This format is most useful when forwarding Windows events in conjunction with im_mseventlog and/or im_msvistalog. The to_syslog_snare() procedure can construct Syslog Snare formatted messages. For more information, see Generating Snare logs.

Example 143. Generating Syslog Snare and Sending Over UDP

In this example, the to_syslog_snare() procedure converts the corresponding fields in the event record to Snare format. The messages are then forwarded over UDP by the om_udp module.

nxlog.conf


  1
2
3
4
5
6
7
8
9
10

  <Extension _syslog>
    Module  xm_syslog
</Extension>

<Output out>
    Module  om_udp
    Host    192.168.1.1
    Port    514
    Exec    to_syslog_snare();
</Output>

NXLog Binary Format

The Binary format is only understood by NXLog. All the fields are preserved when the data is sent in this format, so there is no need to parse it again. It can be applied in cases when direct viewing, editing, or forwarding data to third-party applications is not required. For more details, see the Preservation of Forwarded Data section in the User Guide.

This format is the recommended format when sending data between NXLog instances as it preserves type information and all the fields. It is also possible to send data between NXLog instances in other formats, but the characteristics of the chosen format needs to be considered. For example, JSON has no datetime type, but it is more compliant to standards and a text based human readable format.

Example 144. Generating Binary Format and Sending Over TCP

With this configuration, NXLog reads a JSON file, converts it to NXLog binary format using the OutputType Binary and forwards over TCP using the om_tcp module. To accept a binary-formatted file, the receiver NXLog module instance can be set to InputType Binary.

nxlog.conf


  1
2
3
4
5
6
7
8
9
10
11

  <Input from_file>
    Module      im_file
    File        "/tmp/input.json"
</Input>

<Output to_tcp>
    Module      om_tcp
    Host        192.168.31.93
    Port        10500
    OutputType  Binary
</Output>

Graylog Extended Log Format (GELF)

The xm_gelf module can be used to generate GELF output.

Example 145. Generating GELF Output

With this configuration, NXLog sends the fields in the event record over UDP in GELF format.

nxlog.conf


  1
2
3
4
5
6
7
8
9
10

  <Extension _gelf>
    Module      xm_gelf
</Extension>

<Output out>
    Module      om_udp
    Host        127.0.0.1
    Port        12201
    OutputType  GELF_UDP
</Output>

JSON

This is one of the most popular formats for interchanging data between various systems. The xm_json module provides procedures for generating JSON messages by using data from the event record.

Example 146. Generating JSON and Sending Over TCP

With this configuration, NXLog sends the fields of the event record over TCP in JSON format.

nxlog.conf


  1
2
3
4
5
6
7
8
9
10

  <Extension json>
    Module    xm_json
</Extension>

<Output out>
    Module    om_tcp
    Host      192.168.1.1
    Port      1514
    Exec      to_json();
</Output>

Windows Event Log

This format can contain the details of both system and application events, which can be helpful while troubleshooting problems in Windows operating systems. The im_mseventlog and im_msvistalog modules collect Windows Event Log messages.

After parsing, Event Log data can be captured and processed by using any operating system.

Example 147. Generating Windows Event Log Messages and Sending Over UDP

With this configuration, NXLog reads Windows Event Log entries and selects only those entries which contain IDs 4624 (an account was successfully logged on) and 4647 (user initiated logoff) using the im_msvistalog module. Sending over UDP is carried out using the om_udp module.

nxlog.conf


  1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

  <Input from_eventlog>
    Module      im_msvistalog
    <QueryXML>
        <QueryList>
            <Query Id="0">
                <Select Path="Security">
                    *[System[Level=0 and (EventID=4624 or EventID=4647)]]
                </Select>
            </Query>
        </QueryList>
    </QueryXML>
</Input>

<Output to_udp>
    Module      om_udp
    Host        192.168.31.93
    Port        10500
</Output>

27.2. Forwarding Over the Network

After the data are converted to the required format as described in the Generating Various Formats section, they can be forwarded using various protocols in different formats, including raw data. For each protocol, there is a trade-off between speed, reliability, compatibility, and security.

UDP

To send logs as UDP datagrams, use the om_udp module.

Warning

UDP packets can be dropped by the operating system because the protocol does not guarantee reliable message delivery. It is recommended to use TCP or TLS/SSL instead if message loss is a concern.

Example 148. Using the om_udp Module

This example provides configurations to forward data to the specified host via UDP.

The configuration below converts and forwards log messages in Graylog Extended Log Format (GELF).

nxlog.conf


  1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

  <Extension gelf>
    Module          xm_gelf
</Extension>

<Input in>
    Module          im_file
    File            "/tmp/input"
</Input>

<Output out>
    Module          om_udp
    Host            192.168.1.1
    Port            514
    OutputType      GELF_UDP
</Output>

The below configuration sample forwards data via UDP in JSON format.

nxlog.conf


  1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

  <Extension json>
    Module          xm_json
</Extension>

<Input in>
    Module          im_file
    File            "/tmp/input"
</Input>

<Output out>
    Module          om_udp
    Host            192.168.1.1
    Port            514
    Exec            to_json();
</Output>

TCP

To send logs over TCP, use the om_tcp module.

Example 149. Using the om_tcp Module

In this example, log messages are forwarded to the specified host via TCP.

The configuration below provides forwarding data as a Syslog message in IETF format.

nxlog.conf


  1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

  <Extension _syslog>
    Module          xm_syslog
</Extension>

<Input in>
    Module          im_file
    File            "/tmp/input"
</Input>

<Output out>
    Module          om_tcp
    Host            192.168.1.1
    Port            1514
    Exec            to_syslog_ietf();
</Output>

The below configuration sample forwards messages without transforming them.

nxlog.conf


  1
2
3
4
5
6
7
8
9
10

  <Input in>
    Module          im_file
    File            "/tmp/input"
</Input>

<Output out>
    Module          om_tcp
    Host            192.168.0.127
    Port            10500
</Output>

SSL/TLS

To send logs over a trusted, secure SSL connection, use the om_ssl module.

Example 150. Using the om_ssl Module

This example provides nearly identical behavior to the TCP example above, but in this case SSL is used to securely transmit the data.

The configuration below enables forwarding raw data over SSL/TLS using a self-signed certificate.

nxlog.conf


  1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

  <Input in>
    Module          im_file
    File            '/tmp/input'
</Input>

<Output out>
    Module          om_ssl
    Host            192.168.0.127
    Port            10500
    OutputType      Binary
# Allows using self-signed certificates
    AllowUntrusted  TRUE
# Certificate from the peer host
    CAFile          /tmp/peer_cert.pem
# Certificate file
    CertFile        /tmp/cert.pem
# Keypair file
    CertKeyFile     /tmp/key.pem
</Output>

The below configuration sample forwards data over SSL/TLS in JSON format using a trusted CA certificate.

nxlog.conf


  1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

  <Input in>
    Module          im_file
    File            '/tmp/input'
</Input>

<Extension json>
    Module          xm_json
</Extension>

<Output out>
    Module          om_ssl
    Host            192.168.0.127
    Port            10500
# Allows using self-signed certificates
    AllowUntrusted  FALSE
# Certificate from the peer host
    CAFile          /tmp/peer_cert.pem
# Certificate file
    CertFile        /tmp/cert.pem
# Keypair file
    CertKeyFile     /tmp/key.pem
    Exec            to_json();
</Output>

HTTP(S)

To send logs over an HTTP or HTTPS connection, use the om_http module.

Example 151. Using the om_http Module

This example provides configurations for forwarding data via HTTP to the specified HTTP address.

Using the below configuration sample, NXLog will send raw data in text form using a POST request for each log message.

nxlog.conf


  1
2
3
4
5
6
7
8
9

  <Input in>
    Module          im_file
    File            '/tmp/input'
</Input>

<Output out>
    Module          om_http
    URL             http://server:8080/
</Output>

The configuration below will forward data in Graylog Extended Log Format (GELF) over HTTPS using a trusted certificate.

nxlog.conf


  1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

  <Extension gelf>
    Module                   xm_gelf
</Extension>

<Input in>
    Module                   im_file
    File                     "/tmp/input"
</Input>

<Output out>
    Module                   om_http
    URL                      https://server:8080/
# Allows using self-signed certificates
    HTTPSAllowUntrusted      FALSE
# Certificate from the peer host
    HTTPSCAFile              /tmp/peer_cert.pem
# Certificate file
    HTTPSCertFile            /tmp/cert.pem
# Keypair file
    HTTPSCertKeyFile         /tmp/key.pem
</Output>

27.3. Sending to Files and Sockets

Files

To store logs in local files, use the om_file module. See also Writing Syslog to File.

Example 152. Using the om_file Module

This configuration writes log messages to the specified file. No additional processing is performed by the output module instance.

nxlog.conf


  1
2
3
4

  <Output out>
    Module  om_file
    File    "/var/log/out.log"
</Output>

Unix Domain Socket

To send logs to a Unix domain socket, use the om_uds module. See also Sending Syslog to the Local Syslog Daemon via /dev/log.

Example 153. Using the om_uds Module

With this configuration, log messages are written to the specified socket without any additional processing.

nxlog.conf


  1
2
3
4

  <Output out>
    Module  om_uds
    UDS     /dev/log
</Output>

27.4. Storing in Databases

The om_dbi and om_odbc modules can be used to store logs in databases. The om_dbi module can be used on POSIX systems where libdbi is available. The om_odbc module, available in NXLog Enterprise Edition, can be used with ODBC compatible databases on Windows, Linux, and Unix.

Example 154. Using the om_dbi Module

This configuration uses libdbi and the pgsql driver to insert events into the specified database. The SQL statement references fields in the event record to be added to the database.

nxlog.conf


  1
2
3
4
5
6
7
8
9
10
11
12

  <Output out>
    Module  om_dbi
    SQL     INSERT INTO log (facility, severity, hostname, timestamp, application, \
                             message) \
            VALUES ($SyslogFacility, $SyslogSeverity, $Hostname, '$EventTime', \
                    $SourceName, $Message)
    Driver  pgsql
    Option  host 127.0.0.1
    Option  username dbuser
    Option  password secret
    Option  dbname logdb
</Output>

Example 155. Using the om_odbc Module

This example inserts events into the database specified by the ODBC connection string. In this case, the sql_exec() and sql_fetch() functions are used to interact with the database.

nxlog.conf


  1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

  <Output out>
    Module              om_odbc
    ConnectionString    DSN=mysql_ds;username=mysql;password=mysql;database=logdb;
    <Exec>
        if ( sql_exec("INSERT INTO log (facility, severity, hostname, timestamp,
                       application, message) VALUES (?, ?, ?, ?, ?, ?)",
                      1, 2, "host", now(), "app", $raw_event) == TRUE )
        {
            if ( sql_fetch("SELECT max(id) as id from log") == TRUE )
            {
                log_info("ID: " + $id);
                if ( sql_fetch("SELECT message from log WHERE id=?", $id) == TRUE )
                    log_info($message);
            }
        }
    </Exec>
</Output>

27.5. Sending to Executables

Using the om_exec module, all messages can be piped to an external program or script which will run until the module (or NXLog) is stopped.

Example 156. Using the om_exec Module

This configuration executes the specified command and writes log messages to its standard input.

nxlog.conf


  1
2
3
4
5

  <Output out>
    Module  om_exec
    Command /usr/bin/someprog
    Arg     -
</Output>

You are here

27. Forwarding and Storing Logs | Log Collection Solutions