- Introduction
- Deployment
- Configuration
- 23. Configuration overview
- 24. NXLog language
- 25. Reading and receiving logs
- 26. Processing logs
- 27. Forwarding and Storing Logs
- 28. Centralized Log Collection
- 29. NXLog failover mode
- 30. High Availability (HA)
- 31. Encrypted transfer
- 32. Reducing bandwidth and data size
- 33. Reliable message delivery
- 34. Compression and Encryption
- OS Support
- Integration
- Troubleshooting
- Enterprise Edition Reference Manual
- NXLog Manager
- NXLog Add-Ons
27. Forwarding and Storing Logs
This chapter discusses the configuration of NXLog outputs, including:
-
converting log messages to various formats,
-
forwarding logs over the network,
-
writing logs to files and sockets,
-
storing logs in databases,
-
sending logs to an executable, and
27.1. Generating Various Formats
The data format used in an outgoing log message must be considered in addition to the transport protocol. If the message cannot be parsed by the receiver, it may be discarded or improperly processed. See also Parsing various log formats.
- Syslog
-
There are two Syslog formats, the older BSD Syslog (RFC 3164) and the newer IETF Syslog (RFC 5424). The transport protocol in Syslog can be UDP, TCP, or SSL. The xm_syslog module provides procedures for generating Syslog messages. For more information, see Generating Syslog.
Example 142. Generating Syslog and Sending Over TCPThis configuration uses the to_syslog_ietf() procedure to convert the corresponding fields in the event record to a Syslog message in IETF format. The result is forwarded over TCP by the om_tcp module.
- Syslog Snare
-
The Snare agent format is a special format on top of BSD Syslog which is used and understood by several tools and log analyzer frontends. This format is most useful when forwarding Windows events in conjunction with im_mseventlog and/or im_msvistalog. The to_syslog_snare() procedure can construct Syslog Snare formatted messages. For more information, see Generating Snare logs.
Example 143. Generating Syslog Snare and Sending Over UDPIn this example, the to_syslog_snare() procedure converts the corresponding fields in the event record to Snare format. The messages are then forwarded over UDP by the om_udp module.
- NXLog Binary Format
-
The Binary format is only understood by NXLog. All the fields are preserved when the data is sent in this format, so there is no need to parse it again. It can be applied in cases when direct viewing, editing, or forwarding data to third-party applications is not required. For more details, see the Preservation of Forwarded Data section in the User Guide.
This format is the recommended format when sending data between NXLog instances as it preserves type information and all the fields. It is also possible to send data between NXLog instances in other formats, but the characteristics of the chosen format needs to be considered. For example, JSON has no datetime type, but it is more compliant to standards and a text based human readable format.
Example 144. Generating Binary Format and Sending Over TCP - Graylog Extended Log Format (GELF)
-
The xm_gelf module can be used to generate GELF output.
- JSON
-
This is one of the most popular formats for interchanging data between various systems. The xm_json module provides procedures for generating JSON messages by using data from the event record.
- Windows Event Log
-
This format can contain the details of both system and application events, which can be helpful while troubleshooting problems in Windows operating systems. The im_mseventlog and im_msvistalog modules collect Windows Event Log messages.
After parsing, Event Log data can be captured and processed by using any operating system.
Example 147. Generating Windows Event Log Messages and Sending Over UDPWith this configuration, NXLog reads Windows Event Log entries and selects only those entries which contain IDs 4624 (an account was successfully logged on) and 4647 (user initiated logoff) using the im_msvistalog module. Sending over UDP is carried out using the om_udp module.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
<Input from_eventlog> Module im_msvistalog <QueryXML> <QueryList> <Query Id="0"> <Select Path="Security"> *[System[Level=0 and (EventID=4624 or EventID=4647)]] </Select> </Query> </QueryList> </QueryXML> </Input> <Output to_udp> Module om_udp Host 192.168.31.93 Port 10500 </Output>
27.2. Forwarding Over the Network
After the data are converted to the required format as described in the Generating Various Formats section, they can be forwarded using various protocols in different formats, including raw data. For each protocol, there is a trade-off between speed, reliability, compatibility, and security.
- UDP
-
To send logs as UDP datagrams, use the om_udp module.
WarningUDP packets can be dropped by the operating system because the protocol does not guarantee reliable message delivery. It is recommended to use TCP or TLS/SSL instead if message loss is a concern. Example 148. Using the om_udp ModuleThis example provides configurations to forward data to the specified host via UDP.
The configuration below converts and forwards log messages in Graylog Extended Log Format (GELF).
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
<Extension gelf> Module xm_gelf </Extension> <Input in> Module im_file File "/tmp/input" </Input> <Output out> Module om_udp Host 192.168.1.1 Port 514 OutputType GELF_UDP </Output>
The below configuration sample forwards data via UDP in JSON format.
- TCP
-
To send logs over TCP, use the om_tcp module.
Example 149. Using the om_tcp ModuleIn this example, log messages are forwarded to the specified host via TCP.
The configuration below provides forwarding data as a Syslog message in IETF format.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
<Extension _syslog> Module xm_syslog </Extension> <Input in> Module im_file File "/tmp/input" </Input> <Output out> Module om_tcp Host 192.168.1.1 Port 1514 Exec to_syslog_ietf(); </Output>
The below configuration sample forwards messages without transforming them.
- SSL/TLS
-
To send logs over a trusted, secure SSL connection, use the om_ssl module.
Example 150. Using the om_ssl ModuleThis example provides nearly identical behavior to the TCP example above, but in this case SSL is used to securely transmit the data.
The configuration below enables forwarding raw data over SSL/TLS using a self-signed certificate.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19
<Input in> Module im_file File '/tmp/input' </Input> <Output out> Module om_ssl Host 192.168.0.127 Port 10500 OutputType Binary # Allows using self-signed certificates AllowUntrusted TRUE # Certificate from the peer host CAFile /tmp/peer_cert.pem # Certificate file CertFile /tmp/cert.pem # Keypair file CertKeyFile /tmp/key.pem </Output>
The below configuration sample forwards data over SSL/TLS in JSON format using a trusted CA certificate.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
<Input in> Module im_file File '/tmp/input' </Input> <Extension json> Module xm_json </Extension> <Output out> Module om_ssl Host 192.168.0.127 Port 10500 # Allows using self-signed certificates AllowUntrusted FALSE # Certificate from the peer host CAFile /tmp/peer_cert.pem # Certificate file CertFile /tmp/cert.pem # Keypair file CertKeyFile /tmp/key.pem Exec to_json(); </Output>
- HTTP(S)
-
To send logs over an HTTP or HTTPS connection, use the om_http module.
Example 151. Using the om_http ModuleThis example provides configurations for forwarding data via HTTP to the specified HTTP address.
Using the below configuration sample, NXLog will send raw data in text form using a POST request for each log message.
1 2 3 4 5 6 7 8 9
<Input in> Module im_file File '/tmp/input' </Input> <Output out> Module om_http URL http://server:8080/ </Output>
The configuration below will forward data in Graylog Extended Log Format (GELF) over HTTPS using a trusted certificate.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21
<Extension gelf> Module xm_gelf </Extension> <Input in> Module im_file File "/tmp/input" </Input> <Output out> Module om_http URL https://server:8080/ # Allows using self-signed certificates HTTPSAllowUntrusted FALSE # Certificate from the peer host HTTPSCAFile /tmp/peer_cert.pem # Certificate file HTTPSCertFile /tmp/cert.pem # Keypair file HTTPSCertKeyFile /tmp/key.pem </Output>
27.3. Sending to Files and Sockets
- Files
-
To store logs in local files, use the om_file module. See also Writing Syslog to File.
- Unix Domain Socket
-
To send logs to a Unix domain socket, use the om_uds module. See also Sending Syslog to the Local Syslog Daemon via /dev/log.
27.4. Storing in Databases
The om_dbi and om_odbc modules can be used to store logs in databases. The om_dbi module can be used on POSIX systems where libdbi is available. The om_odbc module, available in NXLog Enterprise Edition, can be used with ODBC compatible databases on Windows, Linux, and Unix.
This configuration uses libdbi and the pgsql
driver to insert events
into the specified database. The SQL statement references fields in
the event record to be added to the database.
1
2
3
4
5
6
7
8
9
10
11
12
<Output out>
Module om_dbi
SQL INSERT INTO log (facility, severity, hostname, timestamp, application, \
message) \
VALUES ($SyslogFacility, $SyslogSeverity, $Hostname, '$EventTime', \
$SourceName, $Message)
Driver pgsql
Option host 127.0.0.1
Option username dbuser
Option password secret
Option dbname logdb
</Output>
This example inserts events into the database specified by the ODBC connection string. In this case, the sql_exec() and sql_fetch() functions are used to interact with the database.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
<Output out>
Module om_odbc
ConnectionString DSN=mysql_ds;username=mysql;password=mysql;database=logdb;
<Exec>
if ( sql_exec("INSERT INTO log (facility, severity, hostname, timestamp,
application, message) VALUES (?, ?, ?, ?, ?, ?)",
1, 2, "host", now(), "app", $raw_event) == TRUE )
{
if ( sql_fetch("SELECT max(id) as id from log") == TRUE )
{
log_info("ID: " + $id);
if ( sql_fetch("SELECT message from log WHERE id=?", $id) == TRUE )
log_info($message);
}
}
</Exec>
</Output>
27.5. Sending to Executables
Using the om_exec module, all messages can be piped to an external program or script which will run until the module (or NXLog) is stopped.