133.17. Raijin (om_raijin)

Table of Contents

This module enables logs to be forwarded to Raijin Database servers for ingestion. It connects to the URL specified in the configuration in either HTTP or HTTPS mode. Raijin accepts HTTP POST requests with multiple JSON records in the request body, assuming that there is a target database table already created on the Raijin side. Although Raijin accepts structured data in flat JSON (i.e. a list of key-value pairs) format, it does not support JSON values comprised of nested data structures or other non-scalar data such as arrays and maps. Raijin currently does not support authorization/SSL but the om_raijin module supports TLS since TLS can be enabled with an HTTP proxy.

133.17.1. Batch processing

Starting with NXLog version 5.2, this module takes advantage of an internal batch processing mechanism which is provided by the NXLog core.

Internal batching works by accumulating multiple successive event records produced by an input or processor module into a batch of records. When a batch is full or after a configurable timeout expires, the batch is forwarded to the next module in the route. For it’s own batching functionality, the om_raijin module now uses the already available batches it receives from the previous module, instead of reorganizing individual event records into batches itself.

Simply put, the new batching functionality works the same as before, with the difference that event records are organized into batches by input and processor modules, whereas in older versions, it was the output modules which did this. Because of this, the FlushInterval and FlushLimit configuration directives have been deprecated for om_raijin. The same functionality is now provided by the BatchFlushInterval and BatchSize directives, which can be used with all input and processor module(s).

Note	To examine the supported platforms, see the list of installer packages in the Available Modules chapter.

Note

This module requires the xm_json extension module to be loaded in order to convert the payload to JSON. If the $raw_event field is empty the fields will be automatically converted to JSON. If $raw_event contains a valid JSON string it will be sent as-is, otherwise a JSON record will be generated in the following structure: {"raw_event":"escaped raw_event content"}

133.17.2. Configuration

The om_raijin module accepts the following directives in addition to the common module directives.

These three om_raijin directives are required:

DBName: This mandatory directive specifies the name of the Raijin database containing the table where logs will be stored.

DBTable: This mandatory directive specifies the name of the database table where log events will be inserted as JSON records.

URL

This mandatory directive is comprised of the REST API URI that Raijin Server uses for receiving bulk, JSON-formatted events via POST requests:

http://<RaijinServerHostname>:<port>/_bulk

If an om_raijin module instance contains multiple URL directives, the hosts will function as a cluster in NXLog failover mode. This directive also supports HTTPS URLs. If the optional port number part of the URL (:<port>) is not explicitly included in the URL, it will default to port 80 for HTTP (or port 443 for HTTPS). The /_bulk endpoint part of the URL is required and will return a 400 Bad Request error if missing.

Note	The Raijin Database REST API listens by default on port 2500. If you plan to use this directive without explicitly specifying a port number, make sure the Raijin Database server is configured to listen on the correct default port corresponding to this directive (either 80 or 443).

The following om_raijin directives are optional.

FlushInterval: This directive has been deprecated. See Batch processing for details.

FlushLimit: This directive has been deprecated. See Batch processing for details.

HTTPSAllowUntrusted: Accepts a boolean value to determine whether the remote connection should be allowed without certificate verification. If set to TRUE, the connection will be allowed even if the remote HTTPS server presents an unknown or self-signed certificate. The default value is FALSE; the remote HTTPS server must present a trusted certificate.

HTTPSCADir: Specifies the path to a directory containing certificate authority (CA) certificates which will be used to check the certificate of the remote HTTPS server. The certificate filenames in this directory must be in the OpenSSL hashed format. A remote’s self-signed certificate (which is not signed by a CA) can also be trusted by including a copy of the certificate in this directory.

HTTPSCAFile: Specifies the path to the certificate authority (CA) certificate which will be used to validate the certificate of the remote HTTPS server. To trust a self-signed certificate presented by the remote host (which is not signed by a CA), provide that certificate instead.

HTTPSCertFile: Specifies the path of the certificate file to be used for the HTTPS handshake.

HTTPSCertKeyFile: Specifies the path of the certificate key file to be used for the HTTPS handshake.

HTTPSCRLDir: Specifies the path to a directory containing certificate revocation lists (CRLs), which will be consulted when checking the certificate of the remote HTTPS server. The certificate filenames in this directory must be in the OpenSSL hashed format.

HTTPSCRLFile: Specifies the path of the certificate revocation list (CRL) which will be consulted when checking the certificate of the remote HTTPS server.

HTTPSKeyPass: Specifies a password for the certificate key file defined in HTTPSCertKeyFile. This directive is not needed for passwordless private keys.

HTTPSSSLCipher: Sets the permitted SSL cipher list, overriding the default. Use the format described in the ciphers man page.

HTTPSSSLCiphersuites: This optional directive can be used to define the permitted SSL cipher list in case the HTTPSSSLProtocol directive is set to TLSv1.3. Use the same format as in the HTTPSSSLCipher directive.

HTTPSSSLCompression: Accepts a boolean value to determine whether data compression should be enabled. The compression algorithm is based on the zlib compression library. If not specified, it defaults to FALSE (compression will be disabled).

Note	Some Linux packages (for example, Debian) use the OpenSSL library provided by the OS and may not support the zlib compression mechanism. The module will emit a warning on startup if compression support is missing. The generic deb/rpm packages are bundled with a zlib-enabled libssl library.

HTTPSSSLProtocol: This directive can be used to set the allowed SSL/TLS protocol(s). It takes a comma-separated list of values which can be any of the following: SSLv2, SSLv3, TLSv1, TLSv1.1, TLSv1.2 and TLSv1.3. By default, the TLSv1.2 and TLSv1.3 protocols are allowed. Note that the OpenSSL library shipped by Linux distributions may not support SSLv2 and SSLv3, and these will not work even if enabled with this directive.

ProxyAddress: Specifies the IP address of the proxy server in case the module needs to connect to a Raijin Database server through a proxy.

Note	The om_raijin module supports only HTTP proxies, i.e. SOCKS4/SOCKS5 proxies are not supported.

ProxyPort: Specifies the port number required to connect to the proxy server.

SNI: Specifies the hostname used for Server Name Indication (SNI) in HTTPS mode.

133.17.3. Examples

Example 781. Sending logs to a Raijin Database server on localhost

This configuration reads log messages from a file and forwards them to a Raijin Database server deployed on localhost.

nxlog.conf


  1
2
3
4
5
6
7
8
9
10

  <Extension json>
    Module              xm_json
</Extension>

<Output raijin_local>
    Module              om_raijin
    URL                 http://localhost:2500/_bulk
    DBName              logdb
    DBTable             logs
</Output>

Example 782. Sending logs to a Raijin Database cluster of servers in NXLog failover mode

This configuration sends logs to a cluster of Raijin Database servers configured for NXLog failover mode. The cluster is comprised of three hosts: http://raijin-1.example.com:2500/_bulk, http://localhost:2500/_bulk, and http://192.168.1.123:2500/_bulk.

nxlog.conf


  1
2
3
4
5
6
7
8
9
10
11
12

  <Extension json>
    Module  xm_json
</Extension>

<Output raijin_cluster>
    Module  om_raijin
    URL     http://raijin-1.example.com:2500/_bulk
    URL     http://localhost:2500/_bulk
    URL     http://192.168.1.123:2500/_bulk
    DBName  logdb
    DBTable logs
</Output>

You are here

133.17. Raijin (om_raijin) | Log Collection Solutions