107. Sumo Logic

Table of Contents

Sumo Logic is a cloud-based service that collects, manages, and analyzes log data. NXLog can be configured to send log data to Sumo Logic in syslog format over TCP, or via a custom HTTP endpoint. Additionally, it can also be configured to send host metrics via HTTP.

Note	Sumo Logic offers different plans for their service including a free plan. Some of the methods described here may require a paid plan.

107.1. Using NXLog to collect data

Sumo Logic accepts data from two types of collectors, installed or hosted. Installed collectors are set up by installing agent software provided by Sumo Logic, whereas hosted collectors are used to send data over TCP or HTTP(S) from agents like NXLog. This section presents some scenarios in which using an NXLog agent with a Sumo Logic hosted collector has an advantage over using a Sumo Logic installed collector.

Flexible log processing

NXLog provides more control at the log processing stage before events are sent to the Sumo Logic service. This is especially useful since Sumo Logic charges per ingested byte. Although installed collectors support processing rules to drop or keep certain events, these are quite basic compared to the flexibility an NXLog agent provides. When used with NXLog, event data can be manipulated to determine which information to keep, rather than dropping an entire event, as well as having full control over the output format. See the topic on Reducing Bandwidth and Data Size for some of the configuration options available when using NXLog.

Support for a wider range of log sources

Sumo Logic collectors are designed to work with standard log formats primarily stored in flat files. When logs were not generated by a Sumo Logic supported log source or are not stored in a supported format, such as data stored in a database, network packet captures, or other types of events which are not in a standardized log format, it may not be possible to process such data with the Sumo logic collector. With its vast selection of input modules, NXLog can easily fill this gap as it can be configured to process a broad spectrum of log sources. Additionally, it can also process log data using custom scripts with support for Go, Java, Perl, Python and Ruby. See the NXLog documentation on Available Modules for a complete list of input, processing, and extension modules.

Platform requirements

The Sumo Logic collector is a Java-based agent that requires a substantial amount of resources. NXLog, with its smaller footprint, is a powerful alternative for use cases where the dependency on Java needs to be eliminated or a less resource intensive agent with more flexibility is preferred. Case in point: when installed on a Microsoft Windows 10 system and processing only the Application Windows Event Log channel, it was observed that the Sumo Logic collector consumes 200+ MB of memory, compared with 4-6 MB of memory usage by the NXLog agent when tested under the same conditions. See the table below for the minimum system requirements for both the NXLog and Sumo Logic agents. Furthermore, the Sumo Logic collector does not support collecting events in the older Microsoft Windows Event Log format as documented in this Sumo Logic article on Windows 2003 Event Logs ingestion. NXLog, on the other hand, specifically supports the Windows XP/2000/2003 event log format through its im_mseventlog input module and can be installed on systems running such older Microsoft Windows releases.

Table 105. NXLog and Sumo Logic minimum system requirements
	NXLog	Sumo Logic Collector
Processor Cores	1	1
Memory	60 MB	512 MB
Disk Space	50 MB	8 GB
Pre-requisites	-	Java 1.8+

107.2. Setting up a hosted collector

For Sumo Logic to receive data over TCP or HTTP(S), a hosted collector must be created from the Sumo Logic web interface. To create a hosted collector for data coming from NXLog:

Log in to the Sumo Logic web interface.
Navigate to Manage Data > Collection.
On the Collection tab, click on Add Collector.
Choose Hosted Collector, fill in the required details, and click Save.

For more details, see the Sumo Logic documentation on how to Configure a Hosted Collector.

107.3. Setting up the digital certificate

To be able to send data to Sumo Logic, a PEM-encoded DigiCert Certification Authority certificate is required. This section describes how to prepare the certificate on Linux and Microsoft Windows to be used by NXLog.

107.3.1. Linux

On Linux, the certificate needs to be downloaded from the DigiCert website and then the OpenSSL tool can be used to convert it.

Download the DigiCert DER encoded certificate:

$ sudo wget -O digicert_ca.der https://www.digicert.com/CACerts/DigiCertHighAssuranceEVRootCA.crt

Convert the certificate to a PEM-encoded certificate:

$ sudo openssl x509 -inform der -in digicert_ca.der -out digicert_ca.crt

Copy digicert_ca.crt to a location accessible by NXLog.

107.3.2. Microsoft Windows

On Microsoft Windows, the certificate can be exported using the Certificates MMC snap-in.

Go to the Windows Start menu, type certmgr.msc, and click Enter.
Expand Trusted Root Certificatation Authorities > Certificates.
From the list of certificates, right click on DigiCert High Assurance EV Root CA and select Open.
Go to the Details tab and click the Copy to File… button.
The Certificate Export Wizard opens, click Next.
Select Base-64 encoded X.509 (.CER) and click Next.
Click on the Browse… button and select a location accessible by NXLog.
Enter a filename e.g. digicert_ca.cer and click Save.
Click Next and then Finish to complete the export.

107.4. Sending logs to Sumo Logic using TCP

Sumo Logic accepts log data as syslog messages in IETF (RFC 5424) format and requires data to be sent using TLS v1.2 over TCP. To be able to use this method, a Cloud Syslog Source must be created in Sumo Logic.

From the Sumo Logic web interface, navigate to Manage Data > Collection.
On the Collection tab, click on Add Source next to the previously created hosted collector.
Select Cloud Syslog and fill in the required details. It is recommended to specify a Source Category to make events from this source easily searchable in Sumo Logic.
Click Save to finish creating the syslog source.
On creation of the new syslog source a dialog is displayed containing the Token, Host, and TCP Port. The NXLog configuration will need these details for sending log data to Sumo Logic.

For further details on configuring a syslog source, see the Sumo Logic documentation on Cloud Syslog Sources.

Note	Syslog messages must be compliant with RFC 5424 or they will be dropped by Sumo Logic. Messages over 64KB will be truncated.

Example 487. Sending syslog messages using TLS encryption

In this configuration, NXLog processes a log file using the im_file module, then generates a syslog message using the xm_syslog module and sends it to Sumo Logic using the om_ssl module. The log data is formatted as JSON using the xm_json module.

The SUMO_TOKEN value needs to be replaced with an actual Sumo Logic Syslog source token. In this example, processing is done in the output instance where the NXLog ID in the structured data of the syslog message is replaced with the token defined by the SUMO_TOKEN constant.

The SUMO_HOST value needs to be replaced with an actual Sumo Logic Syslog source host URL.

The SUMO_PORT value needs to be replaced with the correct port for the host defined by the SUMO_HOST constant.

The CA_FILE value needs to be replaced with the actual path to the DigiCert Certification Authority certificate.


  1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

  define SUMO_TOKEN    xxxxxxxxxxxxxxxxxxxx@41123
define SUMO_HOST     syslog.collection.<YOUR_DEPLOYMENT>.sumologic.com
define SUMO_PORT     6514
define CA_FILE       /opt/nxlog/cert/digicert_ca.crt

<Extension json>
    Module           xm_json
</Extension>

<Extension syslog>
    Module           xm_syslog
</Extension>

<Input file>
    Module           im_file
    File             "/var/log/file"
    Exec             parse_syslog();
    Exec             $Message = to_json();
</Input>

<Output sumo_tcp>
    Module           om_ssl
    Host             %SUMO_HOST%:%SUMO_PORT%
    CAFile           %CA_FILE%
    Exec             to_syslog_ietf();
    Exec             $raw_event =~ s/(\[NXLOG@14506.*?\])//g; \
                     $raw_event = replace($raw_event, \
                     '{', '[%SUMO_TOKEN%] {', 1);
</Output>

Output sample

This sample of the syslog header will be sent to Sumo Logic. The structured data contains the Sumo Logic token as specified in the NXLog configuration.

<13>1 2020-12-04T11:33:47 NXLog-Ubuntu-1 systemd 1 -
[xxxxxxxxxxxxxxxxxxxx@41123]

This sample JSON-formatted event will be sent as part of the syslog message to the Sumo Logic service.

{
  "EventReceivedTime": "2020-12-04 13:37:47",
  "SourceModuleName": "file",
  "SourceModuleType": "im_file",
  "SyslogFacilityValue": 1,
  "SyslogFacility": "USER",
  "SyslogSeverityValue": 5,
  "SyslogSeverity": "NOTICE",
  "SeverityValue": 2,
  "Severity": "INFO",
  "Hostname": "NXLog-Ubuntu-1",
  "EventTime": "2020-12-04 11:33:47",
  "SourceName": "systemd",
  "ProcessID": "1",
  "Message": "Started Run anacron jobs."
}

Example 488. Sending Windows Event Log events as syslog messages

In this configuration, NXLog processes Windows Event Log events using the im_msvistalog module, generates a syslog message using the xm_syslog module, and sends it to Sumo Logic using the om_ssl module. The log data is formatted as JSON using the xm_json module.

The SUMO_TOKEN value needs to be replaced with an actual Sumo Logic Syslog source token. In this example, processing is done in the output instance where the token defined by the SUMO_TOKEN constant is inserted before the JSON data.

The SUMO_HOST value needs to be replaced with an actual Sumo Logic Syslog source host URL.

The SUMO_PORT value needs to be replaced with the correct port for the host defined by the SUMO_HOST constant.

The CA_FILE value needs to be replaced with the actual path to the DigiCert Certification Authority certificate.


  1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

  define SUMO_TOKEN    xxxxxxxxxxxxxxxxxxxx@41123
define SUMO_HOST     syslog.collection.<YOUR_DEPLOYMENT>.sumologic.com
define SUMO_PORT     6514
define CA_FILE       C:\Program Files\nxlog\cert\digicert_ca.cer

<Extension json>
    Module           xm_json
</Extension>

<Extension syslog>
    Module           xm_syslog
</Extension>

<Input eventlog>
    Module           im_msvistalog
    <QueryXML>
        <QueryList>
            <Query Id='0'>
                <Select Path="System">*[System[(Level=1  or Level=2)]]</Select>
            </Query>
        </QueryList>
    </QueryXML>
    Exec             $Message = to_json();
</Input>

<Output sumo_tcp>
    Module           om_ssl
    Host             %SUMO_HOST%:%SUMO_PORT%
    CAFile           %CA_FILE%
    Exec             to_syslog_ietf();
    Exec             $raw_event =~ s/(\[.*])//g; \
                     $raw_event = replace($raw_event, \
                     '{', '[%SUMO_TOKEN%] {', 1);
</Output>

Output sample

This syslog header sample will be sent to Sumo Logic. The structured data contains the Sumo Logic token as specified in the NXLog configuration.

<11>1 2020-12-04T11:31:38.031887Z Hopper VBoxNetLwf 0 -
[xxxxxxxxxxxxxxxxxxxx@41123]

This sample JSON-formatted event will be sent as part of the syslog message to the Sumo Logic service.

{
  "EventTime": "2020-12-07 11:30:19",
  "Hostname": "Hopper",
  "Keywords": "36028797018963968",
  "EventType": "ERROR",
  "SeverityValue": 4,
  "Severity": "ERROR",
  "EventID": 12,
  "SourceName": "VBoxNetLwf",
  "TaskValue": 0,
  "RecordNumber": 44758,
  "ExecutionProcessID": 0,
  "ExecutionThreadID": 0,
  "Channel": "System",
  "Message": "The driver detected an error on \\Device\\VBoxNetLwf.",
  "Data": "\\Device\\VBoxNetLwf",
  "EventData.Binary": "00000C0001000000000000000C0004",
  "EventReceivedTime": "2020-12-07 11:31:38",
  "SourceModuleName": "eventlog",
  "SourceModuleType": "im_msvistalog"
}

107.5. Sending data to Sumo Logic using HTTPS

Logs and host metrics can be sent to Sumo Logic over HTTP(S) using a unique URL generated for each source. For NXLog to be able to send data over HTTPS, an HTTP Logs & Metrics Source must be created in Sumo Logic.

From the Sumo Logic web interface, navigate to Manage Data > Collection
On the Collection tab, click on Add Source next to the previously created hosted collector.
Select HTTP Logs & Metrics and fill in the required details. It is recommended to specify a Source Category to make events from this source easily searchable in Sumo Logic.
Click Save to finish creating the HTTP source.
On creation of the new HTTP source a dialog is displayed containing the HTTP Source Address. The NXLog configuration will need this URL for sending log data to Sumo Logic.

For further details on configuring an HTTP source, see the Sumo Logic documentation on HTTP Logs and Metrics Sources.

107.5.1. Sending log data

Example 489. Sending logs in batches over HTTPS

In this configuration, NXLog POST log data to Sumo Logic using the om_http module. To send data in batches, the BatchMode directive in the output instance needs to be set to multiline to specify that each log record should be separated by a new line.

Sumo Logic accepts batched requests up to 1MB of uncompressed data. For simplicity, this example uses the default NXLog batch settings. For further configuration options, see the BatchSize and BatchFlushInterval directives.

Data can be sent to Sumo Logic as plain uncompressed or compressed by the deflate or gzip method. The om_http module supports data compression based on the zlib compression library (deflate). In the configuration below, the HTTPSSSLCompression directive specifies that compression should be used when sending data to Sumo Logic. If this directive is not specified or set to FALSE, data will be sent uncompressed.

When sending data over HTTP(S), Sumo Logic accepts additional optional headers to configure custom settings related to the log records. For more information, see the Sumo Logic documentation on Supported HTTP Headers. The om_http module supports specifying additional headers by using the AddHeader directive. In the configuration below, the X-Sumo-Category header is added with the value my-category.

The SUMO_URL value needs to be replaced with an actual Sumo Logic HTTP source URL.

The CA_FILE value needs to be replaced with the actual path to the DigiCert Certification Authority certificate.


  1
2
3
4
5
6
7
8
9
10
11

  define SUMO_URL            https://<YOUR_SUMO_ENDPOINT>/receiver/v1/http/<UNIQUE_CODE>
define CA_FILE             /opt/nxlog/cert/digicert_ca.crt

<Output sumo_http>
    Module                 om_http
    URL                    %SUMO_URL%
    BatchMode              multiline
    HTTPSSSLCompression    TRUE
    AddHeader              X-Sumo-Category: my-category
    HTTPSCAFile            %CA_FILE%
</Output>

Note	It may take a few minutes for data to be shown in the Sumo Logic web interface. If not, see the Sumo Logic documentation on troubleshooting HTTP Sources.

107.5.2. Sending host metrics

In Sumo Logic terms, host metrics are a set of data points that measure the value of a property over time. For example, host metrics can be used to monitor the availability and performance of an application, or the resource usage of a host. For more information, see the Overview of Metrics in the Sumo Logic documentation.

Sumo Logic supports metrics in the Graphite, Carbon 2.0, and Prometheus formats. NXLog can be configured to send data in any of these formats, either by reading preformatted records from a file, or by using an Exec block to output the data in the desired format.

For more information on the requirements of host metrics, refer to the Sumo Logic documentation on Uploading Metrics to an HTTP Source and Metric Formats.

Example 490. Sending host metrics over HTTPS

This configuration illustrates how NXLog can retrieve Microsoft Windows performance counters using the im_winperfcount module and send the data to Sumo Logic in the Carbon 2.0 format using the om_http module.

The input instance is configured to poll the host’s available memory every 30 seconds. The field containing the retrieved value is renamed to MemFreeBytes to make it easier for referencing in the output instance.

The Exec block in the output instance builds the metric in the required Carbon 2.0 format. Sumo Logic accepts timestamps in seconds or milliseconds, therefore in this example, the NXLog timestamp is converted from microseconds to milliseconds.

The SUMO_URL value needs to be replaced with an actual Sumo Logic HTTP source URL.

The CA_FILE value needs to be replaced with the actual path to the DigiCert Certification Authority certificate.


  1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

  define SUMO_URL      https://<YOUR_SUMO_ENDPOINT>/receiver/v1/http/<UNIQUE_CODE>
define CA_FILE       C:\Program Files\nxlog\cert\digicert_ca.cer

<Input memory_count>
    Module           im_winperfcount
    Counter          \Memory\Available Bytes
    PollInterval     30
    Exec             rename_field("\\Memory\\Available Bytes", "MemFreeBytes");
</Input>

<Output sumo_http>
    Module           om_http
    URL              %SUMO_URL%
    BatchMode        multiline
    HTTPSCAFile      %CA_FILE%
    ContentType      application/vnd.sumologic.carbon2
    <Exec>
        $raw_event = "metric=Mem_Free " + "host=" + $Hostname + \
                     "  " + $MemFreeBytes + " " + \
                     integer($EventTime)*1/1000;
    </Exec>
</Output>

Output sample

The following is a sample of the data that will be sent to Sumo Logic. In this example, the metric is named Mem_Free and represents the available free memory in bytes.

metric=Mem_Free host=Hopper  1008345088 1608040330345
metric=Mem_Free host=Hopper  1017278464 1608040340346
metric=Mem_Free host=Hopper  1056428032 1608040350346

Note	It may take a few minutes for data to appear in the Sumo Logic web interface. If not, see the Sumo Logic documentation on troubleshooting HTTP Sources.

107.6. Verifying data in Sumo Logic

Reception of log data can be verified using the Sumo Logic web interface. One way to do this is to search for events using the name of the collector and source. Navigate to Manage Data > Collection, and on the Collection tab click on the Open in Log Search icon next to the respective log source.

This image displays the NXLog hosted collector and a list of sources in the Sumo Logic web interface. The icon to open the log search is highlighted.

This image displays events filtered by the collector and source.

This image displays a syslog event in JSON format that was sent to Sumo Logic via HTTPS.

This image displays a Windows Event Log event in JSON format that was sent to Sumo Logic as a syslog message.

Events can be exported from Sumo Logic in CSV format, in which event fields are output as comma-separated values. The following is a sample syslog message exported from Sumo Logic. The original event was sent by NXLog via HTTPS.

"_messagetimems","_messagetime","_raw","_collector","_size","_source","_sourcecategory","_sourcehost","_sourcename"
"1607341156001","12/07/2020 12:39:16.001 +0100","{""EventReceivedTime"":""2020-12-07T12:39:16.001994+01:00"",""SourceModuleName"":""file"",""SourceModuleType"":""im_file"",""SyslogFacilityValue"":1,""SyslogFacility"":""USER"",""SyslogSeverityValue"":5,""SyslogSeverity"":""NOTICE"",""SeverityValue"":2,""Severity"":""INFO"",""Hostname"":""NXLog-Ubuntu-1"",""EventTime"":""2020-12-07T12:30:37.000000+01:00"",""SourceName"":""systemd"",""ProcessID"":""1"",""Message"":""Started Fingerprint Authentication Daemon.""}","NXLog","414","NXLog-Ubuntu1","linux-http","19.168.0.100","Http Input"

Host Metrics can be viewed in the Sumo Logic web interface by clicking on the + New button to open a Metrics tab, and adding a metric query for the desired property. The image below shows host metrics sent by NXLog for available memory (Bytes) over a period of time. When sending the data, the metric was named Mem_Free.

You are here

107. Sumo Logic | Log Collection Solutions

107. Sumo Logic

107.1. Using NXLog to collect data

107.2. Setting up a hosted collector

107.3. Setting up the digital certificate

107.3.1. Linux

107.3.2. Microsoft Windows

107.4. Sending logs to Sumo Logic using TCP

107.5. Sending data to Sumo Logic using HTTPS

107.5.1. Sending log data

107.5.2. Sending host metrics

107.6. Verifying data in Sumo Logic