Solarwinds Loggly

This guide explains how to collect and parse logs with NXLog so that they can be forwarded to Loggly.

Solarwinds Loggly is a cloud-based log analysis and monitoring service. NXLog can be configured to send log data to Loggly in Syslog format over TCP, or via the Loggly API using HTTP(S).

The Loggly Customer Token

Loggly requires a customer token to be included with each event sent to its service. This token is an alpha-numeric string which is generated when a Loggly account is created and it can be found on the Source Setup > Customer Tokens page of the Loggly web interface. For more information, refer to the Loggly documentation about the Customer Token.

Example Loggly customer token: 69af9e44-e002-4f67-8777-7b3b93ab0297

The NXLog configuration will need this token for sending log data to Loggly.

Sending Logs to Loggly Using TCP

Syslog is the most common way to send data to Loggly. NXLog can be configured to send data in IETF Syslog (RFC5424) format using the xm_syslog module.

When using this method, the customer token and any custom tags need to be included in the structured data section of the Syslog message.

Example 1. Sending logs over TCP

In the configuration below, NXLog processes a log file using the im_file module, generates a Syslog message using the xm_syslog module, and sends it to Loggly using the om_tcp module. The log data is formatted as JSON using the xm_json module. Logs are given the tag linux to make them easily searchable in Loggly.

The CUSTOMER_TOKEN value needs to be replaced with an actual Loggly customer token. In this example, processing is done in the output instance where the NXLog ID in the structured data of the Syslog message is replaced with the customer token defined by the CUSTOMER_TOKEN constant.

define CUSTOMER_TOKEN xxxx-xxx-xxxx-xxxx-xxxx

<Extension json>
    Module    xm_json
</Extension>

<Extension syslog>
    Module    xm_syslog
</Extension>

<Input file>
    Module    im_file
    File      "/var/log/file"
    Exec      parse_syslog();
    Exec      $Message = to_json();
</Input>

<Output loggly_tcp>
    Module    om_tcp
    Host      logs-01.loggly.com:514
    Exec      to_syslog_ietf();
    Exec      $raw_event =~ s/(\[NXLOG@14506.*?\])//g; \
              $raw_event = replace($raw_event, \
              '{', '[%CUSTOMER_TOKEN%@41058 tag="linux"] {', 1);
</Output>

Output Sample

The following is a sample of the Syslog header that will be sent to Loggly. The structured data includes the customer token and tag as specified in the NXLog configuration.

<13>1 2020-11-26T06:41:43 NXLog-Ubuntu-1 dbus-daemon 3540 -
[xxxx-xxxx-xxxx-xxxx-xxxx@41058 tag="linux"]

The JSON formatted log sample below will be sent as part of the Syslog message to the Loggly service.

{
  "EventReceivedTime": "2020-11-26 08:22:43",
  "SourceModuleName": "file",
  "SourceModuleType": "im_file",
  "SyslogFacilityValue": 1,
  "SyslogFacility": "USER",
  "SyslogSeverityValue": 5,
  "SyslogSeverity": "NOTICE",
  "SeverityValue": 2,
  "Severity": "INFO",
  "Hostname": "NXLog-Ubuntu-1",
  "EventTime": "2020-11-26 06:41:43",
  "SourceName": "dbus-daemon",
  "ProcessID": "3540",
  "Message": "Successfully activated service 'org.gnome.Terminal'"
}

Example 2. Sending logs using TLS Encryption

Logs can be sent securely to Loggly using TLS encryption. To be able to make use of TLS, the Loggly certificate file needs to be downloaded and placed in a location that can be accessed by NXLog.

The configuration below uses the im_msvistalog module to collect events from the Windows Event Log, generates a Syslog message using the xm_syslog module, and sends it to Loggly using the om_ssl module. The event data is formatted in JSON using the xm_json module. Events are given the tag windows to make them easily searchable in Loggly.

The CUSTOMER_TOKEN value needs to be replaced with an actual Loggly customer token. In this example, processing is done in the output instance where the customer token defined by the CUSTOMER_TOKEN constant is inserted before the JSON data.

The CERTDIR value needs to be replaced with the actual path to the Loggly certificate.

define CERTDIR C:\Program Files\nxlog\cert

define CUSTOMER_TOKEN xxxx-xxx-xxxx-xxxx-xxxx

<Extension json>
    Module            xm_json
</Extension>

<Extension syslog>
    Module            xm_syslog
</Extension>

<Input eventlog>
    Module            im_msvistalog
    <QueryXML>
        <QueryList>
            <Query Id='0'>
                <Select Path="System">*[System[(Level=1  or Level=2)]]</Select>
            </Query>
        </QueryList>
    </QueryXML>
    Exec              $Message = to_json();
</Input>

<Output loggly_tls>
    Module            om_ssl
    Host              logs-01.loggly.com:6514
    CAFile            %CERTDIR%\logs-01.loggly.com_sha12.crt
    AllowUntrusted    FALSE
    Exec              to_syslog_ietf();
    Exec              $raw_event =~ s/(\[.*])//g; \
                      $raw_event = replace($raw_event, \
                      '{', '[%CUSTOMER_TOKEN%@41058 tag="windows"] {', 1);
</Output>

Output Sample

The following is a sample of the Syslog header that will be sent to Loggly. The structured data includes the customer token and tag as specified in the NXLog configuration.

<11>1 2020-11-26T11:09:30.592060Z Hopper VBoxNetLwf 0 -
[xxxx-xxxx-xxxx-xxxx-xxxx@41058 tag="windows"]

The JSON formatted sample event below will be sent as part of the Syslog message to the Loggly service.

{
  "EventTime": "2020-11-26 12:09:30",
  "Hostname": "Hopper",
  "Keywords": "36028797018963968",
  "EventType": "ERROR",
  "SeverityValue": 4,
  "Severity": "ERROR",
  "EventID": 12,
  "SourceName": "VBoxNetLwf",
  "TaskValue": 0,
  "RecordNumber": 44000,
  "ExecutionProcessID": 0,
  "ExecutionThreadID": 0,
  "Channel": "System",
  "Message": "The driver detected an error on \\Device\\VBoxNetLwf.",
  "Data": "\\Device\\VBoxNetLwf",
  "EventData.Binary": "00000C0001000000000000000C0004",
  "EventReceivedTime": "2020-11-26 13:19:02",
  "SourceModuleName": "eventlog",
  "SourceModuleType": "im_msvistalog"
}

Sending Logs to Loggly Using HTTPS

NXLog Enterprise Edition exclusive feature

As part of their API, Loggly provides two HTTP(S) endpoints that accept log data, one for sending single log records and another to send logs in batches. NXLog can be configured to send logs to Loggly over HTTPS using the om_http module. Data can be sent as plaintext, JSON, or any of the log formats supported by automated parsing. Refer to the Loggly documentation for a list of supported log types.

When logs are sent over HTTPS, the Loggly customer token and any custom tags need to be included as part of the URL.

To send data to Loggly using HTTPS, the Loggly certificate file needs to be downloaded and placed in a location accessible to NXLog.

Example 3. Sending individual logs over HTTPS

In the configuration below, NXLog will send a POST request for every log record it processes. The events are given the tag http to make them easily searchable in Loggly. This method is suited for low volume log sources where a considerable amount of time elapses between each new event. For high volume logging, see the configuration for sending logs in batches.

The CUSTOMER_TOKEN value needs to be replaced with an actual Loggly customer token.

The HTTPSCAFile directive needs to specify the actual path and filename of the Loggly certificate.

define CUSTOMER_TOKEN xxxx-xxx-xxxx-xxxx-xxxx

<Output loggly_http>
    Module         om_http
    URL            https://logs-01.loggly.com/inputs/%CUSTOMER_TOKEN%/tag/http/
    HTTPSCAFile    /opt/nxlog/cert/logs-01.loggly.com_sha12.crt
</Output>

Example 4. Sending logs in batches over HTTPS

In the configuration below, NXLog will POST log data to Loggly in batches. The events are given the tag http to make them easily searchable in Loggly. The BatchMode directive in the output block needs to be set to multiline, to specify that each log record should be separated by a new line.

The Loggly bulk endpoint supports up to 1MB per event and a maximum of 5MB per batch. For simplicity, this example uses the default NXLog batch settings. For further configuration options, see the BatchSize and BatchFlushInterval directives.

The CUSTOMER_TOKEN value needs to be replaced with an actual Loggly customer token.

The HTTPSCAFile directive needs to specify the actual path and filename of the Loggly certificate.

define CUSTOMER_TOKEN xxxx-xxx-xxxx-xxxx-xxxx

<Output loggly_http>
    Module         om_http
    URL            https://logs-01.loggly.com/bulk/%CUSTOMER_TOKEN%/tag/http/
    BatchMode      multiline
    HTTPSCAFile    /opt/nxlog/cert/logs-01.loggly.com_sha12.crt
</Output>

It may take a few minutes for data to be indexed and shown in the Loggly web interface. If not, see the Loggly documentation on troubleshooting HTTP.

Verifying Data in Loggly

Reception of log data can be verified using the Loggly web interface. One way to do this is to search for events using the custom tag as specified in the NXLog configuration. Navigate to the Logs > Log Explorer page and type tag:<mytag> in the search field. More advanced searches can be done on other event fields. For more information, refer to the Loggly search query language documentation.

The image below shows events filtered by the linux tag.

The image below shows a Syslog event in JSON format that was sent to Loggly as a Syslog message over TCP.

The image below shows a Windows Event Log event in JSON format that was sent to Loggly as a Syslog message.

The image below shows a Syslog event in JSON format that was sent to Loggly via HTTPS.

Events can be exported from Loggly in three formats:

JSON - all event fields are returned as a JSON object
CSV - all event fields are returned as comma-separated values
Raw - log data is returned in its original format

The following is a sample Windows Event Log event exported from Loggly in JSON format. The original event was sent to Loggly by NXLog as a Syslog message.

{
  "total_events": 1,
  "page": 0,
  "events": [
    {
      "timestamp": 1606388970592,
      "unparsed": null,
      "event": {
        "syslog": {
          "severity": "Error",
          "facility": "user-level messages",
          "timestamp": "2020-11-26T11:09:30.592060Z",
          "appName": "VBoxNetLwf",
          "pid": 0,
          "priority": "11",
          "host": "Hopper"
        },
        "json": {
          "ExecutionProcessID": 0,
          "EventID": 12,
          "EventTime": "2020-11-26 12:09:30",
          "EventData_Binary": "00000C0001000000000000000C0004",
          "Severity": "ERROR",
          "RecordNumber": 44031,
          "SourceModuleType": "im_msvistalog",
          "SourceName": "VBoxNetLwf",
          "EventType": "ERROR",
          "SourceModuleName": "eventlog",
          "Hostname": "Hopper",
          "ExecutionThreadID": 0,
          "TaskValue": 0,
          "Keywords": "36028797018963968",
          "SeverityValue": 4,
          "Message": "The driver detected an error on \\Device\\VBoxNetLwf.",
          "Data": "\\Device\\VBoxNetLwf",
          "Channel": "System",
          "EventReceivedTime": "2020-11-26 13:19:02"
        }
      }
    }
  ]
}

Disclaimer

While we endeavor to keep the information in this topic up to date and correct, NXLog makes no representations or warranties of any kind, express or implied about the completeness, accuracy, reliability, suitability, or availability of the content represented here. We update our screenshots and instructions on a best-effort basis.

Last revision: 07 December 2020