129. Internal logs

Table of Contents

When issues arise while configuring or maintaining an NXLog instance, a stepwise troubleshooting approach (moving from the most likely and simple cases to the more complex and rare ones) generally yields favorable results. The first step is always to inspect the internal log which NXLog generates.

129.1. Default settings

By default, NXLog generates log messages about its own operations. These messages are essential for troubleshooting problems, and should be checked first if NXLog is not functioning as expected.

These internal messages are written to the file defined in the LogFile directive in nxlog.conf. On Windows that file is C:\Program Files\nxlog\data\nxlog.log; on Linux, /opt/nxlog/var/log/nxlog/nxlog.log. If this directive is not specified, internal logging is disabled.

129.2. Enable internal logging

Internal logging is enabled by default after installation, but may be disabled if edits were made to nxlog.conf, or if the LogFile directive points to a path which is unavailable. Enable internal logging by editing nxlog.conf to ensure the LogFile directive is set to an available path.

Note	Some Windows applications (WordPad, for example) cannot open the log file while the NXLog process is running because of exclusive file locking. Use a viewer that does not lock the file, like Notepad.

129.3. Raise the severity level of logged events

By default, internal logs are generated with a log level of INFO. To get detailed information about NXLog’s operations, the log level can be raised to DEBUG level. This level of detail reliably produces a large amount of log messages, and so we recommend setting this log level only for sustained troubleshooting sessions.

To raise the log level temporarily (until NXLog is restarted).

On Linux, send SIGUSR2:

# kill -SIGUSR2 $PID

On Windows, send service control command 201:

> sc control nxlog 201

To raise the log level for an extended troubleshooting session.

On all systems, set the LogLevel directive to DEBUG, then restart NXLog.

129.4. Send customized log messages to the internal log

It may be helpful to add extra logging statements to your configuration using the log_info() procedure.

The generated messages will be visible:

in the file defined in the LogFile global directive
in the input from the im_internal module
on standard output when running NXLog in the foreground with the -f command line switch

Example 592. Writing specific fields and values to the internal log

This configuration uses the log_info() procedure to send values to the internal log. Log messages are accepted over UDP on port 514. If keyword is found in the unparsed message, an INFO level message will be generated.

nxlog.conf


  1
2
3
4
5
6
7
8

  <Input in>
    Module  im_udp
    Port    514
    <Exec>
        if $raw_event =~ /keyword/
            log_info("FOUND KEYWORD IN MSG: [" + $raw_event + "]");
    </Exec>
</Input>

129.5. Send all fields to the internal log

Example 593. Send all fields to the internal log

In this configuration, the to_json() procedure from the xm_json module is used to send all the fields to the internal log.

nxlog.conf


  1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

  <Extension _syslog>
    Module  xm_syslog
</Extension>

<Extension _json>
    Module  xm_json
</Extension>

<Input in>
    Module  im_tcp
    Host    0.0.0.0
    Port    1514
    <Exec>
        parse_syslog_bsd();

        # Dump $raw_event
        log_info("raw_event is: " + $raw_event);

        # Dump fields in JSON
        log_info("Other fields are: " + to_json());
    </Exec>
</Input>

Output sample

{
  "MessageSourceAddress": "127.0.0.1",
  "EventReceivedTime": "2012-05-18 13:11:35",
  "SourceModuleName": "in",
  "SourceModuleType": "im_tcp",
  "SyslogFacilityValue": 3,
  "SyslogFacility": "DAEMON",
  "SyslogSeverityValue": 3,
  "SyslogSeverity": "ERR",
  "SeverityValue": 4,
  "Severity": "ERROR",
  "Hostname": "host",
  "EventTime": "2010-10-12 12:49:06",
  "SourceName": "app",
  "ProcessID": "12345",
  "Message": "test message"
}

129.6. Send debug dump to the internal log

A simple way to quickly get a more complete picture of NXLog’s current status is to dump debug info into the internal log. This information can be helpful in determining, for example, why an input module is not sending to an output module. Normally, internal events are written to the log file configured with the LogFile directive.

On Linux, send SIGUSR1 to the application:

# kill -SIGUSR1 $PID

On Windows, send the service control command "200" to the application:

> sc control nxlog 200

Dumped debug info example

2017-03-29 10:05:19 INFO event queue has 2 events;jobgroup with priority 10;job
of module in/im_file, events: 0;job of module out/om_null, events: 0;non-module
job, events: 0;jobgroup with priority 99;non-module job, events: 0;[route 1]; -
in: type INPUT, status: RUNNING queuesize: 0; - out: type OUTPUT, status:
RUNNING queuesize: 0;

Note

The status is the most important piece of information in the dumped log entries. A status of PAUSED means the input module is not able to send because the output module queue is full. In such a case the queuesize for the corresponding output(s) would be over 99. A status of STOPPED means the module is fully stopped, usually due to an error.

129.7. Send internal log to STDOUT

Run NXLog in the foreground to print internal logs to the standard output and standard error streams, which are both visible in the terminal.

Use nxlog -f to run NXLog in the foreground.

129.8. Send internal log to an existing route

NXLog can also write internal log data into a normal route using the im_internal module. Internal log messages can then be forwarded like any other log source.

Tip	Local logging is more fault-tolerant than routed logging, and is therefore recommended for troubleshooting.

Note	It is not possible to use a log level higher than INFO with the im_internal module. `DEBUG` level messages can only be written to the local log file.

129.9. Send information to an external file

Sending the internal log or other information to external files can also be useful while troubleshooting.

In this example configuration, the file_write() procedure (from the xm_fileop module) is used to dump information to an external file.

nxlog.conf


  1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

  <Extension _fileop>
    Module  xm_fileop
</Extension>

<Extension _syslog>
    Module  xm_syslog
</Extension>

<Input in>
    Module  im_tcp
    Host    0.0.0.0
    Port    1514
    <Exec>
        parse_syslog_bsd();

        # Debug $SyslogSeverity and $Hostname fields
        file_write("/tmp/debug.txt",
                   "Severity: " + $SyslogSeverity +
                   ", Hostname: " + $Hostname + "\n");
    </Exec>
</Input>

You are here

129. Internal logs | Log Collection Solutions