125. Common issues | Log Collection Solutions

When configured to read from a file in the /var/log directory on Linux, NXLog may log the following error:

This error occurs because NXLog does not have permission to read the file with the configured User and Group. See the Reading Rsyslog log files for more information about using NXLog to read files from the /var/log directory.

When collecting events from the Windows Event Log, the user running NXLog may not have sufficient permissions to access certain channels. When NXLog is running as a service, this applies to the user that the service is configured to run under. In this case, the NXLog log file shows errors such as:

When this error occurs, the user needs to be granted access to read the specified channel for NXLog to be able to collect events from it.

For default Event Log channels, it is sufficient to add the user to the built-in Event Log Readers group by following these steps:

If the error persists, permission needs to be granted using the Windows Registry.

When using the im_tcp and im_ssl modules to transfer data over the network, firewalls and other network issues can prevent successful connections. This can result in Connection refused errors.

To resolve this issue:

Authentication issues basically stem from unsuccessful verification of the certificate against the CA on the client and server sides. For example, the specified CA file may be non-existent or the configuration may point to the wrong CA file.

A typical authentication error message is provided below:

Another example of the verification error message:

Usage of a self-signed certificate produces the following message:

The first step in solving such problems is to set the AllowUntrusted directive to TRUE and restart the agent. The manager instance should establish connection with the agent.

Additionally, CA and certificate files can be verified using the openssl tool. To verify the CA file, use the command below:

To verify the server’s certificate against the CA certificate specified by the CAFile directive, use the following command:

Instead of the CA file, the path to the CA directory can be specified per the command below:

The following error is generated by the agent when it presents a certificate that cannot be verified by NXLog Manager:

Certificates created with an incorrect or incompatible "certificate purpose" produce the following error:

To examine certificates, use the following command:

If you are using Logstash and find that log entries are concatenated, make sure that you are using the json_lines codec in your Logstash server configuration.

The default json codec in Logstash sometimes fails to parse log entries passed from NXLog. Switch to the json_lines codec for better reliability.

This happens when NXLog tries to evaluate a directive, but the required log data is not available in the current context. This causes any dependent operations to fail and the directive to terminate as well as the following error to be logged:

This error occurs when attempting to access a field from the Exec directive of a Schedule block. The log data is not available in the current context. Log data is never available to a scheduled Exec directive because its execution is not triggered by a log message.

An attempt to access a field can occur directly with a field assignment, or indirectly by calling a function or procedure that accesses log data.

NXLog can send one log stream to multiple outputs. This can be configured by either using the same input in multiple routes or using multiple outputs in the same route (see Routes). By default, when one of the outputs fails NXLog will stop sending logs to all outputs. This is caused by NXLog’s flow control mechanism, which is designed to prevent messages from being lost. Flow control pauses an Input or Processor module when the next module in the route is not ready to accept data.

In some cases, it is preferred for NXLog to continue sending logs to the remaining active output and discard logs for the failed output. The simplest solution is to disable flow control. This can be done globally with the global FlowControl directive, or for the corresponding Input (and Processor, if any) modules only, with the module FlowControl directive.

To retain the improved message durability provided by flow control, it is possible to instead explicitly specify when to drop logs by using a separate route for each output that may fail. Add a pm_buffer module instance to that route, and configure the buffer to drop logs when it reaches a certain size. The output that follows the buffer can fail without causing any Input or Processor module before the buffer to pause processing. For examples, see the Using Buffers section.

When a system nears or exceeds its open files limit, significant performance penalties are typically quick to follow. LSOF (List Open Files) is a common debugging tool found on the majority of Linux systems and can reveal a great deal about the running system.

On Linux, run the following command to see, for example, which files NXLog has open:

This example returns the number of open files:

To check NXLog’s system limits use the following command:

On Systems not using /proc, check the system’s open file limit with:

or with:

When trying to view the NXLog’s internal log file on Windows, you may receive an error message indicating that the log file is in use by another application and cannot be accessed.

To resolve this issue, either: