130. Common issues

Table of Contents

Common issues are easily resolved by internal logs to identify typical symptoms, finding the corresponding description of the symptom below, and then following the suggested remediation steps.

130.1. NXLog fails to start

You may receive this error message in the log file when NXLog fails to start (line break added):

nxlog failed to start: Invalid keyword: ÿþ# at \
C:\Program Files (x86)\nxlog\conf\nxlog.conf:1

This issue occurs because the NXLog configuration file has been saved in either UTF-16 text encoding, or UTF-8 text encoding with a BOM header.

Open the configuration file in a text editor and save it using ASCII encoding or plain UTF-8.

Tip	On Windows, you can use Notepad to correct the text encoding of this file.

130.2. Permission errors

130.2.1. "Permission denied"

When configured to read from a file in the /var/log directory on Linux, NXLog may log the following error:

ERROR failed to open /var/log/messages;Permission denied

This error occurs because NXLog does not have permission to read the file with the configured User and Group. See the Reading Rsyslog log files for more information about using NXLog to read files from the /var/log directory.

130.2.2. Windows Event Log error: "access denied" or "ignoring source"

When collecting events from the Windows Event Log, the user running NXLog may not have sufficient permissions to access certain channels. When NXLog is running as a service, this applies to the user that the service is configured to run as. In this case, the NXLog log file shows errors such as:

WARNING [im_msvistalog|windows] failed to subscribe to msvistalog events,access denied [error code: 5]: Access is denied.
WARNING [im_msvistalog|windows] Invalid channel: 'Security': Access is denied.

WARNING [im_msvistalog|windows] ignoring source as it cannot be subscribed to (error code: 5)

When this error occurs, the user needs to be granted access to read the specified channel. For default Event Log channels, it is usually sufficient to add the user to the built-in Event Log Readers group by following these steps:

Open the Computer Management MMC snap-in by going to the Windows Start menu, type compmgmt.msc and press Enter.
Expand System Tools > Local Users and Groups > Groups.
Double-click on the Event Log Readers group and add the NXLog user to it.

If the error persists, permission needs to be granted using Group Policy for the default Windows Event Log channels or the Windows Registry for other channels. Permissions are specified using the Security Descriptor Definition Language (SDDL).

The first step is to retrieve the SID of the NXLog user. From a command prompt, run the following command:
```
> wmic useraccount where name='<username>' get sid
```
For default Windows Event Log channels:
1. Open command prompt with an admin user and run the following command afer replacing <channel_name> with the actual channel name.
  > wevtutil gl <channel_name>
  In this example, the security channel was chosen:
2. Take note of the channelAccess value.
3. Open the Group Policy Editor by going to the Windows Start menu, then type gpedit.msc and press Enter.
4. Expand Computer Configuration > Administrative Templates > Windows Components > Event Log Service.
5. Select the required channel from the list, for example Security. Double-click on the Configure log access policy to edit it.
6. Select the Enabled option.
7. Under Log Access, enter the channelAccess value retrieved above.
8. Append the Log Access value with the permission for the NXLog service user. Add the following permission to grant the user read access:
  (A;;0x1;;;<user_sid>)
  Here, A means allow and 0x1 means read. You will need to replace <user_sid> with the SID retrieved in step 1.
9. From a command prompt run the following command to apply the updated policy:
  > gpupdate /force

For other Windows Event Log channels:

Open the Registry Editor by going to the Windows Start menu, type regedit and press Enter.

Expand the following registry key:

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\WINEVT\Channels

From the list of keys, find the channel shown in the error and click on it.
In the right pane, double click on the ChannelAccess value to modify it.
Append the permission for the NXLog service user to the existing value. Add the following permission to grant the user read access:
```
(A;;0x1;;;<user_sid>)
```
Here, A means allow and 0x1 means read. You will need to replace <user_sid> with the SID retrieved in step 1.
Repeat these steps for each channel showing the error.

Restart Windows. This step is important because the new permissions will not be applied until Windows has been restarted.

Note	Your channel selection might be stored in a different registry key than the one specified above. If so, you will need to do some research to determine the correct registry key for each required channel.

Warning

These steps require altering the Windows Registry and should be executed with care. Incorrect modifications could potentially render the system unusable.

130.3. Connection errors

130.3.1. "Connection refused" with im_tcp or im_ssl

When using the im_tcp and im_ssl modules to transfer data over the network, firewalls and other network issues can prevent successful connections. This can result in Connection refused errors.

To resolve this issue:

Check that no firewall, gateway, or other network issue is blocking the connection
Verify that the system can resolve the host name used in the Host directive of the configuration file

130.3.2. Certificate/TLS issues

Authentication issues basically stem from unsuccessful verification of the certificate against the CA on the client and server sides. For example, the specified CA file may be non-existent or the configuration may point to the wrong CA file.

A typical authentication error message is provided below:

SSL certificate problem: unable to get local issuer certificate

Another example of the verification error message:

ERROR SSL certificate verification failed: unable to get local issuer
certificate (err: 20)

Usage of a self-signed certificate produces the following message:

ERROR SSL certificate verification failed: self signed certificate (err: 18)

The first step in solving such problems is to set the AllowUntrusted directive to TRUE and restart the agent. The manager instance should establish connection with the agent.

Additionally, CA and certificate files can be verified using the openssl tool. To verify the CA file, use the command below:

# openssl s_client -CAfile <path_to_CA_file> -connect <host:port>

To verify the server’s certificate against the CA certificate specified by the CAFile directive, use the following command:

# openssl s_client -connect <host:port> -cert <path_to_cert_file> -key <path_to_key_file> -CAfile <path_to_CA_file> -verify 1

Instead of the CA file, the path to the CA directory can be specified per the command below:

# openssl s_client -connect <host:port> -cert <path_to_cert_file> -key <path_to_key_file> -CApath  <path_to_CA_dir> -verify 1

The following error is generated by the agent when it presents a certificate that cannot be verified by NXLog Manager:

ERROR remote ssl socket was reset? (SSL_ERROR_SSL with errno=9); End of file found

Certificates created with an incorrect or incompatible "certificate purpose" produce the following error:

ERROR SSL certificate verification failed: unsupported certificate purpose (err: 26)

To examine certificates, use the following command:

# openssl x509 -text -in <path_to_cert_file>

130.4. Log format errors

130.4.1. Log entries are concatenated with Logstash

If you are using Logstash and find that log entries are concatenated, make sure that you are using the json_lines codec in your Logstash server configuration.

The default json codec in Logstash sometimes fails to parse log entries passed from NXLog. Switch to the json_lines codec for better reliability.

130.5. Data missing errors

130.5.1. "Missing logdata" error

This happens when NXLog tries to evaluate a directive, but the required log data is not available in the current context. This causes any dependent operations to fail and the directive to terminate as well as the following error to be logged:

missing record, assignment possibly after drop()

This error occurs when attempting to access a field from the Exec directive of a Schedule block. The log data is not available in the current context. Log data is never available to a scheduled Exec directive because its execution is not triggered by a log message.

An attempt to access a field can occur directly with a field assignment, or indirectly by calling a function or procedure that accesses log data.

130.6. Processing unexpectedly paused or stopped

130.6.1. Processing stops if an output fails

NXLog can send one log stream to multiple outputs. This can be configured by either using the same input in multiple routes or using multiple outputs in the same route (see Routes). By default, when one of the outputs fails NXLog will stop sending logs to all outputs. This is caused by NXLog’s flow control mechanism, which is designed to prevent messages from being lost. Flow control pauses an Input or Processor module when the next module in the route is not ready to accept data.

In some cases, it is preferred for NXLog to continue sending logs to the remaining active output and discard logs for the failed output. The simplest solution is to disable flow control. This can be done globally with the global FlowControl directive, or for the corresponding Input (and Processor, if any) modules only, with the module FlowControl directive.

Note	With flow control disabled, an Input or Processor module will continue to process logs even if the next module’s buffers are full (and the logs will be dropped).

To retain the improved message durability provided by flow control, it is possible to instead explicitly specify when to drop logs by using a separate route for each output that may fail. Add a pm_buffer module instance to that route, and configure the buffer to drop logs when it reaches a certain size. The output that follows the buffer can fail without causing any Input or Processor module before the buffer to pause processing. For examples, see the Using buffers section.

130.6.2. Check open files and limits

When a system nears or exceeds its open files limit, significant performance penalties are typically quick to follow. LSOF (List Open Files) is a common debugging tool found on the majority of Linux systems and can reveal a great deal about the running system.

On Linux, run the following command to see, for example, which files NXLog has open:

$ lsof -u nxlog

This example returns the number of open files:

$ lsof -Fn -u nxlog | sort | uniq | wc -l

To check NXLog’s system limits use the following command:

$ cat /proc/$(sudo cat /opt/nxlog/var/run/nxlog/nxlog.pid)/limits

On Systems not using /proc, check the system’s open file limit with:

$ sysctl kern.maxfiles

or with:

$ sysctl fs.file-max

Note	There is no Windows equivalent to `lsof`. You can use Windows Process Explorer from Microsoft’s Windows Sysinternals to inspect which program has files or directories open.

130.6.2.1. systemd and open files limit

There are certain cases where systemd ignores system level file limits. This can cause "too many files open" errors.

2019-01-22 15:26:37 ERROR SSL error, failed to load ca cert from '/opt/nxlog/var/lib/nxlog/cert/agent-ca.pem', reason: Too many open files, system lib,
system lib

This scenario requires edits to the service file or an override. To check NXLog’s system limits use the following command:

$ cat /proc/$(cat /opt/nxlog/var/run/nxlog/nxlog.pid)/limits

On Systems not using /proc, check the system’s open file limit:

$ sysctl kern.maxfiles

To adjust limits for nxlog, create /etc/systemd/system/nxlog.service.d/override.conf and add the following definition:


  1
2

  [Service]
LimitNOFILE=100000

Update the service settings with:

$ systemctl daemon-reload

130.6.3. Log file is in use by another application

When trying to view the NXLog’s internal log file on Windows, you may receive an error message indicating that the log file is in use by another application and cannot be accessed.

To resolve this issue, either:

Open the log file with an application that does not use exclusive locking (such as Notepad)

or
Stop NXLog before opening the log file

You are here

130. Common issues | Log Collection Solutions