125. Common issues
Common issues are easily resolved by internal logs to identify typical symptoms, finding the corresponding description of the symptom below, and then following the suggested remediation steps.
125.1. NXLog fails to start
You may receive this error message in the log file when NXLog fails to start (line break added):
nxlog failed to start: Invalid keyword: ÿþ# at \
C:\Program Files (x86)\nxlog\conf\nxlog.conf:1
This issue occurs because the NXLog configuration file has been saved in either UTF-16 text encoding, or UTF-8 text encoding with a BOM header.
Open the configuration file in a text editor and save it using ASCII encoding or plain UTF-8.
Tip
|
On Windows, you can use Notepad to correct the text encoding of this file. |
125.2. Permission errors
125.2.1. "Permission denied"
When configured to read from a file in the /var/log directory on Linux, NXLog may log the following error:
ERROR failed to open /var/log/messages;Permission denied
This error occurs because NXLog does not have permission to read the
file with the configured User
and Group
. See the Reading Rsyslog log files for more information about using NXLog to read files from
the /var/log directory.
125.2.2. Windows Event Log error: "ignoring source"
When collecting events from the Windows Event Log, the user running NXLog may not have sufficient permissions to access certain channels. When NXLog is running as a service, this applies to the user that the service is configured to run under. In this case, the NXLog log file shows errors such as:
2020-01-10 13:43:30 WARNING ignoring source as it cannot be subscribed to (error code: 5)
When this error occurs, the user needs to be granted access to read the specified channel for NXLog to be able to collect events from it.
For default Event Log channels, it is sufficient to add the user to the built-in Event Log Readers group by following these steps:
-
Open the Computer Management MMC snap-in by going to the Windows Start menu, type compmgmt.msc and press Enter.
-
Expand System Tools > Local Users and Groups > Groups.
-
Double-click on the Event Log Readers group and add the NXLog user to it.
If the error persists, permission needs to be granted using the Windows Registry.
-
The first step is to retrieve the SID of the NXLog user. From a command prompt run the following command:
> wmic useraccount where name='<username>' get sid
-
Open the Registry Editor by going to the Windows Start menu, type regedit and press Enter.
-
Expand the following registry key:
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\WINEVT\Channels
-
From the list of keys, find the channel shown in the error and click on it.
-
In the right pane, double click on the
ChannelAccess
value to modify it. -
Permissions are specified using the Security Descriptor Definition Language (SDDL) and need to be appended to the existing value. Add the following permission to give read access to the user:
(A;;0x1;;;<user-sid>)
Where
A
means Allow,0x1
means Read, and<user-sid>
needs to be replaced with the SID retrieved in step 1. -
Repeat steps 4-6 for each channel showing the error.
-
Restart the machine. This step is important as the new permissions will not be applied until the machine is restarted.
Note
|
There may be channels which are saved in a different registry key than the one specified above. In such cases research has to be made to determine the location of the respective channel. |
Warning
|
These steps require altering the Windows Registry and should be executed with care. Incorrect modifications could potentially render the system unusable. |
125.3. Connection errors
125.3.1. "Connection refused" with im_tcp or im_ssl
When using the im_tcp and im_ssl modules to
transfer data over the network, firewalls and other network issues can
prevent successful connections. This can result in Connection refused
errors.
To resolve this issue:
-
Check that no firewall, gateway, or other network issue is blocking the connection
-
Verify that the system can resolve the host name used in the
Host
directive of the configuration file
125.3.2. Certificate/TLS issues
Authentication issues basically stem from unsuccessful verification of the certificate against the CA on the client and server sides. For example, the specified CA file may be non-existent or the configuration may point to the wrong CA file.
A typical authentication error message is provided below:
SSL certificate problem: unable to get local issuer certificate
Another example of the verification error message:
ERROR SSL certificate verification failed: unable to get local issuer certificate (err: 20)
Usage of a self-signed certificate produces the following message:
ERROR SSL certificate verification failed: self signed certificate (err: 18)
The first step in solving such problems is to set the AllowUntrusted directive to TRUE and restart the agent. The manager instance should establish connection with the agent.
Additionally, CA and certificate files can be verified using the openssl tool. To verify the CA file, use the command below:
# openssl s_client -CAfile <path_to_CA_file> -connect <host:port>
To verify the server’s certificate against the CA certificate specified by the CAFile directive, use the following command:
# openssl s_client -connect <host:port> -cert <path_to_cert_file> -key <path_to_key_file> -CAfile <path_to_CA_file> -verify 1
Instead of the CA file, the path to the CA directory can be specified per the command below:
# openssl s_client -connect <host:port> -cert <path_to_cert_file> -key <path_to_key_file> -CApath <path_to_CA_dir> -verify 1
The following error is generated by the agent when it presents a certificate that cannot be verified by NXLog Manager:
ERROR remote ssl socket was reset? (SSL_ERROR_SSL with errno=9); End of file found
Certificates created with an incorrect or incompatible "certificate purpose" produce the following error:
ERROR SSL certificate verification failed: unsupported certificate purpose (err: 26)
To examine certificates, use the following command:
# openssl x509 -text -in <path_to_cert_file>
125.4. Log format errors
125.4.1. Log entries are concatenated with Logstash
If you are using Logstash and find that log entries are concatenated,
make sure that you are using the json_lines
codec in your Logstash
server configuration.
The default json
codec in Logstash sometimes fails to parse log
entries passed from NXLog. Switch to the json_lines
codec
for better reliability.
125.5. Data missing errors
125.5.1. "Missing logdata" error
This happens when NXLog tries to evaluate a directive, but the required log data is not available in the current context. This causes any dependent operations to fail and the directive to terminate as well as the following error to be logged:
missing record, assignment possibly after drop()
This error occurs when attempting to access a field from the Exec directive of a Schedule block. The log data is not available in the current context. Log data is never available to a scheduled Exec directive because its execution is not triggered by a log message.
An attempt to access a field can occur directly with a field assignment, or indirectly by calling a function or procedure that accesses log data.
125.6. Processing unexpectedly paused or stopped
125.6.1. Processing stops if an output fails
NXLog can send one log stream to multiple outputs. This can be configured by either using the same input in multiple routes or using multiple outputs in the same route (see Routes). By default, when one of the outputs fails NXLog will stop sending logs to all outputs. This is caused by NXLog’s flow control mechanism, which is designed to prevent messages from being lost. Flow control pauses an Input or Processor module when the next module in the route is not ready to accept data.
In some cases, it is preferred for NXLog to continue sending logs to the remaining active output and discard logs for the failed output. The simplest solution is to disable flow control. This can be done globally with the global FlowControl directive, or for the corresponding Input (and Processor, if any) modules only, with the module FlowControl directive.
Note
|
With flow control disabled, an Input or Processor module will continue to process logs even if the next module’s buffers are full (and the logs will be dropped). |
To retain the improved message durability provided by flow control, it is possible to instead explicitly specify when to drop logs by using a separate route for each output that may fail. Add a pm_buffer module instance to that route, and configure the buffer to drop logs when it reaches a certain size. The output that follows the buffer can fail without causing any Input or Processor module before the buffer to pause processing. For examples, see the Using Buffers section.
125.6.2. Check open files and limits
When a system nears or exceeds its open files limit, significant performance penalties are typically quick to follow. LSOF (List Open Files) is a common debugging tool found on the majority of Linux systems and can reveal a great deal about the running system.
On Linux, run the following command to see, for example, which files NXLog has open:
$ lsof -u nxlog
This example returns the number of open files:
$ lsof -Fn -u nxlog | sort | uniq | wc -l
To check NXLog’s system limits use the following command:
$ cat /proc/$(sudo cat /opt/nxlog/var/run/nxlog/nxlog.pid)/limits
On Systems not using /proc
, check the system’s open file limit with:
$ sysctl kern.maxfiles
or with:
$ sysctl fs.file-max
Note
|
There is no Windows equivalent to lsof . You can use Windows Process Explorer from Microsoft’s Windows Sysinternals to inspect which program has files or
directories open.
|
125.6.2.1. systemd and open files limit
There are certain cases where systemd ignores system level file limits. This can cause "too many files open" errors.
2019-01-22 15:26:37 ERROR SSL error, failed to load ca cert from '/opt/nxlog/var/lib/nxlog/cert/agent-ca.pem', reason: Too many open files, system lib,
system lib
This scenario requires edits to the service file or an override. To check NXLog’s system limits use the following command:
$ cat /proc/$(cat /opt/nxlog/var/run/nxlog/nxlog.pid)/limits
On Systems not using /proc
, check the system’s open file limit:
$ sysctl kern.maxfiles
To adjust limits for nxlog, create /etc/systemd/system/nxlog.service.d/override.conf
and add the following definition:
1
2
[Service]
LimitNOFILE=100000
Update the service settings with:
$ systemctl daemon-reload
125.6.3. Log file is in use by another application
When trying to view the NXLog’s internal log file on Windows, you may receive an error message indicating that the log file is in use by another application and cannot be accessed.
To resolve this issue, either:
-
Open the log file with an application that does not use exclusive locking (such as Notepad)
or
-
Stop NXLog before opening the log file