Ask questions. Get answers. Find technical product solutions from passionate experts in the NXLog community.
README: how to ask questions effectively
gahorvath Nxlog ✓ created
Hi All,please adhere to the guidelines below to get faster responses to your queries.1 State the problem clearly:I am trying to collect logs from a log file using im_file and NXLog reads the whole file after each restart.2 Provide your configurationin a code block:
```
configuration text
```3 provide the contents of nxlog.log in a code block2023-04-12 08:00:27 INFO [CORE|main] configuration OK
2023-04-12 08:00:27 INFO [CORE|main] nxlog-5.7.7898 (68bb24e7e@REL_v5.7) started on Linux
2023-04-12 08:00:27 INFO [om_batchcompress|to_relay] connecting to lab1.home(192.168.14.52):2514
2023-04-12 08:00:27 INFO [om_batchcompress|to_relay] tcp connection established with lab1.home(192.168.14.52):2514
2023-04-12 08:00:27 INFO [xm_admin|admin] connecting to lab1.home(192.168.14.52):4041
2023-04-12 08:00:27 INFO [xm_admin|admin] tcp connection established with lab1.home(192.168.14.52):4041
2023-04-12 08:00:28 INFO [im_fim|fim] Module 'fim': FIM scan started
2023-04-12 08:00:28 ERROR [im_fim|fim] Module 'fim' could not open file '/opt/nxlog/bin/upgrade-nxsec.sh': Permission denied
2023-04-12 08:00:28 INFO [im_fim|fim] Module 'fim': FIM scan finished in 0.05 seconds. Scanned folders: 15 Scanned files: 102 Read file bytes: 156746314 provide environment descriptionNXLog CE is running on Windows 2022 server Package version is 3.2.23295 provide relevant detailsThis configuration has been working before we updated from version 2.11Now it does not6 if your problem involves parsing data provide samples of your input and expected output as well as what you actually getMy data looks like this:{"message": “message1”, "time": Thu 20 Apr 10:48:43 CEST 2023}I am getting an error:> error message hereI would like to receive the time in UNIX time (seconds since Epoch)Using this format enables us to get started much faster, and perhaps address your issue in the very first response we writeWould that not be great? :) Thanks for your cooperation!Gabor
gahorvath Nxlog ✓ created
apr_sockaddr_info failed
NOurdine created
Hello there i try to forward logs radius to my elastic siem , i got this error when executing nxlog.exe -f :2024-10-07 11:16:37 INFO nxlog-ce-3.2.2329 started2024-10-07 11:16:37 INFO reconnecting in 1 seconds2024-10-07 11:16:37 ERROR apr_sockaddr_info failed for 172.19.14.51 # IP du serveur distant:514; Unknow Host. Here is the config (without ip for privacy) : <Extension _syslog> Module xm_syslog</Extension><Extension xml> Module xm_xml</Extension><Input radius_log> Module im_file File "D:\NPS\NPS Logs\IN2410.log" SavePos TRUE ReadFromLast TRUE PollInterval 1 Exec parse_xml("/Event"); </Input><Output remote_syslog> Module om_udp Host # IP SRV FORWARD Port 514</Output><Route radius_to_remote> Path radius_log => remote_syslog</Route> Is it possible to get everything in the source folder not just one ? Because the name change everymonth , example : october :IN2410.logNovember: IN2411.log December: IN2412.log etc Please Thnaks a lot
NOurdine created
NxLog CE version 2.X statement of Supporting windows server 2022
Ken1 created
Hi,We are using NXLog-CE version 2.10.2150 and we prepare to deploy NX log agent in Windows server 2022.Is Nxlog 2.10.2150 supporting log collection for Windows server 2022?If the version is not supported, what is the minimum NX log agent to support Windows server 2022?Thanks.
Ken1 created
The NXLOG Process 100% CPU Load after Server restart.
Wolfgang E created
Hi NXLOG Community,the NXLOG Process 100% CPU Load after Server restart on Windows Systems.The Problemcause is simple but i dont understand why.A Handfull of Servers whit much logfiles (nxlogconfiguration use Wildcard for logfolder(s)).The Logfiles are written in the Cachefile and looks okay. But it feels like digs through all log files.Small footage of a Configuration:<Input catalina>
Module im_file
File 'C:\Tomcat\logs\\catalina*.log'
Recursive True
PollInterval 30
SavePos True
CloseWhenIdle True
<Exec>
#parse date
if $raw_event =~ /(\d\d\-\S\S\S-\d\d\d\d \d\d:\d\d:\d\d)/ $EventTime = strptime($1, '%d-%b-%Y%t%H:%M:%S');
# Now set the severity to something custom. This defaults to 'INFO' if unset.
if $raw_event =~ /\sSEVERE\s/ $Severity = 'CRITICAL';
else if $raw_event =~ /\sERROR\s/ $Severity = 'ERROR';
else if $raw_event =~ /\sWARN\s/ $Severity = 'WARNING';
else $Severity = 'INFO';
# The facility can be also set, otherwise the default value is 'USER'.
$SyslogFacility = 'AUDIT';
</Exec>
</Input>We use the Last CE Version.: nxlog-ce-3.2.2329Thanks, regards Wolfgang
Wolfgang E created
nxlog platform start up issue
EH_272573 created
Have installed nx onprem on ubuntu 22.04.5 LTS and cant login after the install finishes. Not sure how many pods are suppose to run but I see a postgress and vault 1c9df1fc6f5d nxlogacr.azurecr.io/vault:1.13.3 3 hours ago Up 2 hours ago (healthy) nxlog-1_2_2-vault-1eef5bec91376 nxlogacr.azurecr.io/postgres:16.3-alpine -c config_file=/e... 3 hours ago Up 2 hours ago (healthy) nxlog-1_2_2-postgres-1 Below is what shows listening and I don't see any web services. dp UNCONN 0 0 10.89.0.1:53 0.0.0.0:* users:(("dnsmasq",pid=1291,fd=4)) udp UNCONN 0 0 127.0.0.53%lo:53 0.0.0.0:* users:(("systemd-resolve",pid=627,fd=13)) udp UNCONN 0 0 [fe80::d433:a2ff:fe02:9e0d]%cni-podman1:53 [::]:* users:(("dnsmasq",pid=1291,fd=10)) tcp LISTEN 0 32 10.89.0.1:53 0.0.0.0:* users:(("dnsmasq",pid=1291,fd=5)) tcp LISTEN 0 4096 127.0.0.53%lo:53 0.0.0.0:* users:(("systemd-resolve",pid=627,fd=14)) tcp LISTEN 0 128 0.0.0.0:22 0.0.0.0:* users:(("sshd",pid=750,fd=3)) tcp LISTEN 0 32 [fe80::d433:a2ff:fe02:9e0d]%cni-podman1:53 [::]:* users:(("dnsmasq",pid=1291,fd=11)) tcp LISTEN 0 128 [::]:22 [::]:* users:(("sshd"I see the following error in nxp.logHA Mode standbyActive Node Address <none>Raft Committed Index 31Raft Applied Index 31Error authenticating: error looking up token: Error making API request.URL: GET http://0.0.0.0:8200/v1/auth/token/lookup-selfCode: 500. Errors:I got the following during the install[2024-09-25 00:28:55] [INFO] Vault container ID: 62f8bd5e1e00[2024-09-25 00:28:55] [INFO] Executing command (/init/bootstrap.sh) in container 62f8bd5e1e00...make: *** [Makefile:231: seed-vault] Error 2
EH_272573 created
NXLog Platform - vault container stuck in bootloop
nervevector created
Hello,I'm attempting to install the NXLog Platform on-prem on a Ubuntu 24.04 LTS VM, but I am running into the following error on the nxlog-1_2_2-vault-1 container:fetch https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.18/community/x86_64/APKINDEX.tar.gz
WARNING: updating and opening https://dl-cdn.alpinelinux.org/alpine/v3.18/main: temporary error (try again later)
WARNING: updating and opening https://dl-cdn.alpinelinux.org/alpine/v3.18/community: temporary error (try again later)
ERROR: unable to select packages:
supervisor (no such package):
required by: world[supervisor]
fetch https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.18/community/x86_64/APKINDEX.tar.gz
WARNING: updating and opening https://dl-cdn.alpinelinux.org/alpine/v3.18/main: temporary error (try again later)
WARNING: updating and opening https://dl-cdn.alpinelinux.org/alpine/v3.18/community: temporary error (try again later)
ERROR: unable to select packages:
supervisor (no such package):
required by: world[supervisor]
fetch https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/APKINDEX.tar.gz
When accessing this link from my host machine, I am able to download the .tar.gz, so it does not seem to be a network issue. Additionally, from the VM I am able to reach the internet perfectly fine to, e.g., run updates and I can cURL the URL from the VM as well.Any assistance on this would be appreciated!Thanks
nervevector created
im_msvistalog problems
cmarsura created
Hi,
seems there is a bug in im_msvistalog module in nxlog-ce-3.0.2272 for Windows.
After service has generated some output, stopping service sometimes give the message box "Broken pipe" and sometimes Nxlog write a malformed configcache.dat and following the start posts an ERROR failed to restore the saved position from bookmark xml (error:15008)"
Tested on Windows 2012 R2 and Windows 2019
<Output out>
Module om_file
File '%LOGDIR%\Output'
</Output>
<Input _im_eventlog>
Module im_msvistalog
SavePos TRUE
<QueryXML>
<QueryList>
<Query Id='1'>
<Select Path='Security'>*</Select>
</Query>
</QueryList>
</QueryXML>
# Query <QueryList>\
# <Query Id='0'>\
# <Select Path="Security">*</Select>\
# </Query>\
# </QueryList>
</Input>
#<Output outNull>
# Module om_null
#</Output>
<Route RouteA>
Path _im_eventlog => out
</Route>
This issue make the module not worth to use because not saving the bookmark of the processed registry record, force it to generate same logs from start over each service restart.
Best.
cmarsura created
Unable to ingest logs from file having 150MB
krishnap created
Hi Team,Need help!We're unable to ingest the logs from the file ‘output.json,’ which is 150MB in size. But i am able to send the logs from file ‘output.json’ which is 10MB in size to the forwarder.Below is the NXLog Community Edition Config file. Please help me is there anything to modify the Config file.define ROOT C:\Program Files\nxlogdefine ADCONTEXT_OUTPUT_DESTINATION_ADDRESS <Forwarder IP>define ADCONTEXT_OUTPUT_DESTINATION_PORT <Port>Moduledir %ROOT%\modulesCacheDir %ROOT%\dataPidfile %ROOT%\data\nxlog.pidSpoolDir %ROOT%\dataLogFile %ROOT%\data\nxlog.log<Extension json> Module xm_json</Extension><Input in_adcontext> Module im_file File "C:\AD\output.json" DirCheckInterval 3600 PollInterval 3600</Input><Output out_chronicle_adcontext> Module om_tcp Host %ADCONTEXT_OUTPUT_DESTINATION_ADDRESS% Port %ADCONTEXT_OUTPUT_DESTINATION_PORT%</Output><Route ad_context_to_chronicle> Path in_adcontext => out_chronicle_adcontext</Route>
krishnap created
Replace function to remove a string
Jay1 created
HiHere is my configuration. However, only Sysmon events are not working because they contain "/operational: " at the beginning of the message, which causes the events to be parsed incorrectlySo I want to know how to remove "/operational: "<Extension syslog> Module xm_syslog</Extension><Input in> Module im_msvistalog ReadFromLast True <QueryXML> <QueryList> <Query Id="0"> <Select Path="Application">*</Select> <Select Path="System">*</Select> <Select Path="Security">*</Select> <Select Path="Windows PowerShell">*</Select> <Select Path="Microsoft-Windows-Sysmon/Operational">*</Select> </Query> </QueryList> </QueryXML> Exec $UnixTime = integer($EventTime)/1000; </Input><Output devo_relay> Module om_tcp Host 192.168.29.133Port 13000 I tried Exec if ($SourceName =~ /Microsoft-Windows-Sysmon\/Operational/) { $Message = replace($Message, "/operational: ", ""); }and Exec if ($Message =~ /\/operational: /) { $Message = replace($Message, "/operational: ", ""); }But all did not work
Jay1 created
Read a log with yesterdays date in the filename
James created
I'm successfully using this config format with im_file to read logs with todays date in the filename:'\\server.domain\Logs\IN' + strftime(now(), "%y%m%d") + '.log'One of our services writes its log for the previous day at 3am on the next day. The filename has yesterdays date. What's the easiest/neatest/most efficient way of reading this log please?
James created
im_tcp with InputType LineBased occasional truncated lines
sgcaveney created
Hi folks,We have an NXLog CE 3.2.2329 configuration using im_tcp with InputType LineBased, receiving blobs of JSON separated by newlines. The received data is passed to xm_perl and then relayed to an output.Under some conditions where there may be thousands of events per second, there is a very rare chance for one event to be truncated at the end yielding an incomplete json blob. We've verified that the input data coming from upstream is valid, fully formed JSON - the emitter encoding logs look OK. The preceeding and following lines are also received and parsed successfully. Has anybody else experienced similar behavior? At this point we suspect a bug in NXLog, considering that it occurs very infrequently (2-3 days between recurrences) and it only occurs during periods of relatively high load (normal load is ~150 events per second, load when issue occurs is ~800/s+)Relevant configuration: <Extension gelf>Module xm_gelf</Extension><Extension perl>Module xm_perlPerlCode /etc/nxlog/scripts/parse_mist.pl</Extension><Input mist_tcp>Module im_tcpHost 0.0.0.0Port 32768InputType LineBased<Exec>perl_call("parse_mist");if $nxlog_internal_drop_log drop();</Exec></Input><Output graylog>Module om_udpHost xPort 12201OutputType GELF</Output><Route graylog>Path mist_tcp => graylog</Route>
sgcaveney created
NXLog Platform Installation on Ubuntu 22.04.4 LTS
jash@techowl.in created
I've successfully installed the NXLog platform on Ubuntu 22.04.4 LTS, but I'm facing an issue with port 443 not being accessible. I’ve allowed port 443 through the firewall, but I still can't access the web console. I’ve checked the service status, and everything seems fine. Is there any configuration or additional steps I might have missed to get port 443 up and running?Any help would be appreciated. Thanks in advance!@Ayodele@2Emeka Nwankwo
jash@techowl.in created
NXLog CE misbehavior (may be bug or multithread issue) - variables lost or corrupt between stages.
tubecleaner created
I have setup with RHEL 7.9 (kernel 3.10.0-1160.108.1.el7.x86_64) / NXLog CE 3.2.2329 / Postgresql 15 vanilla / libdbi 8.4 with configuration:User nxlog
Group nxlog
include /etc/nxlog/nxlog.d/*.conf
LogFile /var/log/nxlog/nxlog.log
LogLevel DEBUG
<Extension charconv>
Module xm_charconv
</Extension>
<Input sqlite3_dbi>
Module im_dbi
SQL SELECT * FROM vw_tp_message
Driver sqlite3
Option dbname term_refdata.db
Option sqlite3_dbdir /app/data/monitoring
Option sqlite3_timeout 1000
PollInterval 5
SavePos TRUE
</Input>
<Output postgres_dbi>
Module om_dbi
SQL INSERT INTO msglog.msglog (facility, severity, hostname, timestamp, application, message)
VALUES ($SyslogFacility, $SyslogSeverity, $Hostname, '$EventTime', $SourceName, $Message)
Driver pgsql
Option host /var/run/postgresql
Option username srv_nxlog
Option dbname MSGLog
Exec $Message = convert($Message, "windows-1251", "utf-8");
</Output>
<Route sqlite_postgres>
Path sqlite3_dbi => postgres_dbi
</Route>There are only 2 records in sqlite database:$ sqlite3 -header /app/data/monitoring/term_refdata.db 'select * from vw_tp_message'
id|SyslogFacility|SyslogSeverity|Hostname|EventTime|SourceName|Message
1293441|USER|INFO|SERVER1-D1-CL|2024-09-12 00:18:22.540|tp_msg|Some text in Windows1251 encoding
1293442|USER|INFO|SERVER1-D1-CL|2024-09-12 00:41:04.677|tp_msg|Another text in Windows1251 encodingAnd this is what I get in nxlog.log:...
2024-09-12 16:54:59 DEBUG logdata missing or undef 'EventTime', setting to NULL
2024-09-12 16:54:59 DEBUG om_dbi SQL: INSERT INTO msglog.msglog (facility, severity, hostname, timestamp, application, message) VALUES ('USER', 'INFO', 'SERVER1-D1-CL', 'NULL', 'tp_msg', 'Some text in Windows1251 encoding')
2024-09-12 16:54:59 ERROR [om_dbi.c:256/om_dbi_write()] -;[om_dbi.c:85/om_dbi_error()] om_dbi failed to execute SQL statement "INSERT INTO msglog.msglog (facility, severity, hostname, timestamp, application, message) VALUES ('USER', 'INFO', 'SERVER1-D1-CL', 'NULL', 'tp_msg', 'Some text in Windows1251 encoding')". ERROR: invalid input syntax for type timestamp: "NULL";LINE 1: ... VALUES ('USER', 'INFO', 'SERVER1-D1-CL', 'NULL', 't...; ^;[errorcode: 0]
...
2024-09-12 16:54:59 DEBUG logdata missing or undef 'EventTime', setting to NULL
2024-09-12 16:54:59 DEBUG om_dbi SQL: INSERT INTO msglog.msglog (facility, severity, hostname, timestamp, application, message) VALUES ('USER', 'INFO', 'SERVER1-D1-CL', 'NULL', 'tp_msg', 'Р?нициализация СЏРґСЂР° после перезапуска')
2024-09-12 16:54:59 ERROR [om_dbi.c:256/om_dbi_write()] -;[om_dbi.c:85/om_dbi_error()] om_dbi failed to execute SQL statement "INSERT INTO msglog.msglog (facility, severity, hostname, timestamp, application, message) VALUES ('USER', 'INFO', 'SERVER1-D1-CL', 'NULL', 'tp_msg', 'Р?нициализация СЏРґСЂР° после перезапуска')". ERROR: invalid input syntax for type timestamp: "NULL";LINE 1: ... VALUES ('USER', 'INFO', 'ABACUS-D1-CL', 'NULL', 't...; ^;[errorcode: 0]
...
2024-09-12 16:54:59 ERROR [expr.c:189/nx_expr_statement_execute()] assignment failed at line 61, character 70 in /etc/nxlog/nxlog.conf. statement execution has been aborted;[expr.c:90/nx_expr_statement_assignment_execute()] -;[expr.c:509/nx_expr_evaluate()] function 'convert' failed at line 61, character 69 in /etc/nxlog/nxlog.conf. expression evaluation has been aborted;[expr.c:279/nx_expr_eval_func()] -;[xm_charconv_funcproc_cb.c:283/nx_expr_func__convert()] -;[str.c:106/_nx_string_new_size()] oversized string (1688710), limit is 1048576 bytes
2024-09-12 16:54:59 DEBUG logdata missing or undef 'EventTime', setting to NULL
2024-09-12 16:54:59 DEBUG om_dbi SQL: INSERT INTO msglog.msglog (facility, severity, hostname, timestamp, application, message) VALUES ('USER', 'INFO', 'SERVER1-D1-CL', 'NULL', 'tp_msg', 'Р В Р’В РВ<C2>
2024-09-12 16:54:59 ERROR [om_dbi.c:256/om_dbi_write()] -;[om_dbi.c:85/om_dbi_error()] om_dbi failed to execute SQL statement "INSERT INTO msglog.msglog (facility, severity, hostname, timestamp, application, message) VALUES ('USER', 'INFO', 'SERVER1-D1-CL', 'NULL', 'tp_msg', 'Р В Р’ВРР<E2><80>
...As you can see, logdata variable $EventTime just disappear and $Message get corrupted after some retries (note: om_dbi retries crazy fast - about 50000 time every few seconds). If I change om_dbi to simple om_file then all works fine. Another note: this configuration worked fine some time since initial setup, but now error appear every time (maybe postgresql query time affects it?)
tubecleaner created
Receiving Windows DHCP logs are missing
Chung Wang created
Please give me some advice. When I used nxlog to receive DHCP server logs, I found that the logs were missing and the nxlog logs showed error messages. INFO inode changed for 'E:\DHCP\DhcpSrvLog-Fri.log' (56→56): reopening possibly rotated file
Chung Wang created
Is it possible to "reset the baseline" for the File Integrity module?
elazur@ecampus.com created
We currently have NXLog running on Windows 2019 with the file integrity module that monitors files in sub directories under a main directly. I.E. Say we have about 20 subdirectories for files under a directory called c:\code. If any file is changed/deleted/added under that directory, an alert message is sent out via the OM_UDP module to our siem. The problem I am looking to resolve, is that I will be copying over about 10 new directories with hundreds of files in each directory, and I don't want NXLog to generate hundreds of alerts because it found new files and directories. Is there a way to “reset” or “refresh” the baseline after the new files/folders are copied over so it will know those files should be there?
elazur@ecampus.com created
Unable to download Community addition
damiany@terrane.net created
I havent been able to download the files for any of the community edition agents. https://nxlog.co/downloads/nxlog-ce#nxlog-community-edition , select the OS version and click Download. 5 sec later I get a pop up stating “Undefined”. I have tried on different Browsers, platforms, and workstations and there is no change in the behavior.
damiany@terrane.net created
NXLog Manager 5.7.5
billychua created
Hi,Could you please advise on how to replace the NXLog-Manager's self-signed certificate for the HTTPS console?The steps in the following article do not apply to version 5.7.5:https://docs.nxlog.co/manager/current/installation/https.htmlPlease help. Thanks.Regards, Billy
billychua created
Nxlog Manager integrate with LDAPS
billychua created
I'm trying to integrate with LDAPS and do have some issue. I'm using the below document and below is the command to import LDAPS certificate into the cacerts keystore. keytool -keystore <PATH_TO_JRE>/lib/security/cacerts -import -alias \ certificate -file <PATH_TO_CERTIFICATE>/certificate.cerWhen i try to import the certificate and they are asking for the keystore password. I assume this keystore was created during installation not sure what is the password.Need to check whether anyone know what is the keystore password?https://docs.nxlog.co/manager/current/users/index.html
billychua created
jQuery vulnerability in nxlog-manager-5.5.5398
sgiliyal created
Greetings!
I have a vulnerability in jQuery to be addressed in the nxlog manager v5 (5.5.5398).
This is related to XSS vulnerabilities from the version of jQuery installed (1.8.3, 1.9.2 ui) .
Will upgrading Manager to 5.6.5633 resolve the issue and update jQuery to 3.5.0+ ?
If not, please can you provide steps to update jQuery manually?
Thanks!
Shyam (on behalf of Shashidhar Ghiliyal)
sgiliyal created