Hi, We are planning to deploy NXLog to thousands of endpoints and need to know when an agent is no longer sending data regularly. Is there an established method for determining NXLog is working normally at scale? Thanks!

Hello, I resisted posting here for a while but am finally at a loss to explain what I'm observing. I'm trying to send nginx access logs to graylog, and am mostly using code adapted from the nxlog ce user guide but I haven't been able to get the fields to successfully capture and arrive in graylog. The nginx server in question logs two sorts of traffic: x.x.x.x - - [04/Feb/2020:03:23:22 +0000] "GET /" 400 271 "-" "-" "-" - These are status checks from a load balancer which I'm wanting to drop. x.x.x.x - <username> [04/Feb/2020:03:23:01 +0000] "POST /rest/api/endpoint HTTP/1.1" 201 508 "-" "okhttp/3.3.0" "-" - This is legitimate traffic to the application behind nginx which I want to parse and capture. I have the following input defined in my config file. <Input nginx_access> Module im_file File '/var/log/nginx/access.log' PollInterval 1 SavePos True ReadFromLast True Recursive False RenameCheck False <Exec> if $raw_event =~ /(?x)^(\S+)\ \S+\ (\S+)\ \[([^\]]+)\]\ \"(\S+)\ (.+)\ HTTP\/\d\.\d\"\ (\S+)\ (\S+)\ \"([^\"]+)\"\ \"([^\"]+)\"\ \"\S+\"$/ { $Hostname = $1; if $2 != '-' $AccountName = $2; $EventTime = parsedate($3); $HTTPMethod = $4; $HTTPURL = $5; $HTTPResponseStatus = $6; if $7 != '-' $FileSize = $7; if $8 != '-' $HTTPReferer = $8; if $9 != '-' $HTTPUserAgent = $9; delete($Message); } else drop(); </Exec> </Input> If I remove the else drop(); then this config forwards only the load balancer lines that I don't want, which indicates to me that the lines I do want are matching my regexp. But as shown, this configuration forwards nothing. I also don't think anything is wrong with my outputs or routes since just straight piping this all to graylog in the message field works fine. If anyone can help me understand what is going on here that would be much appreciated. Thanks

Hey y'all. Does anyone know where to find the GPG key used to sign the NXLog Enterprise Edition (trial) RPM packages? Key ID 9354d2051da9e40e, or just 1da9e40e for short. If someone at NXLog reads this, maybe this page would be a good site to link it from, if it's not immediately available in the download package. I figured if I post this question here, maybe someone else will find it useful in the future. (Wisdom of the Ancients, and all that) Thanks :)

I am currently running into an issue receiving syslog over ssl/tls. I cannot figure it out for the life of me! Version: CE-2.10.2150 Error: INFO SSL connection accepted from IP_ADDRESS:PORT ERROR SSL certificate verification failed: unsupported certificate purpose (err: 26) WARNING SSL connection closed from IP_ADDRESS:PORT Config: <Input in> Module im_ssl Host 0.0.0.0 Port 516 AllowUntrusted TRUE CAFile %CERTDIR%%CA-PEM% CertFile %CERTDIR%%CRT% CertKeyFile %CERTDIR%%KEY% KeyPass %PASSWORD% </Input>

Hi, I am using NXLog to run a script via im_exec and the internal restart true feature which restarts my script when it ends. The thing I'm noticing is that it's filling up the nxlog.log file with error conditions because the script stopped. ERROR Module test got EOF, process exited? The script is restarting deliberately so I don't need to know about these specific messages from this specific module. How can I suppress error logging from a specific module? Have a wonderful day! NOTE: I was able to get it working a different way. The solution involves executing a script on startup to get the initial useful data and send it to a log file. From there NXLog searches for that file and copies it to gateway.log. The input module im_file then reads that file which initiates the heartbeat. The way that im_file works is that it caches the file and won’t read from it again unless the file changes(modifying last modified doesn’t appear to count as a change). To that end I added logic that would periodically delete and re-copy the file. In this way im_file has a new file to read from, and thus send the heartbeat. While NXLog logs when the new files are read, they are logged as WARNING, not ERROR, so you can set the LogLevel to ERROR <Extension _filedelete> Module xm_fileop <Schedule> Every 25 sec <Exec> if file_exists('%GATEWAY%.log') file_remove('%GATEWAY%.log'); if file_exists('%GATEWAY%') file_copy('%GATEWAY%', '%GATEWAY%.log'); </Exec> </Schedule> </Extension> <Input ipGateway> Module im_file PollInterval 60 ReadFromLast false SavePos false File '%GATEWAY%.log' <Exec> $gateway = $raw_event; </Exec> </Input> This is not meant to be some kind of groundbreaking discovery. I only post it here in case someone else ends up trying to figure out this requirement. Hope this helps!

Attempting to log to a syslog server based upon the specified filtered log ID's. When I use this config, I do not get any errors in the error log when the service starts. However I do not get anything to my syslog server. Not sure if this a problem with the in or output and would love some feedback. Panic Soft #NoFreeOnExit TRUE define ROOT C:\Program Files (x86)\nxlog define CERTDIR %ROOT%\cert define CONFDIR %ROOT%\conf define LOGDIR %ROOT%\data define LOGFILE %LOGDIR%\nxlog.log LogFile %LOGFILE% Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data <Extension _syslog> Module xm_syslog </Extension> define HighEventIds 4618, 4649, 4719, 4765, 4766, 4794, 4897, 4964, 5124, 1102 define MediumEventIds 4621, 4675, 4692, 4693, 4706, 4713, 4714, 4715, 4716, 4724, 4727, 4735, 4737, 4739, 4754, 4755, 4764, 4764, 4780, 4816, 4865, 4866, 4867, 4868, 4870, 4882, 4885, 4890, 4892, 4896, 4906, 4907, 4908, 4912, 4960, 4961, 4962, 4963, 4965, 4976, 4977, 4978, 4983, 4984, 5027, 5028, 5029, 5030, 5035, 5037, 5038, 5120, 5121, 5122, 5123, 5376, 5377, 5453, 5480, 5483, 5484, 5485, 6145, 6273, 6274, 6275, 6276, 6277, 6278, 6279, 6280, 24586, 24592, 24593, 24594 define LowEventIds 4608, 4609, 4610, 4611, 4612, 4614, 4615, 4616, 4624, 4625, 4634, 4647, 4648, 4656, 4657, 4658, 4660, 4661, 4662, 4663, 4672, 4673, 4674, 4688, 4689, 4690, 4691, 4696, 4697, 4698, 4699, 4700, 4701, 4702, 4704, 4705, 4707, 4717, 4718, 4720, 4722, 4723, 4725, 4726, 4728, 4729, 4730, 4731, 4732, 4733, 4734, 4738, 4740, 4741, 4742, 4743, 4744, 4745, 4746, 4747, 4748, 4749, 4750, 4751, 4752, 4753, 4756, 4757, 4758, 4759, 4760, 4761, 4762, 4767, 4768, 4769, 4770, 4771, 4772, 4774, 4775, 4776, 4778, 4779, 4781, 4783, 4785, 4786, 4787, 4788, 4789, 4790, 4869, 4871, 4872, 4873, 4874, 4875, 4876, 4877, 4878, 4879, 4880, 4881, 4883, 4884, 4886, 4887, 4888, 4889, 4891, 4893, 4894, 4895, 4898, 5136, 5137 <Input events> Module im_msvistalog <QueryXML> <QueryList> <Query Id="0" Path="Directory Service"> <Select Path="Directory Service">*[System[Provider[ @Name='Microsoft-Windows-ActiveDirectory_DomainService']]] </Select> </Query> </QueryList> </QueryXML> <Exec> if $EventID NOT IN (%HighEventIds%) and $EventID NOT IN (%MediumEventIds%) and $EventID NOT IN (%LowEventIds%) drop(); </Exec> </Input> <Output udp> Module om_udp Host 172.17.103.13 Port 514 Exec to_syslog_snare(); </Output> <Route uds_to_udp> Path events => udp </Route> <Extension _charconv> Module xm_charconv AutodetectCharsets iso8859-2, utf-8, utf-16, utf-32 </Extension> <Extension _exec> Module xm_exec </Extension> <Extension _fileop> Module xm_fileop # Check the size of our log file hourly, rotate if larger than 5MB &lt;Schedule&gt; Every 1 hour Exec if (file_exists('%LOGFILE%') and \ (file_size('%LOGFILE%') &gt;= 5M)) \ file_cycle('%LOGFILE%', 8); &lt;/Schedule&gt; # Rotate our log file every week on Sunday at midnight &lt;Schedule&gt; When @weekly Exec if file_exists('%LOGFILE%') file_cycle('%LOGFILE%', 8); &lt;/Schedule&gt; </Extension>

I am looking for the buster Debian package. I already tried stretch version but it complained about unmet dependencies related to libssl. The following packages have unmet dependencies: nxlog-ce : Depends: libperl5.24 (>= 5.24.0) but it is not installable Depends: libssl1.0.2 (>= 1.0.2d) but it is not installable root@debian:~# dpkg --search libssl libssl1.1:amd64: /usr/share/doc/libssl1.1 libssl1.1:amd64: /usr/share/doc/libssl1.1/changelog.Debian.gz libssl-dev:amd64: /usr/lib/x86_64-linux-gnu/pkgconfig/libssl.pc libssl1.1:amd64: /usr/share/doc/libssl1.1/NEWS.Debian.gz libssl-dev:amd64: /usr/share/doc/libssl-dev/changelog.gz libssl1.1:amd64: /usr/lib/x86_64-linux-gnu/libssl.so.1.1 libssl-dev:amd64: /usr/share/doc/libssl-dev libssl-dev:amd64: /usr/lib/x86_64-linux-gnu/libssl.a libssl-dev:amd64: /usr/share/doc/libssl-dev/copyright libssl1.1:amd64: /usr/share/doc/libssl1.1/copyright libssl-dev:amd64: /usr/lib/x86_64-linux-gnu/libssl.so libssl-dev:amd64: /usr/share/doc/libssl-dev/changelog.Debian.gz android-libboringssl: /usr/lib/x86_64-linux-gnu/android/libssl.so.0 libssl1.1:amd64: /usr/share/doc/libssl1.1/changelog.gz root@debian:~# dpkg --search libperl libperl5.28:amd64: /usr/share/doc/libperl5.28/changelog.Debian.gz libperl5.28:amd64: /usr/lib/x86_64-linux-gnu/libperl.so.5.28 libperl5.28:amd64: /usr/share/doc/libperl5.28 libperl5.28:amd64: /usr/lib/x86_64-linux-gnu/libperl.so.5.28.1 libperl5.28:amd64: /usr/share/doc/libperl5.28/copyright I think it's because of compile, so I tried to compiled it myself on a Debian buster but I stuck on ./configure which can not find libcrypto.(libssl-dev installed, libraries exist in lib path, ... ) I appreciate if anyone can share the binary package for buster release. Thank you.

Hi, I'm trying to execute the following bash command using NXLog's im_exec input module: /usr/sbin/route -n | grep 'UG[ \t]' | awk '{print $2}' From the Linux terminal that command will give the local gateway. When I attempt to run the command in NXLog I get the following: 2020-01-24 13:43:09 WARNING im_exec process /usr/sbin/route exited 2020-01-24 13:43:10 ERROR subprocess '8114' returned a non-zero exit value of 3 Input module I'm using to execute the command: <Input Bash> Module im_exec Restart true Command /usr/sbin/route Arg -n | grep 'UG[ \t]' | awk '{print $2}' Arg grep 'UG[ \t]' Arg awk '{print $2}' Arg sleep 30 <Exec> $gateway = $raw_event; </Exec> </Input> I've tried various combinations of configuring the Arg's but so far no luck. If I try putting everything in the Command field I get this error: /usr/sbin/route -n |grep 'UG[ \t]'| awk '{print $2}';sleep 30;No such file or directory Can someone help me with the correct syntax? NOTE: It seems like the problem is with using the | character in the command. Anyone know if NXLog will accept this argument or what syntax I need to use? Thanks in advance!!

I am following the nxlog to splunk guide here: https://nxlog.co/documentation/nxlog-user-guide/splunk.html. Specifically, section '93.3. Sending Specific Log Types for Splunk to Parse'. When testing, even using the config from the page, I am still getting an error (see further below) <Input eventxml> Module im_msvistalog Channel Security CaptureEventXML TRUE Exec $raw_event = $EventXML; </Input> <Output splunk_hec> Module om_http URL https://127.0.0.1:8088/services/collector/raw AddHeader Authorization: Splunk c6580856-29e8-4abf-8bcb-ee07f06c80b3 </Output> This generates this error: ERROR invalid keyword: CaptureEventXML at C:\Program Files (x86)\nxlog\conf\nxlog.conf Any ideas? thanks

Hello, I'm using nxlog community to send logs from my firewalls through syslog. My output is like that: <output log_to_file> module om_file File 'c:\datas\firewall_' + $MessageSourceAddress + '.log' <output> If my firewalls 192.168.1.1 and 192.168.1.2 are correctly configured, the result will be two files: c:\datas\firewall_192.168.1.1.log and c:\datas\firewall_192.168.1.2.log My problem is now to rotate this file on a daily basis. I've tried to apply the command rotate_to but it seems that it applies only on the first file. How can I do rotation on multiple files with names based on a variable ? Thank you !

nxlog-ce 2.9.1716 on Windows 10/Server 2016. The usage of om_udp seems to cause nxlog.exe to listen on ephemeral port. om_tcp does not cause this. I can't find anything in documentation that explains this behavior. Please help. MK

After using kvp parser i've got variables with spaces in names. For example: "$Event Time" or "$Source Name". I'm interested in two things: How i can interact with this variable names? For example i'm trying construction "$EventTime = $Event Time;" with many shield variations: ",',),], etc, but this not work. it is possible to prevent this situation? Massage format example below: "DeviceEvent: Virus found,IP Address: 10.X.X.X,Computer name: xxx-xxx,Source: Auto-Protect scan,Risk name: Infostealer.Gampass,Occurrences: 1,File path: X:\xxxx_xxx.exe,Description: ,Actual action: Moved back,Requested action: Quarantined,Secondary action: Deleted,Event time: 2020-01-21 17:24:58,Event Insert Time: 2020-01-21 17:27:06,End Time: 2020-01-21 17:59:17,Last update time: 2020-01-21 18:01:07,Domain Name: xxxx,Group Name: XXXX,Server Name: xx-xxx,User Name: SYSTEM,Source Computer Name: ,Source Computer IP: ,Disposition: Reputation was not used in this detection.........."

Hello, I am sending a message with hostname to my syslog server, my conf is as follows: define ROOT C: \ Program Files (x86) \ nxlog Moduledir% ROOT% \ modules CacheDir% ROOT% \ data Pidfile% ROOT% \ data \ nxlog.pid SpoolDir% ROOT% \ data LogFile% ROOT% \ data \ nxlog.log <Extension _syslog> Module xm_syslog </Extension> <Input in> Module im_msvistalog <Exec> parse_syslog (); $ Message = "hostnamexxx" + $ Message; to_syslog_ietf (); </Exec> </Input> <Output out> Om_udp module Host xx.xxx.xx Port 514 Exec to_syslog_ietf (); </Output> <Route 1> Path in => out </Route> My log is coming with the message correctly: Feb 12 23:11:34 DESKTOP-XXXXX Microsoft-Windows-Eventlog [964] hostnamexxxxINFO 1102 The audit log was cleared. Subject: Security ID: # xxxxxxxx-1001 Account Name: Admin Domain Name: DESKTOP-XXXXX Logon ID: 0xD438A However, the message "hostnamexxxx" is coming in the middle of the log, as you can see above. This is disturbing my parser, is there any way I can put this "hostnamexxxx" message last in my log? Example: Feb 12 23:11:34 DESKTOP-XXXXX Microsoft-Windows-Eventlog [964] INFO 1102 The audit log was cleared. Subject: Security ID: # xxxxxxxx-1001 Account Name: Admin Domain Name: DESKTOP-XXXXX Logon ID: 0xD438A hostnamexxxx Thanks

Hello everyone, I'm having trouble architecturing something with hmac verification, any help of yours would be welcome. I'm trying to setup an architecture with three clients/servers and using hmac/hmac_check to guarantee the integrity of the logs. Logs 1 are created by client 1 and sent to client 2, which check their integrity, logs 2 are created by client 2, and both logs 1 and 2 are send in the end to client 3 which finally check for integrity for both of them. Here is a "beautiful" scheme to illustrate my words: client 1 ---hmac(logs 1)---> client 2 ---hmac_check(logs 1) + hmac(logs 2)---> client 3 ---hmac_check(logs 1) + hmac_check(logs2) + hmac(logs 3)-->... I would have multiple routes on each clients and I would use different instances of each processor on each route to avoid having errors like "processor X already used in route A and was not load in the route B". Also, I would be using batchcompress between client 1 and 2 but UDP between client 2 and 3. I'm wondering how you would do this thing? Would you open multiple UDP ports on client 3 to receive independantly logs coming from client 1 and client 2 and check the hmac independantly or would you send those two logs on the same network port and check them with the same hmac_check processor? And would you use multiple routes to process independantly logs coming from different clients because of the hmac integrity check? Thank you in advance, Kind Regards,

Say I wanted to run a PowerShell script on an interval using the im_exec module how would I do that? <Input powershell> Module im_exec Command "C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe" Arg "script goes here" <Exec> $output = $raw_event; </Exec> </Input> I can make the interval work by creating pauses in the code but then it appears as though the script process would run forever which is undesirable for efficiency and stability reasons. Is there a way to run the code on a interval, say every fifteen minutes? Thoughts? Thanks in advance!!

Hi, It's basically what the title implies. I want to run a command that produces a string and save it as a variable to be used later in various output blocks. So far the command is working and I can send the output to a om_file logfile but I don't know how to assign the value to a variable to be used elsewhere in the config. <Input powershell> Module im_exec Command "C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe" Arg "$IP = Get-WmiObject Win32_NetworkAdapterConfiguration | where {$_.IPEnabled};$IP.DefaultIPGateway" <Exec> parse_json(); $EventTime = parsedate($EventTime); </Exec> </Input> Thanks!!

Hello, The documentation about the support of SNARE format (https://nxlog.co/documentation/nxlog-user-guide/snare.html) describes how the account name should be passed. However, the function to_syslog_snare(), puts N/A in that field instead of the username in the Windows event. This happens both in the example output (https://nxlog.co/documentation/nxlog-user-guide/snare.html#generating-snare) and with the latest nxlog community edition. Is this a bug or a paying feature of the enterprise edition? Sincerely

Hi all, I'm trying to deploy NXLog Enterprise to a couple of Windows domain controllers, pointed to Graylog to audit security. As part of this, we need the "ResolveSID" feature so have gone Enterprise edition. Unfortunately only got a 1 year sub approved which doesn't allow enterprise support :( However with Enterprise edition, the only Security events that NXLog sends to Graylog are "Event log automatic backup" events when the .evtx files get rotated - nothing else from Security (all other sources seem OK). The interesting thing here is that the Community edition doesn't have this problem - security events are forwarded just fine. I've also tried the 32bit v4 Enterprise MSI (since the Community edition is 32bit), but it exhibits the same behaviour as above. The v3 Enterprise edition seems to mostly work, but ignores "ResolveSID TRUE" (it reads the setting ok, I've tested this by changing it to a non-boolean value to test that it read it to complain about it, and it did, but when set to TRUE, it still sends unresolved numeric SIDs through for Event ID 4627 "Group membership information" events) Does anyone have any further troubleshooting tips for either of these problems? Ideally I'd like to get v4 working. Thanks

I am trying to use nxlog to read from a text file and send to a syslog collector from Rapid7. I have read through Rapid7's documentation found at https://insightidr.help.rapid7.com/docs/nxlog I have verified that the input is working by outputting to a text file and verifying there was output, but it will not output to syslog. This is traversing a firewall and I have the proper rules in place, I can also see that the traffic is not getting to the firewall as there are no packets dropped or captured at the firewall so my only logical conclusion is nxlog is not sending the output to syslog. ICMP (Ping) traffic goes through the firewall so there is connectivity. Is there a debug log mode for nxlog where I can get more details on what is happening. Here is my config file: Panic Soft #NoFreeOnExit TRUE define ROOT C:\Program Files (x86)\nxlog define CERTDIR %ROOT%\cert define CONFDIR %ROOT%\conf define LOGDIR %ROOT%\data define LOGFILE %LOGDIR%\nxlog.log LogFile %LOGFILE% Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data <Extension _syslog> Module xm_syslog </Extension> <Extension _charconv> Module xm_charconv AutodetectCharsets iso8859-2, utf-8, utf-16, utf-32 </Extension> <Extension fileop> Module xm_fileop </Extension> <Input Ping_Audit> Module im_file File 'C:\ping-logs\audit.log' SavePos TRUE ReadFromLast TRUE PollInterval 1 <Exec> if $raw_event =~ /^#. drop(); else { to_syslog_bsd(); } </Exec> </Input> <Output Rapid7_5004> Module om_udp Host 192.168.251.201 Port 5004 </Output> <Route 1> Path Ping_Audit => Rapid7_5004 </Route> <Extension _exec> Module xm_exec </Extension> <Extension _fileop> Module xm_fileop # Check the size of our log file hourly, rotate if larger than 5MB &lt;Schedule&gt; Every 1 hour Exec if (file_exists('%LOGFILE%') and \ (file_size('%LOGFILE%') &gt;= 5M)) \ file_cycle('%LOGFILE%', 8); &lt;/Schedule&gt; # Rotate our log file every week on Sunday at midnight &lt;Schedule&gt; When @weekly Exec if file_exists('%LOGFILE%') file_cycle('%LOGFILE%', 8); &lt;/Schedule&gt; </Extension>

Hi, I'm using NXLog on Windows to send Event Log data into Google BigQuery but I'm not seeing my data. There appears to be no errors in nxlog.log When I use this: <Output out> Module om_file File 'C:\Program Files (x86)\nxlog\data\nxlogfile.log' </Output> I can see the data from the event log being logged. Is there a module that logs all data as it's being transmitted? Thanks in advance!!