I have a file with multiple log lines, but I'm only interested in one type that has 6 fields in CSV format. I want to discard all the rest. So I have this [partial] file: <Extension csv> Module xm_csv Fields $time, $date, $host, $from, $ip, $loginfo, $color FieldTypes string, string, string, string, string, string, integer Delimiter | </Extension> <Input M2P_In> Module im_file File "C:\\M2PLogs\\log*" SavePos TRUE <Exec> if $raw_event =~ /^#/ drop(); else { csv->parse_csv(); if ( not defined $color ) drop(); $message = $raw_event; $raw_event = to_json(); } </Exec> </Input> In this case, if the line doesn't have 6 fields, I understand the $color field will be undefined. But it doesn't work, I get both lines in output: the correct one being processed and the rest in plain text. Perhaps I'm following the wrong approach, so I'm also open for alternatives. Could you please help?

Hi everybody! I have a problem with collecting logs. Сlient application logs: 2020-01-09 15:24:54 INFO connected to server OK 2020-01-09 15:25:22 INFO reconnecting in 1 seconds 2020-01-09 15:25:22 ERROR remote ssl socket was reset? (SSL_ERROR_SYSCALL with errno=9); End of file found TCP dump at the moment error: C: Client Hello S: Server Hello, Certificate, Certificate Request, Server Hello Done C: Certificate, Client Key Exchange, Certificat Verify, Change Cipher Spec, Encrypted Handshake Message S: New Session Ticket, Change Cipher Spec, Encrypted Handshake Message C: Application Data S: Encrypted Alert And part of the data segment is looped, the infinitely the same fragment of data is stored in the log file on the server side. How I may to detect the cause of this problem? I hope you help me, please. May be, I need to correct deep parameters of network settings? Thank you!

Hi! Please help, Is it possible to collect single line logs from file, without newline characters (SAP for example) with NxLog? In Splunk Heavy Forwarder this function work with Line Breaker(Regex) LINE_BREAKER = ([23])[A-Z][A-Z][A-Z0-9]\d{14}00

Hi All, I recently upgraded a machine (used for evaluating graylog) from CentOS 7 to CentOS 8 and it turns out that nxlog was removed in the process. Now I can't find it in the CentOS 8 dnf packages. Is nxlog available for Centos 8? and if so, who can I install it? Many thanks, Oren

Hi there, I'm just trying to find the Community Edition Windows installer MSI hashes/checksums (MD5, SHA-256 etc.) on your website but I'm struggling. I also see that the installer itself doesn't have a digital signature. Can you point me to where the installer hashes are if you have them? I want to be able to verify the integrity of my download before deployment. Best regards!

I have download an eval copy of nxlog EE, and am trying to send Microsoft DNS logs to my Splunk Cloud instance. I've read thru various documentation, but am getting an error "ERROR SSL certificate verification failed: self signed certificate in certificate chain (err: 19)", which is confusing me, because I am sending this to Splunk Cloud over HEC, which should have proper cert chain, so I'm not sure where it's getting a self-signed cert from. Here is what my config file looks like (not the whole thing, just the points of interest): <Input dns_analytical> Module im_etw Provider Microsoft-Windows-DNSServer </Input> <Output splunk_out> Module om_http URL https://http-inputs-xxx.splunkcloud.com/services/collector AddHeader Authorization: Splunk <auth key from Splunk HEC> ... </Output> <Route splunk> Path dns_analytical => splunk_out </Route> I could use on my local on-prem heavy forwarder but I would really like to send directly to Splunk Cloud over HEC, since it will minimize some moving parts.

We are utilizing nxlog win with the im_msvistalog module. Fairly simplified json output file (om_file) with an hourly rotation. AWS Kinesis Tap is configured to read from this file as a source. We continue to experience strange situations where nxlog will appear to be running healthy as a service, but will be writing 0 lines to the output file as if the mswineventlog back end is just not functioning. When this occurs the nxlog configuration schedule will executed as expected, and rotate 0KB files into the destination folder. This is running on two domain controllers with wineventlog seeing anywhere from 20-30 individual logs per second. Is anyone utilizing this type of configuration and experiencing any similar issues? Strangely enough a search in the forum for "kinesis" was returned with 0 results. Panic Soft #NoFreeOnExit TRUE define ROOT C:\Program Files (x86)\nxlog define CERTDIR %ROOT%\cert define CONFDIR %ROOT%\conf define LOGDIR %ROOT%\data define LOGFILE %LOGDIR%\nxlog.log LogFile %LOGFILE% Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data <Extension _json> Module xm_json </Extension> <Extension _exec> Module xm_exec </Extension> <Extension _fileop> Module xm_fileop # Check the size of our log file hourly, rotate if larger than 5MB <Schedule> Every 1 hour Exec if (file_exists('%LOGFILE%') and \ (file_size('%LOGFILE%') >= 5M)) \ file_cycle('%LOGFILE%', 8); </Schedule> # Rotate our log file every week on Sunday at midnight <Schedule> When @weekly Exec if file_exists('%LOGFILE%') file_cycle('%LOGFILE%', 8); </Schedule> </Extension> #Get logs from Windows EventLog API <Input eventlog> Module im_msvistalog </Input> #Send logs to customized file <Output out> #define OUTDIR C:\Program Files (x86)\logs\data_nxlogs define OUTDIR D:\nxlog-logs Exec create_var('offset'); Module om_file File '%OUTDIR%\output.json' <Schedule> When @hourly <Exec> if not dir_exists('%OUTDIR%\Older_Logs') dir_make('%OUTDIR%\Older_Logs'); rotate_to('%OUTDIR%\Older_Logs\' +strftime(now(), '%m_%d_%Y-%H_%M.log')); </Exec> </Schedule> <Exec> $EventTime = strftime($EventTime, '%Y-%m-%d %H:%M:%S %z'); to_json(); </Exec> #Check back the previous logs and delete all logs older than 24hrs <Schedule> When @hourly <Exec> #log_info(now()); #log_info(now() - 86400); set_var('offset', now() - 86400); if file_exists('D:\nxlog-logs\Older_Logs\' +strftime(get_var('offset'), '%m_%d_%Y-%H_%M.log')) file_remove('D:\nxlog-logs\Older_Logs\' +strftime(get_var('offset'), '%m_%d_%Y-%H_%M.log')); </Exec> </Schedule> </Output> <Route eventlog_to_out> Path eventlog => out </Route>

I can't get NXlog to read hidden files in Windows server 2012 R2. The source files are in this hidden folder: C:\ProgramData. There are no errors in the log file even if i put the debug mode on. I've double checked the path in input module and it is correct. The NXLog version is 4.4.4347. I also tried to look at Windows logs but no sign of problems there either. Any ideas on what to look next?

Scenario: Two inputs are configured. One is incorrect and results in a 400 response from the endpoint defined in om_http. The other input can send events without generating an error. Observed behavior: NXLog will continue to resend an event that failed which causes a loop condition that prevents any other events from being sent. Is there a way to change this behavior? Thanks!

Hi All, Is there a better solution to capturing IIS logs across multiple sites on a single server than adding an input per site in nxlog.conf? Thanks, Matt.

currently running 2.10.2150 on windows with a config that reads a debug DNS log (on c:), parses the logs, drops 99% of the logs, and writes the remainder out with file_write() we are seeing this memory pool allocation error and looking for info about if it's a known issue, something that we should be changing the config to deal with, or what's happening. our current config is something very close to (filenames may vary): Panic Soft #NoFreeOnExit TRUE define ROOT C:\Program Files (x86)\nxlog define CERTDIR %ROOT%\cert #define CONFDIR %ROOT%\conf define CONFDIR "C:\Program Files (x86)\nxlog\conf define LOGDIR %ROOT%\data #define LOGFILE %LOGDIR%\nxlog.log define LOGFILE %LOGDIR%\nxlog.log LogFile %LOGFILE% #LogLevel DEBUG define OUTFILE H:\dnsadvlogs\dns-filtered.log Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data <Extension _syslog> Module xm_syslog </Extension> <Extension _json> Module xm_json </Extension> <Extension _charconv> Module xm_charconv AutodetectCharsets iso8859-2, utf-8, utf-16, utf-32 </Extension> <Extension _exec> Module xm_exec </Extension> <Extension _fileop> Module xm_fileop # Check the log file size every hour and rotate if larger than 5 MB &lt;Schedule&gt; Every 1 hour &lt;Exec&gt; if (file_exists('%LOGFILE%') and file_size('%LOGFILE%') &gt;= 5M) file_cycle('%LOGFILE%', 8); &lt;/Exec&gt; &lt;/Schedule&gt; # Rotate log file every week on Sunday at midnight &lt;Schedule&gt; When @weekly Exec if file_exists('%LOGFILE%') file_cycle('%LOGFILE%', 8); &lt;/Schedule&gt; </Extension> <Extension _fileop2> Module xm_fileop # Rotate log file every week on Sunday at midnight &lt;Schedule&gt; Every 5 min Exec if file_exists('%OUTFILE') file_cycle('%OUTFILE%', 8); &lt;/Schedule&gt; </Extension> <Extension win_dns_parser> Module xm_multiline # look for a date at the start of the line HeaderLine /^\d+/\d+/\d+/ #filter blank and header lines from the input Exec if $raw_event =~ /^\s*$/ drop(); Exec if $raw_event =~ /^DNS Server log file creation/ drop(); Exec if $raw_event =~ /^Log file wrap at / drop(); Exec if $raw_event =~ /^Message logging key/ drop(); Exec if $raw_event =~ /^\sField # Information Values/ drop(); Exec if $raw_event =~ /^\s------- ----------- ------/ drop(); Exec if $raw_event =~ /^\s1 Date/ drop(); Exec if $raw_event =~ /^\s2 Time/ drop(); Exec if $raw_event =~ /^\s3 Thread ID$/ drop(); Exec if $raw_event =~ /^\s4 Context$/ drop(); Exec if $raw_event =~ /^\s5 Internal packet identifier$/ drop(); Exec if $raw_event =~ /^\s6 UDP/TCP indicator$/ drop(); Exec if $raw_event =~ /^\s7 Send/Receive indicator$/ drop(); Exec if $raw_event =~ /^\s8 Remote IP$/ drop(); Exec if $raw_event =~ /^\s9 Xid (hex)$/ drop(); Exec if $raw_event =~ /^\s10 Query/Response R = Response$/ drop(); Exec if $raw_event =~ /^\sblank = Query$/ drop(); Exec if $raw_event =~ /^\s11 Opcode Q = Standard Query$/ drop(); Exec if $raw_event =~ /^\sN = Notify$/ drop(); Exec if $raw_event =~ /^\sU = Update$/ drop(); Exec if $raw_event =~ /^\s*? = Unknown$/ drop(); Exec if $raw_event =~ /^\s12 [ Flags (hex)$/ drop(); Exec if $raw_event =~ /^\s13 Flags (char codes) A = Authoritative Answer$/ drop(); Exec if $raw_event =~ /^\sT = Truncated Response$/ drop(); Exec if $raw_event =~ /^\sD = Recursion Desired$/ drop(); Exec if $raw_event =~ /^\sR = Recursion Available$/ drop(); Exec if $raw_event =~ /^\s14 ResponseCode ]$/ drop(); Exec if $raw_event =~ /^\s15 Question Type$/ drop(); Exec if $raw_event =~ /^\s16 Question Name/ drop(); </Extension> <Input win_dns> Module im_file #File "H:\dnsadvlogs\dns.log" File "H:\dnsadvlogs\dns.log" # for testing we want to re-read from the start of the file each time SavePos False ReadFromLast True InputType win_dns_parser <Exec> $Message = $raw_event; $Message = replace($Message, "\r", ""); </Exec> </Input> <Output win_dns_trimmed> Module om_file File "H:\dnsadvlogs\dns-trimmed.log" File &quot;H:\dnsadvlogs\dns-trimmed.log&quot; &lt;Exec&gt; # manipulate the log entry here, not in input so that we can do other things with the raw input as well $Message =~ /^(?&lt;timestamp&gt;\d+\/\d+\/\d+ \d+\:\d+\:\d+\s+\S+)\s+(?&lt;pid&gt;\d+)\s+(?&lt;win_dns_type&gt;[^ ]*)\s+(?&lt;win_dns_packetID&gt;[^ ]*)\s+(?&lt;win_dns_protocol&gt;[^ ]*)\s+(?&lt;win_dns_direction&gt;[^ ]*)\s+(?&lt;win_dns_IP&gt;[^ ]*)\s+(?&lt;win_dns_hexID&gt;[^ ]*) (?&lt;win_dns_qr&gt;.) (?P&lt;win_dns_opcode&gt;.) \[(?&lt;win_dns_flags_hex&gt;\S+) (?&lt;win_dns_flags&gt;.*) (?&lt;win_dns_resultcode&gt;\S+)\]\s+(?&lt;win_dns_recordType&gt;\S+)\s+(?&lt;win_dns_query&gt;\S*).*ANSWER SECTION:.(?&lt;answer&gt;.+)\s+AUTHORITY SECTION:/s; $timestamp=$1; $pid=$2; $win_dns_type=$3; $win_dns_packetID=$4; $win_dns_protocol=$5; $win_dns_direction=$6; $win_dns_IP=$7; $win_dns_hexID=$8; $win_dns_qr=$9; $win_dns_opcode=$10; $win_dns_flags_hex=$11; $win_dns_flags=$12; $win_dns_resultcode=$13; $win_dns_recordType=$14; $win_dns_query=$15; $answer=$16; # drop messages if they are not replies (since the replies contain the query info) if $win_dns_qr == &quot; &quot; drop(); # drop logs that have no answer info #if $answer =~ /^\s+empty\s+$/ drop(); # drop logs that don't parse (if we don't have a requestion IP address, the log is worthless to UBA) if not defined $win_dns_IP drop(); # drop logs from dnsmasq caching servers. #if $win_dns_IP IN (&quot;10.16.169.32&quot;,&quot;10.49.58.4&quot;,&quot;10.49.58.3&quot;,&quot;10.16.6.22&quot;,&quot;199.47.139.239&quot;,&quot;199.47.139.238&quot;,&quot;199.47.139.182&quot;) drop(); # for the first pass, just filter the logs, don't change the format # this greatly simplifies the Splunk changes needed as the log parsing doesn't need to change $orig = replace($raw_event, &quot;\r&quot;, &quot;\r\n&quot;) + &quot;\r\n&quot;; file_write(&quot;%OUTFILE%&quot;,$orig); drop(); delete($orig); delete($Message); delete($EventReceivedTime); delete($SourceModuleName); delete($SourceModuleType); delete($pid); delete($win_dns_type); delete($win_dns_packetID); delete($win_dns_protocol); delete($win_dns_direction); delete($win_dns_hexID); delete($win_dns_qr); delete($win_dns_opcode); delete($win_dns_flags_hex); delete($win_dns_flags); $answer = replace($answer,&quot;(3)&quot;,&quot;.&quot;); $answer = replace($answer,&quot;(6)&quot;,&quot;.&quot;); $answer = replace($answer,&quot;(9)&quot;,&quot;&quot;); $answer = replace($answer,&quot;(0)&quot;,&quot;&quot;); $answer =~ s/\[\S\S\S\S\]//g; $win_dns_query = replace($win_dns_query,&quot;(3)&quot;,&quot;.&quot;); $win_dns_query = replace($win_dns_query,&quot;(6)&quot;,&quot;.&quot;); $win_dns_query = replace($win_dns_query,&quot;(9)&quot;,&quot;&quot;); $win_dns_query =~ s/\(0\)UDP//; $win_dns_query =~ s/\(0\)TCP//; $win_dns_query =~ s/\[\S\S\S\S\]//g; rename_field(&quot;win_dns_query&quot;,&quot;q&quot;); rename_field(&quot;win_dns_resultcode&quot;,&quot;rc&quot;); rename_field(&quot;win_dns_IP&quot;,&quot;ip&quot;); rename_field(&quot;win_dns_recordType&quot;,&quot;type&quot;); if $rc == &quot;NXDOMAIN&quot; delete($answer); to_json(); &lt;/Exec&gt; </Output> <Route win_dns_route> Path win_dns => win_dns_trimmed </Route>

Hello, I've read a few posts about ingesting multiple .evt files using the im_msvista module. Is there a way to do this? I can point to one, but I need to look at several. thank you! Franz

i am getting a cab error when loading on server 2016

Hello, I want to know if it's possible to have a dynamic variable like that : if .... { $1 = $2 ; } Thanks !

Hello! How can i get logs from shared folder by domain user? Should i start nxlog service under this user or there is another way?

Hi everybody, I would like to centralize sysmon events from endpoints using WEF and then forward "forwarded events" from collector server to graylog. Can i do it using Community edition of Nxlog ? If yes, can i concerve originated enpoint source ? which format does i need to use (SYSLOG or GELF) ? Thanks

Hello, I parameterized as seen in the examples the nxlog configuration file for the logs of my Windows 2016 servers, but when I restart the services with them. In the nxlog files I find this: nxlog failed to start: Expected </Extension_gelf> but saw </Extension> at C:\Program Files (x86)\nxlog\conf\nxlog.conf:48 nxlog failed to start: Expected </Extension_gelf> but saw </Extension> at C:\Program Files (x86)\nxlog\conf\nxlog.conf:48 nxlog failed to start: Expected </Extension2> but saw </Extension> at C:\Program Files (x86)\nxlog\conf\nxlog.conf:48 Do you have ideas to list the errors? thank you in advance

Hello, When using sysmon and nxlog, nxlog.exe is triggering alerts for suspicious file characteristics from sigma https://github.com/Neo23x0/sigma/blob/master/rules/windows/sysmon/sysmon_susp_file_characteristics.yml nxlog.exe binary (others?) have missing properties fields like product, fileversion, company, description. Having those along binary signature would be great! Thanks a lot for great work!

Hi, My app runs on Windows and it writes log to the file a.log. When I run NXLog to watch this file (to send log to the server), My app gets CFileException::sharingViolation trying to write log to a.log file. It does not happen always, but when the log is big sometimes it happens and my app can not write log to a.log file. Searching the internet, I found 'You should open the file with CFile::shareDenyNone' to avoid this problem. So my questions is : Is NXLog opens the file with ' CFile::shareDenyNone'? How can I avoid this problem? Here's my nxlog.conf 's input configuration. <Input watchfile> Module im_file File 'C:\Log\a.log' Exec $Message = $raw_event; Exec if file_name() =~ /.\(.)/ $SourceName = $1; SavePos TRUE Recursive TRUE CloseWhenIdle TRUE </Input> Any comment will be appreciated. Thanks Harry

Hi Guy's, We are using nxlog community to ship server logs to a lake, powershell logs are collected on the server but we want to exclude shipping them, after much googling i thought it would be as easy as adding this line to the conf: Exec if ($SourceName == 'PowerShell (PowerShell)' drop(); This does not work, can any experts in NX out there advise? Thanks, O.