Hi, I'm using the latest version community edition of nxlog to send data from Server 2016. The server started going crazy (CPU/Messages per second) and the eventlog has this. Warning 8/26/2018 8:21 EvntAgnt 3006 None Error reading log event record. Handle specified is 1313800280. Return code from ReadEventLog is 87. Warning 8/26/2018 8:21 EvntAgnt 3006 None Error reading log event record. Handle specified is 1313800280. Return code from ReadEventLog is 87. we were getting 31,000 of these events per second. Microsoft had this to say... https://support.microsoft.com/en-us/help/177199/bug-readeventlog-fails-with-error-87 So, is there a different way to configure the nxlog so this won't occur? We just rebooted for now but I'm sure stopping/starting the service would have fixed it but they didn't know.

Hi, I'm trying to install the nxlog msi package (community edition) on Windows Server 2012 but installer abnormally terminates with error 1603. I'm running it with an administrative account, and tried it either using windows gui and command line (msiexec with /qn switch) Any ideas?

I am trying to collect Windows Defender Eventlogs on our win clients and forward them to our central log server (graylog). I have included the Windows Defender Log in the config files search query like this: <Select Path="Microsoft-Windows-Windows Defender/Operational">*</Select>\ This seems to work basically as there are messages coming through to the logserver (mostly of EventID 1150/1151) but the important ones (1116,1117,...) dont show up. I already tried to select 1116 and 1117 specifically in the query and provoked some entries of that type, but to no avail. The events are in the eventlog, but they dont make it out of the client system. Searching the web didn't get me anywhere either, so i'm trying my luck with the nxlog community. Any help would be very much appreciated! Thanks

I am using a config of Output syslog1> Module om_tcp Host syslog.server.name Port 6514 Exec to_syslog_ietf(); OutputType Syslog_TLS </Output> and a source of <Input eventlog_application> Module im_msvistalog Query <QueryList> <Query Id="0"> <Select Path="Application">*</Select> </Query> </QueryList> <Exec> $Message = 'EventID[' + $EventID + '] Log[' + $Channel + '] Type[' + $EventType + '] Domain[' + $Domain + '] User[' + $AccountName + '] ' + $Message; $SyslogFacilityValue = 3; </Exec> </Input> but when the message arrives as the syslog server the ident field does NOT include the PID. I expaect something like SourceName[PID]: the text of the message but what I get is SourceName: the text of the message How can I get the PID of the SourceName?

We are currently using Nxlog enterprise 4.0.3735 to send events logs(windows events - > json out) from windows server to central log server. I want to estimate system requirements(cpu,memory,disk) needed by nxlog agent to handle 4000 to 5000 events per seconds. I m also concerned about what happens when the agent gets high (unexpected) number of events more than expected during peak load times Are logs buffered some where on disk and sent later or something similar when link between the nxlog agent and central logserver is broken for some reason ?

Hello all, I wanted to syslog the Windows logs to one of my SIEM in snare format. when i configure the nxlog as given below, the hostname is there in header. but i wanted to use any of the IP address of the server in header. current logs format: Dec 15 11:18:52 ad-server.ad.cap62.test MSWinEventLog 1 Directory Service 1742824 Tue Dec 15 11:18:52 2015 103 NTDS ISAM N/A N/A Information ad-server.ad.cap62.test Knowledge Consistency Checker NTDS (500) NTDSA: The database engine stopped the instance (0). 458 the format i need: Dec 15 11:18:52 10.10.10.1 MSWinEventLog 1 Directory Service 1742824 Tue Dec 15 11:18:52 2015 103 NTDS ISAM N/A N/A Information ad-server.ad.cap62.test Knowledge Consistency Checker NTDS (500) NTDSA: The database engine stopped the instance (0). 458 config file: define ROOT C:\Program Files (x86)\nxlog Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data LogFile %ROOT%\data\nxlog.log <Extension syslog> Module xm_syslog </Extension> <Input internal> Module im_internal </Input> <Input in> Module im_msvistalog </Input> <Output out> Module om_udp Host <IP Address of host> Port 514 Exec to_syslog_snare(); </Output> <Route 1> Path eventlog, in => out </Route> Please anyone help me with that?

I have seen one question on this topic in 2014, has anyone managed to implement a solution that allows events to be sent to multiple servers in sequence such that duplicate events are not sent. I am sending events on to a Nifi cluster and would really rather not send the traffic back through the nginx load balancer a 2nd time.

This is a complex question about a complex problem, but please feel free to read i anyway :-) We use NXLog to read the Windows eventlog and also csv files. We send the data to a linux loghost which does some regexp-based parsing. We now encounter problems with missing fields. Example 1: A csv file with three columns A, B and C. It looks like this: #A,B,C a,b,c 1,2,3 x,y,z NXLog reads this file, uses an xm_csv module to parse the content, uses an xm_json module to convert it to JSON, uses an xm_syslog module to further convert it to syslog and finally sends it to the syslog server. At first glance this works fine. Here is the result from the syslog server: ...SourceModuleType:im_file,A:a,B:b,C:c,Hostname:... ...SourceModuleType:im_file,A:1,B:2,C:3,Hostname:... ...SourceModuleType:im_file,A:x,B:y,C:z,Hostname:... However, as soon as we have empty values in a csv row, we run into problems: #A,B,C a,b, 1,,3 ,y, leads to: ...SourceModuleType:im_file,A:a,B:b,Hostname:... ...SourceModuleType:im_file,A:1,C:3,Hostname:... ...SourceModuleType:im_file,B:y,Hostname:... All the fields that are empty in the csv file are now absent in the syslog message. (And this is a huge issue for our regexp parser.) Interestingly, this: #A,B,C "a","b","" 1,"",3 ,y,"" leads to: ...SourceModuleType:im_file,A:a,B:b,C:,Hostname:... ...SourceModuleType:im_file,A:1,B:,C:3,Hostname:... ...SourceModuleType:im_file,B:y,C:,Hostname:... So it looks like NXLog treats an empty string in a different way than "nothing". (However, this is of limited value, as we are dealing with csv files created by applications, such as Exchange Server logfiles.) The same behaviour not only applies to csv-based file inputs but also to the Windows eventlog input. Example 2: Windows security log Windows event 4624 (successful login) includes the two fields "TargetUserName" and "TargetDomainName". If users log in to a system using "DOMAIN\username" as their username, everything works fine: ...,TargetUserName:Administrator,TargetDomainName:DEMO,TargetLogonId:... However, if a user uses the UPN (user.name@domain.org) to log in, Windows writes the UPN into the "TargetUserName" field and leaves the TargetDomainField empty. This results in: ...,TargetUserName:Administrator@demo.local,TargetLogonId:... The "TargetDomainName" field is missing. I have already spent a lot of time to troubleshoot this issue, but still haven't found THE solution. This is what I found out so far: The parse_csv() function of the xm_csv extension module does or does not create an NXLog field for each value in each row. If there is a value, such as in 1,2,3 a field with the respective value is generated. For empty strings, such as in 1,"",3 a field is generated as well, with an empty string as its value. But for "nothing", such as in 1,,3 no field is generated, and this seems to be the root cause of our problem. Both to_json() and to_kvp() add all existing NXLog fields to the message, even the ones having "undef" values. But of course, fields that don't exist do not appear in the message. I could not find a way to distinguish between an NXLog field that is present but has an "undef" value and a field that is not present. The if defined($A) construct returns false in both cases. There is a (not so elegant) solution for the problem that applies to csv files only: Before calling parse_csv() all fields can be initialized manually, like this: $A = ""; $B = ""; $C = ""; parse_csv(); However, this does not apply to the Windows eventlog input, because the fields differ between Windows event ids. So finally, the questions: Does anybody have a (config-based) solution for this problem? Is a change in NXLog behaviour needed to resolve the root cause? (I hope NXLog staff is reading this post.)

Hello: I am trying to send an event related to terminal server logins to a logstash instance. When the message associated with the event contains an IPv4 address it works without issue. When the source network address in the message is IPv6 (containing a percent symbol), nxlog fails to parse the message with the error "ERROR_EVT_UNRESOLVED true". I have tried doing a replace of the % but it hasn't had and effect. Thanks!

Hello there, I am having some issues with NxLog using xm_xml. The regex seems to match fine, so I think it's something else. When I try to run it, I get a completely blank file. Here is my config <Extension multiline> Module xm_multiline HeaderLine /^\s*&lt;Obj RefId=&quot;[0-9][0-9]?[0-9]?[0-9]?&quot;&gt;/ #EndLine /^\s*&lt;/entry&gt;/ </Extension> <Extension _xml> Module xm_xml </Extension> <Extension _json> Module xm_json </Extension> <Input in3> Module im_file File &quot;C:\\Users\\administrator\\Desktop\\2016.xml&quot; InputType multiline SavePos FALSE ReadFromLast FALSE Exec if $raw_event !~ /^\s*&lt;Obj RefId=&quot;[0-9][0-9]?[0-9]?[0-9]?&quot;&gt;/ drop(); Exec parse_xml(); Exec to_json(); </Input> <Output out3> Module om_file File &quot;C:\\Users\\administrator\\Desktop\\testxml.txt&quot; </Output> Sample Data: <Obj RefId="0"> <TN RefId="0"> <T>System.Diagnostics.EventLogEntry#Security/Microsoft-Windows-Security-Auditing/4673</T> <T>System.Diagnostics.EventLogEntry#Security/Microsoft-Windows-Security-Auditing</T> <T>System.Diagnostics.EventLogEntry</T> <T>System.ComponentModel.Component</T> <T>System.MarshalByRefObject</T> <T>System.Object</T> </TN> <ToString>System.Diagnostics.EventLogEntry</ToString> <Props> <S N="MachineName">testServer.ad.testDomain.com</S> <BA N="Data" /> <I32 N="Index">447206</I32> <S N="Category">(13056)</S> <I16 N="CategoryNumber">13056</I16> <I32 N="EventID">4673</I32> <Obj N="EntryType" RefId="1"> <TN RefId="1"> <T>System.Diagnostics.EventLogEntryType</T> <T>System.Enum</T> <T>System.ValueType</T> <T>System.Object</T> </TN> <ToString>SuccessAudit</ToString> <I32>8</I32> </Obj> <S N="Message">A privileged service was called._x000D__x000A__x000D__x000A_Subject:_x000D__x000A__x0009_Security ID:_x0009__x0009_S-1-5-21-26028188-150678075-188441444-157239_x000D__x000A__x0009_Account Name:_x0009__x0009_testAccount_x000D__x000A__x0009_Account Domain:_x0009__x0009_testDomain_x000D__x000A__x0009_Logon ID:_x0009__x0009_0x2053a6e4_x000D__x000A__x000D__x000A_Service:_x000D__x000A__x0009_Server:_x0009_Security_x000D__x000A__x0009_Service Name:x0009-_x000D__x000A__x000D__x000A_Process:_x000D__x000A__x0009_Process ID:_x0009_0x1770_x000D__x000A__x0009_Process Name:_x0009_C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe_x000D__x000A__x000D__x000A_Service Request Information:_x000D__x000A__x0009_Privileges:_x0009__x0009_SeCreateGlobalPrivilege</S> <S N="Source">Microsoft-Windows-Security-Auditing</S> <Obj N="ReplacementStrings" RefId="2"> <TN RefId="2"> <T>System.String[]</T> <T>System.Array</T> <T>System.Object</T> </TN> <LST> <S>S-1-5-21-26028188-150678075-188441444-157239</S> <S>testAccount</S> <S>testDomain</S> <S>0x2053a6e4</S> <S>Security</S> <S>-</S> <S>SeCreateGlobalPrivilege</S> <S>0x1770</S> <S>C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe</S> </LST> </Obj> <I64 N="InstanceId">4673</I64> <DT N="TimeGenerated">2018-08-14T08:34:37-04:00</DT> <DT N="TimeWritten">2018-08-14T08:34:37-04:00</DT> <Nil N="UserName" /> <Nil N="Site" /> <Nil N="Container" /> </Props> <MS> <I32 N="EventID">4673</I32> </MS> </Obj>

I have NXLog configured by the GrayLog Sidecar collector. The generated config will be available here for a week : NXLog generated config I use multiline to get my full stack traces. It is working fine But sometimes I get a message with two line, and my delimiters are present, twice. See this : Message in GrayLog, hosted on imgur I am wondering why, and if there is a solution.

Hello All. I am at my wits end here. I am trying to capture specific data in a line of text so I can display it in a Dashboard. I am trying to capture the data from a .txt file. In the file the line looks like this: Aug 09 09:00:08.076 CCSPAVCS01 VCS: [0x0000068c] ThrowingIncrementPolicy::increment1. newActiveCalls = 33 maxCalls = 220 The data I want is "newActiveCalls = 33". Basically, I am interested in all occurrences of that. The section in NxLog for the log (called "messages.txt") basically looks like this: <Input ccsp_logs> Module im_file File "C:\temp\messages.txt" Exec parse_syslog(); Exec to_json(); </Input> The output looks like this: {"EventReceivedTime":"2018-08-13 11:25:52","SourceModuleName":"ccsp_logs","SourceModuleType":"im_file","SyslogFacilityValue":1,"SyslogFacility":"USER","SyslogSeverityValue":5,"SyslogSeverity":"NOTICE","SeverityValue":2,"Severity":"INFO","Hostname":"CCSPAVCS01","EventTime":"2018-08-09 09:00:08","SourceName":"VCS","Message":"[0x0000068c] ThrowingIncrementPolicy::increment1. newActiveCalls = 33 maxCalls = 220 "} As you can see, the data I want (newActiveCalls = 33) is encapulated in the "Message" field created by NxLog. I have tried numerous things (to many to list here). Does anyone know a way to capture that specific data so I may extract the numeral (in this case, 33; which is the important part) to create a quantified graphic for my servers? (In this case, I am using Kibana. The end result being a graph showing: Server-1 with 33 active calls, Server-2 with 20 active calls, etc.) Thank you all for your time and help.

Hello, I have a CSV column that has returns in it. If I try to run NxLog, it errors out saying it expected 16 columns and got 0 (for the blank lines), got 1 (when there was one entry), etc. The configuration I have works if I open up the CSV in excel or something and replace the returns with a ; instead. I was wondering if there was a way to do that with NxLog. Maybe something like Exec $Message = replace($column11, "\r\n", ";"); but i cannot seem to get NxLog to run correctly because it's spitting out the errors stated above. Any help would be great. Thanks

Hi, im new to NXlog, would like to understand few things in Nxlog. 1.what if a windows server which has nxlog running on it and lost connectivity to network? will the logs stored in buffer, if so what is the size of it and where is the location of it.

I'm using the OutputType GELF_TCP my questions are 1) Why nxlog sends short_message field and not the message field? 2) Why nxlog is truncating the log content to 65 char? I can fix it by using the following Exec directive but why this at the first place? Exec $ShortMessage = $raw_event Output Config ########## <Output default_out> Module om_tcp Host 10.30.0.85 Port 12203 OutputType GELF_TCP </Output> Now, with this config the event record I receive on the receiving end at 10.30.0.85 is following { "version": "1.1", "_EventReceivedTime": "2018-08-03 16:08:08", "_SourceModuleName": "dnsDebug_in", "_SourceModuleType": "im_file", "host": "srvr2012", "short_message": "3/08/2018 4:07:12 PM 04C4 PACKET 00000039F019C100 UDP Rcv 10.30", "timestamp": 1533276488, "level": 6 }

I am looking into the possibility of being able to be notified if nxlog cannot output to redis. Has anyone managed to setup anything that writes to redis and if it cannot it sends an email.

Hi, I encountered the following error suddenly in Nxlog: ERROR ### ASSERTION FAILED at line 74 in om_http.c/om_http_timeout_event(): "modconf->timeout_event == NULL" I have an INPUT that reads from multiple files and then send to an output, on_http, which makes a request to an ElasticSearch instance. Do you have more details on what can cause this error?

This is so strange, I was looking at the previous questions regarding the type of failure and we're seeing it too, but not related to IIS logs but ftp logs. Here I'll include a sample of the logs and part of the config. 2018-07-25 12:42:02 ERROR if-else failed at line 449, character 236 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; procedure 'parse_csv' failed at line 449, character 90 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; Not enough fields in CSV input, expected 12, got 1 in input 'java.lang.NullPointerException' 2018-07-25 12:42:02 ERROR if-else failed at line 449, character 236 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; procedure 'parse_csv' failed at line 449, character 90 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; Not enough fields in CSV input, expected 12, got 8 in input '20180725 12:42:01 ERROR Exception Caught in channel 1189859467: ' 2018-07-25 12:42:02 ERROR if-else failed at line 449, character 236 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; procedure 'parse_csv' failed at line 449, character 90 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; Not enough fields in CSV input, expected 12, got 5 in input 'java.net.ConnectException: Connection refused: connect: /xx.xx.xx.xx:22' Windows FTP events log: <Input FTP_Logs> Module im_file File "D:\GAMFT\logs\goanywhere_*" SavePos TRUE Exec if $raw_event =~/^#|(\s+at\s+)/ drop();\ else\ {\ w3c_ftp-&gt;parse_csv();\ $EventTime = parsedate($date + &quot; &quot; + $time);\ $SourceName = &quot;WINFTP&quot;;\ $raw_event = to_json();\ } the config looks right too me. Suggestions?

Hello, I currently am sending ms eventlogs from a server to our syslog server, and this is working fine. So I wanted to add to the nxlog conf file to allow it to collect the IIS W3C logs from this server as well and send them to the syslog server. A lot of the support examples out there added scripting to parse the w3c logs to json format. I do not want to do this. Just need to the raw logs to go to our syslog server. Below is my conf file. I'd appreciate pointers and/or the pointing out of errors. define ROOT C:\Program Files (x86)\nxlog Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data LogFile %ROOT%\data\nxlog.log <Extension syslog> Module xm_syslog </Extension> <Extension w3c> Module xm_csv Fields $date, $time, $s-sitename, $s-computername, $s_ip, $cs_method, $cs_uri_stem, $cs_uri_query, $s_port, $cs_username, $c_ip, $cs-version, $cs_User_Agent, $cs(Cookie), $cs_Referer, $cs-host, $sc_status, $sc_substatus, $sc_win32_status, $sc-bytes, $cs-bytes, $time_taken FieldTypes string, string, string, string, string, string, string, string, integer, string, string, string, string, string, string, string, integer, integer, integer, integer, integer, integer Delimiter ' ' </Extension> <Input EventLogs> Module im_msvistalog </Input> <Input IIS> Module im_file File "C:\inetpub\logs\LogFiles\W3SVC1\u_ex*" SavePos TRUE UndefValue - </Input> <Output Syslog> Module om_udp Host 172.16.x.x Port 514 Exec to_syslog_snare(); </Output> <Route 1> Path EventLogs, IIS => Syslog </Route>

When attempting to input IIS logs I see the following error: 'ERROR invalid keyword: w3c->parse_csv()'. Has anyone come across this before and knows how I can parse IIS log into a JSON format to input to redis my config <Extension _exec> Module xm_exec </Extension> <Extension _json> Module xm_json </Extension> <Extension _csv> Module xm_csv Fields $date, $time, $s_ip, $cs_method, $cs_uri_stem, $cs_uri_query, $s_port, $cs_username, $c_ip, $cs(User_Agent), $cs(Referer), $sc_status, $sc_substatus, $sc_win32_status, $time_taken FieldTypes string, string, string, string, string, string, integer, string, string, string, string, string, string, string, integer </Extension> <Extension w3c> Module xm_csv #Fields: date time s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs(User-Agent) cs(Referer) sc-status sc-substatus sc-win32-status time-taken Fields $date, $time, $s_ip, $cs_method, $cs_uri_stem, $cs_uri_query, $s_port, $cs_username, $c_ip, $cs(User_Agent), $cs(Referer), $sc_status, $sc_substatus, $sc_win32_status, $time_taken FieldTypes string, string, string, string, string, string, integer, string, string, string, string, string, string, string, integer Delimiter ' ' QuoteChar ''' </Extension> <Extension w3cinput> Module xm_w3c Delimiter ' ' </Extension> <Input iis_logs> Module im_file File "C:\inetpub\logs\W3SVC2\u_ex*.log" ReadFromLast True Recursive True SavePos True Exec if $raw_event =~ /^#/ drop(); else { w3c->parse_csv(); $EventTime = parsedate($date + " " + $time); $SourceName = "IIS"; to_json(); } </Input> <output out_iis> Module om_redis Host # Port # Key "iis" </Output> <Route 1> Path iis_logs => out_iis </Route>