I'm trying to ws_ftp logs that are xml formatted as such : <?xml version="1.0" encoding="utf-8" ?> <log> <entry> <log_time> xxxx </log_time> <description>data <![CDATA[Connection established]]></description> <service>data </service> <sessionid> xxxx </sessionid> <type>0</type> <severity>0</severity> <lstnconnaddr> ip </lstnconnaddr> <cliconnaddr>ip </cliconnaddr> <sguid> xxx </sguid> </entry> ... with this configuration : define ROOT C:\Program Files\nxlog <Extension gelf> Module xm_gelf </Extension> Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data LogFile %ROOT%\data\nxlog.log <Extension multiline> Module xm_multiline HeaderLine /^<event>/ EndLine /^</event>/ </Extension> <Extension xmlparser> Module xm_xml </Extension> <Extension json> Module xm_json </Extension> <Input in> Module im_file File "PATH TO .XML" SavePos FALSE PollInterval 1 ReadFromLast FALSE InputType multiline <Exec> $EventData = extract_xml("/log/entry"); if $EventData == "" { delete($EventData); } to_json(); </Exec> </Input> <Output out> Module om_file File "C:\\temp\\output_test.txt" </Output> <Route 1> Path in => out </Route> i can't get past this error : ERROR Couldn't parse Exec block at C:\Program Files\nxlog\conf\nxlog.conf:34; couldn't parse statement at line 35, character 40 in C:\Program Files\nxlog\conf\nxlog.conf; function 'extract_xml()' does not exist or takes different arguments ERROR module 'in' has configuration errors, not adding to route '1' at C:\Program Files\nxlog\conf\nxlog.conf:51 how can i get further?

I have an XML that I am trying to strip some data out so it can be pre-processed by software on another machine. XML file is being generated on a windows logging to an XML file. I am at bit of a lost I have tried too many things to list here. What I am trying to do is remove our domain name and our domain email address before it is sent to the machine to be pre-processed. I only want the username. Any records that have host\domain.com don’t need to be sent and I figured out how to drop that data. (number 4). If it helps I am running nxlog-ce-3.0.2284. Here is an example of event: (Removed bunch of xml fields for clarity of this post) 1: <Event><Timestamp data_type="4">05/25/2022 12:45:43.806</Timestamp><Userid data_type="1">DOMAIN\username</Userid><IP-Address data_type="3">x.x.x.x</IP-Address><Endtimestamp data_type="5">05/25/2022 12:46:43.806</Endtimestamp> 2: <Event><Timestamp data_type="4">05/25/2022 12:45:43.806</Timestamp><Userid data_type="1">username</Userid><IP-Address data_type="3">x.x.x.x</IP-Address><Endtimestamp data_type="5">05/25/2022 12:46:43.806</Endtimestamp> 3: <Event><Timestamp data_type="4">05/25/2022 12:45:43.806</Timestamp><Userid data_type="1">username@domain.com</Userid><IP-Address data_type="3">x.x.x.x</IP-Address><Endtimestamp data_type="5">05/25/2022 12:46:43.806</Endtimestamp> 4: <Event><Timestamp data_type="4">05/25/2022 12:45:43.806</Timestamp><Userid data_type="1">host\domain.com</Userid><IP-Address data_type="3">x.x.x.x</IP-Address><Endtimestamp data_type="5">05/25/2022 12:46:43.806</Endtimestamp> Nxlog.conf: #NoFreeOnExit TRUE define ROOT C:\Program Files\nxlog define CERTDIR %ROOT%\cert define CONFDIR %ROOT%\conf define LOGDIR %ROOT%\data define LOGFILE %LOGDIR%\nxlog.log LogFile %LOGFILE% Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data <Extension xmlparser> Module xm_xml </Extension> <Extension json> Module xm_json </Extension> <Input in> Module im_file File "C:\LogFiles\log*.log" InputType LineBased Exec $Message = $raw_event; SavePos TRUE ReadFromLast TRUE <Exec> Discard everything that doesn't seem to be an xml event if $raw_event !~ /^<Event>/ drop(); if $raw_event =~ /^(.+)host(.+)/ drop(); parse_xml(); Convert to JSON to_json(); </Exec> </Input> <Output out> Module om_udp Host yy.xx.xx.xx Port 514 </Output> <Route 1> Path in => out </Route>

Hello, We are using NXLog extensively and just recently started seeing some parsing issues, so far specifically on Windows 2012r2 using Windows Event Forwarding, but could be others. It appears to be something with processing self-closed tags at first glance, but I've done a little bit of testing myself and couldn't directly reproduce the problem (so far), so figured I'd come here for guidance. Specific details are included below. Current Behavior NXLog appears to be improperly parsing empty, self-closed XML tags. Expected Behavior NXLog properly ignores empty, self-closed XML tags. NXLog Version: nxlog-ce-2.10.2150 NXLog Configuration File: ## NXLog configuration file define ROOT C:\Program Files (x86)\nxlog define LOGFILE %ROOT%\data\nxlog.log Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data LogFile %LOGFILE% # Rotate agent logs on the local system such that only the last 4 files are kept <Extension fileop> Module xm_fileop # Check the size of our log file every hour and rotate if it is larger than 1M <Schedule> Every 1 hour Exec if (file_size('%LOGFILE%') >= 1M) file_cycle('%LOGFILE%', 4); </Schedule> # Rotate our log file every week on sunday at midnight <Schedule> When @weekly Exec file_cycle('%LOGFILE%', 4); </Schedule> </Extension> <Extension syslog> Module xm_syslog </Extension> <Extension json> Module xm_json </Extension> # Agent logs <Input internal> Module im_internal </Input> # OS logs <Input eventlog> Module im_msvistalog # Drop EventID 5156 logs when application name is nxlog.exe Exec if ($SourceName == 'Microsoft-Windows-Security-Auditing')\ AND ($EventID == 5156)\ AND ($Application =~ /nxlog.exe$/)\ drop(); </Input> <Input forwardedEvents> Module im_msvistalog Query <QueryList> \ <Query Id="0"> \ <Select Path="ForwardedEvents">*</Select>\ </Query> \ </QueryList> </Input> # SIEM port 3514 is listening for JSON-encoded IETF style syslog messages # OutputType Syslog_TLS required to enable the octet-framing described in RFC5425 <Output out> Module om_tcp Host XXX.XXX.XXX.XXX Port 3514 OutputType Syslog_TLS Exec $Message = to_json(); # Remove param-value pairs from structured data header with names > 32 characters. Exec if ($SourceName == 'Microsoft-Windows-GroupPolicy') {\ if ($EventID == 5017) OR ($EventID == 6017) OR ($EventID == 7017)\ delete($OperationElaspedTimeInMilliSeconds);\ if ($EventID == 5116) OR ($EventID == 6116) OR ($EventID == 7116)\ delete($GpsvcInitTimeElapsedInMilliseconds);\ if ($EventID == 5126) OR ($EventID == 6126) OR ($EventID == 7126)\ delete($GPODownloadTimeElapsedInMilliseconds);\ if ($EventID == 5257) OR ($EventID == 6257) OR ($EventID == 7257)\ delete($PolicyDownloadTimeElapsedInMilliseconds);\ if ($EventID == 5351) OR ($EventID == 6351) OR ($EventID == 7351)\ delete($WinlogonReturnTimeElapsedInMilliseconds);\ } Exec to_syslog_ietf(); </Output> <Route 1> Path internal, eventlog, forwardedEvents => out </Route> Windows Version: Windows 2012r2 (potentially others, but confirmed for sure on this one) Example Exported XML from EventViewer <?xml version="1.0" encoding="utf-8" standalone="yes"?> <Events><Event xmlns='http://schemas.microsoft.com/win/2004/08/events/event' xml:lang='en-US'><System><Provider Name='Microsoft-Windows-Security-Auditing' Guid='{54849625-5478-4994-A5BA-3E3B0328C30D}'/><EventID>4688</EventID><Version>2</Version><Level>0</Level><Task>13312</Task><Opcode>0</Opcode><Keywords>0x8020000000000000</Keywords><TimeCreated SystemTime='2019-10-23T14:01:56.684789600Z'/><EventRecordID>5238702734</EventRecordID><Correlation/><Execution ProcessID='4' ThreadID='3948'/><Channel>Security</Channel><Computer>hostname.domain</Computer><Security/></System><EventData><Data Name='SubjectUserSid'>S-1-5-18</Data><Data Name='SubjectUserName'>workstation$</Data><Data Name='SubjectDomainName'>DOMAIN</Data><Data Name='SubjectLogonId'>0x3e7</Data><Data Name='NewProcessId'>0x1754</Data><Data Name='NewProcessName'>C:\Windows\System32\wbem\WmiApSrv.exe</Data><Data Name='TokenElevationType'>%%1936</Data><Data Name='ProcessId'>0x2f8</Data><Data Name='CommandLine'/><Data Name='TargetUserSid'>S-1-0-0</Data><Data Name='TargetUserName'>-</Data><Data Name='TargetDomainName'>-</Data><Data Name='TargetLogonId'>0x0</Data></EventData><RenderingInfo Culture='en-US'><Message>A new process has been created...</Message><Level>Information</Level><Task>Process Creation</Task><Opcode>Info</Opcode><Channel>Security</Channel><Provider>Microsoft Windows security auditing.</Provider><Keywords><Keyword>Audit Success</Keyword></Keywords></RenderingInfo></Event></Events> Example Data as Received on the wire: <14>1 2019-10-21T15:44:36.650065-04:00 hostname.domain Microsoft-Windows-Security-Auditing 4 - [NXLOG@14506 Keywords="-9214364837600034816" EventType="AUDIT_SUCCESS" EventID="4688" ProviderGuid="{54849625-5478-4994-A5BA-3E3B0328C30D}" Version="2" Task="13312" OpcodeValue="0" RecordNumber="355111132" ThreadID="5020" Channel="Security" Category="Process Creation" Opcode="Info" SubjectUserSid="S-1-5-18" SubjectUserName="workstation$" SubjectDomainName="DOMAIN" SubjectLogonId="0x3e7" NewProcessId="0x13a8" NewProcessName="C:\\Windows\\System32\\wbem\\WmiPrvSE.exe" TokenElevationType="%%1936" CommandLine\'/><Data_Name=\'TargetUserSid="S-1-0-0" TargetUserName="workstation$" TargetDomainName="DOMAIN" TargetLogonId="0x3e4" EventReceivedTime="2019-10-21 15:45:38" SourceModuleName="eventlog" SourceModuleType="im_msvistalog"] {"EventTime":"2019-10-21 15:44:36","Hostname":"hostname.domain","Keywords":-9214364837600034816,"EventType":"AUDIT_SUCCESS","SeverityValue":2,"Severity":"INFO","EventID":4688,"SourceName":"Microsoft-Windows-Security-Auditing","ProviderGuid":"{54849625-5478-4994-A5BA-3E3B0328C30D}","Version":2,"Task":13312,"OpcodeValue":0,"RecordNumber":355111132,"ProcessID":4,"ThreadID":5020,"Channel":"Security","Message":"A new process has been created...","Category":"Process Creation","Opcode":"Info","SubjectUserSid":"S-1-5-18","SubjectUserName":"workstation$","SubjectDomainName":"domain","SubjectLogonId":"0x3e7","NewProcessId":"0x13a8","NewProcessName":"C:\\Windows\\System32\\wbem\\WmiPrvSE.exe","TokenElevationType":"%%1936","CommandLine'/><Data Name='TargetUserSid":"S-1-0-0","TargetUserName":"workstation$","TargetDomainName":"DOMAIN","TargetLogonId":"0x3e4","EventReceivedTime":"2019-10-21 15:45:38","SourceModuleName":"eventlog","SourceModuleType":"im_msvistalog"} NOTE: I know the above two examples don't reference the exact same event. They are just for illustration and reference purposes You should see that in the XML exported from EventViewer, there's an empty and self-closed 'CommandLine' tag. However, in the example pulled from the wire, you should see that the 'CommandLine' tag has been mangled and parsed as if it were part of the 'TargetUserSid' tag and there's some XML data there. Have you seen similar behavior when parsing logs either through the ForwardedEvents (Windows Event Forwarding) or otherwise? Addendum: We have also seen this behavior with other empty, self-closed XML Fields such as a 'TargetDomainName' field, so it does not seem to be specifically related to the CommandLine tag itself.

Hi guys, I wanted to use NXLog to send data from a XML file into GrayLog. My first test worked fine with a simple XML file. But with a Nessus Report (in XML) it don't work, because this file is more complexe. Do you have an example configuration nxlog.conf to help me ? My actual nxlog.conf : define ROOT C:\Program Files (x86)\nxlog Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data LogFile %ROOT%\data\nxlog.log <Extension gelf> Module xm_gelf </Extension> <Extension multiline> Module xm_multiline HeaderLine /^<event>/ EndLine /^</event>/ </Extension> <Extension xmlparser> Module xm_xml </Extension> <Extension json> Module xm_json </Extension> <Input in> Module im_file File "C:\Program Files\Graylog\collector-sidecar\generated\exempleFichier.xml" SavePos FALSE ReadFromLast FALSE InputType multiline <Exec> # Discard everything that doesn't seem to be an xml event if $raw_event !~ /^<event>/ drop(); # Parse the xml event parse_xml(); # Rewrite some fields $EventTime = parsedate($timestamp); delete($timestamp); delete($EventReceivedTime); # Convert to JSON to_json(); </Exec> </Input> <Output out> Module om_udp Host xx.xx.xx.xx Port 12201 </Output> <Route 1> Path in => out </Route> And a (verry little) preview of my Nessus File... <Report name="Scan_Nessus" xmlns:cm="http://www.nessus.org/cm"> <ReportHost name="192.168.1.1"><HostProperties> <tag name="traceroute-hop-8">192.168.1.1</tag> <tag name="LastUnauthenticatedResults">1490484150</tag> <tag name="Credentialed_Scan">false</tag> <tag name="policy-used">policies_scan</tag> <tag name="patch-summary-total-cves">1</tag> <tag name="os">other</tag> <tag name="system-type">general-purpose</tag> <tag name="operating-system">CISCO IOS</tag> <tag name="traceroute-hop-7">?</tag> <tag name="traceroute-hop-6">192.168.1.1</tag> <tag name="traceroute-hop-5">192.168.1.2</tag> <tag name="traceroute-hop-4">192.168.1.3</tag> <tag name="HOST_END">Sun Mar 26 00:22:30 2017</tag> <tag name="host-ip">192.168.1.1</tag> <tag name="HOST_START">Sun Mar 26 00:20:19 2017</tag> </HostProperties> <ReportItem port="123" svc_name="ntp" protocol="udp" severity="0" pluginID="10884" pluginName="Network Time Protocol (NTP) Server Detection" pluginFamily="Service detection"> <cpe>cpe:/a:ntp:ntp</cpe> <description>An NTP server with an insecure configuration is listening on port 123. It provides information about its version, current date, current time, and possibly system information.</description> <fname>ntp_open.nasl</fname> <plugin_modification_date>2017/03/21</plugin_modification_date> <plugin_name>Network Time Protocol (NTP) Server Detection</plugin_name> <plugin_publication_date>2015/03/20</plugin_publication_date> <plugin_type>remote</plugin_type> <risk_factor>None</risk_factor> <script_version>$Revision: 1.27 $</script_version> <see_also>http://www.ntp.org</see_also> <solution>n/a</solution> <synopsis>An NTP server with an insecure configuration is listening on the remote host.</synopsis> <plugin_output>   Version : unknown </plugin_output> </ReportItem> Thanks !

I am having no luck with a simple parsing of EVT log files. Is there an easy way to read in EVT (Binary Log files) and output them in Syslog Format? This is the config file I am using: (I Used python evtx to extract into text XML) However that yields XML attributes which apparently are not parse-able. Problem Set: Give 3 files (System.evt, Application.evt, and Security.EVT) parse the EVT format into Syslog_BSD(or IETF) formats.     <Extension multiline>     Module    xm_multiline     HeaderLine    /^<event>/     EndLine    /^</event>/ </Extension> <Extension syslog> Module    xm_syslog </Extension> <Extension xmlparser> Module    xm_xml </Extension> <Extension json> Module    xm_json </Extension> <Extension fileop> Module xm_fileop </Extension> <Input in>     Module im_file     File "%ROOT%/test.xml" #    File "/tmp/cab.xml"     SavePos    FALSE     ReadFromLast FALSE     InputType    multiline     <Exec>       # Discard everything that doesn't seem to be an xml event          if $raw_event !~ /^<event>/ drop();       # Parse the xml event       parse_xml(); to_syslog_ietf();       # Rewrite some fields        $EventTime = parsedate($timestamp);       delete($timestamp);       delete($EventReceivedTime);       # Convert to JSON       to_json();     </Exec> </Input> <Output out>     Module  om_file     File    "%ROOT%/out.log"     Exec    parse_xml();     Exec     log_info("FIELD" +  to_json()); </Output> <Route 1>     Path    in => out </Route>

I've seen some posts from about a year ago that NXLog is unable to parse attributes using xm_xml, I just wanted to check if this is still true? I am running NXLog as a service on Windows machines and want to be able to parse the following message, is it possible? <log4j:event logger="com.sentry.test.LogContextListener" timestamp="1437661699866" level="TRACE" thread="localhost-startStop-1"> <log4j:message><![CDATA[This is a trace message about how we should use C#]]></log4j:message> </log4j:event>

Hi, I'm trying to use the xm_multiline module with nxlog to forward content of a logfile to logstash The log contains different xml elements which are properly indented (opening and closing elements are located at the start of the line) . E.g. <data version="x" xmlns:bla="http://www.example.com/bla"> <val:InfoSet> ... ... ... </val:InfoSet> </data> <message ...> <ns>bla</ns> ... ... </message> Because the elements have different names, I can only use < and </ to find the start and end line. I was hoping a filter like this should be enough to select the correct lines: HeaderLine /^</ EndLine /^<// But somehow nxlog gets confused with the / in the regex pattern. I also tried escaping which dindn't help. More testing showed that it needs at least one letter. I tried to specify all letters via regex but that didn't work: HeaderLine /^<[a-z]/ Only way that seems to work is to specify all letters in the square braket (with the exception of the lettern, which breaks).  HeaderLine /^<[abcdefghijklmopqrstuvwxyz]/ (left out n) Here all my test results. These lines worked: HeaderLine /^<m/ EndLine /^</m/ HeaderLine /^<m/ EndLine /^<\/m/ HeaderLine /^<[abcdefghijklm]/ EndLine /^<\/[abcdefghijklm]/ HeaderLine /^<[abcdefghijklmo]/ EndLine /^<\/[abcdefghijklmo]/ HeaderLine /^<[abcdefghijklmopqrstuvwxyz]/ (left out n) EndLine /^<\/[abcdefghijklmopqrstuvwxyz]/ HeaderLine /^<[abcdefghijklmopqrstuvwxyz]/ (left out n + not escaped EndLine /^</[abcdefghijklmopqrstuvwxyz]/ These lines didn't work: HeaderLine /^</ EndLine /^</m/ HeaderLine /^<[a-z]/ EndLine /^</m/ HeaderLine /^<\w/ EndLine /^</m/ HeaderLine /^<[abcdefghijklmn]/ EndLine /^<\/[abcdefghijklmn]/ HeaderLine /^<[bcdefghijklmn]/ EndLine /^<\/[bcdefghijklmn]/ HeaderLine /^<[abcdefghijklmopqrstuvwxyzn]/ EndLine /^<\/[abcdefghijklmopqrstuvwxyzn]/ HeaderLine /^<[abcdefghijklmnopqrstuvwxyz]/ EndLine /^</[abcdefghijklmnopqrstuvwxyz]/ Right now I still have a problem because many of my bessages start with <n. I think this is a bug in the module. Can you confirm so I can open a ticket? Thanks   Fyi, this is a duplicate of http://stackoverflow.com/questions/27429234/which-headerline-and-endline-for-multiline-xml-with-different-elements