I'm trying to ws_ftp logs that are xml formatted as such : xxxx data data xxxx 0 0 ip ip xxx ... with this configuration : define ROOT C:\Program Files\nxlog Module xm_gelf Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data LogFile %ROOT%\data\nxlog.log Module xm_multiline HeaderLine /^/ EndLine /^/ Module xm_xml Module xm_json Module im_file File "PATH TO .XML" SavePos FALSE PollInterval 1 ReadFromLast FALSE InputType multiline $EventData = extract_xml("/log/entry"); if $EventData == "" { delete($EventData); } to_json(); Module om_file File "C:\\temp\\output_test.txt" Path in => out i can't get past this error : > ERROR Couldn't parse Exec block at C:\Program Files\nxlog\conf\nxlog.conf:34; couldn't parse statement at line 35, character 40 in C:\Program Files\nxlog\conf\nxlog.conf; function 'extract_xml()' does not exist or takes different arguments > ERROR module 'in' has configuration errors, not adding to route '1' at C:\Program Files\nxlog\conf\nxlog.conf:51 how can i get further?

I have an XML that I am trying to strip some data out so it can be pre-processed by software on another machine. XML file is being generated on a windows logging to an XML file. I am at bit of a lost I have tried too many things to list here. What I am trying to do is remove our domain name and our domain email address before it is sent to the machine to be pre-processed. I only want the username. Any records that have host\domain.com don’t need to be sent and I figured out how to drop that data. (number 4). If it helps I am running nxlog-ce-3.0.2284.
Here is an example of event: (Removed bunch of xml fields for clarity of this post) 1: <Event><Timestamp data_type="4">05/25/2022 12:45:43.806</Timestamp><Userid data_type="1">DOMAIN\username</Userid><IP-Address data_type="3">x.x.x.x</IP-Address><Endtimestamp data_type="5">05/25/2022 12:46:43.806</Endtimestamp> 2: <Event><Timestamp data_type="4">05/25/2022 12:45:43.806</Timestamp><Userid data_type="1">username</Userid><IP-Address data_type="3">x.x.x.x</IP-Address><Endtimestamp data_type="5">05/25/2022 12:46:43.806</Endtimestamp> 3: <Event><Timestamp data_type="4">05/25/2022 12:45:43.806</Timestamp><Userid data_type="1">username@domain.com</Userid><IP-Address data_type="3">x.x.x.x</IP-Address><Endtimestamp data_type="5">05/25/2022 12:46:43.806</Endtimestamp> 4: <Event><Timestamp data_type="4">05/25/2022 12:45:43.806</Timestamp><Userid data_type="1">host\domain.com</Userid><IP-Address data_type="3">x.x.x.x</IP-Address><Endtimestamp data_type="5">05/25/2022 12:46:43.806</Endtimestamp>

Nxlog.conf: #NoFreeOnExit TRUE

define ROOT C:\Program Files\nxlog define CERTDIR %ROOT%\cert define CONFDIR %ROOT%\conf define LOGDIR %ROOT%\data

define LOGFILE %LOGDIR%\nxlog.log LogFile %LOGFILE%

Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data

<Extension xmlparser> Module xm_xml </Extension>

<Extension json> Module xm_json </Extension>

<Input in> Module im_file File "C:\LogFiles\log*.log" InputType LineBased Exec $Message = $raw_event; SavePos TRUE
ReadFromLast TRUE <Exec>

Discard everything that doesn't seem to be an xml event

if $raw_event !~ /^<Event>/ drop(); if $raw_event =~ /^(.+)host(.+)/ drop();

parse_xml();

Convert to JSON

    to_json();

</Exec> </Input>

<Output out> Module om_udp Host yy.xx.xx.xx Port 514 </Output>

<Route 1> Path in => out </Route>

Hello, We are using NXLog extensively and just recently started seeing some parsing issues, so far specifically on Windows 2012r2 using Windows Event Forwarding, but could be others. It appears to be something with processing self-closed tags at first glance, but I've done a little bit of testing myself and couldn't directly reproduce the problem (so far), so figured I'd come here for guidance. Specific details are included below. **Current Behavior** NXLog appears to be improperly parsing empty, self-closed XML tags. **Expected Behavior** NXLog properly ignores empty, self-closed XML tags. **NXLog Version**: nxlog-ce-2.10.2150 **NXLog Configuration File**: ``` ## NXLog configuration file define ROOT C:\Program Files (x86)\nxlog define LOGFILE %ROOT%\data\nxlog.log Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data LogFile %LOGFILE% # Rotate agent logs on the local system such that only the last 4 files are kept Module xm_fileop # Check the size of our log file every hour and rotate if it is larger than 1M Every 1 hour Exec if (file_size('%LOGFILE%') >= 1M) file_cycle('%LOGFILE%', 4); # Rotate our log file every week on sunday at midnight When @weekly Exec file_cycle('%LOGFILE%', 4); Module xm_syslog Module xm_json # Agent logs Module im_internal # OS logs Module im_msvistalog # Drop EventID 5156 logs when application name is nxlog.exe Exec if ($SourceName == 'Microsoft-Windows-Security-Auditing')\ AND ($EventID == 5156)\ AND ($Application =~ /nxlog.exe$/)\ drop(); Module im_msvistalog Query \ \ *\ \ # SIEM port 3514 is listening for JSON-encoded IETF style syslog messages # OutputType Syslog_TLS required to enable the octet-framing described in RFC5425 Module om_tcp Host XXX.XXX.XXX.XXX Port 3514 OutputType Syslog_TLS Exec $Message = to_json(); # Remove param-value pairs from structured data header with names > 32 characters. Exec if ($SourceName == 'Microsoft-Windows-GroupPolicy') {\ if ($EventID == 5017) OR ($EventID == 6017) OR ($EventID == 7017)\ delete($OperationElaspedTimeInMilliSeconds);\ if ($EventID == 5116) OR ($EventID == 6116) OR ($EventID == 7116)\ delete($GpsvcInitTimeElapsedInMilliseconds);\ if ($EventID == 5126) OR ($EventID == 6126) OR ($EventID == 7126)\ delete($GPODownloadTimeElapsedInMilliseconds);\ if ($EventID == 5257) OR ($EventID == 6257) OR ($EventID == 7257)\ delete($PolicyDownloadTimeElapsedInMilliseconds);\ if ($EventID == 5351) OR ($EventID == 6351) OR ($EventID == 7351)\ delete($WinlogonReturnTimeElapsedInMilliseconds);\ } Exec to_syslog_ietf(); Path internal, eventlog, forwardedEvents => out ``` **Windows Version**: Windows 2012r2 (potentially others, but confirmed for sure on this one) **Example Exported XML from EventViewer** ``` 4688201331200x80200000000000005238702734Securityhostname.domainS-1-5-18workstation$DOMAIN0x3e70x1754C:\Windows\System32\wbem\WmiApSrv.exe%%19360x2f8S-1-0-0--0x0A new process has been created...InformationProcess CreationInfoSecurityMicrosoft Windows security auditing.Audit Success ``` **Example Data as Received on the wire**: ``` 1 2019-10-21T15:44:36.650065-04:00 hostname.domain Microsoft-Windows-Security-Auditing 4 - [NXLOG@14506 Keywords="-9214364837600034816" EventType="AUDIT_SUCCESS" EventID="4688" ProviderGuid="{54849625-5478-4994-A5BA-3E3B0328C30D}" Version="2" Task="13312" OpcodeValue="0" RecordNumber="355111132" ThreadID="5020" Channel="Security" Category="Process Creation" Opcode="Info" SubjectUserSid="S-1-5-18" SubjectUserName="workstation$" SubjectDomainName="DOMAIN" SubjectLogonId="0x3e7" NewProcessId="0x13a8" NewProcessName="C:\\Windows\\System32\\wbem\\WmiPrvSE.exe" TokenElevationType="%%1936" CommandLine\'/>

Hi guys,

I wanted to use NXLog to send data from a XML file into GrayLog.

My first test worked fine with a simple XML file. But with a Nessus Report (in XML) it don't work, because this file is more complexe. Do you have an example configuration nxlog.conf to help me ?

My actual nxlog.conf :

define ROOT C:\Program Files (x86)\nxlog

Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log

<Extension gelf>
Module xm_gelf
</Extension>

<Extension multiline>

    Module	xm_multiline
    HeaderLine	/^<event>/
    EndLine	/^</event>/
</Extension>

<Extension xmlparser>
    Module	xm_xml
</Extension>

<Extension json>
    Module	xm_json
</Extension>

<Input in>
    Module	im_file
    File	"C:\Program Files\Graylog\collector-sidecar\generated\exempleFichier.xml"
    SavePos	FALSE
    ReadFromLast FALSE
    InputType	multiline
    <Exec>
      # Discard everything that doesn't seem to be an xml event   
      if $raw_event !~ /^<event>/ drop();

      # Parse the xml event
      parse_xml();

      # Rewrite some fields 
      $EventTime = parsedate($timestamp);
      delete($timestamp);
      delete($EventReceivedTime);

      # Convert to JSON
      to_json();
    </Exec>
</Input>

<Output out>
    Module      om_udp
    Host        xx.xx.xx.xx
    Port        12201
</Output>

<Route 1>
    Path        in => out
</Route>

And a (verry little) preview of my Nessus File...

<Report name="Scan_Nessus" xmlns:cm="http://www.nessus.org/cm">
<ReportHost name="192.168.1.1"><HostProperties>
<tag name="traceroute-hop-8">192.168.1.1</tag>
<tag name="LastUnauthenticatedResults">1490484150</tag>
<tag name="Credentialed_Scan">false</tag>
<tag name="policy-used">policies_scan</tag>
<tag name="patch-summary-total-cves">1</tag>
<tag name="os">other</tag>
<tag name="system-type">general-purpose</tag>
<tag name="operating-system">CISCO IOS</tag>
<tag name="traceroute-hop-7">?</tag>
<tag name="traceroute-hop-6">192.168.1.1</tag>
<tag name="traceroute-hop-5">192.168.1.2</tag>
<tag name="traceroute-hop-4">192.168.1.3</tag>
<tag name="HOST_END">Sun Mar 26 00:22:30 2017</tag>
<tag name="host-ip">192.168.1.1</tag>
<tag name="HOST_START">Sun Mar 26 00:20:19 2017</tag>
</HostProperties>
<ReportItem port="123" svc_name="ntp" protocol="udp" severity="0" pluginID="10884" pluginName="Network Time Protocol (NTP) Server Detection" pluginFamily="Service detection">
<cpe>cpe:/a:ntp:ntp</cpe>
<description>An NTP server with an insecure configuration is listening on port 123.
It provides information about its version, current date, current time, and possibly system information.</description>
<fname>ntp_open.nasl</fname>
<plugin_modification_date>2017/03/21</plugin_modification_date>
<plugin_name>Network Time Protocol (NTP) Server Detection</plugin_name>
<plugin_publication_date>2015/03/20</plugin_publication_date>
<plugin_type>remote</plugin_type>
<risk_factor>None</risk_factor>
<script_version>$Revision: 1.27 $</script_version>
<see_also>http://www.ntp.org</see_also>
<solution>n/a</solution>
<synopsis>An NTP server with an insecure configuration is listening on the remote host.</synopsis>
<plugin_output>
  Version : unknown
</plugin_output>
</ReportItem>

Thanks !

I am having no luck with a simple parsing of EVT log files.

Is there an easy way to read in EVT (Binary Log files) and output them in Syslog Format?

This is the config file I am using: (I Used python evtx to extract into text XML) However that yields XML attributes which apparently are not parse-able.

Problem Set:

Give 3 files (System.evt, Application.evt, and Security.EVT) parse the EVT format into Syslog_BSD(or IETF) formats.

 

 

<Extension multiline>
    Module    xm_multiline
    HeaderLine    /^<event>/
    EndLine    /^</event>/
</Extension>

<Extension syslog>
Module    xm_syslog
</Extension>

<Extension xmlparser>
Module    xm_xml
</Extension>

<Extension json>
Module    xm_json
</Extension>

<Extension fileop>
Module xm_fileop
</Extension>

<Input in>
    Module im_file
    File "%ROOT%/test.xml"
#    File "/tmp/cab.xml"
    SavePos    FALSE
    ReadFromLast FALSE
    InputType    multiline
    <Exec>
      # Discard everything that doesn't seem to be an xml event   
      if $raw_event !~ /^<event>/ drop();

      # Parse the xml event
      parse_xml(); to_syslog_ietf();

      # Rewrite some fields 
      $EventTime = parsedate($timestamp);
      delete($timestamp);
      delete($EventReceivedTime);

      # Convert to JSON
      to_json();
    </Exec>
</Input>

<Output out>
    Module  om_file
    File    "%ROOT%/out.log"
    Exec    parse_xml();
    Exec     log_info("FIELD" +  to_json());
</Output>

<Route 1>
    Path    in => out
</Route>

I've seen some posts from about a year ago that NXLog is unable to parse attributes using xm_xml, I just wanted to check if this is still true?

I am running NXLog as a service on Windows machines and want to be able to parse the following message, is it possible?

<log4j:event logger="com.sentry.test.LogContextListener" timestamp="1437661699866" level="TRACE" thread="localhost-startStop-1"> <log4j:message><![CDATA[This is a trace message about how we should use C#]]></log4j:message> </log4j:event>

Hi,

I'm trying to use the xm_multiline module with nxlog to forward content of a logfile to logstash The log contains different xml elements which are properly indented (opening and closing elements are located at the start of the line) . E.g.

<data
    version="x"
    xmlns:bla="http://www.example.com/bla">
    <val:InfoSet>
      ...
        ...
          ...
    </val:InfoSet>
</data>

<message  ...>
    <ns>bla</ns>
    ...
        ...
</message>

Because the elements have different names, I can only use < and </ to find the start and end line. I was hoping a filter like this should be enough to select the correct lines:

HeaderLine  /^</
EndLine     /^<//

But somehow nxlog gets confused with the / in the regex pattern. I also tried escaping which dindn't help. More testing showed that it needs at least one letter. I tried to specify all letters via regex but that didn't work:

HeaderLine  /^<[a-z]/

Only way that seems to work is to specify all letters in the square braket (with the exception of the lettern, which breaks). 

HeaderLine  /^<[abcdefghijklmopqrstuvwxyz]/    (left out n)

Here all my test results.

These lines worked:

HeaderLine  /^<m/
EndLine     /^</m/

HeaderLine  /^<m/
EndLine     /^<\/m/

HeaderLine  /^<[abcdefghijklm]/
EndLine     /^<\/[abcdefghijklm]/

HeaderLine  /^<[abcdefghijklmo]/
EndLine     /^<\/[abcdefghijklmo]/

HeaderLine  /^<[abcdefghijklmopqrstuvwxyz]/    (left out n)
EndLine     /^<\/[abcdefghijklmopqrstuvwxyz]/

HeaderLine  /^<[abcdefghijklmopqrstuvwxyz]/ (left out n + not escaped
EndLine     /^</[abcdefghijklmopqrstuvwxyz]/

These lines didn't work:

HeaderLine  /^</
EndLine     /^</m/

HeaderLine  /^<[a-z]/
EndLine     /^</m/

HeaderLine  /^<\w/
EndLine     /^</m/

HeaderLine  /^<[abcdefghijklmn]/
EndLine     /^<\/[abcdefghijklmn]/

HeaderLine  /^<[bcdefghijklmn]/
EndLine     /^<\/[bcdefghijklmn]/

HeaderLine  /^<[abcdefghijklmopqrstuvwxyzn]/
EndLine     /^<\/[abcdefghijklmopqrstuvwxyzn]/

HeaderLine  /^<[abcdefghijklmnopqrstuvwxyz]/
EndLine     /^</[abcdefghijklmnopqrstuvwxyz]/

Right now I still have a problem because many of my bessages start with <n. I think this is a bug in the module. Can you confirm so I can open a ticket? Thanks
 

Fyi, this is a duplicate of http://stackoverflow.com/questions/27429234/which-headerline-and-endline-for-multiline-xml-with-different-elements