Ask questions. Get answers. Find technical product solutions from passionate experts in the NXLog community.

Parsing Windows Event Log CSV with NXLog (into JSON)
Hi All,  I am looking to use nxlog to transform a CSV formatted input from an SMB share into a json formatted line-by-line output for parsing by further handlers of our logging information. The CSV in question is an export of Windows Event Logs from a domain controller. An example of the CSV I am trying to parse is: "Index","TimeGenerated","InstanceId","EntryType","UserName","MachineName","Category","Field1","Field2","Field3","Field4","Field5","Field6","Field7","Field8","Field9","Field10","Field11","Field12","Field13","Field14","Field15","Field16","Field17" "3297643","20170914-00:00:01","4768","SuccessAudit",,"DC1.Office.ExampleDomain.com","(11111)","Test","Office.ExampleDomain.com","S-1-5-21-910167743-1234567890-1234567890-1388","krbtgt","S-1-5-21-910167743-1234567890-1234567890-502","0x40800000","0x0","0x17","2","10.0.0.1","55393","","","",,, "3297644","20170914-00:00:31","4768","SuccessAudit",,"DC1.Office.ExampleDomain.com","(11111)","Test","Office.ExampleDomain.com","S-1-5-21-910167743-1234567890-1234567890-1388","krbtgt","S-1-5-21-910167743-1234567890-1234567890-502","0x40800000","0x0","0x17","2","10.0.0.1","45086","","","",,, "3297645","20170914-00:01:01","4768","SuccessAudit",,"DC1.Office.ExampleDomain.com","(11111)","Test","Office.ExampleDomain.com","S-1-5-21-910167743-1234567890-1234567890-1388","krbtgt","S-1-5-21-910167743-1234567890-1234567890-502","0x40800000","0x0","0x17","2","10.0.0.1","35822","","","",,, "3297646","20170914-00:01:31","4768","SuccessAudit",,"DC1.Office.ExampleDomain.com","(11111)","Test","Office.ExampleDomain.com","S-1-5-21-910167743-1234567890-1234567890-1388","krbtgt","S-1-5-21-910167743-1234567890-1234567890-502","0x40800000","0x0","0x17","2","10.0.0.1","44883","","","",,, "3297647","20170914-00:02:01","4768","SuccessAudit",,"DC1.Office.ExampleDomain.com","(11111)","Test","Office.ExampleDomain.com","S-1-5-21-910167743-1234567890-1234567890-1388","krbtgt","S-1-5-21-910167743-1234567890-1234567890-502","0x40800000","0x0","0x17","2","10.0.0.1","48917","","","",,, "3297648","20170914-00:02:31","4768","SuccessAudit",,"DC1.Office.ExampleDomain.com","(11111)","Test","Office.ExampleDomain.com","S-1-5-21-910167743-1234567890-1234567890-1388","krbtgt","S-1-5-21-910167743-1234567890-1234567890-502","0x40800000","0x0","0x17","2","10.0.0.1","58464","","","",,, "3297649","20170914-00:03:01","4768","SuccessAudit",,"DC1.Office.ExampleDomain.com","(11111)","Test","Office.ExampleDomain.com","S-1-5-21-910167743-1234567890-1234567890-1388","krbtgt","S-1-5-21-910167743-1234567890-1234567890-502","0x40800000","0x0","0x17","2","10.0.0.1","51655","","","",,, "3297651","20170914-00:03:23","4732","SuccessAudit",,"DC1.Office.ExampleDomain.com","(11111)","CN=DC1,OU=Users,OU=___,DC=DC1,DC=Office,DC=ExampleDomain,DC=com","S-1-5-21-2131238190-1946908106-23540016-118539","TestUser","DC1","S-1-5-21-910167743-1234567890-1234567890-14815","S-1-5-21-910167743-1234567890-1234567890-1318","$SECURITY_ACCOUNT","DC1","0x134b47790","-",,,,,,, "3297650","20170914-00:03:23","4735","SuccessAudit",,"DC1.Office.ExampleDomain.com","(11111)","TestUser"","DC1","S-1-5-21-910167743-1234567890-1234567890-14815","S-1-5-21-910167743-1234567890-1234567890-1318","$SECURITY_ACCOUNT","DC1","0x134b47790","-","-","-",,,,,,, To keep things simple, I have left out the SMB part of the requirements and set up a configuration as follows: User root Group root define ROOT /opt/nxsec/ #NoFreeOnExit TRUE define CERTDIR /opt/nxsec/var/lib/nxlog/cert define CONFDIR /opt/nxsec/var/lib/nxlog define LOGDIR /opt/nxsec/var/log/nxlog define LOGFILE "%LOGDIR%/nxlog.log" SpoolDir /opt/nxsec/var/spool/nxlog PidFile /opt/nxsec/var/run/nxlog/nxlog.pid CacheDir /opt/nxsec/var/spool/nxlog ModuleDir /opt/nxsec/lib/nxlog/modules <Extension json> Module xm_json </Extension> <Extension csv> Module xm_csv     Fields      $Index, $TimeGenerated, $InstanceId, $EntryType, $UserName, $MachineName, $Category, $Field1, $Field2, $Field3, $Field4, $Field5, $Field6, $Field7, $Field8, $Field9, $Field10, $Field11, $Field12, $Field13, $Field14, $Field15, $Field16, $Field17 EscapeControl FALSE </Extension> <Input in> Module im_file File "/test/test.csv" InputType LineBased PollInterval 1 Exec csv->parse_csv(); Exec $Message = to_json(); </Input> <Output out> Module om_file File "/test/output.json" Sync TRUE </Output> <Route 1> Path in => out </Route> NXLog -v validates this file correctly and when run, nxlog does not indicate any errors or log any errors. Using strace, I can see that it even reads the source file, however, it is not writing to the output file. I have tried various permutations of this configuration, including moving the Exec $Message = json->to_json(); line to the output module, but no matter what I do I cannot seem to get the CSV parsed and written back out again. No crashes happen and no log messages appear from nxlog, however. Is there something I am doing wrong? Does anyone have a self-contained, complete working example to parse a Windows Event Log CSV export?

avhk created
Replies: 2
View post »
last updated