Hi All, 

I am looking to use nxlog to transform a CSV formatted input from an SMB share into a json formatted line-by-line output for parsing by further handlers of our logging information. The CSV in question is an export of Windows Event Logs from a domain controller. An example of the CSV I am trying to parse is:

"Index","TimeGenerated","InstanceId","EntryType","UserName","MachineName","Category","Field1","Field2","Field3","Field4","Field5","Field6","Field7","Field8","Field9","Field10","Field11","Field12","Field13","Field14","Field15","Field16","Field17"
"3297643","20170914-00:00:01","4768","SuccessAudit",,"DC1.Office.ExampleDomain.com","(11111)","Test","Office.ExampleDomain.com","S-1-5-21-910167743-1234567890-1234567890-1388","krbtgt","S-1-5-21-910167743-1234567890-1234567890-502","0x40800000","0x0","0x17","2","10.0.0.1","55393","","","",,,
"3297644","20170914-00:00:31","4768","SuccessAudit",,"DC1.Office.ExampleDomain.com","(11111)","Test","Office.ExampleDomain.com","S-1-5-21-910167743-1234567890-1234567890-1388","krbtgt","S-1-5-21-910167743-1234567890-1234567890-502","0x40800000","0x0","0x17","2","10.0.0.1","45086","","","",,,
"3297645","20170914-00:01:01","4768","SuccessAudit",,"DC1.Office.ExampleDomain.com","(11111)","Test","Office.ExampleDomain.com","S-1-5-21-910167743-1234567890-1234567890-1388","krbtgt","S-1-5-21-910167743-1234567890-1234567890-502","0x40800000","0x0","0x17","2","10.0.0.1","35822","","","",,,
"3297646","20170914-00:01:31","4768","SuccessAudit",,"DC1.Office.ExampleDomain.com","(11111)","Test","Office.ExampleDomain.com","S-1-5-21-910167743-1234567890-1234567890-1388","krbtgt","S-1-5-21-910167743-1234567890-1234567890-502","0x40800000","0x0","0x17","2","10.0.0.1","44883","","","",,,
"3297647","20170914-00:02:01","4768","SuccessAudit",,"DC1.Office.ExampleDomain.com","(11111)","Test","Office.ExampleDomain.com","S-1-5-21-910167743-1234567890-1234567890-1388","krbtgt","S-1-5-21-910167743-1234567890-1234567890-502","0x40800000","0x0","0x17","2","10.0.0.1","48917","","","",,,
"3297648","20170914-00:02:31","4768","SuccessAudit",,"DC1.Office.ExampleDomain.com","(11111)","Test","Office.ExampleDomain.com","S-1-5-21-910167743-1234567890-1234567890-1388","krbtgt","S-1-5-21-910167743-1234567890-1234567890-502","0x40800000","0x0","0x17","2","10.0.0.1","58464","","","",,,
"3297649","20170914-00:03:01","4768","SuccessAudit",,"DC1.Office.ExampleDomain.com","(11111)","Test","Office.ExampleDomain.com","S-1-5-21-910167743-1234567890-1234567890-1388","krbtgt","S-1-5-21-910167743-1234567890-1234567890-502","0x40800000","0x0","0x17","2","10.0.0.1","51655","","","",,,
"3297651","20170914-00:03:23","4732","SuccessAudit",,"DC1.Office.ExampleDomain.com","(11111)","CN=DC1,OU=Users,OU=___,DC=DC1,DC=Office,DC=ExampleDomain,DC=com","S-1-5-21-2131238190-1946908106-23540016-118539","TestUser","DC1","S-1-5-21-910167743-1234567890-1234567890-14815","S-1-5-21-910167743-1234567890-1234567890-1318","$SECURITY_ACCOUNT","DC1","0x134b47790","-",,,,,,,
"3297650","20170914-00:03:23","4735","SuccessAudit",,"DC1.Office.ExampleDomain.com","(11111)","TestUser"","DC1","S-1-5-21-910167743-1234567890-1234567890-14815","S-1-5-21-910167743-1234567890-1234567890-1318","$SECURITY_ACCOUNT","DC1","0x134b47790","-","-","-",,,,,,,

To keep things simple, I have left out the SMB part of the requirements and set up a configuration as follows:

User root
Group root
define ROOT /opt/nxsec/
#NoFreeOnExit TRUE
define CERTDIR /opt/nxsec/var/lib/nxlog/cert
define CONFDIR /opt/nxsec/var/lib/nxlog
define LOGDIR /opt/nxsec/var/log/nxlog
define LOGFILE "%LOGDIR%/nxlog.log"

SpoolDir /opt/nxsec/var/spool/nxlog
PidFile /opt/nxsec/var/run/nxlog/nxlog.pid
CacheDir /opt/nxsec/var/spool/nxlog
ModuleDir /opt/nxsec/lib/nxlog/modules

<Extension json>
    Module      xm_json
</Extension>

<Extension csv>
    Module      xm_csv
    Fields      $Index, $TimeGenerated, $InstanceId, $EntryType, $UserName, $MachineName, $Category, $Field1, $Field2, $Field3, $Field4, $Field5, $Field6, $Field7, $Field8, $Field9, $Field10, $Field11, $Field12, $Field13, $Field14, $Field15, $Field16, $Field17
    EscapeControl FALSE
</Extension>
<Input in>
    Module im_file
    File "/test/test.csv"
    InputType     LineBased        
    PollInterval  1
    Exec csv->parse_csv();
    Exec $Message = to_json();
</Input>

<Output out>
    Module  om_file
    File    "/test/output.json"
    Sync    TRUE
</Output>

<Route 1>
    Path  in => out
</Route>

NXLog -v validates this file correctly and when run, nxlog does not indicate any errors or log any errors. Using strace, I can see that it even reads the source file, however, it is not writing to the output file.

I have tried various permutations of this configuration, including moving the Exec $Message = json->to_json(); line to the output module, but no matter what I do I cannot seem to get the CSV parsed and written back out again. No crashes happen and no log messages appear from nxlog, however.

Is there something I am doing wrong? Does anyone have a self-contained, complete working example to parse a Windows Event Log CSV export?