11
responses

Hi All, 

I am looking to use nxlog to transform a CSV formatted input from an SMB share into a json formatted line-by-line output for parsing by further handlers of our logging information. The CSV in question is an export of Windows Event Logs from a domain controller. An example of the CSV I am trying to parse is:

"Index","TimeGenerated","InstanceId","EntryType","UserName","MachineName","Category","Field1","Field2","Field3","Field4","Field5","Field6","Field7","Field8","Field9","Field10","Field11","Field12","Field13","Field14","Field15","Field16","Field17"
"3297643","20170914-00:00:01","4768","SuccessAudit",,"DC1.Office.ExampleDomain.com","(11111)","Test","Office.ExampleDomain.com","S-1-5-21-910167743-1234567890-1234567890-1388","krbtgt","S-1-5-21-910167743-1234567890-1234567890-502","0x40800000","0x0","0x17","2","10.0.0.1","55393","","","",,,
"3297644","20170914-00:00:31","4768","SuccessAudit",,"DC1.Office.ExampleDomain.com","(11111)","Test","Office.ExampleDomain.com","S-1-5-21-910167743-1234567890-1234567890-1388","krbtgt","S-1-5-21-910167743-1234567890-1234567890-502","0x40800000","0x0","0x17","2","10.0.0.1","45086","","","",,,
"3297645","20170914-00:01:01","4768","SuccessAudit",,"DC1.Office.ExampleDomain.com","(11111)","Test","Office.ExampleDomain.com","S-1-5-21-910167743-1234567890-1234567890-1388","krbtgt","S-1-5-21-910167743-1234567890-1234567890-502","0x40800000","0x0","0x17","2","10.0.0.1","35822","","","",,,
"3297646","20170914-00:01:31","4768","SuccessAudit",,"DC1.Office.ExampleDomain.com","(11111)","Test","Office.ExampleDomain.com","S-1-5-21-910167743-1234567890-1234567890-1388","krbtgt","S-1-5-21-910167743-1234567890-1234567890-502","0x40800000","0x0","0x17","2","10.0.0.1","44883","","","",,,
"3297647","20170914-00:02:01","4768","SuccessAudit",,"DC1.Office.ExampleDomain.com","(11111)","Test","Office.ExampleDomain.com","S-1-5-21-910167743-1234567890-1234567890-1388","krbtgt","S-1-5-21-910167743-1234567890-1234567890-502","0x40800000","0x0","0x17","2","10.0.0.1","48917","","","",,,
"3297648","20170914-00:02:31","4768","SuccessAudit",,"DC1.Office.ExampleDomain.com","(11111)","Test","Office.ExampleDomain.com","S-1-5-21-910167743-1234567890-1234567890-1388","krbtgt","S-1-5-21-910167743-1234567890-1234567890-502","0x40800000","0x0","0x17","2","10.0.0.1","58464","","","",,,
"3297649","20170914-00:03:01","4768","SuccessAudit",,"DC1.Office.ExampleDomain.com","(11111)","Test","Office.ExampleDomain.com","S-1-5-21-910167743-1234567890-1234567890-1388","krbtgt","S-1-5-21-910167743-1234567890-1234567890-502","0x40800000","0x0","0x17","2","10.0.0.1","51655","","","",,,
"3297651","20170914-00:03:23","4732","SuccessAudit",,"DC1.Office.ExampleDomain.com","(11111)","CN=DC1,OU=Users,OU=___,DC=DC1,DC=Office,DC=ExampleDomain,DC=com","S-1-5-21-2131238190-1946908106-23540016-118539","TestUser","DC1","S-1-5-21-910167743-1234567890-1234567890-14815","S-1-5-21-910167743-1234567890-1234567890-1318","$SECURITY_ACCOUNT","DC1","0x134b47790","-",,,,,,,
"3297650","20170914-00:03:23","4735","SuccessAudit",,"DC1.Office.ExampleDomain.com","(11111)","TestUser"","DC1","S-1-5-21-910167743-1234567890-1234567890-14815","S-1-5-21-910167743-1234567890-1234567890-1318","$SECURITY_ACCOUNT","DC1","0x134b47790","-","-","-",,,,,,,

To keep things simple, I have left out the SMB part of the requirements and set up a configuration as follows:

User root
Group root
define ROOT /opt/nxsec/
#NoFreeOnExit TRUE
define CERTDIR /opt/nxsec/var/lib/nxlog/cert
define CONFDIR /opt/nxsec/var/lib/nxlog
define LOGDIR /opt/nxsec/var/log/nxlog
define LOGFILE "%LOGDIR%/nxlog.log"

SpoolDir /opt/nxsec/var/spool/nxlog
PidFile /opt/nxsec/var/run/nxlog/nxlog.pid
CacheDir /opt/nxsec/var/spool/nxlog
ModuleDir /opt/nxsec/lib/nxlog/modules

<Extension json>
    Module      xm_json
</Extension>

<Extension csv>
    Module      xm_csv
    Fields      $Index, $TimeGenerated, $InstanceId, $EntryType, $UserName, $MachineName, $Category, $Field1, $Field2, $Field3, $Field4, $Field5, $Field6, $Field7, $Field8, $Field9, $Field10, $Field11, $Field12, $Field13, $Field14, $Field15, $Field16, $Field17
    EscapeControl FALSE
</Extension>
<Input in>
    Module im_file
    File "/test/test.csv"
    InputType     LineBased        
    PollInterval  1
    Exec csv->parse_csv();
    Exec $Message = to_json();
</Input>

<Output out>
    Module  om_file
    File    "/test/output.json"
    Sync    TRUE
</Output>

<Route 1>
    Path  in => out
</Route>

NXLog -v validates this file correctly and when run, nxlog does not indicate any errors or log any errors. Using strace, I can see that it even reads the source file, however, it is not writing to the output file.

I have tried various permutations of this configuration, including moving the Exec $Message = json->to_json(); line to the output module, but no matter what I do I cannot seem to get the CSV parsed and written back out again. No crashes happen and no log messages appear from nxlog, however.

Is there something I am doing wrong? Does anyone have a self-contained, complete working example to parse a Windows Event Log CSV export?

AskedSeptember 28, 2017 - 4:34pm

Answers (2)

I guess the reason it is not reading anything is because it has saved the position in configcache.dat. You will want to use "ReadFromLast FALSE" and "SavePos FALSE" so that it restarts reading from the beginning of the file every time you (re)start it. The latter is not needed with nxlog-processor.

Comments (2)

With the above config it does not seem to rewrite $raw_event so you should see the same in the output.

If you want json, use this:

Exec to_json();

or

Exec $raw_event = to_json();

The two are equivalent.

This does not have any effect unless you use $Message later:

Exec $Message = to_json();

 

Comments (7)

  • avhk's picture

    OK, so, I changed my input block to:

    <Input in>
        Module im_file
        File "/test/test.csv"
        SavePos       FALSE
        ReadFromLast  FALSE
        InputType     LineBased        
        PollInterval  1
        Exec csv->parse_csv();
        Exec $raw_event = json->to_json();
    </Input>

    This works. So my next step is that I need to be able to not re-read the file every time nxlog starts, that is, I need nxlog to save its position and only read new events, unless the source file is completely overwritten.

    Can nxlog do this?

     

  • avhk's picture

    Ahh sorry I did not explain myself very well, when I remove these values or set them to true, nothing is written to the output file. Am I missing something else to make this happen? Thanks for your help!

    Edit: In this case, running strace -f on nxlog in foreground mode, I can see nxlog is seeking to the end of the file and not reading it. 

    516   open("/test/test.csv", O_RDONLY|O_CLOEXEC) = 8
    516   lseek(8, 2895, SEEK_SET)          = 2895
    516   fstat(8, {st_mode=S_IFREG|0755, st_size=2895, ...}) = 0

    However I have deleted /opt/nxsec/var/spool/nxlog/* to ensure there is no cache and I am running in a docker container, so it should be a clean instance anyway. I'm not sure why nxlog thinks it has already read the file, because it should not have I don't think.

     

  • b0ti's picture
    (NXLog)

    If you delete configcache.dat then obviously with ReadFromLast set to TRUE it will seek to the end of the file when you start it. Otherwise if there is a saved position it should seek there.

     

  • avhk's picture

    Sorry, I meant SavePos is set to TRUE but ReadFromLast is set to False. I deleted configcache.dat just to be sure the last position was forgotten, so it should start from the beginning.

    Except it doesn't seem to.

  • b0ti's picture
    (NXLog)

    There are several internal unit tests to check that it works well with the various combinations of ReadFromLast and SavePos. If you think it doesn't please create a simple conf (without any CSV parsing and such) and a script that generates the data that can be used to reproduce the issue or provide the instruction/commands needed for this.

  • avhk's picture

    OK, I've tried it again with SavePos TRUE and ReadFromLast FALSE explicitly, with a clean test case and it seems to work. I'll try it in staging and see how we go! Thanks for your help so far :)