1
answer

Hi, I am trying to read logs (csv format) from Service Now and send it to ELK stack. I need some help in writing the input module, so that I can properly send the logs to ELK stack.
My input file contains 5 fields, but field3 has multiline input. I tried many methods and it doesnot work as per expectations. Can someone please help in writing proper input/output module to my stack.

Input file sample as follows:

Created,Level,Message,Source,Created by
7/22/2019 3:00,Warning,"org.mozilla.javascript.EcmaError: Cannot convert null to an object.
Caused by error in sys_script.914d69890a0a3c1101310dab6c2ebf01.script at line 1

==> 1: geamBlockCI();
2: function geamBlockCI() {
3: var user = gs.getUser();
4: //gs.log('**** 1 User'+ user,'Test');
",Evaluator,admin
7/22/2019 3:00,Warning,"org.mozilla.javascript.EcmaError: Cannot convert null to an object.
Caused by error in sys_script.914d69890a0a3c1101310dab6c2ebf01.script at line 1

==> 1: geamBlockCI();
2: function geamBlockCI() {
3: var user = gs.getUser();
4: //gs.log('**** 1 User'+ user,'Test');
",Evaluator,admin

AskedAugust 15, 2019 - 9:50pm

Comments (2)

  • abasha's picture

    Thank you. When I am trying to execute it throws me following error:
    2019-08-15 17:22:53 ERROR Couldn't parse Exec block at C:\Program Files (x86)\nxlog\conf\nxlog.conf:33; couldn't parse statement at line 40, character 20 in C:\Program Files (x86)\nxlog\conf\nxlog.conf; procedure 'parse_csv()' does not exist or takes different arguments
    2019-08-15 17:22:53 ERROR module 'filein' has configuration errors, not adding to route 'parse_xml' at C:\Program Files (x86)\nxlog\conf\nxlog.conf:55
    2019-08-15 17:22:53 ERROR route parse_xml is not functional without input modules, ignored at C:\Program Files (x86)\nxlog\conf\nxlog.conf:55
    2019-08-15 17:22:53 WARNING no routes defined!
    2019-08-15 17:22:53 WARNING not starting unused module filein
    2019-08-15 17:22:53 WARNING not starting unused module fileout
    2019-08-15 17:22:53 INFO nxlog-ce-2.9.1716 started

    Here is my complete config file:
    ## Please set the ROOT to the folder your nxlog was installed into,
    ## otherwise it will not start.

    #define ROOT C:\Program Files\nxlog
    define ROOT C:\Program Files (x86)\nxlog

    Moduledir %ROOT%\modules
    CacheDir %ROOT%\data
    Pidfile %ROOT%\data\nxlog.pid
    SpoolDir %ROOT%\data
    LogFile %ROOT%\data\nxlog.log

    <Extension multiline>
    Module xm_multiline
    HeaderLine /^\d{1,2}\/\d{1,2}\/\d{4}\s/
    </Extension>

    <Extension json>
    Module xm_json
    </Extension>

    <Extension csv>
    Module xm_csv
    Fields $Created,$Level,$Message,$Source,$CreatedBy
    </Extension>

    <Input filein>
    Module im_file
    File 'e:\ServiceNow\iCentral_MS-DSC-PROD1\agent\export\snow_test.csv'
    InputType multiline
    ReadFromLast TRUE
    SavePos TRUE
    <Exec>
    # Ignore top line
    if $raw_event =~ /Created,Level,Message,Source,Created by/ drop();

    # Convert Newline and Tab to printed character
    $raw_event =~ s/\R/\\r\\n/g;
    $raw_event =~ s/\t/\\t/g;

    # Parse $raw_event as CSV
    parse_csv();

    # Convert to JSON
    to_json();
    </Exec>
    </Input>

    <Output fileout>
    Module om_file
    File 'c:\tmp\test.log'
    </Output>

    <Route parse_xml>
    Path filein => fileout
    </Route>

    August 15, 2019 - 11:24pm
  • abasha's picture

    Hello I was able to fix it,

    # Parse $raw_event as CSV
    csv->parse_csv();

    Thanks.

    August 16, 2019 - 12:00am

Answer (1)

You probably want to use the xm_multiline module.
Something like the following.

<Extension multiline>
    Module          xm_multiline
    # Detect date ##/##/####
    HeaderLine      /^\d{1,2}\/\d{1,2}\/\d{4}\s/
</Extension>
<Extension json>
    Module          xm_json
</Extension>
<Extension csv>
    Module          xm_csv
    Fields          $Created,$Level,$Message,$Source,CreatedBy
</Extension>

<Input filein>
    Module          im_file
    File            "/opt/nxlog/etc/multi.log"
    InputType       multiline
    ReadFromLast    TRUE
    SavePos         TRUE
    <Exec>
        # Ignore top line
        if $raw_event =~ /Created,Level,Message,Source,Created by/ drop();

        # Convert Newline and Tab to printed character
        $raw_event =~ s/\R/\\r\\n/g;
        $raw_event =~ s/\t/\\t/g;

        # Parse $raw_event as CSV
        parse_csv();

        # Convert to JSON
        to_json();
    </Exec>
</Input>

<Output fileout>
    Module          om_file
    File            '/tmp/out.log'
</Output>

<Route parse_xml>
    Path            filein => fileout
</Route>

Output:

{"EventReceivedTime":"2019-08-14T22:12:21.404463-04:00","SourceModuleName":"filein","SourceModuleType":"im_file","Created":"7/22/2019 3:00","Level":"Warning","Message":"org.mozilla.javascript.EcmaError: Cannot convert null to an object.\\r\\nCaused by error in sys_script.914d69890a0a3c1101310dab6c2ebf01.script at line 1\\r\\n\\r\\n==> 1: geamBlockCI();\\r\\n2: function geamBlockCI() {\\r\\n3: var user = gs.getUser();\\r\\n4: //gs.log('**** 1 User'+ user,'Test');\\r\\n","Source":"Evaluator","CreatedBy":"admin"}
{"EventReceivedTime":"2019-08-14T22:12:21.404601-04:00","SourceModuleName":"filein","SourceModuleType":"im_file","Created":"7/22/2019 3:00","Level":"Warning","Message":"org.mozilla.javascript.EcmaError: Cannot convert null to an object.\\r\\nCaused by error in sys_script.914d69890a0a3c1101310dab6c2ebf01.script at line 1\\r\\n\\r\\n==> 1: geamBlockCI();\\r\\n2: function geamBlockCI() {\\r\\n3: var user = gs.getUser();\\r\\n4: //gs.log('**** 1 User'+ user,'Test');\\r\\n","Source":"Evaluator","CreatedBy":"admin"}
AnsweredAugust 15, 2019 - 10:48pm