Need help in writing input module | Log collection solutions

#1 abasha 4 years ago

Input file sample as follows:

Created,Level,Message,Source,Created by 7/22/2019 3:00,Warning,"org.mozilla.javascript.EcmaError: Cannot convert null to an object. Caused by error in sys_script.914d69890a0a3c1101310dab6c2ebf01.script at line 1

==> 1: geamBlockCI(); 2: function geamBlockCI() { 3: var user = gs.getUser(); 4: //gs.log('**** 1 User'+ user,'Test'); ",Evaluator,admin 7/22/2019 3:00,Warning,"org.mozilla.javascript.EcmaError: Cannot convert null to an object. Caused by error in sys_script.914d69890a0a3c1101310dab6c2ebf01.script at line 1

==> 1: geamBlockCI(); 2: function geamBlockCI() { 3: var user = gs.getUser(); 4: //gs.log('**** 1 User'+ user,'Test'); ",Evaluator,admin

Permalink

#4 Zhengshi Nxlog ✓ 4 years ago

#1 abasha

Hi, I am trying to read logs (csv format) from Service Now and send it to ELK stack. I need some help in writing the input module, so that I can properly send the logs to ELK stack. My input file contains 5 fields, but field3 has multiline input. I tried many methods and it doesnot work as per expectations. Can someone please help in writing proper input/output module to my stack. Input file sample as follows: Created,Level,Message,Source,Created by 7/22/2019 3:00,Warning,"org.mozilla.javascript.EcmaError: Cannot convert null to an object. Caused by error in sys_script.914d69890a0a3c1101310dab6c2ebf01.script at line 1 ==> 1: geamBlockCI(); 2: function geamBlockCI() { 3: var user = gs.getUser(); 4: //gs.log('**** 1 User'+ user,'Test'); ",Evaluator,admin 7/22/2019 3:00,Warning,"org.mozilla.javascript.EcmaError: Cannot convert null to an object. Caused by error in sys_script.914d69890a0a3c1101310dab6c2ebf01.script at line 1 ==> 1: geamBlockCI(); 2: function geamBlockCI() { 3: var user = gs.getUser(); 4: //gs.log('**** 1 User'+ user,'Test'); ",Evaluator,admin

You probably want to use the xm_multiline module.
Something like the following.

<Extension multiline>
    Module          xm_multiline
    # Detect date ##/##/####
    HeaderLine      /^\d{1,2}\/\d{1,2}\/\d{4}\s/
</Extension>
<Extension json>
    Module          xm_json
</Extension>
<Extension csv>
    Module          xm_csv
    Fields          $Created,$Level,$Message,$Source,CreatedBy
</Extension>

<Input filein>
    Module          im_file
    File            "/opt/nxlog/etc/multi.log"
    InputType       multiline
    ReadFromLast    TRUE
    SavePos         TRUE
    <Exec>
        # Ignore top line
        if $raw_event =~ /Created,Level,Message,Source,Created by/ drop();

        # Convert Newline and Tab to printed character
        $raw_event =~ s/\R/\\r\\n/g;
        $raw_event =~ s/\t/\\t/g;

        # Parse $raw_event as CSV
        parse_csv();

        # Convert to JSON
        to_json();
    </Exec>
</Input>

<Output fileout>
    Module          om_file
    File            '/tmp/out.log'
</Output>

<Route parse_xml>
    Path            filein => fileout
</Route>

Output:

{"EventReceivedTime":"2019-08-14T22:12:21.404463-04:00","SourceModuleName":"filein","SourceModuleType":"im_file","Created":"7/22/2019 3:00","Level":"Warning","Message":"org.mozilla.javascript.EcmaError: Cannot convert null to an object.\\r\\nCaused by error in sys_script.914d69890a0a3c1101310dab6c2ebf01.script at line 1\\r\\n\\r\\n==> 1: geamBlockCI();\\r\\n2: function geamBlockCI() {\\r\\n3: var user = gs.getUser();\\r\\n4: //gs.log('**** 1 User'+ user,'Test');\\r\\n","Source":"Evaluator","CreatedBy":"admin"}
{"EventReceivedTime":"2019-08-14T22:12:21.404601-04:00","SourceModuleName":"filein","SourceModuleType":"im_file","Created":"7/22/2019 3:00","Level":"Warning","Message":"org.mozilla.javascript.EcmaError: Cannot convert null to an object.\\r\\nCaused by error in sys_script.914d69890a0a3c1101310dab6c2ebf01.script at line 1\\r\\n\\r\\n==> 1: geamBlockCI();\\r\\n2: function geamBlockCI() {\\r\\n3: var user = gs.getUser();\\r\\n4: //gs.log('**** 1 User'+ user,'Test');\\r\\n","Source":"Evaluator","CreatedBy":"admin"}