Need help in writing input module
Hi, I am trying to read logs (csv format) from Service Now and send it to ELK stack. I need some help in writing the input module, so that I can properly send the logs to ELK stack. My input file contains 5 fields, but field3 has multiline input. I tried many methods and it doesnot work as per expectations. Can someone please help in writing proper input/output module to my stack.
Input file sample as follows:
Created,Level,Message,Source,Created by 7/22/2019 3:00,Warning,"org.mozilla.javascript.EcmaError: Cannot convert null to an object. Caused by error in sys_script.914d69890a0a3c1101310dab6c2ebf01.script at line 1
==> 1: geamBlockCI(); 2: function geamBlockCI() { 3: var user = gs.getUser(); 4: //gs.log('**** 1 User'+ user,'Test'); ",Evaluator,admin 7/22/2019 3:00,Warning,"org.mozilla.javascript.EcmaError: Cannot convert null to an object. Caused by error in sys_script.914d69890a0a3c1101310dab6c2ebf01.script at line 1
==> 1: geamBlockCI(); 2: function geamBlockCI() { 3: var user = gs.getUser(); 4: //gs.log('**** 1 User'+ user,'Test'); ",Evaluator,admin
You probably want to use the xm_multiline module.
Something like the following.
<Extension multiline>
Module xm_multiline
# Detect date ##/##/####
HeaderLine /^\d{1,2}\/\d{1,2}\/\d{4}\s/
</Extension>
<Extension json>
Module xm_json
</Extension>
<Extension csv>
Module xm_csv
Fields $Created,$Level,$Message,$Source,CreatedBy
</Extension>
<Input filein>
Module im_file
File "/opt/nxlog/etc/multi.log"
InputType multiline
ReadFromLast TRUE
SavePos TRUE
<Exec>
# Ignore top line
if $raw_event =~ /Created,Level,Message,Source,Created by/ drop();
# Convert Newline and Tab to printed character
$raw_event =~ s/\R/\\r\\n/g;
$raw_event =~ s/\t/\\t/g;
# Parse $raw_event as CSV
parse_csv();
# Convert to JSON
to_json();
</Exec>
</Input>
<Output fileout>
Module om_file
File '/tmp/out.log'
</Output>
<Route parse_xml>
Path filein => fileout
</Route>
Output:
{"EventReceivedTime":"2019-08-14T22:12:21.404463-04:00","SourceModuleName":"filein","SourceModuleType":"im_file","Created":"7/22/2019 3:00","Level":"Warning","Message":"org.mozilla.javascript.EcmaError: Cannot convert null to an object.\\r\\nCaused by error in sys_script.914d69890a0a3c1101310dab6c2ebf01.script at line 1\\r\\n\\r\\n==> 1: geamBlockCI();\\r\\n2: function geamBlockCI() {\\r\\n3: var user = gs.getUser();\\r\\n4: //gs.log('**** 1 User'+ user,'Test');\\r\\n","Source":"Evaluator","CreatedBy":"admin"}
{"EventReceivedTime":"2019-08-14T22:12:21.404601-04:00","SourceModuleName":"filein","SourceModuleType":"im_file","Created":"7/22/2019 3:00","Level":"Warning","Message":"org.mozilla.javascript.EcmaError: Cannot convert null to an object.\\r\\nCaused by error in sys_script.914d69890a0a3c1101310dab6c2ebf01.script at line 1\\r\\n\\r\\n==> 1: geamBlockCI();\\r\\n2: function geamBlockCI() {\\r\\n3: var user = gs.getUser();\\r\\n4: //gs.log('**** 1 User'+ user,'Test');\\r\\n","Source":"Evaluator","CreatedBy":"admin"}
