Need help in writing input module | Log collection solutions

#1 abasha 6 years ago

Hi, I am trying to read logs (csv format) from Service Now and send it to ELK stack. I need some help in writing the input module, so that I can properly send the logs to ELK stack. My input file contains 5 fields, but field3 has multiline input. I tried many methods and it doesnot work as per expectations. Can someone please help in writing proper input/output module to my stack.

Input file sample as follows:

Created,Level,Message,Source,Created by 7/22/2019 3:00,Warning,"org.mozilla.javascript.EcmaError: Cannot convert null to an object. Caused by error in sys_script.914d69890a0a3c1101310dab6c2ebf01.script at line 1

==> 1: geamBlockCI(); 2: function geamBlockCI() { 3: var user = gs.getUser(); 4: //gs.log('**** 1 User'+ user,'Test'); ",Evaluator,admin 7/22/2019 3:00,Warning,"org.mozilla.javascript.EcmaError: Cannot convert null to an object. Caused by error in sys_script.914d69890a0a3c1101310dab6c2ebf01.script at line 1

==> 1: geamBlockCI(); 2: function geamBlockCI() { 3: var user = gs.getUser(); 4: //gs.log('**** 1 User'+ user,'Test'); ",Evaluator,admin

Permalink

#4 Zhengshi Nxlog ✓ 6 years ago

#1 abasha

Input file sample as follows:

==> 1: geamBlockCI(); 2: function geamBlockCI() { 3: var user = gs.getUser(); 4: //gs.log('**** 1 User'+ user,'Test'); ",Evaluator,admin

You probably want to use the `xm_multiline` module. Something like the following. ``` Module xm_multiline # Detect date ##/##/#### HeaderLine /^\d{1,2}\/\d{1,2}\/\d{4}\s/ Module xm_json Module xm_csv Fields $Created,$Level,$Message,$Source,CreatedBy Module im_file File "/opt/nxlog/etc/multi.log" InputType multiline ReadFromLast TRUE SavePos TRUE # Ignore top line if $raw_event =~ /Created,Level,Message,Source,Created by/ drop(); # Convert Newline and Tab to printed character $raw_event =~ s/\R/\\r\\n/g; $raw_event =~ s/\t/\\t/g; # Parse $raw_event as CSV parse_csv(); # Convert to JSON to_json(); Module om_file File '/tmp/out.log' Path filein => fileout ``` Output: ``` {"EventReceivedTime":"2019-08-14T22:12:21.404463-04:00","SourceModuleName":"filein","SourceModuleType":"im_file","Created":"7/22/2019 3:00","Level":"Warning","Message":"org.mozilla.javascript.EcmaError: Cannot convert null to an object.\\r\\nCaused by error in sys_script.914d69890a0a3c1101310dab6c2ebf01.script at line 1\\r\\n\\r\\n==> 1: geamBlockCI();\\r\\n2: function geamBlockCI() {\\r\\n3: var user = gs.getUser();\\r\\n4: //gs.log('**** 1 User'+ user,'Test');\\r\\n","Source":"Evaluator","CreatedBy":"admin"} {"EventReceivedTime":"2019-08-14T22:12:21.404601-04:00","SourceModuleName":"filein","SourceModuleType":"im_file","Created":"7/22/2019 3:00","Level":"Warning","Message":"org.mozilla.javascript.EcmaError: Cannot convert null to an object.\\r\\nCaused by error in sys_script.914d69890a0a3c1101310dab6c2ebf01.script at line 1\\r\\n\\r\\n==> 1: geamBlockCI();\\r\\n2: function geamBlockCI() {\\r\\n3: var user = gs.getUser();\\r\\n4: //gs.log('**** 1 User'+ user,'Test');\\r\\n","Source":"Evaluator","CreatedBy":"admin"} ```