parse logfile with comma in nxlog to logstash

#1 deepap 2 years ago

Am trying to send the file seperated with commas as below "33,adasdad,null,03May2021,notification,SG,null,null"

In nxlog.conf ---> i wrote

<Extension log> Module xm_multiline HeaderLine /^<event>/ EndLine /^</\event>/ </Extension>

<Input apptype> Module im_file File "testlog.log" <Exec> $message = $raw_event; to_json; </Exec> </Input>

<Output out> Module om_tcp Host localhost:port# Exec to_json(); </Output>

My logstash confgurariotn

input{ tcp { port => "port# codec => multiline{ pattern => "^[%{TIMESTAMP_ISO8601}]" negate => true what => "previous" } } } filter {} output { stdout {codec =>rubydebug} }

Output am gettingis as attached

{ "@timestamp" => "2021-05-03T09:47:14.575Z", "message" => ""33,adasdad,null,03May2021,notification,SG,null,null", "@version" => "1", "tags" => [ [0] "multiline", [1] "multiline_codec_max_lines_reached" ], "host" => "<localhost>", "port" => <port#>, "EventReceivedTime" => "2021-05-03T17:47:14.472325+08:00", "SourceModuleName" => "<apptype>", "SourceModuleType" => "im_file", }

Permalink

#2 rafDeactivated Nxlog ✓ 2 years ago

#1 deepap

Hi Am trying to send the file seperated with commas as below "33,adasdad,null,03May2021,notification,SG,null,null" In nxlog.conf ---> i wrote <Extension log> Module xm_multiline HeaderLine /^<event>/ EndLine /^</\event>/ </Extension> <Input apptype> Module im_file File "testlog.log" <Exec> $message = $raw_event; to_json; </Exec> </Input> <Output out> Module om_tcp Host localhost:port# Exec to_json(); </Output> My logstash confgurariotn input{ tcp { port => "port# codec => multiline{ pattern => "^[%{TIMESTAMP_ISO8601}]" negate => true what => "previous" } } } filter {} output { stdout {codec =>rubydebug} } Output am gettingis as attached { "@timestamp" => "2021-05-03T09:47:14.575Z", "message" => ""33,adasdad,null,03May2021,notification,SG,null,null", "@version" => "1", "tags" => [ [0] "multiline", [1] "multiline_codec_max_lines_reached" ], "host" => "<localhost>", "port" => <port#>, "EventReceivedTime" => "2021-05-03T17:47:14.472325+08:00", "SourceModuleName" => "<apptype>", "SourceModuleType" => "im_file", }

Hi,

You haven't mentioned what effect you'd like to achieve - I can only assume you want to extract data from limited by ",", right?

If that's the case, it's a typical CSV format. Please, refer to xm_csv docs to explore the possibilities here.

Good luck,
Rafal