xm_multiline issue | View post | Log collection solutions

jd01 4 years ago

Hi all :)

(I'm new to nxlog)

I'm currently facing issues handling logs which are being sent to nxlog via syslog line by line. Basically after looking at documentation i found out that possibly xm_multiline can help me out.

Raw log example:

2020.05.20 15:22:37:481 CEST | Info       | HTTP

Body text part 1

2020.05.20 15:22:37:502 CEST | Info | HTTP

Body text part 2

2020.05.20 15:22:37:502 CEST | Info | HTTP

Body text part 3

2020.05.20 15:22:37:502 CEST |Debug | HTTP

Body text part 4

2020.05.20 15:22:37:502 CEST | Info | HTTP

I'm using the following headerline /^\d\d\d\d.\d\d.\d\d\s+\d\d:\d\d:\d\d:\d\d\d/ to capture the event into one.

<Extension charconv>
   Module      xm_charconv
   AutodetectCharsets utf-8, euc-jp, utf-16, utf-32, iso8859-2
</Extension>
<Extension json>
    Module	xm_json
</Extension>
<Extension multiline_header>
	Module xm_multiline
	HeaderLine /^\d\d\d\d.\d\d.\d\d\s+\d\d:\d\d:\d\d:\d\d\d/
</Extension>
<Input log_udp>
	Module  im_udp
    Host    0.0.0.0
    Port    5140
	InputType multiline_header
	Exec $type = 'mylog';
	Exec $Message = $raw_event;
</Input>
<Output log_out>
       Module om_udp
       Host 1.1.1.1
       Port 514
	   Exec  $raw_event = to_json();
</Output>
<Route forward_xsp> 
      Path log_udp=> log_out 
</Route>

Transforming the log into json.

The expected output would be:

Event no. 1

2020.05.20 15:22:37:481 CEST | Info | HTTP

Body text part 1

Event no. 2

2020.05.20 15:22:37:502 CEST | Info | HTTP

Body text part 2.

etc.

The issue end result:

Event no. 1

2020.05.20 15:22:37:481 CEST | Info | HTTP Body text part 2020.05.20 15:22:37:502 CEST | Info | HTTP Body text part 2

Event no. 2

Event no. 3

the successive timestamp headerline is ignored and the logs are grouped by the second. (see above) :( am i doing anything wrong ? do you guys have any suggestions on how to tackle this type of logs.