xm_multiline issue

View thread

jd01
Hi all :) (I'm new to nxlog) I'm currently facing issues handling logs which are being sent to nxlog via syslog *line by line*. Basically after looking at documentation i found out that possibly xm_multiline can help me out. **Raw log example:** 2020.05.20 15:22:37:481 CEST | Info | HTTP Body text part 1 2020.05.20 15:22:37:502 CEST | Info | HTTP Body text part 2 2020.05.20 15:22:37:502 CEST | Info | HTTP Body text part 3 2020.05.20 15:22:37:502 CEST |Debug | HTTP Body text part 4 2020.05.20 15:22:37:502 CEST | Info | HTTP **I'm using the following headerline /^\d\d\d\d.\d\d.\d\d\s+\d\d:\d\d:\d\d:\d\d\d/ to capture the event into one.** Module xm_charconv AutodetectCharsets utf-8, euc-jp, utf-16, utf-32, iso8859-2 Module xm_json Module xm_multiline HeaderLine /^\d\d\d\d.\d\d.\d\d\s+\d\d:\d\d:\d\d:\d\d\d/ Module im_udp Host 0.0.0.0 Port 5140 InputType multiline_header Exec $type = 'mylog'; Exec $Message = $raw_event; Module om_udp Host 1.1.1.1 Port 514 Exec $raw_event = to_json(); Path log_udp=> log_out Transforming the log into json. **The expected output would be:** Event no. 1 ------------------------------------------------------------------------------------------------------ --------------------------------------------------------------- 2020.05.20 15:22:37:481 CEST | Info | HTTP Body text part 1 ------------------------------------------------------------------------------------------------------ --------------------------------------------------------------- Event no. 2 ------------------------------------------------------------------------------------------------------ --------------------------------------------------------------- 2020.05.20 15:22:37:502 CEST | Info | HTTP Body text part 2. ------------------------------------------------------------------------------------------------------ --------------------------------------------------------------- etc. **The issue end result:** -------------------------------------------------------------------------------------------------------------------------------------------------------- Event no. 1 2020.05.20 15:22:37:481 CEST | Info | HTTP Body text part 2020.05.20 15:22:37:502 CEST | Info | HTTP Body text part 2 ------------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------------ Event no. 2 2020.05.20 15:22:38:481 CEST | Info | HTTP Body text part 2020.05.20 15:22:38:502 CEST | Info | HTTP Body text part 2 2020.05.20 15:22:38:481 CEST | Info | HTTP Body text part 2020.05.20 15:22:38:502 CEST | Info | HTTP Body text part 2 ------------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------------ Event no. 3 2020.05.20 15:22:39:481 CEST | Info | HTTP Body text part 2020.05.20 15:22:39:502 CEST | Info | HTTP Body text part 2 2020.05.20 15:22:39:481 CEST | Info | HTTP Body text part 2020.05.20 15:22:38:502 CEST | Info | HTTP Body text part 2 2020.05.20 15:22:39:481 CEST | Info | HTTP Body text part 2020.05.20 15:22:39:502 CEST | Info | HTTP Body text part 2 2020.05.20 15:22:39:481 CEST | Info | HTTP Body text part 2020.05.20 15:22:38:502 CEST | Info | HTTP Body text part 2 2020.05.20 15:22:39:481 CEST | Info | HTTP Body text part 2020.05.20 15:22:39:502 CEST | Info | HTTP Body text part 2 2020.05.20 15:22:39:481 CEST | Info | HTTP Body text part 2020.05.20 15:22:38:502 CEST | Info | HTTP Body text part 2 2020.05.20 15:22:39:481 CEST | Info | HTTP Body text part 2020.05.20 15:22:39:502 CEST | Info | HTTP Body text part 2 2020.05.20 15:22:39:481 CEST | Info | HTTP Body text part 2020.05.20 15:22:38:502 CEST | Info | HTTP Body text part 2 ------------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------------ the successive timestamp headerline is ignored and the logs are grouped by the second. (see above) :( am i doing anything wrong ? do you guys have any suggestions on how to tackle this type of logs.