xm_multiline issue
Tags:
xm_multiline
#1
jd01
Hi all :)
(I'm new to nxlog)
I'm currently facing issues handling logs which are being sent to nxlog via syslog *line by line*. Basically after looking at documentation i found out that possibly xm_multiline can help me out.
**Raw log example:**
2020.05.20 15:22:37:481 CEST | Info | HTTP
Body text part 1
2020.05.20 15:22:37:502 CEST | Info | HTTP
Body text part 2
2020.05.20 15:22:37:502 CEST | Info | HTTP
Body text part 3
2020.05.20 15:22:37:502 CEST |Debug | HTTP
Body text part 4
2020.05.20 15:22:37:502 CEST | Info | HTTP
**I'm using the following headerline /^\d\d\d\d.\d\d.\d\d\s+\d\d:\d\d:\d\d:\d\d\d/ to capture the event into one.**
Module xm_charconv
AutodetectCharsets utf-8, euc-jp, utf-16, utf-32, iso8859-2
Module xm_json
Module xm_multiline
HeaderLine /^\d\d\d\d.\d\d.\d\d\s+\d\d:\d\d:\d\d:\d\d\d/
Module im_udp
Host 0.0.0.0
Port 5140
InputType multiline_header
Exec $type = 'mylog';
Exec $Message = $raw_event;
Module om_udp
Host 1.1.1.1
Port 514
Exec $raw_event = to_json();
Path log_udp=> log_out
Transforming the log into json.
**The expected output would be:**
Event no. 1
------------------------------------------------------------------------------------------------------
---------------------------------------------------------------
2020.05.20 15:22:37:481 CEST | Info | HTTP
Body text part 1
------------------------------------------------------------------------------------------------------
---------------------------------------------------------------
Event no. 2
------------------------------------------------------------------------------------------------------
---------------------------------------------------------------
2020.05.20 15:22:37:502 CEST | Info | HTTP
Body text part 2.
------------------------------------------------------------------------------------------------------
---------------------------------------------------------------
etc.
**The issue end result:**
--------------------------------------------------------------------------------------------------------------------------------------------------------
Event no. 1
2020.05.20 15:22:37:481 CEST | Info | HTTP Body text part 2020.05.20 15:22:37:502 CEST | Info | HTTP Body text part 2
------------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------------
Event no. 2
2020.05.20 15:22:38:481 CEST | Info | HTTP Body text part 2020.05.20 15:22:38:502 CEST | Info | HTTP Body text part 2 2020.05.20 15:22:38:481 CEST | Info | HTTP Body text part 2020.05.20 15:22:38:502 CEST | Info | HTTP Body text part 2
------------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------------
Event no. 3
2020.05.20 15:22:39:481 CEST | Info | HTTP Body text part 2020.05.20 15:22:39:502 CEST | Info | HTTP Body text part 2 2020.05.20 15:22:39:481 CEST | Info | HTTP Body text part 2020.05.20 15:22:38:502 CEST | Info | HTTP Body text part 2 2020.05.20 15:22:39:481 CEST | Info | HTTP Body text part 2020.05.20 15:22:39:502 CEST | Info | HTTP Body text part 2 2020.05.20 15:22:39:481 CEST | Info | HTTP Body text part 2020.05.20 15:22:38:502 CEST | Info | HTTP Body text part 2 2020.05.20 15:22:39:481 CEST | Info | HTTP Body text part 2020.05.20 15:22:39:502 CEST | Info | HTTP Body text part 2 2020.05.20 15:22:39:481 CEST | Info | HTTP Body text part 2020.05.20 15:22:38:502 CEST | Info | HTTP Body text part 2 2020.05.20 15:22:39:481 CEST | Info | HTTP Body text part 2020.05.20 15:22:39:502 CEST | Info | HTTP Body text part 2 2020.05.20 15:22:39:481 CEST | Info | HTTP Body text part 2020.05.20 15:22:38:502 CEST | Info | HTTP Body text part 2
------------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------------
the successive timestamp headerline is ignored and the logs are grouped by the second. (see above) :( am i doing anything wrong ? do you guys have any suggestions on how to tackle this type of logs.
#1
jd01
Hi all :)
(I'm new to nxlog)
I'm currently facing issues handling logs which are being sent to nxlog via syslog *line by line*. Basically after looking at documentation i found out that possibly xm_multiline can help me out.
**Raw log example:**
2020.05.20 15:22:37:481 CEST | Info | HTTP
Body text part 1
2020.05.20 15:22:37:502 CEST | Info | HTTP
Body text part 2
2020.05.20 15:22:37:502 CEST | Info | HTTP
Body text part 3
2020.05.20 15:22:37:502 CEST |Debug | HTTP
Body text part 4
2020.05.20 15:22:37:502 CEST | Info | HTTP
**I'm using the following headerline /^\d\d\d\d.\d\d.\d\d\s+\d\d:\d\d:\d\d:\d\d\d/ to capture the event into one.**
Module xm_charconv
AutodetectCharsets utf-8, euc-jp, utf-16, utf-32, iso8859-2
Module xm_json
Module xm_multiline
HeaderLine /^\d\d\d\d.\d\d.\d\d\s+\d\d:\d\d:\d\d:\d\d\d/
Module im_udp
Host 0.0.0.0
Port 5140
InputType multiline_header
Exec $type = 'mylog';
Exec $Message = $raw_event;
Module om_udp
Host 1.1.1.1
Port 514
Exec $raw_event = to_json();
Path log_udp=> log_out
Transforming the log into json.
**The expected output would be:**
Event no. 1
------------------------------------------------------------------------------------------------------
---------------------------------------------------------------
2020.05.20 15:22:37:481 CEST | Info | HTTP
Body text part 1
------------------------------------------------------------------------------------------------------
---------------------------------------------------------------
Event no. 2
------------------------------------------------------------------------------------------------------
---------------------------------------------------------------
2020.05.20 15:22:37:502 CEST | Info | HTTP
Body text part 2.
------------------------------------------------------------------------------------------------------
---------------------------------------------------------------
etc.
**The issue end result:**
--------------------------------------------------------------------------------------------------------------------------------------------------------
Event no. 1
2020.05.20 15:22:37:481 CEST | Info | HTTP Body text part 2020.05.20 15:22:37:502 CEST | Info | HTTP Body text part 2
------------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------------
Event no. 2
2020.05.20 15:22:38:481 CEST | Info | HTTP Body text part 2020.05.20 15:22:38:502 CEST | Info | HTTP Body text part 2 2020.05.20 15:22:38:481 CEST | Info | HTTP Body text part 2020.05.20 15:22:38:502 CEST | Info | HTTP Body text part 2
------------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------------
Event no. 3
2020.05.20 15:22:39:481 CEST | Info | HTTP Body text part 2020.05.20 15:22:39:502 CEST | Info | HTTP Body text part 2 2020.05.20 15:22:39:481 CEST | Info | HTTP Body text part 2020.05.20 15:22:38:502 CEST | Info | HTTP Body text part 2 2020.05.20 15:22:39:481 CEST | Info | HTTP Body text part 2020.05.20 15:22:39:502 CEST | Info | HTTP Body text part 2 2020.05.20 15:22:39:481 CEST | Info | HTTP Body text part 2020.05.20 15:22:38:502 CEST | Info | HTTP Body text part 2 2020.05.20 15:22:39:481 CEST | Info | HTTP Body text part 2020.05.20 15:22:39:502 CEST | Info | HTTP Body text part 2 2020.05.20 15:22:39:481 CEST | Info | HTTP Body text part 2020.05.20 15:22:38:502 CEST | Info | HTTP Body text part 2 2020.05.20 15:22:39:481 CEST | Info | HTTP Body text part 2020.05.20 15:22:39:502 CEST | Info | HTTP Body text part 2 2020.05.20 15:22:39:481 CEST | Info | HTTP Body text part 2020.05.20 15:22:38:502 CEST | Info | HTTP Body text part 2
------------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------------
the successive timestamp headerline is ignored and the logs are grouped by the second. (see above) :( am i doing anything wrong ? do you guys have any suggestions on how to tackle this type of logs.
Hi and welcome aboard. :)
Could you try to use your regex pattern without start of the string assertion? If I get structure of your logs correctly it should do the trick.
Regards, Arch