5
responses

Hi all :)

(I'm new to nxlog)

I'm currently facing issues handling logs which are being sent to nxlog via syslog line by line. Basically after looking at documentation i found out that possibly xm_multiline can help me out.

Raw log example:

2020.05.20 15:22:37:481 CEST | Info       | HTTP

Body text part 1

2020.05.20 15:22:37:502 CEST | Info | HTTP

Body text part 2

2020.05.20 15:22:37:502 CEST | Info | HTTP

Body text part 3

2020.05.20 15:22:37:502 CEST |Debug | HTTP

Body text part 4

2020.05.20 15:22:37:502 CEST | Info | HTTP

I'm using the following headerline /^\d\d\d\d.\d\d.\d\d\s+\d\d:\d\d:\d\d:\d\d\d/ to capture the event into one.

<Extension charconv>
   Module      xm_charconv
   AutodetectCharsets utf-8, euc-jp, utf-16, utf-32, iso8859-2
</Extension>
<Extension json>
    Module  xm_json
</Extension>
<Extension multiline_header>
    Module xm_multiline
    HeaderLine /^\d\d\d\d.\d\d.\d\d\s+\d\d:\d\d:\d\d:\d\d\d/
</Extension>
<Input log_udp>
    Module  im_udp
    Host    0.0.0.0
    Port    5140
    InputType multiline_header
    Exec $type = 'mylog';
    Exec $Message = $raw_event;
</Input>
<Output log_out>
       Module om_udp
       Host 1.1.1.1
       Port 514
       Exec  $raw_event = to_json();
</Output>
<Route forward_xsp> 
      Path log_udp=> log_out 
</Route> 

Transforming the log into json.

The expected output would be:

Event no. 1

------------------------------------------------------------------------------------------------------

2020.05.20 15:22:37:481 CEST | Info | HTTP

Body text part 1

------------------------------------------------------------------------------------------------------

Event no. 2

------------------------------------------------------------------------------------------------------

2020.05.20 15:22:37:502 CEST | Info | HTTP

Body text part 2.

------------------------------------------------------------------------------------------------------

etc.

The issue end result:


Event no. 1

2020.05.20 15:22:37:481 CEST | Info | HTTP Body text part 2020.05.20 15:22:37:502 CEST | Info | HTTP Body text part 2

------------------------------------------------------------------------------------------------------

Event no. 2

2020.05.20 15:22:38:481 CEST | Info | HTTP Body text part 2020.05.20 15:22:38:502 CEST | Info | HTTP Body text part 2 2020.05.20 15:22:38:481 CEST | Info | HTTP Body text part 2020.05.20 15:22:38:502 CEST | Info | HTTP Body text part 2

------------------------------------------------------------------------------------------------------

Event no. 3

2020.05.20 15:22:39:481 CEST | Info | HTTP Body text part 2020.05.20 15:22:39:502 CEST | Info | HTTP Body text part 2 2020.05.20 15:22:39:481 CEST | Info | HTTP Body text part 2020.05.20 15:22:38:502 CEST | Info | HTTP Body text part 2 2020.05.20 15:22:39:481 CEST | Info | HTTP Body text part 2020.05.20 15:22:39:502 CEST | Info | HTTP Body text part 2 2020.05.20 15:22:39:481 CEST | Info | HTTP Body text part 2020.05.20 15:22:38:502 CEST | Info | HTTP Body text part 2 2020.05.20 15:22:39:481 CEST | Info | HTTP Body text part 2020.05.20 15:22:39:502 CEST | Info | HTTP Body text part 2 2020.05.20 15:22:39:481 CEST | Info | HTTP Body text part 2020.05.20 15:22:38:502 CEST | Info | HTTP Body text part 2 2020.05.20 15:22:39:481 CEST | Info | HTTP Body text part 2020.05.20 15:22:39:502 CEST | Info | HTTP Body text part 2 2020.05.20 15:22:39:481 CEST | Info | HTTP Body text part 2020.05.20 15:22:38:502 CEST | Info | HTTP Body text part 2

------------------------------------------------------------------------------------------------------

the successive timestamp headerline is ignored and the logs are grouped by the second. (see above) :( am i doing anything wrong ? do you guys have any suggestions on how to tackle this type of logs.

AskedMay 20, 2020 - 5:06pm

Answers (2)

Jocelyn,

Using this config...

<Extension charconv>
   Module      xm_charconv
   AutodetectCharsets utf-8, euc-jp, utf-16, utf-32, iso8859-2
</Extension>

<Extension json>
    Module  xm_json
</Extension>

<Extension multiline_header>
    Module xm_multiline
    HeaderLine /^\d\d\d\d.\d\d.\d\d\s+\d\d:\d\d:\d\d:\d\d\d/
</Extension>

<Input log_udp>
    Module  im_file
    SavePos FALSE
    ReadFromLast FALSE
    File '/tmp/in.txt' 
    InputType multiline_header
    Exec $type = 'mylog';
    Exec $Message = $raw_event;
</Input>

<Output log_out>
       Module om_file
       File '/tmp/out.txt' 
       Exec  $raw_event = to_json();
</Output>

<Route forward_xsp> 
      Path log_udp=> log_out 
</Route>

And this input...

2020.05.20 15:22:37:481 CEST | Info       | HTTP
Body text part 1
2020.05.20 15:22:37:502 CEST | Info | HTTP
Body
sd text part 2
2020.05.20 15:22:37:502 CEST | Info | HTTP
Body te
dsdds
sddsxt part 3
2020.05.20 15:22:37:502 CEST |Debug | HTTP
Body text part 4
ddsdssd
sddsds

I get...

{"EventReceivedTime":"2020-05-21 13:10:48","SourceModuleName":"log_udp","SourceModuleType":"im_file","type":"mylog","Message":"2020.05.20 15:22:37:481 CEST | 
Info       | HTTP\nBody text part 1"}
{"EventReceivedTime":"2020-05-21 13:10:48","SourceModuleName":"log_udp","SourceModuleType":"im_file","type":"mylog","Message":"2020.05.20 15:22:37:502 CEST | 
Info | HTTP\nBody\nsd text part 2"}
{"EventReceivedTime":"2020-05-21 13:10:48","SourceModuleName":"log_udp","SourceModuleType":"im_file","type":"mylog","Message":"2020.05.20 15:22:37:502 CEST | 
Info | HTTP\nBody te\ndsdds\nsddsxt part 3"}
{"EventReceivedTime":"2020-05-21 13:10:48","SourceModuleName":"log_udp","SourceModuleType":"im_file","type":"mylog","Message":"2020.05.20 15:22:37:502 CEST |D
ebug | HTTP\nBody text part 4\nddsdssd\nsddsds"}

Comments (2)

  • jd01's picture

    nice :) you used the input file module. Thanks for this i'll give it a try and let you know :)

    UPDATE: This worked for me with the im_file module, i did some amendments to the headerline since nxlog was adding another timestamp when listening to syslog and writing to a file for first time. thanks for the help guys find config below:

    <Extension json>
    Module xm_json
    </Extension>
    <Extension multiline_header>
    Module xm_multiline
    HeaderLine /[\d\t .:]+ [1A-Za-z-]+ [MXZa-z-]+ [\d.\/-]+ [\d:]+ CEST \| Info/
    </Extension>
    #5140 udp listenner
    <Input udp_log_listenner>
    Module im_udp
    Host 0.0.0.0
    Port 5140
    </Input>
    #writes input to a file
    <Output log_raw_to_file>
    Module om_file
    File 'location of raw file'
    </Output>
    #Headerline and transformation
    <Input raw_udp_file>
    Module im_file
    SavePos FALSE
    ReadFromLast FALSE
    File 'location of raw file'
    InputType multiline_header
    Exec $type = 'mylog';
    Exec $Message = $raw_event;
    </Input>
    #output of the transformation
    <Output log_json_out>
    Module om_file
    File 'location of final log events'
    Exec $raw_event = to_json();
    </Output>

    #1st route
    <Route udp_to_file>
    Path udp_log_listenner => log_raw_to_file
    </Route>
    #2nd route
    <Route udp_file_to_json_file>
    Path raw_udp_file => log_json_out
    </Route>