issue with multilining with empty line as header

#1 _asp_ 7 years ago

Hi,

I have following log:

23.08.2016 22:00:00: [20740] INFO: Line 1
23.08.2016 22:00:00: [20740] Line 2
23.08.2016 22:00:00: [20740] Line 3
23.08.2016 22:00:00: [20740] Line 4
23.08.2016 22:00:00: [20740] Line 5

23.08.2016 22:00:00: [20745] INFO: Line 1
23.08.2016 22:00:00: [20745] Line 2
23.08.2016 22:00:00: [20745] Line 3
23.08.2016 22:00:00: [20745] Line 4
23.08.2016 22:00:00: [20745] Line 5

Each multiline log line is beginning with an empty line. So I tried to use the empty line as header:

<Extension multilineEmtpyLine>
Module xm_multiline
HeaderLine /^$/
</Extension>

<Input foo>
   Module im_file
   File "C:/logfile/foo.log"

   #enabling multilining
   InputType multilineEmtpyLine
   SavePos TRUE
   Exec $Message = $raw_event;
</Input>

<Output localTCP>
   Module om_tcp
    Host localhost
   Port 5544

   Exec $tmpmessage = $Message; delete($Message); rename_field("tmpmessage","message");
   Exec $raw_event = to_json();

   # Uncomment for debug output
   Exec file_write('c:\nxlog\nxlog_localtcp_debug_output.log', $raw_event + "\n");
</Output>

<Route nxlogLocal>
#Path topbeat_debug, ttp_debug => localTCP
Path foo=> localTCP
</Route>

As I see in debug output and logstash each source line will be transmitted as single line. Multilining is not working.

How can I get it work?

thanks, Andreas

Permalink

#2 adm Nxlog ✓ 7 years ago

#1 _asp_

Hi, I have following log: 23.08.2016 22:00:00: [20740] INFO: Line 1 23.08.2016 22:00:00: [20740] Line 2 23.08.2016 22:00:00: [20740] Line 3 23.08.2016 22:00:00: [20740] Line 4 23.08.2016 22:00:00: [20740] Line 5 23.08.2016 22:00:00: [20745] INFO: Line 1 23.08.2016 22:00:00: [20745] Line 2 23.08.2016 22:00:00: [20745] Line 3 23.08.2016 22:00:00: [20745] Line 4 23.08.2016 22:00:00: [20745] Line 5 Each multiline log line is beginning with an empty line. So I tried to use the empty line as header: <Extension multilineEmtpyLine> Module xm_multiline HeaderLine /^$/ </Extension> <Input foo> Module im_file File "C:/logfile/foo.log" #enabling multilining InputType multilineEmtpyLine SavePos TRUE Exec $Message = $raw_event; </Input> <Output localTCP> Module om_tcp Host localhost Port 5544 Exec $tmpmessage = $Message; delete($Message); rename_field("tmpmessage","message"); Exec $raw_event = to_json(); # Uncomment for debug output Exec file_write('c:\nxlog\nxlog_localtcp_debug_output.log', $raw_event + "\n"); </Output> <Route nxlogLocal> #Path topbeat_debug, ttp_debug => localTCP Path foo=> localTCP </Route> As I see in debug output and logstash each source line will be transmitted as single line. Multilining is not working. How can I get it work? thanks, Andreas

Here is an example to show you that it works:

<Extension multi>
    Module      xm_multiline
    HeaderLine  /^$/
</Extension>

<Extension json>
    Module      xm_json
</Extension>

<Input in>
    Module      im_file
    File        "tmp/input.txt"
    ReadFromLast FALSE
    InputType   multi
    Exec        $Message = $raw_event;
</Input>

<Output out>
    Module      om_file
    File        "tmp/multi.out"
    Exec        to_json();
</Output>

<Route 2>
    Path        in => out
</Route>

Input:


event 1 first line

event 2 first line
event 2 second line

event 3 first line

The output produced is the following:

{"EventReceivedTime":"2016-08-25 13:41:59","SourceModuleName":"in","SourceModuleType":"im_file","Message":"\nevent 1 first line"}
{"EventReceivedTime":"2016-08-25 13:41:59","SourceModuleName":"in","SourceModuleType":"im_file","Message":"\nevent 2 first line\nevent 2 second line"}
{"EventReceivedTime":"2016-08-25 13:41:59","SourceModuleName":"in","SourceModuleType":"im_file","Message":"\nevent 3 first line"}

You can see that event 2 is in a single json record.