issue with multilining with empty line as header
Hi,
I have following log:
23.08.2016 22:00:00: [20740] INFO: Line 1
23.08.2016 22:00:00: [20740] Line 2
23.08.2016 22:00:00: [20740] Line 3
23.08.2016 22:00:00: [20740] Line 4
23.08.2016 22:00:00: [20740] Line 5
23.08.2016 22:00:00: [20745] INFO: Line 1
23.08.2016 22:00:00: [20745] Line 2
23.08.2016 22:00:00: [20745] Line 3
23.08.2016 22:00:00: [20745] Line 4
23.08.2016 22:00:00: [20745] Line 5
Each multiline log line is beginning with an empty line. So I tried to use the empty line as header:
<Extension multilineEmtpyLine>
Module xm_multiline
HeaderLine /^$/
</Extension>
<Input foo>
Module im_file
File "C:/logfile/foo.log"
#enabling multilining
InputType multilineEmtpyLine
SavePos TRUE
Exec $Message = $raw_event;
</Input>
<Output localTCP>
Module om_tcp
Host localhost
Port 5544
Exec $tmpmessage = $Message; delete($Message); rename_field("tmpmessage","message");
Exec $raw_event = to_json();
# Uncomment for debug output
Exec file_write('c:\nxlog\nxlog_localtcp_debug_output.log', $raw_event + "\n");
</Output>
<Route nxlogLocal>
#Path topbeat_debug, ttp_debug => localTCP
Path foo=> localTCP
</Route>
As I see in debug output and logstash each source line will be transmitted as single line. Multilining is not working.
How can I get it work?
thanks, Andreas
Hi,
I have following log:
23.08.2016 22:00:00: [20740] INFO: Line 1
23.08.2016 22:00:00: [20740] Line 2
23.08.2016 22:00:00: [20740] Line 3
23.08.2016 22:00:00: [20740] Line 4
23.08.2016 22:00:00: [20740] Line 5
23.08.2016 22:00:00: [20745] INFO: Line 1
23.08.2016 22:00:00: [20745] Line 2
23.08.2016 22:00:00: [20745] Line 3
23.08.2016 22:00:00: [20745] Line 4
23.08.2016 22:00:00: [20745] Line 5
Each multiline log line is beginning with an empty line. So I tried to use the empty line as header:
<Extension multilineEmtpyLine>
Module xm_multiline
HeaderLine /^$/
</Extension>
<Input foo>
Module im_file
File "C:/logfile/foo.log"
#enabling multilining
InputType multilineEmtpyLine
SavePos TRUE
Exec $Message = $raw_event;
</Input>
<Output localTCP>
Module om_tcp
Host localhost
Port 5544
Exec $tmpmessage = $Message; delete($Message); rename_field("tmpmessage","message");
Exec $raw_event = to_json();
# Uncomment for debug output
Exec file_write('c:\nxlog\nxlog_localtcp_debug_output.log', $raw_event + "\n");
</Output>
<Route nxlogLocal>
#Path topbeat_debug, ttp_debug => localTCP
Path foo=> localTCP
</Route>
As I see in debug output and logstash each source line will be transmitted as single line. Multilining is not working.
How can I get it work?
thanks, Andreas
Here is an example to show you that it works:
<Extension multi>
Module xm_multiline
HeaderLine /^$/
</Extension>
<Extension json>
Module xm_json
</Extension>
<Input in>
Module im_file
File "tmp/input.txt"
ReadFromLast FALSE
InputType multi
Exec $Message = $raw_event;
</Input>
<Output out>
Module om_file
File "tmp/multi.out"
Exec to_json();
</Output>
<Route 2>
Path in => out
</Route>
Input:
event 1 first line event 2 first line event 2 second line event 3 first line
The output produced is the following:
{"EventReceivedTime":"2016-08-25 13:41:59","SourceModuleName":"in","SourceModuleType":"im_file","Message":"\nevent 1 first line"}
{"EventReceivedTime":"2016-08-25 13:41:59","SourceModuleName":"in","SourceModuleType":"im_file","Message":"\nevent 2 first line\nevent 2 second line"}
{"EventReceivedTime":"2016-08-25 13:41:59","SourceModuleName":"in","SourceModuleType":"im_file","Message":"\nevent 3 first line"}
You can see that event 2 is in a single json record.
