xm_multiline not detecting header unless there is a new line at end of file
I have the following configuration:
define DEFAULT_OUTPUTS null_output
include c:/Program Files (x86)/nxlog/conf/nxlog.conf.d/\\*.default
<Extension app_log_multiline>
Module xm_multiline
HeaderLine /^[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}\.[0-9]{3}/
</Extension>
<Input app_logs>
Module im_file
InputType app_log_multiline
RenameCheck true
File "C:\\ProgramData\\App\\EventLog\\TestFile.log"
Exec $type = 'windows_log_files';
Exec $Message = $raw_event;
Exec $file_path = file_name();
</Input>
<Route r_app_logs>
Path app_logs => testfile
</Route>
and
<Output testfile>
Module om_file
Exec to_json();
File "C:\\testlogoutput.txt"
</Output>
If the log file contains the following lines:
2016-09-15 02:00:37.906 TestEvent
2016-09-15 02:00:42.251 TestEvent2
with an empty line after the second log line, the output is correct:
{
"EventReceivedTime":"2016-09-19 22:35:32",
"SourceModuleName":"app_logs",
"SourceModuleType":"im_file",
"type":"windows_log_files",
"Message":"2016-09-15 02:00:37.906 TestEvent",
"file_path":"C:\\ProgramData\\App\\EventLog\\TestFile.log"
}
{
"EventReceivedTime":"2016-09-19 22:35:32",
"SourceModuleName":"app_logs",
"SourceModuleType":"im_file",
"type":"windows_log_files",
"Message":"2016-09-15 02:00:42.251 TestEvent2",
"file_path":"C:\\ProgramData\\App\\EventLog\\TestFile.log"
}
However, if the log file contains the same two lines as above, without an empty line after the second log line, the output becomes:
{
"EventReceivedTime":"2016-09-19 22:40:59",
"SourceModuleName":"app_logs",
"SourceModuleType":"im_file",
"type":"windows_log_files",
"Message":"2016-09-15 02:00:37.906 TestEvent\r\n2016-09-15 02:00:42.251 TestEvent2",
"file_path":"C:\\ProgramData\\App\\EventLog\\TestFile.log"
}
Basically, unless there is an empty line after the final log event, it seems that multiline is combining the last two lines into one, even though the HeaderLine is matched correctly.
Did I configure something wrong?
> with an empty line after the second log line
Actually I think you meant no newline after the second log line.
Looks like this is a bug. When there is no newline the data is flushed by im_file after a period of time and xm_multiline does not properly separate the two, instead it concatenates the incomplete line to the previous.