xm_multiline not detecting header unless there is a new line at end of file

Tags:

#1 scalesleaf

I have the following configuration:

 

define DEFAULT_OUTPUTS null_output

include c:/Program Files (x86)/nxlog/conf/nxlog.conf.d/\\*.default

<Extension app_log_multiline>
  Module xm_multiline
  HeaderLine /^[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}\.[0-9]{3}/
</Extension>

<Input app_logs>
  Module im_file
  InputType app_log_multiline
  RenameCheck true
  File "C:\\ProgramData\\App\\EventLog\\TestFile.log"
  Exec $type = 'windows_log_files';
  Exec $Message = $raw_event;
  Exec $file_path = file_name();
</Input>

<Route r_app_logs>
  Path app_logs => testfile
</Route>

and

<Output testfile>
  Module om_file
  Exec to_json();
  File "C:\\testlogoutput.txt"
</Output>

If the log file contains the following lines:

2016-09-15 02:00:37.906 TestEvent
2016-09-15 02:00:42.251 TestEvent2

with an empty line after the second log line, the output is correct:

{
    "EventReceivedTime":"2016-09-19 22:35:32",
    "SourceModuleName":"app_logs",
    "SourceModuleType":"im_file",
    "type":"windows_log_files",
    "Message":"2016-09-15 02:00:37.906 TestEvent",
    "file_path":"C:\\ProgramData\\App\\EventLog\\TestFile.log"
}
{
    "EventReceivedTime":"2016-09-19 22:35:32",
    "SourceModuleName":"app_logs",
    "SourceModuleType":"im_file",
    "type":"windows_log_files",
    "Message":"2016-09-15 02:00:42.251 TestEvent2",
    "file_path":"C:\\ProgramData\\App\\EventLog\\TestFile.log"
}

 

However, if the log file contains the same two lines as above, without an empty line after the second log line, the output becomes:

{
    "EventReceivedTime":"2016-09-19 22:40:59",
    "SourceModuleName":"app_logs",
    "SourceModuleType":"im_file",
    "type":"windows_log_files",
    "Message":"2016-09-15 02:00:37.906 TestEvent\r\n2016-09-15 02:00:42.251 TestEvent2",
    "file_path":"C:\\ProgramData\\App\\EventLog\\TestFile.log"
}

Basically, unless there is an empty line after the final log event, it seems that multiline is combining the last two lines into one, even though the HeaderLine is matched correctly.
Did I configure something wrong?

#2 b0ti Nxlog ✓
#1 scalesleaf
I have the following configuration:   define DEFAULT_OUTPUTS null_output include c:/Program Files (x86)/nxlog/conf/nxlog.conf.d/\\*.default <Extension app_log_multiline>   Module xm_multiline   HeaderLine /^[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}\.[0-9]{3}/ </Extension> <Input app_logs>   Module im_file   InputType app_log_multiline   RenameCheck true   File "C:\\ProgramData\\App\\EventLog\\TestFile.log"   Exec $type = 'windows_log_files';   Exec $Message = $raw_event;   Exec $file_path = file_name(); </Input> <Route r_app_logs>   Path app_logs => testfile </Route> and <Output testfile>   Module om_file   Exec to_json();   File "C:\\testlogoutput.txt" </Output> If the log file contains the following lines: 2016-09-15 02:00:37.906 TestEvent 2016-09-15 02:00:42.251 TestEvent2 with an empty line after the second log line, the output is correct: {     "EventReceivedTime":"2016-09-19 22:35:32",     "SourceModuleName":"app_logs",     "SourceModuleType":"im_file",     "type":"windows_log_files",     "Message":"2016-09-15 02:00:37.906 TestEvent",     "file_path":"C:\\ProgramData\\App\\EventLog\\TestFile.log" } {     "EventReceivedTime":"2016-09-19 22:35:32",     "SourceModuleName":"app_logs",     "SourceModuleType":"im_file",     "type":"windows_log_files",     "Message":"2016-09-15 02:00:42.251 TestEvent2",     "file_path":"C:\\ProgramData\\App\\EventLog\\TestFile.log" }   However, if the log file contains the same two lines as above, without an empty line after the second log line, the output becomes: {     "EventReceivedTime":"2016-09-19 22:40:59",     "SourceModuleName":"app_logs",     "SourceModuleType":"im_file",     "type":"windows_log_files",     "Message":"2016-09-15 02:00:37.906 TestEvent\r\n2016-09-15 02:00:42.251 TestEvent2",     "file_path":"C:\\ProgramData\\App\\EventLog\\TestFile.log" } Basically, unless there is an empty line after the final log event, it seems that multiline is combining the last two lines into one, even though the HeaderLine is matched correctly. Did I configure something wrong?

> with an empty line after the second log line

Actually I think you meant no newline after the second log line.

Looks like this is a bug. When there is no newline the data is flushed by im_file after a period of time and xm_multiline does not properly separate the two, instead it concatenates the incomplete line to the previous.