Nxlog - Miliseconds difference in Event Timestamp

Tags:

#1 Avin
Hello Team,

We are using below nxlog config to parse one application log file to Graylog. Nxlog is properly parsing file line by line to Graylog. But we are observing mismatch in timestamp (in 500 to 900 milliseconds) for actual event and showing in Graylog. Please let us know how to fix this issue.
=====================================================================
<Input itmlog>
Module im_file
File "/opt/bin/applogs2*.txt"
SavePos TRUE
ReadFromLast FALSE
InputType multiline
PollInterval 1
Exec if $raw_event =~ /^(\d{4}-\d{2}-\d{2}\s+\d{2}:\d{2}\d{2}\.\d{3})/ $EventTime = parsedate($1 + "Z");
Exec if $raw_event =~ /^.*([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12})\|([^\|]*)\|([^\|]+)\|([\S\s]*)/\
{ \
$UUID = $1; \
$Plugin = $2; \
$Severity = $3; \
$Message = $4; \
} \
else if $raw_event =~ /^\d{4}\-\d{2}\-\d{2} \d{2}\:\d{2}\:\d{2}[\.\d]* ([^\|]+)\|([^\|]+)\|([\S\s]*)/\
{ \
$Plugin = $1; \
$Severity = $2; \
$Message = $3; \
} \
else \
{ \
$Message = $raw_message; \
}
Exec $facility = 'APP_UAT';
#Exec $EventTime = strftime($EventTime, "%Y-%m-%dT%H:%M:%SZ");
Exec $Hostname = %IP%;
#Exec $FullMessage = '';
</Input>
========================================================================================
#2 KlevinDeactivated Nxlog ✓
#1 Avin
Hello Team,

We are using below nxlog config to parse one application log file to Graylog. Nxlog is properly parsing file line by line to Graylog. But we are observing mismatch in timestamp (in 500 to 900 milliseconds) for actual event and showing in Graylog. Please let us know how to fix this issue.
=====================================================================
<Input itmlog>
Module im_file
File "/opt/bin/applogs2*.txt"
SavePos TRUE
ReadFromLast FALSE
InputType multiline
PollInterval 1
Exec if $raw_event =~ /^(\d{4}-\d{2}-\d{2}\s+\d{2}:\d{2}\d{2}\.\d{3})/ $EventTime = parsedate($1 + "Z");
Exec if $raw_event =~ /^.*([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12})\|([^\|]*)\|([^\|]+)\|([\S\s]*)/\
{ \
$UUID = $1; \
$Plugin = $2; \
$Severity = $3; \
$Message = $4; \
} \
else if $raw_event =~ /^\d{4}\-\d{2}\-\d{2} \d{2}\:\d{2}\:\d{2}[\.\d]* ([^\|]+)\|([^\|]+)\|([\S\s]*)/\
{ \
$Plugin = $1; \
$Severity = $2; \
$Message = $3; \
} \
else \
{ \
$Message = $raw_message; \
}
Exec $facility = 'APP_UAT';
#Exec $EventTime = strftime($EventTime, "%Y-%m-%dT%H:%M:%SZ");
Exec $Hostname = %IP%;
#Exec $FullMessage = '';
</Input>
========================================================================================
Hello Sir, At first view i see the regex process may affect the load. Since one module ( Input, output, processor ) use a single thread of the cpu, so you can split the by splitting the modules. For example: Module im_file File "/opt/bin/applogs2*.txt" SavePos TRUE ReadFromLast FALSE InputType multiline PollInterval 1 Module pm_null Exec if $raw_event =~ /^(\d{4}-\d{2}-\d{2}\s+\d{2}:\d{2}\d{2}\.\d{3})/ $EventTime = parsedate($1 + "Z"); Exec if $raw_event =~ /^.*([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12})\|([^\|]*)\|([^\|]+)\|([\S\s]*)/\ { \ $UUID = $1; \ $Plugin = $2; \ $Severity = $3; \ $Message = $4; \ } \ else if $raw_event =~ /^\d{4}\-\d{2}\-\d{2} \d{2}\:\d{2}\:\d{2}[\.\d]* ([^\|]+)\|([^\|]+)\|([\S\s]*)/\ { \ $Plugin = $1; \ $Severity = $2; \ $Message = $3; \ } \ else \ { \ $Message = $raw_message; \ } Exec $facility = 'APP_UAT'; #Exec $EventTime = strftime($EventTime, "%Y-%m-%dT%H:%M:%SZ"); Exec $Hostname = %IP%; #Exec $FullMessage = ''; Path itmlog => rewrite => output Also you can add om_file to see if the same date is written in the file . Another suggestion is to remove the `PollInterval 1` in order for a lower 0.5 sec will be applied. Sincerely Klevin