Nxlog - Miliseconds difference in Event Timestamp
Tags:
#1
Avin
Hello Team,
We are using below nxlog config to parse one application log file to Graylog. Nxlog is properly parsing file line by line to Graylog. But we are observing mismatch in timestamp (in 500 to 900 milliseconds) for actual event and showing in Graylog. Please let us know how to fix this issue.
=====================================================================
<Input itmlog>
Module im_file
File "/opt/bin/applogs2*.txt"
SavePos TRUE
ReadFromLast FALSE
InputType multiline
PollInterval 1
Exec if $raw_event =~ /^(\d{4}-\d{2}-\d{2}\s+\d{2}:\d{2}\d{2}\.\d{3})/ $EventTime = parsedate($1 + "Z");
Exec if $raw_event =~ /^.*([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12})\|([^\|]*)\|([^\|]+)\|([\S\s]*)/\
{ \
$UUID = $1; \
$Plugin = $2; \
$Severity = $3; \
$Message = $4; \
} \
else if $raw_event =~ /^\d{4}\-\d{2}\-\d{2} \d{2}\:\d{2}\:\d{2}[\.\d]* ([^\|]+)\|([^\|]+)\|([\S\s]*)/\
{ \
$Plugin = $1; \
$Severity = $2; \
$Message = $3; \
} \
else \
{ \
$Message = $raw_message; \
}
Exec $facility = 'APP_UAT';
#Exec $EventTime = strftime($EventTime, "%Y-%m-%dT%H:%M:%SZ");
Exec $Hostname = %IP%;
#Exec $FullMessage = '';
</Input>
========================================================================================
We are using below nxlog config to parse one application log file to Graylog. Nxlog is properly parsing file line by line to Graylog. But we are observing mismatch in timestamp (in 500 to 900 milliseconds) for actual event and showing in Graylog. Please let us know how to fix this issue.
=====================================================================
<Input itmlog>
Module im_file
File "/opt/bin/applogs2*.txt"
SavePos TRUE
ReadFromLast FALSE
InputType multiline
PollInterval 1
Exec if $raw_event =~ /^(\d{4}-\d{2}-\d{2}\s+\d{2}:\d{2}\d{2}\.\d{3})/ $EventTime = parsedate($1 + "Z");
Exec if $raw_event =~ /^.*([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12})\|([^\|]*)\|([^\|]+)\|([\S\s]*)/\
{ \
$UUID = $1; \
$Plugin = $2; \
$Severity = $3; \
$Message = $4; \
} \
else if $raw_event =~ /^\d{4}\-\d{2}\-\d{2} \d{2}\:\d{2}\:\d{2}[\.\d]* ([^\|]+)\|([^\|]+)\|([\S\s]*)/\
{ \
$Plugin = $1; \
$Severity = $2; \
$Message = $3; \
} \
else \
{ \
$Message = $raw_message; \
}
Exec $facility = 'APP_UAT';
#Exec $EventTime = strftime($EventTime, "%Y-%m-%dT%H:%M:%SZ");
Exec $Hostname = %IP%;
#Exec $FullMessage = '';
</Input>
========================================================================================
#1
Avin
Hello Team,
We are using below nxlog config to parse one application log file to Graylog. Nxlog is properly parsing file line by line to Graylog. But we are observing mismatch in timestamp (in 500 to 900 milliseconds) for actual event and showing in Graylog. Please let us know how to fix this issue.
=====================================================================
<Input itmlog>
Module im_file
File "/opt/bin/applogs2*.txt"
SavePos TRUE
ReadFromLast FALSE
InputType multiline
PollInterval 1
Exec if $raw_event =~ /^(\d{4}-\d{2}-\d{2}\s+\d{2}:\d{2}\d{2}.\d{3})/ $EventTime = parsedate($1 + "Z");
Exec if $raw_event =~ /^.([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12})|([^|])|([^|]+)|([\S\s])/
{
$UUID = $1;
$Plugin = $2;
$Severity = $3;
$Message = $4;
}
else if $raw_event =~ /^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}[.\d] ([^|]+)|([^|]+)|([\S\s]*)/
{
$Plugin = $1;
$Severity = $2;
$Message = $3;
}
else
{
$Message = $raw_message;
}
Exec $facility = 'APP_UAT';
#Exec $EventTime = strftime($EventTime, "%Y-%m-%dT%H:%M:%SZ");
Exec $Hostname = %IP%;
#Exec $FullMessage = '';
</Input>
========================================================================================
Hello Sir,
At first view i see the regex process may affect the load.
Since one module ( Input, output, processor ) use a single thread of the cpu, so you can split the by splitting the modules. For example:
<Input itmlog>
Module im_file
File "/opt/bin/applogs2*.txt"
SavePos TRUE
ReadFromLast FALSE
InputType multiline
PollInterval 1
</Input>
<Processor rewrite>
Module pm_null
Exec if $raw_event =~ /^(\d{4}-\d{2}-\d{2}\s+\d{2}:\d{2}\d{2}\.\d{3})/ $EventTime = parsedate($1 + "Z");
Exec if $raw_event =~ /^.*([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12})\|([^\|]*)\|([^\|]+)\|([\S\s]*)/\
{ \
$UUID = $1; \
$Plugin = $2; \
$Severity = $3; \
$Message = $4; \
} \
else if $raw_event =~ /^\d{4}\-\d{2}\-\d{2} \d{2}\:\d{2}\:\d{2}[\.\d]* ([^\|]+)\|([^\|]+)\|([\S\s]*)/\
{ \
$Plugin = $1; \
$Severity = $2; \
$Message = $3; \
} \
else \
{ \
$Message = $raw_message; \
}
Exec $facility = 'APP_UAT';
#Exec $EventTime = strftime($EventTime, "%Y-%m-%dT%H:%M:%SZ");
Exec $Hostname = %IP%;
#Exec $FullMessage = '';
</Processor>
<Route syslog_to_file>
Path itmlog => rewrite => output
</Route>
Also you can add om_file to see if the same date is written in the file .
Another suggestion is to remove the PollInterval 1 in order for a lower 0.5 sec will be applied.
Sincerely Klevin
