1
response

Hello Team,

We are using below nxlog config to parse one application log file to Graylog. Nxlog is properly parsing file line by line to Graylog. But we are observing mismatch in timestamp (in 500 to 900 milliseconds) for actual event and showing in Graylog. Please let us know how to fix this issue.
=====================================================================
<Input itmlog>
Module im_file
File "/opt/bin/applogs2*.txt"
SavePos TRUE
ReadFromLast FALSE
InputType multiline
PollInterval 1
Exec if $raw_event =~ /^(\d{4}-\d{2}-\d{2}\s+\d{2}:\d{2}\d{2}\.\d{3})/ $EventTime = parsedate($1 + "Z");
Exec if $raw_event =~ /^.*([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12})\|([^\|]*)\|([^\|]+)\|([\S\s]*)/\
{ \
$UUID = $1; \
$Plugin = $2; \
$Severity = $3; \
$Message = $4; \
} \
else if $raw_event =~ /^\d{4}\-\d{2}\-\d{2} \d{2}\:\d{2}\:\d{2}[\.\d]* ([^\|]+)\|([^\|]+)\|([\S\s]*)/\
{ \
$Plugin = $1; \
$Severity = $2; \
$Message = $3; \
} \
else \
{ \
$Message = $raw_message; \
}
Exec $facility = 'APP_UAT';
#Exec $EventTime = strftime($EventTime, "%Y-%m-%dT%H:%M:%SZ");
Exec $Hostname = %IP%;
#Exec $FullMessage = '';
</Input>
========================================================================================

AskedAugust 17, 2022 - 8:16am

Answer (1)

Hello Sir,

At first view i see the regex process may affect the load.

Since one module ( Input, output, processor ) use a single thread of the cpu, so you can split the by splitting the modules. For example:

<Input itmlog>
    Module im_file
    File "/opt/bin/applogs2*.txt"
    SavePos TRUE
    ReadFromLast FALSE
    InputType multiline
    PollInterval 1

</Input>

<Processor rewrite>
    Module      pm_null
    Exec if $raw_event =~ /^(\d{4}-\d{2}-\d{2}\s+\d{2}:\d{2}\d{2}\.\d{3})/ $EventTime = parsedate($1 + "Z");
    Exec if $raw_event =~ /^.*([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12})\|([^\|]*)\|([^\|]+)\|([\S\s]*)/\
    { \
    $UUID = $1; \
    $Plugin = $2; \
    $Severity = $3; \
    $Message = $4; \
    } \
    else if $raw_event =~ /^\d{4}\-\d{2}\-\d{2} \d{2}\:\d{2}\:\d{2}[\.\d]* ([^\|]+)\|([^\|]+)\|([\S\s]*)/\
    { \
    $Plugin = $1; \
    $Severity = $2; \
    $Message = $3; \
    } \
    else \
    { \
    $Message = $raw_message; \
    }
    Exec $facility = 'APP_UAT';
    #Exec $EventTime = strftime($EventTime, "%Y-%m-%dT%H:%M:%SZ");
    Exec $Hostname = %IP%;
    #Exec $FullMessage = '';
</Processor>


<Route syslog_to_file>
    Path        itmlog => rewrite => output
</Route>

Also you can add om_file to see if the same date is written in the file .

Another suggestion is to remove the PollInterval 1 in order for a lower 0.5 sec will be applied.

Sincerely Klevin