Possible to Set Interval for Log Deduplication?

#1 steven.su 2 years ago

Hi team, from the guide below we know that the Nxlog is able to deduplicate the log by some specific fields ("CheckFields"). https://nxlog.co/documentation/nxlog-user-guide/pm_norepeat.html

May I know if it is possible to set the interval for it? Let's take below configuration as example. Is it possible to set the interval = 10s, so that the log with the same fields (Hostname, SourceName, Message) will be suppressed for only every 10 seconds. This is doable from Logstash (throttle -> period), but we wanna confirm if it is capable in Nxlog as well. Thank you.

<Input uds> Module im_uds UDS /dev/log </Input>

<Processor norepeat> Module pm_norepeat CheckFields Hostname, SourceName, Message </Processor>

<Output file> Module om_file File "/var/log/messages" </Output>

<Route uds_to_file> Path uds => norepeat => file </Route>

Permalink

#2 NenadMDeactivated Nxlog ✓ 2 years ago

#1 steven.su

Hi team, from the guide below we know that the Nxlog is able to deduplicate the log by some specific fields ("CheckFields"). https://nxlog.co/documentation/nxlog-user-guide/pm_norepeat.html May I know if it is possible to set the interval for it? Let's take below configuration as example. Is it possible to set the interval = 10s, so that the log with the same fields (Hostname, SourceName, Message) will be suppressed for only every 10 seconds. This is doable from Logstash (throttle -> period), but we wanna confirm if it is capable in Nxlog as well. Thank you. <Input uds> Module im_uds UDS /dev/log </Input> <Processor norepeat> Module pm_norepeat CheckFields Hostname, SourceName, Message </Processor> <Output file> Module om_file File "/var/log/messages" </Output> <Route uds_to_file> Path uds => norepeat => file </Route>

The time for the de-duplication in the pm_norepeat is limited to 1s by default: "The module waits one second for duplicated messages to arrive. If duplicates are detected, the first message is forwarded, the rest are dropped, and a message containing "last message repeated n times" is sent instead."

At the moment NXLog Enterprise Edition doesn't have an option to extend this interval. However, there is another module - pm_evcorr available in the NXLog CE and the NXLog EE version. This module uses the Interval directive for setting the time interval for the events that mach certain correlation formula. This module also exports the defined get_prev_event_data() function. ie. get_prev_event_data("raw_event") will retrieve the value of previous event and it can be used to compare it with the value of the current one.

PS: You shared the link that points to the NXLog EE documentation page. Please tell us which product do you use: NXLog EE or CE?