Remove Duplicates Help Needed
Hello All,
I am trying to use the pm_norepeat module to remove duplicate log messages that sometimes flood my logs. I am apparently not grasping how this works as the duplicate records are still flooding through the logs when I attempt to use the pm_norepeat function.
Can anyone advise on what I am doing wrong? Is there a different way to accomplish de-duplification of messages with the im_msvistalog Exec to_syslog_bsd() modules ?????
Here are my configuration file statements:
define ROOT C:\Program Files (x86)\nxlog
define ROOT_STRING C:\Program Files (x86)\\nxlog
Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log
<Extension syslog>
Module xm_syslog
</Extension>
<Input in>
Module im_msvistalog
Exec to_syslog_bsd();
ReadFromLast TRUE
SavePos TRUE
Query <QueryList>\
<Query Id="0">\
<Select Path="Application">*</Select>\
<Select Path="System">*</Select>\
<Select Path="Security">*</Select>\
</Query>\
</QueryList>
</Input>
<Processor norepeat>
Module pm_norepeat
CheckFields Hostname, SourceName, Message
</Processor>
<Output out>
Module om_udp
Host xxxxx.xxxxxxxxxxapp.com
Port 12345
</Output>
<Route 1>
Path in => norepeat => out
</Route>
pm_norepeat checks whether two or more successive event records have the same $Message field. Converting to syslog is irrelevant here since to_syslog_bsd() only rewrites the $raw_event field.
If you want to deduplicate the events based on other fields you should list those in CheckFields.