1
response

Hello All,

I am trying to use the pm_norepeat module to remove duplicate log messages that sometimes flood my logs. I am apparently not grasping how this works as the duplicate records are still flooding through the logs when I attempt to use the pm_norepeat function.

Can anyone advise on what I am doing wrong?  Is there a different way to accomplish de-duplification of messages with the im_msvistalog Exec to_syslog_bsd() modules ?????

Here are my configuration file statements:

define ROOT C:\Program Files (x86)\nxlog
define ROOT_STRING C:\Program Files (x86)\\nxlog
  
Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log

<Extension syslog>
    Module      xm_syslog
</Extension>
 
<Input in>
    Module      im_msvistalog
    Exec    to_syslog_bsd();   
    ReadFromLast TRUE
    SavePos     TRUE
    Query       <QueryList>\
                    <Query Id="0">\
                        <Select Path="Application">*</Select>\
                        <Select Path="System">*</Select>\
                        <Select Path="Security">*</Select>\
                    </Query>\
                </QueryList>    
</Input>

<Processor norepeat>
   Module    pm_norepeat
   CheckFields Hostname, SourceName, Message
</Processor>

<Output out>
   Module      om_udp
    Host xxxxx.xxxxxxxxxxapp.com
    Port 12345
</Output>

<Route 1>
    Path in => norepeat => out
</Route>

AskedMarch 28, 2016 - 9:00pm

Answer (1)

pm_norepeat checks whether two or more successive event records have the same $Message field. Converting to syslog is irrelevant here since to_syslog_bsd() only rewrites the $raw_event field.

If you want to deduplicate the events based on other fields you should list those in CheckFields.