Remove Duplicates Help Needed

Tags:

#1 Zendog

Hello All,

I am trying to use the pm_norepeat module to remove duplicate log messages that sometimes flood my logs. I am apparently not grasping how this works as the duplicate records are still flooding through the logs when I attempt to use the pm_norepeat function.

Can anyone advise on what I am doing wrong?  Is there a different way to accomplish de-duplification of messages with the im_msvistalog Exec to_syslog_bsd() modules ?????

Here are my configuration file statements:

define ROOT C:\Program Files (x86)\nxlog
define ROOT_STRING C:\Program Files (x86)\\nxlog
  
Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log

<Extension syslog>
    Module      xm_syslog
</Extension>
 
<Input in>
    Module      im_msvistalog
    Exec    to_syslog_bsd();   
    ReadFromLast TRUE
    SavePos     TRUE
    Query       <QueryList>\
                    <Query Id="0">\
                        <Select Path="Application">*</Select>\
                        <Select Path="System">*</Select>\
                        <Select Path="Security">*</Select>\
                    </Query>\
                </QueryList>    
</Input>

<Processor norepeat>
   Module    pm_norepeat
   CheckFields Hostname, SourceName, Message
</Processor>

<Output out>
   Module      om_udp
    Host xxxxx.xxxxxxxxxxapp.com
    Port 12345
</Output>

<Route 1>
    Path in => norepeat => out
</Route>

#2 adm Nxlog ✓ (Last updated )
#1 Zendog
Hello All, I am trying to use the pm_norepeat module to remove duplicate log messages that sometimes flood my logs. I am apparently not grasping how this works as the duplicate records are still flooding through the logs when I attempt to use the pm_norepeat function. Can anyone advise on what I am doing wrong?  Is there a different way to accomplish de-duplification of messages with the im_msvistalog Exec to_syslog_bsd() modules ????? Here are my configuration file statements: define ROOT C:\Program Files (x86)\nxlog define ROOT_STRING C:\Program Files (x86)\\nxlog    Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data LogFile %ROOT%\data\nxlog.log <Extension syslog>     Module      xm_syslog </Extension>   <Input in>     Module      im_msvistalog     Exec    to_syslog_bsd();        ReadFromLast TRUE     SavePos     TRUE     Query       <QueryList>\                     <Query Id="0">\                         <Select Path="Application">*</Select>\                         <Select Path="System">*</Select>\                         <Select Path="Security">*</Select>\                     </Query>\                 </QueryList>     </Input> <Processor norepeat>    Module    pm_norepeat    CheckFields Hostname, SourceName, Message </Processor> <Output out>    Module      om_udp     Host xxxxx.xxxxxxxxxxapp.com     Port 12345 </Output> <Route 1>     Path in => norepeat => out </Route>

pm_norepeat checks whether two or more successive event records have the same $Message field. Converting to syslog is irrelevant here since to_syslog_bsd() only rewrites the $raw_event field.

If you want to deduplicate the events based on other fields you should list those in CheckFields.