I'm trying to avoid having duplicate logs send to my OSSIM server. I tried using the pm_norepeat module but to no avail.
Here the line I added in nxlog.conf file :

<Processor sans_doublons>
Module pm_norepeat

<Route route_windows_logs>
Path in_windows_events => sans_doublons => out_alienvault_csv

I also tried adding "CheckFields raw_event" in the processor, but I still get duplicate logs.
Does anyone know what could be the problem ?


AskedJuly 17, 2018 - 10:48am

Answer (1)

Does anyone know what could be the problem ?

The records is not identical or there are other records between but it is hard to tell without actually looking at it.

Comments (3)

  • CharlesCharles's picture

    There isn't much activity on my windows and i'm only sending the "application" event logs. The logs received in my OSSIM server are identical duplicates, same time, same seconds, same message, same informations, etc.

  • Jean's picture

    Hello CharlesCharles, this is Jean,

    In the documentation, they say that the module considere that a messages are duplicated if they are consecutives during a one second window.
    Maybe your duplicated messages are consecutives (you have two following message that are identical), but they are not processed/emitted during the same second.

    You should check the timestamp of your messages, maybe the rate of creation of your duplicated messages is not high enough.