My team is currently experiencing an issue with duplicate logs being produced in NXLog's outbound syslog feed. NXLog is reading from a flat file and sending a syslog feed to another machine for processing. Running a TCPdump for the incoming data on that second machine shows that multiple copies of the same log are being sent by NXlog. Moreover, the duplicates are not sent sequentially, but instead are sent almost exactly 5 minutes apart as show by five minute gaps between the "EventReceivedTime" syslog header values. We have reviewed the flat file and confirmed that it does not produce the duplicates, and thus must be related to the configuration of NXLog and the way it reads the flat file. I'd appreciate any insight others might have on what is causing this problem. The NXLog configuration is as follows: Panic Soft #NoFreeOnExit TRUE define ROOT D:\NX Log\Program Files define CERTDIR %ROOT%\cert define CONFDIR %ROOT%\conf define LOGDIR %ROOT%\data define LOGFILE %LOGDIR%\nxlog.log LogFile %LOGFILE% Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data <Extension _syslog> Module xm_syslog </Extension> <Extension _charconv> Module xm_charconv AutodetectCharsets iso8859-2, utf-8, utf-16, utf-32 </Extension> <Extension _exec> Module xm_exec </Extension> <Extension _fileop> Module xm_fileop # Check the size of our log file hourly, rotate if larger than 5MB &lt;Schedule&gt; Every 1 hour Exec if (file_exists('%LOGFILE%') and \ (file_size('%LOGFILE%') &gt;= 5M)) \ file_cycle('%LOGFILE%', 8); &lt;/Schedule&gt; # Rotate our log file every week on Sunday at midnight &lt;Schedule&gt; When @weekly Exec if file_exists('%LOGFILE%') file_cycle('%LOGFILE%', 8); &lt;/Schedule&gt; </Extension> <Input in> Module im_file File "D:[Ingested File Path].cef" InputType LineBased SavePos TRUE ReadFromLast TRUE PollInterval 1 </Input> <Processor norepeat> Module pm_norepeat </Processor> <Output out> Module om_ssl Host [Second Machine IP] Port 515 Exec to_syslog_ietf(); Allows using self-signed certificates AllowUntrusted FALSE Certificate from the peer host CAFile D:\[CA File Path].pem Certificate file CertFile D:\[Certificate File Path].pem Keypair file CertKeyFile D:\[Key File Path].pem </Output> <Route sitecollector> Path in => norepeat => out </Route>

I'm using NXLog to read a log file and send it to Logstash.  This works fine, but some of the log lines are duplicates.  They're in separate lines, but the content is exactly the same.  I can't change the way the logs are written to the log file, so the only way is to fix it either with NXLog before it gets send, or in Logstash when it arrives, which I prefer not to do. I see NXLog does have a function for this, but it's not working for me.  I've tried this in my config file. <Processor norepeat> Module pm_norepeat </Processor> <Route 1> Path in => norepeat => out </Route> This is abviously not working for me, am I maybe missing something here?