Log Duplication and Line Re-Reading

My team is currently experiencing an issue with duplicate logs being produced in NXLog's outbound syslog feed. NXLog is reading from a flat file and sending a syslog feed to another machine for processing. Running a TCPdump for the incoming data on that second machine shows that multiple copies of the same log are being sent by NXlog.

AskedFebruary 23, 2021 - 4:12pm

Remove duplicates in text file

I'm using NXLog to read a log file and send it to Logstash.  This works fine, but some of the log lines are duplicates.  They're in separate lines, but the content is exactly the same.  I can't change the way the logs are written to the log file, so the only way is to fix it either with NXLog before it gets send, or in Logstash when it arrives, which I prefer not to do.

I see NXLog does have a function for this, but it's not working for me.  I've tried this in my config file.

AskedJuly 27, 2015 - 2:38pm