Remove duplicates in text file

I'm using NXLog to read a log file and send it to Logstash.  This works fine, but some of the log lines are duplicates.  They're in separate lines, but the content is exactly the same.  I can't change the way the logs are written to the log file, so the only way is to fix it either with NXLog before it gets send, or in Logstash when it arrives, which I prefer not to do.

I see NXLog does have a function for this, but it's not working for me.  I've tried this in my config file.

AskedJuly 27, 2015 - 2:38pm

NXLog Parsing XML

I've seen some posts from about a year ago that NXLog is unable to parse attributes using xm_xml, I just wanted to check if this is still true?

I am running NXLog as a service on Windows machines and want to be able to parse the following message, is it possible?

AskedJuly 23, 2015 - 5:02pm

nxLog Community Edition License details

I am working in Product development company in india. I have downloaded nxlog community edition and setup to transfer our logs into logstash server.

Our environment details as below.

OS - Windows 7 Professional.

nxLog installed as service.  

Everything working as we expected.

Now what i want to ask is, can we take this setup into our production environment? 

AskedJuly 22, 2015 - 7:59am

om_ssl loses data

I have a stack of  nxlog -> om_ssl -> tcp input ->logstash.  And after i do kill -9 i lose a bunch of messages. i looked up buffered data in logstash and switch it to 1. tested it and if i read file i can lose about max 5 messages. So i tried to look up in nxlog, data_timeout is there 30 sec.  so as i looked up with data lose over time  it looks it might be it. Is there a chance to change this value after installing rpm ?  or i have to compile it with changed value in the code.  

AskedJuly 21, 2015 - 12:12pm