How to add a field for the file offset?

Post a different question
2
responses

We are pushing logs from file with the im_file module to logstash and then to elasticsearch. However, some of these logs only have second accuracy, and therefore not returned in order by elasticsearch when sorted by just the time. To get around this problem, we would like to add the position of the log event to a field that we store (for example beginning line number or byte offset within the source file). Is there a way I can add a separate field for this in the events pushed by nxlog, either in the input or output configuration?

AskedNovember 18, 2016 - 1:11am

Answer (1)

It is not possible to retrieve the file offset or line number but you could get around this by adding a serial number using statistical counters or variables.

Also note that the $EventReceivedTime field is automatically set when the line is read. This contains a millisecond precision value.

AnsweredNovember 18, 2016 - 9:20am

Comments (1)

  • scalesleaf's picture

    I tried that $EventReceivedTime, and it wouln't work for the case when the service is restarted, and lots of logs are read in at once since the $EventReceivedTime would be the same.

    Statistical counters seems to be the way to go, but I was wondering what the behavior of the COUNT counter was, if I kept adding 1, and it reached MAX_INT value. Would it throw, or would it jsut rotate and start from 0 again?

    November 21, 2016 - 8:54pm