1
answer

I'm looking at ways to get high precision (with fractions of seconds) timestamps out of nxlog. If application provides these, it's of course easy to get this data into nxlog, but it's not easy to get it out - strftime function you can format timestamps with doesn't support fractions of seconds. Am I right?

It's even more problematic if application doesn't provide high precision timestamps - which is the case with most of unix programs using im_uds or all data from im_kernel for example. Even EventReceivedTime timestamp used in case of IETF syslog format isn't high precision.

Now, before anyone says that these wouldn't very precise timestamps anyway - that's not what I really care about. But as in the path to the log analysis events can be reordered (with redundant message brokers and stuff), it is critical to have high precision timestamps so correct order of the messages can be restored.

 Ideally I'd like to use BSD syslog with high precision timestamps, but in any way it seems to be impossible at the moment. Or am I overlooking something?

AskedMarch 9, 2015 - 8:53am

Answer (1)

strftime() does not provide a way to output the fractional part because this is missing from the underlying C function call and the POSIX API.

Internally all datetime fields store a microsecond precision value (e.g. EventTime, EventReceivedTime, etc). Unfortunately due to the above limitations of strftime() it's not easy to convert this to a human readable form. A solution you might consider - this is what some people use - is to convert it into an integer value:

Exec $EventTime = integer($EventTime);

Adding another formatter function that should properly support timezones and fractional parts is on the roadmap.

BSD syslog can't have fractional seconds as it would break the standard. The newer IETF syslog already has that.

AnsweredMarch 9, 2015 - 12:21pm

Comments (2)

  • ttyserial's picture

    Although "Exec $EventTime = integer($EventTime);" seemed to be a good idea at first, it broke timestamps completely for sending syslog over network. Seems that to_syslog_ietf() isn't able to parse an integer in the EventTime field, and puts time.now() into there while sending a message. Which might be a completely different time compared to original event time of course. Yes, I know that BSD syslog with fractional seconds breaks the standard, but that seems something that a lot of vendors do. Btw, is there a way to disable adding structured data into IETF format messages?

    March 10, 2015 - 12:21pm
  • adm's picture
    (NXLog)

    What I meant is that the integer format can be transferred in the message body, e.g. in JSON:

    Exec $EvtTime = integer($EventTime); $Message = to_json(); to_syslog_bsd();

    There are several vendors with products that break standards compliance, the notorious example is Cisco and they are the authors of RFC3164...

    It's not possible to disable structured data part in IETF format but you can nuke that with a proper regexp:

    Exec to_syslog_ietf(); $raw_event =~ s/____//;

     

     

    March 10, 2015 - 5:58pm