How to convert local time to UTC before sending logs to Logstash
Tags:
#1
achechen
I have the following output config:
<Output out> Module om_tcp Host 10.36.52.62 Port 12201 Exec $EventTime = strftime($EventTime, '%Y-%m-%d %H:%M:%S %Z'); \ to_json(); </Output>
Which is sending the EventTime in the local time zone of the server. This is how it looks like at Logstash side:
{
"message" => "{\"EventTime\":\"2016-03-03 03:07:29 Central Standard Time\",\"EventTimeWritten\":\"2016-03-03 03:07:29\",\"Hostname\":\"testwin2012\",\"EventType\":\"INFO\",\"SeverityValue\":2,\"Severity\":\"INFO\",\"SourceName\":\"Service Control Manager\",\"FileName\":\"System\",\"EventID\":7036,\"CategoryNumber\":0,\"RecordNumber\":34297,\"Message\":\"The nxlog service entered the running state. \",\"EventReceivedTime\":\"2016-03-03 03:07:30\",\"SourceModuleName\":\"eventlog\",\"SourceModuleType\":\"im_mseventlog\"}\r",
"@version" => "1",
"@timestamp" => "2016-03-03T09:07:34.479Z",
"host" => "testwin2012",
"port" => 49632,
"type" => "windows",
"EventTime" => "2016-03-03 03:07:29 Central Standard Time",
"EventTimeWritten" => "2016-03-03 03:07:29",
"SeverityValue" => 2,
"Severity" => "INFO",
"SourceName" => "Service Control Manager",
"FileName" => "System",
"EventID" => 7036,
"CategoryNumber" => 0,
"RecordNumber" => 34297,
"Message" => "The nxlog service entered the running state. "
}
I have to do a lot of expensive operations in Logstash to convert the timestamp into UTC. I have to convert "Central Standard Time" to Joda, which requires me to take that string, put it into a seperate field, prepare a dictionary, use an expensive translate operation on that new field and put it back to the timestamp field. Is there any way to make nxlog convert the EventTime field into UTC before sending?
#1
achechen
I have the following output config:
<Output out>
Module om_tcp
Host 10.36.52.62
Port 12201
Exec $EventTime = strftime($EventTime, '%Y-%m-%d %H:%M:%S %Z'); \
to_json();
</Output>
Which is sending the EventTime in the local time zone of the server. This is how it looks like at Logstash side:
{
"message" => "{\"EventTime\":\"2016-03-03 03:07:29 Central Standard Time\",\"EventTimeWritten\":\"2016-03-03 03:07:29\",\"Hostname\":\"testwin2012\",\"EventType\":\"INFO\",\"SeverityValue\":2,\"Severity\":\"INFO\",\"SourceName\":\"Service Control Manager\",\"FileName\":\"System\",\"EventID\":7036,\"CategoryNumber\":0,\"RecordNumber\":34297,\"Message\":\"The nxlog service entered the running state. \",\"EventReceivedTime\":\"2016-03-03 03:07:30\",\"SourceModuleName\":\"eventlog\",\"SourceModuleType\":\"im_mseventlog\"}\r",
"@version" => "1",
"@timestamp" => "2016-03-03T09:07:34.479Z",
"host" => "testwin2012",
"port" => 49632,
"type" => "windows",
"EventTime" => "2016-03-03 03:07:29 Central Standard Time",
"EventTimeWritten" => "2016-03-03 03:07:29",
"SeverityValue" => 2,
"Severity" => "INFO",
"SourceName" => "Service Control Manager",
"FileName" => "System",
"EventID" => 7036,
"CategoryNumber" => 0,
"RecordNumber" => 34297,
"Message" => "The nxlog service entered the running state. "
}
I have to do a lot of expensive operations in Logstash to convert the timestamp into UTC. I have to convert "Central Standard Time" to Joda, which requires me to take that string, put it into a seperate field, prepare a dictionary, use an expensive translate operation on that new field and put it back to the timestamp field. Is there any way to make nxlog convert the EventTime field into UTC before sending?
For now this is how most people deal with this:
Exec $EventReceivedTime = integer($EventReceivedTime) / 1000000; to_json();
The upcoming NXLog EE (and then NXLog CE) release will be able to do this:
<Extension json>
Module xm_json
DateFormat YYYY-MM-DDThh:mm:ss.sUTC
</Extension>
This is currently being tested.
