JSON Formatting questions | Log collection solutions

#1 gportnoy 5 years ago

I don't believe this is currently possible, but hopefully someone can correct me if it's already implemented.

Ability to reorder fields in the raw message that gets sent to output module. The way things currently work any fields that are generated during the processing are added on to the end of the message when it gets forwarded to the output destination. The challenge for me is that I generate both a timestamp and a hostname field which then appear at the end of the message. Both of those fields are important for processing during the ingestion of the data on the remote side. Splunk for example by default only reads a certain limited number of characters into each message in order to find a timestamp and host field. I'd love an ability to reorder (or just move to front of the message) the fields that are generated.
JSON templating. For use with another pipeline, we have a requirement for a very specific JSON structure that must wrap each message. It's several levels nested and certain fields have to be present in the right place and in the right order for the event message to be accepted/processed. I tried faking it with json flatten and unflatten functions, but they aren't precise enough. Is there a way to define a specific template that should wrap all the messages before being sent to the output?

Permalink

#2 b0ti Nxlog ✓ 5 years ago

#1 gportnoy

I don't believe this is currently possible, but hopefully someone can correct me if it's already implemented. Ability to reorder fields in the raw message that gets sent to output module. The way things currently work any fields that are generated during the processing are added on to the end of the message when it gets forwarded to the output destination. The challenge for me is that I generate both a timestamp and a hostname field which then appear at the end of the message. Both of those fields are important for processing during the ingestion of the data on the remote side. Splunk for example by default only reads a certain limited number of characters into each message in order to find a timestamp and host field. I'd love an ability to reorder (or just move to front of the message) the fields that are generated. JSON templating. For use with another pipeline, we have a requirement for a very specific JSON structure that must wrap each message. It's several levels nested and certain fields have to be present in the right place and in the right order for the event message to be accepted/processed. I tried faking it with json flatten and unflatten functions, but they aren't precise enough. Is there a way to define a specific template that should wrap all the messages before being sent to the output?

There is no way to reorder the fields currently. If you need sophisticated text processing I suggest using xm_perl (or xm_python) that allows much more than what you can do within the Exec block using nxlog functions, e.g. reformatting and dealing with nested JSON structures.

Here is a trick that we have used with Splunk that may help you with the timestamp issue in 1.:

    <Exec>
        # Set $raw_event field to timestamp that Splunk will recognize
        $raw_event = strftime($EventTime, "%D %r");

        # Remove $EventTime field rather than include it in key/value pairs
        delete($EventTime);

        # Append all remaining internal fields to $raw_event as KVP
        $raw_event = $raw_event + "\n" + to_kvp();