2
responses

I don't believe this is currently possible, but hopefully someone can correct me if it's already implemented.

1. Ability to reorder fields in the raw message that gets sent to output module. The way things currently work any fields that are generated during the processing are added on to the end of the message when it gets forwarded to the output destination. The challenge for me is that I generate both a timestamp and a hostname field which then appear at the end of the message. Both of those fields are important for processing during the ingestion of the data on the remote side. Splunk for example by default only reads a certain limited number of characters into each message in order to find a timestamp and host field. I'd love an ability to reorder (or just move to front of the message) the fields that are generated.

2. JSON templating. For use with another pipeline, we have a requirement for a very specific JSON structure that must wrap each message. It's several levels nested and certain fields have to be present in the right place and in the right order for the event message to be accepted/processed. I tried faking it with json flatten and unflatten functions, but they aren't precise enough. Is there a way to define a specific template that should wrap all the messages before being sent to the output?

AskedMarch 4, 2019 - 3:38pm

Answers (2)

There is no way to reorder the fields currently. If you need sophisticated text processing I suggest using xm_perl (or xm_python) that allows much more than what you can do within the Exec block using nxlog functions, e.g. reformatting and dealing with nested JSON structures.

Here is a trick that we have used with Splunk that may help you with the timestamp issue in 1.:

    <Exec>
        # Set $raw_event field to timestamp that Splunk will recognize
        $raw_event = strftime($EventTime, "%D %r");

        # Remove $EventTime field rather than include it in key/value pairs
        delete($EventTime);

        # Append all remaining internal fields to $raw_event as KVP
        $raw_event = $raw_event + "\n" + to_kvp();

I recently ran into this problem. I am using nxlog-ce to log windows event logs, and in order to capture the full details I am using to_JSON to export the values. The primary reason I look at logs is to see the content of the $Message. However, in JSON mode the first fields to be displayed are $EventTime and $HostName, both of which are redundant since the syslog has the time and hostname on the left. My solution was to use the built-in functions to rename and assign the values I want into the first two positions.

In other words I swap $EventTime with $Message, and $Hostname with $AccountName.

Here's how I did it.

 <Input from_eventlog>

    Module  im_msvistalog

    Exec    $Hostname = hostname(); # fix hostname

    # Swap Message and EventTime (if they exist) 
    Exec    if $Message if $EventTime { $tmp=$EventTime; rename_field("Message","oldMessage"); \
            rename_field("EventTime","Message"); $Message=$oldMessage; \
            rename_field("oldMessage","EventTime"); $EventTime=$tmp; }

    # Swap Hostname and AccountName (if they exist)
    Exec    if $HostName if $AccountName { $tmp=$AccountName; rename_field("Hostname","oldHostname"); \
            rename_field("AccountName","Hostname"); $HostName=$oldHostname; \
            rename_field("oldHostname","AccountName"); $AccountName=$tmp; }

    # Normalize SourceName
    Exec    $SourceName = replace($SourceName," ","-");
    Exec    $SourceName = replace($SourceName,"_","-");

    Exec $Message = to_json(); 
</Input>