Can we avoid to_syslog_ietf to include all fields in forwarded message

Post a different question
3
responses

Hello everyone,
I'm new to NXLOG and I'm trying to understand how event creation and forwarding work.

I have a question about the parse_json() function or maybe the to_syslog_ietf() function, I'm not sure.
Is it possible to use parse_json($Message) then to_syslog_ietf() to forward $Message without all the fields obtained with parse_json included in the forwarded syslog event?

Here are more explanations:
- I receive a syslog event and use parse_syslog_ietf()
- I get $Message which is json formated, so I use parse_json($Message)
- As I want to forward a specific field (let's say MsgToForward) I define $Message = $MsgToForward
- Then use to_syslog_ietf to forward it through syslog.

In the forwarded event everything is ok, except that all the fields in the json (the original $Message field) are included in the forwarded message.
But I don't understand why because if I put a log_info($raw_event) after the parse_syslog and after the parse_json, $raw_event has not changed.
So is it to_parse-ietf() function which includes all the fields available? Can we avoid that?

For instance the original event is:

2020-12-29 03:05:19 INFO <7>1 2020-12-29T03:05:19.692869 admin-PC send-logs 3556 send-logs.exe - {"user":"NT AUTHORITY\\SYSTEM","name":"main","programSource":"send-logs","msgToForward":"Send-logs started"};

Then the forwarded message:

2020-12-29 03:05:19 INFO <7>1 2020-12-29T03:05:19.692869 admin-PC send-logs 3556 send-logs.exe [NXLOG@14506 MessageSourceAddress="127.0.0.1" EventReceivedTime="2020-12-29 03:05:19" SourceModuleName="inSyslog" SourceModuleType="im_udp" user="NT AUTHORITY\\SYSTEM" name="main" programSource="send-logs" msgToForward="Send-logs started"] Send-logs started

We can see the fields available in the json in the first event (user, name, programSource and msgToForward) are added in the second event. But there are useless here, so I'm trying to have this result :

2020-12-29 03:05:19 INFO <7>1 2020-12-29T03:05:19.692869 admin-PC send-logs 3556 send-logs.exe [NXLOG@14506 MessageSourceAddress="127.0.0.1" EventReceivedTime="2020-12-29 03:05:19" SourceModuleName="inSyslog" SourceModuleType="im_udp"] Send-logs started

Is there a way to avoid parse_json or to_syslog_ietf to add these fields? (The number of fields and their names are not always the same.)
Or should I just not use parse_json and use regexp instead to get the value of MsgToForward?

My configuration if needed.

<Input  inSyslog>
  Module      im_udp
  Host        127.0.0.1
  Port        514
</Input>

<Output outSyslog2>
    Module          om_udp
    Host            127.0.0.1
    Port            1514
      Exec            parse_syslog_ietf(); parse_json($Message); $Message = $MsgToForward; to_syslog_ietf();
</Output>

<Route  Syslog>
    Path        inSyslog => outSyslog2
</Route>

Any help is welcomed. Thank you :)
Whok.

AskedDecember 29, 2020 - 5:02am

Comments (3)

  • Arkadiy's picture
    (NXLog)

    Hello Whok.

    You should try and use $raw_event instead of $Message so your code should look like this:

    Exec            parse_syslog_ietf(); parse_json($Message); $raw_event = $MsgToForward; to_syslog_ietf();

    Hope it solve your issue.
    Best regards, Arch

    December 29, 2020 - 8:30am
  • WW_809295's picture

    Hello Arch, thank you for your help :)

    Unfortunatly, setting $raw_event instead of $Message was the first thing I tried, but it makes the json fields duplicated. Here is the output:

    <7>1 2020-12-29T10:55:18.608844Z admin-PC send-logs 5580 - [NXLOG@14506 MessageSourceAddress="127.0.0.1" EventReceivedTime="2020-12-29 10:55:18" SourceModuleName="inSyslog" SourceModuleType="im_udp" name="main" user="NT AUTHORITY\\SYSTEM" programSource="send-logs" msgToForward="Send-logs started"] {"user":"NT AUTHORITY\\SYSTEM","name":"main","programSource":"send-logs","msgToForward":"Send-logs started"}

    The json fields are first key=value formated in the information added by to_syslog_ietf (I think). Then the whole json is appended. I suppose it re-uses the already created field $Message to create the new syslog event, and as I don't change it, it adds the full json. That's why I was trying to change $Message instead of $raw_event.

    Again thank you. If you have more ideas, please let me know.

    Whok.

    December 29, 2020 - 12:27pm
  • konstantinos's picture
    (NXLog)

    Hi Whok,

    The to_syslog_ietf() procedure is used to generate the $raw_event field from all fields in the event record

    There is a special module where you can define which fields to keep: xm_rewrite, however this is available only in the EE version.

    I see two options:

    Exec parse_syslog_ietf(); parse_json($Message); $Message = $MsgToForward; delete($user); delete($name); delete($programSource); to_syslog_ietf();
    • An alternative would be to parse the json part using a regex and avoid extracting all fields with parse_json():
    <Exec>
      parse_syslog();
      parse_syslog($Message);
      if $Message =~ /.*msgToForward":"(.*)".*/ $Message=$1;
      to_syslog_ietf();
</Exec>

    Kind regards,

    Konstantinos

    January 4, 2021 - 11:45am

Answers (0)