1
answer

Dear community,

I'm currently working on parsing MS Exchange logs and sending them via GELF to my graylog instance.

I'd like to convert the sender- and recipient-address field to lowercase. Sounds pretty easy, in fact, I need help :(

my current config looks like this (below). Any help is appreciated.

I've tried to work with "Exec       $sender-address = lc($sender-address);" within the input as well as Output backet - neither did work.

define BASEDIR C:\Program Files\Microsoft\Exchange Server\V14\TransportRoles\Logs\MessageTracking

<Extension csv>
Module xm_csv
Fields $date-time, $client-ip, $client-hostname, $server-ip, $server-hostname, $source-context, $connector-id, $exchange_source, $event-id, $internal-message-id, $message-id, $recipient-address, $recipient-status, $total-bytes, $recipient-count, $related-recipient-address, $reference, $message-subject, $sender-address, $return-path, $message-info, $directionality, $tenant-id, $original-client-ip, $original-server-ip, $custom-data
FieldTypes string, string, string, string, string, string, string, string, string, integer, string, string, string, integer, integer, string, string, string, string, string, string, string, string, string, string, string
Delimiter ,
</Extension>

<Input in_exchange>
Module im_file
File '%BASEDIR%\MSGTRK????????*-*.LOG'
SavePos TRUE
Exec if $raw_event =~ /HealthMailbox/ drop();
Exec if $raw_event =~ /^#/ drop();
Exec csv->parse_csv();
</Input>

<Output out_exchange>
Module om_udp
Host graylog.local
Port 12203
OutputType GELF
Exec $SourceName = 'exchange_msgtrk_log';
</Output>

<Route exchange>
Path in_exchange => out_exchange
</Route>

AskedOctober 15, 2015 - 6:40pm

Answer (1)

You can do something like this:

Exec rename_field('sender-address', 'sender_address'); $sender_address = lc($sender_address);

rename_field() is needed because the dash causes issues in field names (i.e. $sender-address is not valid). You can rename it back if you want. Else you should specify names that don't contain a dash in xm_csv's Fields.

AnsweredOctober 15, 2015 - 8:12pm

Comments (11)

  • nomoresecrets's picture

    Thanks, that worked well.

    I've renamed my fields to "_" and then "lc($...)" did the job :)

    October 16, 2015 - 9:48am
  • kilamzh's picture

    Can someone post working config please. I have renamed my Fields to "_" but not sure where to add "Exec $sender_address = lc(sender_address)
    Thank you.

    May 22, 2018 - 9:13pm
  • Malik.Z's picture

    Thank you for quick reply,

    I've been through this document many times, but I still struggle.

    My config looks exactly like one above, except, I already change Fields to "_"

    -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
    define BASEDIR C:\Program Files\Microsoft\Exchange Server\V14\TransportRoles\Logs\MessageTracking

    <Extension csv>
    Module xm_csv
    Fields $date_time, $client_ip, $client_hostname, $server_ip, $server_hostname, $source_context, $connector_id, $exchange_source, $event_id, $internal_message_id, $message_id, $recipient_address, $recipient_status, $total_bytes, $recipient_count, $related_recipient_address, $reference, $message_subject, $sender_address, $return_path, $message_info, $directionality, $tenant_id, $original_client_ip, $original_server_ip, $custom_data
    Delimiter ,
    </Extension>

    <Input in_exchange>
    Module im_file
    File '%BASEDIR%\MSGTRK????????*-*.LOG'
    SavePos TRUE
    Exec if $raw_event =~ /HealthMailbox/ drop();
    Exec if $raw_event =~ /^#/ drop();
    Exec csv->parse_csv();
    </Input>

    <Output out_exchange>
    Module om_udp
    Host "graylog IP"
    Port "Graylog port"
    OutputType GELF
    Exec $sender_address = lc(serder_address)
    Exec $recipient_address = lc(recipient_address)
    Exec $SourceName = 'exchange_msgtrk_log';
    </Output>

    <Route exchange>
    Path in_exchange => out_exchange
    </Route>
    ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
    Works fine if I remove this two lines
    Exec $sender_address = lc(serder_address)
    Exec $recipient_address = lc(recipient_address)

    Please advice.
    Thank you.

    May 23, 2018 - 12:05am
  • b0ti's picture
    (NXLog)

    A semi-colon is needed to terminate statements, i.e.:

    Exec $sender_address = lc($sender_address);
    

    The $ sign is also needed to reference fields.

    May 23, 2018 - 12:10am
  • Malik.Z's picture

    Thank you.
    Must be something else wrong with my setup. Still not working.
    Config is below, if you can see something wrong, please let me know.

    define ROOT C:\Program Files (x86)\nxlog
    define BASEDIR "path"\MessageTracking

    Moduledir %ROOT%\modules
    CacheDir %ROOT%\data
    Pidfile %ROOT%\data\nxlog.pid
    SpoolDir %ROOT%\data
    LogFile %ROOT%\data\nxlog.log

    <Extension csv>
    Module xm_csv
    Fields $date_time, $client_ip, $client_hostname, $server_ip, $server_hostname, $source_context, $connector_id, $source, $event_id, $internal_message_id, $message_id, $network_message_id, $recipient_address, $recipient_status, $total_bytes, $recipient_count, $related_recipient_address, $reference, $message_subject, $sender_address, $return_path, $message_info, $directionality, $tenant_id, $original_client_ip, $original_server_ip, $custom_data
    Delimiter ,
    </Extension>

    <Input in_exchange>
    Module im_file
    File '%BASEDIR%\MSGTRK????????*-*.LOG' # Exports all logs in Directory
    SavePos TRUE
    Exec if $raw_event =~ /HealthMailbox/ drop();
    Exec if $raw_event =~ /^#/ drop();
    Exec csv->parse_csv();
    </Input>

    <Output out_exchange>
    Module om_udp
    Host "graylog IP"
    Port "graylog port"
    Exec $sender_address = lc($sender_address);
    Exec $recipient_address = lc($recipient_address);
    Exec $SourceName = 'exchange_msgtrk_log';
    OutputType GELF
    </Output>

    <Route exchange>
    Path in_exchange => out_exchange
    </Route>

    Thank you.

    May 23, 2018 - 7:32pm
  • b0ti's picture
    (NXLog)

    The error message in nxlog.log should tell you what's wrong and where.

    May 23, 2018 - 8:28pm
  • Malik.Z's picture

    2018-05-23 12:00:26 WARNING stopping nxlog service
    2018-05-23 12:00:26 WARNING nxlog-ce received a termination request signal, exiting...
    2018-05-23 12:00:32 ERROR Invalid OutputType 'GELF' at C:\Program Files (x86)\nxlog\conf\nxlog.conf:33
    2018-05-23 12:00:32 ERROR module 'out_exchange' has configuration errors, not adding to route 'exchange' at C:\Program Files (x86)\nxlog\conf\nxlog.conf:37
    2018-05-23 12:00:32 ERROR route exchange is not functional without output modules, ignored at C:\Program Files (x86)\nxlog\conf\nxlog.conf:37
    2018-05-23 12:00:32 WARNING no routes defined!
    2018-05-23 12:00:32 WARNING not starting unused module in_exchange
    2018-05-23 12:00:32 WARNING not starting unused module out_exchange
    2018-05-23 12:00:32 INFO nxlog-ce-2.9.1716 started
    2018-05-23 13:25:04 WARNING stopping nxlog service

    May 23, 2018 - 11:17pm
  • Malik.Z's picture

    so I guess i'm missing

    <Extension gelf>
    Module xm_gelf
    </Extension>

    ?

    May 23, 2018 - 11:57pm
  • Zhengshi's picture
    (NXLog)

    Yes. Without the extension loaded, the type doesn't exist yet.

    May 24, 2018 - 12:10am