3
responses

Is there any way of splitting very long log messages in half or smaller portions? We are currently forwarding logs with NXLog to a SIEM system that has a 8kb limit on the messages and what goes beyond that limit is truncated and we don't want that. I tried to read the manual but did not find anything related to my problem. Help please?

AskedMarch 27, 2019 - 10:00am

Answer (1)

Is there any way of splitting very long log messages in half or smaller portions? We are currently forwarding logs with NXLog to a SIEM system that has a 8kb limit on the messages and what goes beyond that limit is truncated and we don't want that. I tried to read the manual but did not find anything related to my problem. Help please?

Depending on your log sources, it may be enough to trim the messages down. We have a lot of im_msvistalog users that do this currently as Windows EventLog messages can be quite wordy.
This can involve removing fields and using Regex to clean up events, removing the unnecessary portion.
Please see the following links.
https://nxlog.co/documentation/nxlog-user-guide-full#reducing-bandwidth
https://nxlog.co/documentation/nxlog-user-guide-full#reducing_trimming

As for actually splitting an event, there are no prepackaged solutions for this currently. We do have something on the roadmap though.
If you are using EE, you could probably use im_perl with logdata_new() if reading from a text file.
Another more universal solution could be to use xm_fileop's file_write() in combination with some regex that could split the line and send one half in current message while placing the other half into a new file using the file_write() and reading it back in with im_file. This is admittedly a bit messy, but should serve the purpose until the split feature is built.
In this same thought, log_info() and im_internal could be used.

Comments (2)

  • JaVa's picture

    Ok. Thanks for your reply. We are using EE and reading the logs from a file but in a RHEL environment. I don't think that it is possible to trim down those messages as our clients want to see the whole message for debug reasons. The over sized logs are some application logs that contain payload so they can be very long. Are those functions (ex. logdata_new) usage explained somewhere?

  • Zhengshi's picture
    (NXLog)

    Sure! Check out the link below. There is an example perl script as well.
    https://nxlog.co/documentation/nxlog-user-guide/im_perl.html

    If you are more familiar with Python :
    https://nxlog.co/documentation/nxlog-user-guide/im_python.html

    Or Ruby:
    https://nxlog.co/documentation/nxlog-user-guide/im_ruby.html