Splitting long log messages?

Tags:

#1 JaVa

Is there any way of splitting very long log messages in half or smaller portions? We are currently forwarding logs with NXLog to a SIEM system that has a 8kb limit on the messages and what goes beyond that limit is truncated and we don't want that. I tried to read the manual but did not find anything related to my problem. Help please?

#2 Zhengshi Nxlog ✓
#1 JaVa
Is there any way of splitting very long log messages in half or smaller portions? We are currently forwarding logs with NXLog to a SIEM system that has a 8kb limit on the messages and what goes beyond that limit is truncated and we don't want that. I tried to read the manual but did not find anything related to my problem. Help please?

Is there any way of splitting very long log messages in half or smaller portions? We are currently forwarding logs with NXLog to a SIEM system that has a 8kb limit on the messages and what goes beyond that limit is truncated and we don't want that. I tried to read the manual but did not find anything related to my problem. Help please?

Depending on your log sources, it may be enough to trim the messages down. We have a lot of im_msvistalog users that do this currently as Windows EventLog messages can be quite wordy.
This can involve removing fields and using Regex to clean up events, removing the unnecessary portion.
Please see the following links.
https://nxlog.co/documentation/nxlog-user-guide-full#reducing-bandwidth
https://nxlog.co/documentation/nxlog-user-guide-full#reducing_trimming

As for actually splitting an event, there are no prepackaged solutions for this currently. We do have something on the roadmap though.
If you are using EE, you could probably use im_perl with logdata_new() if reading from a text file.
Another more universal solution could be to use xm_fileop's file_write() in combination with some regex that could split the line and send one half in current message while placing the other half into a new file using the file_write() and reading it back in with im_file. This is admittedly a bit messy, but should serve the purpose until the split feature is built.
In this same thought, log_info() and im_internal could be used.