im_tcp with InputType LineBased occasional truncated lines
Hi folks,
We have an NXLog CE 3.2.2329 configuration using im_tcp with InputType LineBased, receiving blobs of JSON separated by newlines. The received data is passed to xm_perl and then relayed to an output.
Under some conditions where there may be thousands of events per second, there is a very rare chance for one event to be truncated at the end yielding an incomplete json blob. We've verified that the input data coming from upstream is valid, fully formed JSON - the emitter encoding logs look OK. The preceeding and following lines are also received and parsed successfully.
Has anybody else experienced similar behavior? At this point we suspect a bug in NXLog, considering that it occurs very infrequently (2-3 days between recurrences) and it only occurs during periods of relatively high load (normal load is ~150 events per second, load when issue occurs is ~800/s+)
Relevant configuration:
<Extension gelf>
Module xm_gelf
</Extension>
<Extension perl>
Module xm_perl
PerlCode /etc/nxlog/scripts/parse_mist.pl
</Extension>
<Input mist_tcp>
Module im_tcp
Host 0.0.0.0
Port 32768
InputType LineBased
<Exec>
perl_call("parse_mist");
if $nxlog_internal_drop_log drop();
</Exec>
</Input>
<Output graylog>
Module om_udp
Host x
Port 12201
OutputType GELF
</Output>
<Route graylog>
Path mist_tcp => graylog
</Route>
Hi @1sgcaveney,
Please note that events can be truncated when a module receives (for the input modules) or sends (for the output modules) data above the Buffer Size (The default buffer size is 65000 bytes). In such cases, a log message such as the following will be returned in the logs:
Data size (67200) is over the free buffer limit (65000), will be truncated.
To mitigate this, please increase the Buffer Size. The buffer size you specify should be large enough to hold the biggest chunk of data you expect the input/output module to write. Note that setting a very large buffer size is unlikely to improve performance and will increase the probability of data loss if NXLog terminates unexpectedly.
Ayodele
