Randomly TCP Output
Hi Folks,
I have a tcp output that has 3 hosts in sequence to send to graylog (failover), but I would like to "randomly" switch the ouputs to better distribute the load on the nodes. In my config example, 'graylog_1' will always receive all events. Is there a bultin solution for processor/output to send randomly to the multiple nodes?
Config example:
<Output out_graylog>
Module om_tcp
FlowControl False
Host 192.168.0.10:514 # graylog_1
Host 192.168.0.11:514 # graylog_2
Host 192.168.0.12:514 # graylog_3
</Output>A viP/loadbalancer for graylog is not the solution I'm looking for, I want to understand the power of nxlog and its customization.
Hi Tulio,
I'm afraid this isn't currently supported.
Multiple Host directives are used to define a fallback list in case the first destination becomes unavailable.
You could do some tricks, but it's unlikely to yield the results I surmise you'd want.
One of those tricks:
- mark each log entry with a random value 0, 1 or 2 in the Input definition (
get_rand()) - define a route with 3 outputs, each with one of your host from above
- create an exec to drop different 2 of the markers on each output
This will still strain NXLog's core, but you would have evenly distributed, and non-overlapping events going out to your 3 addresses.
I haven't tested this, so YMMV :-D
Also, you may or may not get higher performance than you'd get with a single output, and this is fragile, and may cause out of order delivery, since the outputs are decoupled.
There is also a significant potential impact on memory usage.
Gabor
