Randomly TCP Output
I have a tcp output that has 3 hosts in sequence to send to graylog (failover), but I would like to "randomly" switch the ouputs to better distribute the load on the nodes. In my config example, 'graylog_1' will always receive all events. Is there a bultin solution for processor/output to send randomly to the multiple nodes?
<Output out_graylog> Module om_tcp FlowControl False Host 192.168.0.10:514 # graylog_1 Host 192.168.0.11:514 # graylog_2 Host 192.168.0.12:514 # graylog_3 </Output>
A viP/loadbalancer for graylog is not the solution I'm looking for, I want to understand the power of nxlog and its customization.
I'm afraid this isn't currently supported.
Host directives are used to define a fallback list in case the first destination becomes unavailable.
You could do some tricks, but it's unlikely to yield the results I surmise you'd want.
One of those tricks:
- mark each log entry with a random value 0, 1 or 2 in the Input definition (
- define a route with 3 outputs, each with one of your host from above
- create an exec to drop different 2 of the markers on each output
This will still strain NXLog's core, but you would have evenly distributed, and non-overlapping events going out to your 3 addresses.
I haven't tested this, so YMMV :-D
Also, you may or may not get higher performance than you'd get with a single output, and this is fragile, and may cause out of order delivery, since the outputs are decoupled.
There is also a significant potential impact on memory usage.