Randomly TCP Output

#1 Tulio Gomes (Last updated )

Hi Folks,

I have a tcp output that has 3 hosts in sequence to send to graylog (failover), but I would like to "randomly" switch the ouputs to better distribute the load on the nodes. In my config example, 'graylog_1' will always receive all events. Is there a bultin solution for processor/output to send randomly to the multiple nodes?

Config example:

<Output out_graylog>
        Module om_tcp
        FlowControl False    
        Host # graylog_1    
        Host # graylog_2    
        Host # graylog_3

A viP/loadbalancer for graylog is not the solution I'm looking for, I want to understand the power of nxlog and its customization.

#2 gahorvath Nxlog ✓

Hi Tulio,

I'm afraid this isn't currently supported. 

Multiple Host directives are used to define a fallback list in case the first destination becomes unavailable. 

You could do some tricks, but it's unlikely to yield the results I surmise you'd want.

One of those tricks:

  • mark each log entry with a random value 0, 1 or 2 in the Input definition (get_rand())
  • define a route with 3 outputs, each with one of your host from above
  • create an exec to drop different 2 of the markers on each output

This will still strain NXLog's core, but you would have evenly distributed, and non-overlapping events going out to your 3 addresses.

I haven't tested  this, so YMMV :-D

Also, you may or may not get higher performance than you'd get with a single output, and this is fragile, and may cause out of order delivery, since the outputs are decoupled.

There is also a significant potential impact on memory usage.