Dynamic filename in log collector

Tags:

#1 ttyserial

When using NXLog as syslog collector, it's a good idea filter messages by hostname and to construct file name/path based on it and info in the message (facility and/or program name). What I thought about is something like this:

<Input tcp_in>
    Module      im_tcp
    Port        514
    Exec        parse_syslog_ietf();
    Exec        if $SyslogFacility == 'MAIL' { \
                    $Filetype = 'mail'; \
                } else if $SyslogFacility == 'FTP' { \
                    $Filetype = 'ftp'; \
                } else { \
                    $Filetype = 'messages'; \
                }
</Input>

<Output remotelog_out>
    Module      om_file
    CreateDir   TRUE
    File        "/var/log/" + $Hostname + "/" + $Filetype + ".log"
</Output>

<Route>
    tcp_in => remotelog_out
</Route>

It works, but performance is ... The problem seems to be that om_file can keep only one file open and for every new message if filename doesn't match already open one, the old one is closed and the new one is opened. This isn't of course optimal - in case of many hosts and log types significant amount of time is spent in closing/opening.

Is there a better way? I'm holding it wrong? :)

#2 adm Nxlog ✓
#1 ttyserial
When using NXLog as syslog collector, it's a good idea filter messages by hostname and to construct file name/path based on it and info in the message (facility and/or program name). What I thought about is something like this: <Input tcp_in> Module im_tcp Port 514 Exec parse_syslog_ietf(); Exec if $SyslogFacility == 'MAIL' { \ $Filetype = 'mail'; \ } else if $SyslogFacility == 'FTP' { \ $Filetype = 'ftp'; \ } else { \ $Filetype = 'messages'; \ } </Input> <Output remotelog_out> Module om_file CreateDir TRUE File "/var/log/" + $Hostname + "/" + $Filetype + ".log" </Output> <Route> tcp_in => remotelog_out </Route> It works, but performance is ... The problem seems to be that om_file can keep only one file open and for every new message if filename doesn't match already open one, the old one is closed and the new one is opened. This isn't of course optimal - in case of many hosts and log types significant amount of time is spent in closing/opening. Is there a better way? I'm holding it wrong? :)

The om_file module closes and opens the file if the name of the file changes. This could be a performance bottleneck indeed but it needs to be profiled to make sure. An fd cache mechnism that's already employed in im_file could be implemented here to help with this.

Get in touch if you would like to get this added under a support contract.