om_perl and xm_perl questions | Log collection solutions

#1 davidatpinger 6 years ago

I've got a route where the input is from im_tcp and I'd like to output the incoming data to a set of files, but select a specific file based on the content of the data.

I can easily do this in om_perl with something that examines the data, figures out the correct file, then stuffs it into that file, but that would appear to require a file open and a file close per event, which seems non-optimal at scale. So - is the code identified by the PerlCode directive basically run for each event, or is it stateful so I can maintain an array of open files and just stuff the event in the correct one?

Alternatively, can I use xm_perl to find the correct filename and set that as an environment valiable that could be consumed by om_file?

I'm open to other efficient methods for accomplishing this, if there are ideas out there. Thanks!

Permalink

#2 b0ti Nxlog ✓ 6 years ago

#1 davidatpinger

I've got a route where the input is from im_tcp and I'd like to output the incoming data to a set of files, but select a specific file based on the content of the data. I can easily do this in om_perl with something that examines the data, figures out the correct file, then stuffs it into that file, but that would appear to require a file open and a file close per event, which seems non-optimal at scale. So - is the code identified by the PerlCode directive basically run for each event, or is it stateful so I can maintain an array of open files and just stuff the event in the correct one? Alternatively, can I use xm_perl to find the correct filename and set that as an environment valiable that could be consumed by om_file? I'm open to other efficient methods for accomplishing this, if there are ideas out there. Thanks!

This should already work out-of-the-box using a dynamic value for the File directive without having to resort to xm_perl or om_perl. Here is an example to illustrate that:

<Output out>
    Module      om_file
    <Exec>
      if $raw_event =~ /foo/ $filename = 'foofile'
      else $filename = 'barfile';
    </Exec>
    File        '/var/log' + $filename + '.log'
</Output>

If this doesn't cut it and you still need xm_perl or om_perl then you should be able to use global variables in the perl code where you could maintain your files in a hash or array. Variables declared inside the method call that you invoke with process() are cleaned up obviously after the call returns but global variables should not.