6
responses

Hey!

I can think of several ways to solve this, but I'm looking to see if anyone has a better idea. I'm getting some logs that are uploaded as a gzipped file. I need to uncompress the file and then send the log content to several destinations, but for this purpose, let's assume that I'm using om_tcp as the destination. I'm considering a couple of options, but has anyone already done something like this or have a brilliant idea for handling it?

Thanks!

AskedApril 12, 2018 - 8:48pm

Comments (2)

  • tape's picture

    Hello,

    could you be more clearer, what you would like to achieve? What is the problem with uncompressing the files, read it with im_file, than send it with om_tcp?

    Regards,

    Peter

  • davidatpinger's picture

    I don't want to have something that has to watch for new files to appear and uncompress them - nxlog is far better than I am at watching for new files.

Answer (1)

Built-in compression support is on the roadmap. Unfortunately this didn't make it into the latest NXLog EE v4.0 release.

Would something like the following work? You can use a script via im_exec that uncompresses the files:

#!/bin/sh

while true; do
 for i in *.log.gz; do
   zcat $i; 
   mv $i /processed/;
 done
 sleep 1;
done

Comments (3)

  • davidatpinger's picture

    That's along the lines of what I've been considering. The files in question will appear in a directory (they are uploaded to a web server), and may be mixed with other files. I don't really want to write something to watch the directory for changes (since nxlog is very good at it and my code is not), so I was thinking of leveraging im_exec. Or, just using im_file, and running everything through xm_perl (the device is a CentOS 6 box, so I can't leverage the new python module - thanks for that, by the way!) to unwrap and ship the lines in each file back to om_tcp. There's several options...just trying to see what's going to be the least painful in the short term. I guess im_exec makes more sense, since im_file would return "lines", which isn't really the right construct for a compressed file. Any idea when compression will appear?

  • b0ti's picture
    (NXLog)

    With im_file you can't transfer arbitrary binary data. This capability could be added I think but it's a little bit out-of-scope (i.e. nxlog is not rsync).

    Built-in compression support might arrive in the NXLog EE later this year (Q3?) but there is no ETA yet.

  • davidatpinger's picture

    Yeah, that makes sense. I'm building a small proof of concept setup to see what I can break with it, and using im_exec as the start of the whole thing.

    I'm looking forward to the compression support! Glad to hear it's at least on the roadmap. Thanks!