Have NXLog always forward file on disk (even if it hasn't changed)

Post a different question
1
answer

I have the following NXLog config below that reads a file from disk and forwards it to another host (syslog). By default, it appears NXLog only forwards the data in the log file if it has changed.

How can you configure 'nxlog.conf' so that it always forwards the file, even if the data hasn't changed?

<Input in>
Module im_file
File "C:\\myfile.txt"
</Input>

<Output out>
Module om_tcp
Host 127.0.0.1
Port 12345
</Output>

<Route 1>
Path in => out
</Route>

AskedDecember 7, 2018 - 5:14pm

Answer (1)

NXLog will only forward events that occur after it has started by default.
There are two directives that you may want to look at. SavePos and ReadFromLast. Set both to false and NXLog will read the entire file. Small warning though, every time you restart NXLog with those directives set, it will read the entire file from the start. You may want to read once, and then reset the directives.

Edit: Added link.
https://nxlog.co/documentation/nxlog-user-guide-full#im_file_config

AnsweredDecember 7, 2018 - 5:23pm

Comments (10)

  • demetri's picture

    I've looked at setting both of those to FALSE. It didn't help with my objective of always sending the file while NXLog is running, even if the contents haven't changed. I have a a schedule running every 60 seconds that regenerates this file - I was hoping to send this file every 60 seconds regardless of whether or not it has changed.

    <Extension _exec>
    Module xm_exec
    <Schedule>
    Every 60 sec
    Exec exec_async("C:\\create_myfile.txt.exe");
    </Schedule>
    </Extension>

    December 7, 2018 - 5:34pm
  • b0ti's picture
    (NXLog)

    What do you mean by if it has changed ? im_file was designed to read append-only files and to only read the appended records. It also handles truncation and detection of rotated log files when it assumes the full content is new.

    December 7, 2018 - 6:19pm
  • demetri's picture

    That design makes sense b0ti.

    But for me, I'd always like to send the file periodically, whether or not any changes have been detected. So for instance, consider this scenario:
    1. myfile has contents "abc" @ 9:00
    2. myfile has contents "abc" @ 9:10
    3. myfile has contents "def" @ 9:20

    I'd like the file to be sent in all 3 cases above via some config settings. Is this possible?

    December 7, 2018 - 6:52pm
  • Zhengshi's picture
    (NXLog)

    I see what you are trying to accomplish.
    Is this script that makes the log something that you have written? If so, it may be easier and more direct to simply write to the file in append mode instead.

    December 7, 2018 - 7:06pm
  • demetri's picture

    Hi,

    Yes I have written this script/exe (code). I thought about appending to the end of the file, but eventually it will get huge.

    I believe I can delete the file before the next scheduled run, which should then 'trick' NXLog to forward the data. But I am a little worried that there might be something 'in memory' that could prevent this.

    December 7, 2018 - 7:14pm
  • b0ti's picture
    (NXLog)

    Truncating the file and rewriting it is an anti-pattern that I would strongly advise against if you don't want data to be missed. As mentioned you should set up log rotation so that the file does not grow unbounded, this can be done by your script or by nxlog. See Log Rotation and Retention for more information.

    December 7, 2018 - 7:23pm
  • demetri's picture

    Thanks for the suggestion. I'm attempting to perform the rotation on the file, but am having issues. Using my config at the beginning of the post, can you give me an example of how to do this?

    Here is what I'm using now, but 'myfile.txt' is locked by NXLog's log rotation mechanism, so I can't write into myfile.txt at times

    <Extension _syslog>
    Module xm_syslog
    </Extension>

    <Extension _exec>
    Module xm_exec
    <Schedule>
    Every 10 sec
    Exec exec_async(<path-to-file-that-generates-C:\\myfile.txt>);
    </Schedule>
    </Extension>

    <Input in>
    Module im_file
    File "C:\\myfile.txt"
    </Input>

    <Output out>
    Module om_tcp
    Host 127.0.0.1
    Port 12345
    </Output>

    <Route 1>
    Path in => out
    </Route>

    <Extension _fileop>
    Module xm_fileop
    </Extension>

    <Output out2>
    Module om_file
    File "C:\\myfile.txt"
    <Schedule>
    Every 6 sec
    <Exec>
    file_cycle(file_name(), 2);
    reopen();
    </Exec>
    </Schedule>
    </Output>

    <Route 2>
    Path in => out2
    </Route>

    thanks.

    December 10, 2018 - 11:07pm
  • Zhengshi's picture
    (NXLog)

    A modification of the second example in the link suggested by @b0ti or the second link I sent should work for your uses.

    Note that you shouldn't need both of the below schedules. First one will check every hour and rotate when the file gets to 1M in size, keeping 7 copies. Second one will rotate daily at midnight and keep 7 copies.

    <Extension fileop>
    Module      xm_fileop

    # Check the log file size every hour and rotate if larger than 1 MB
    <Schedule>
        Every   1 hour
        Exec    if (file_size('C:\\myfile.txt') >= 1M) \
                    file_cycle('C:\\myfile.txt', 7);
    </Schedule>

    # Rotate log file every week on Sunday at midnight
    <Schedule>
        When    @daily
        Exec    file_cycle('C:\\myfile.txt', 7);
    </Schedule>
</Extension>

    December 10, 2018 - 11:10pm