9
responses

I'm struggling with rotating syslog files at midnight so that they are named "YYYY-MM-DD.log" (and contain log records for that date).

I wonder if anyone has an example of the best way to achieve this.  The manual could use an example like this.

Thanks

Phil

 

 

AskedMarch 26, 2017 - 8:01am

Comments (9)

  • atmosx's picture

    Hi Biggsy,

    There are ways to rotate files. You can go like:

    <Extension _fileop>
        Module      xm_fileop
        # Rotate our log file every day
        <Schedule>
          When @daily
          Exec    if file_exists(%LOGFILE%) file_cycle(%LOGFILE%, 8);
        </Schedule>
    </Extension>

     

    This will rename the file, to <filename>.1 which is the standard format for log rotation. Then you can use "om_exec" module to manage to rename the file to what you want. To get yesterday's date you can use this bash command $(date  --date="1 days ago" +"%Y-%m-%d").

     

    UPDATE: You can try using:

     strftime(now() - 24*60*60, "%Y%m%d%H%M%S");

  • Biggsy's picture

    Many thanks, atmosx.

    I am trying to replace the free verison of Kiwi syslog server - it has some (probably deliberate) performance limitations.  A replacement is better than changing all the backend processing of logs. Coming from Kiwi I guess I was looking for an easy way to name the initial log (at nxlog startup) with today's date and rollover to the next date automatically at midnight.

    For now, I've named the output file as "today.log" and implemented the following, based on your suggestion.

     

    <Output out>
        Module      om_file
        Exec         parse_syslog_bsd();
        Exec        $raw_event = $EventReceivedTime + "\t" + $SyslogFacility + "." + $SyslogSeverity + "\t" + $MessageSourceAddress + "\t" + $raw_event;
        File        "%FILENAME%"
        <Schedule>
            When    @daily
            <Exec>

                $DatedFile = strftime(now() - 86400, "%Y-%m-%d") + ".log";
                out->rotate_to("D:\syslog\logs\" + $DatedFile);

            </Exec>
            </Schedule>   
    </Output>

     

    I've never seen that means of performing arithmetic on dates/times before, so thanks again.  I guess I'll see if it works tonight :-)

     

  • Biggsy's picture

    This didn't work:

    <Schedule>
            When    @daily
            <Exec>

                $DatedFile = strftime(now() - 86400, "%Y-%m-%d") + ".log";
                out->rotate_to("D:\syslog\logs\" + $DatedFile);

            </Exec>
            </Schedule> 

    I changed it to the below but still no joy - the arithmetic throws an error: 

    "... statement execution has been aborted; missing logdata, assignment possibly after drop()".

        <Schedule>

            When    @daily
            <Exec>

                $DatedFile = "D:\syslog\logs\" + strftime(now() - 86400, "%Y-%m-%d") + ".log";
                out->rotate_to($DatedFile);

            </Exec>
            </Schedule>   

    I think I'll have to look at an external Powershell script to do the rename to yesterday's date.

     

  • Biggsy's picture

    What I have now is similar to something I found on the mailing list:

    <Output out>
        Module      om_file
        Exec         parse_syslog_bsd();
        Exec        $raw_event = $EventReceivedTime + "\t" + $SyslogFacility + "." + $SyslogSeverity + "\t" + $MessageSourceAddress + "\t" + $raw_event;
        File        ("%OUTDIR%\\" + strftime(now(), "%Y-%m-%d")+ ".log")
    </Output>

    I can't believe it was that simple.  Am I missing something?

     

    My concern was that 2017-04-06.log would be left open once 2017-04-07.log started to be written but that doesn't appear to be the case.

     

    I also guess the overhead of comparing the date to the currently open file name isn't going to have a big impact on performance.  Would that be a valid assumption?

     

  • b0ti's picture
    (NXLog)

    $DatedFile is a field in the event record which does not exist in the scope of <Schedule>. There are module variables for this purpose but this should work:

    out->rotate_to("D:\syslog\logs\" + strftime(now() - 86400, "%Y-%m-%d") + ".log");

     

  • Biggsy's picture

    Thank you, b0ti.  I see the problem now.  If the date arithmetic works I might revert to that.

     

    Are there any disadvantages to the way I have it now:

     

    <Output out>
        Module      om_file
        Exec         parse_syslog_bsd();
        Exec        $raw_event = $EventReceivedTime + "\t" + $SyslogFacility + "." + $SyslogSeverity + "\t" + $MessageSourceAddress + "\t" + $raw_event;
        File        ("%OUTDIR%\\" + strftime(now(), "%Y-%m-%d")+ ".log")
    </Output>

     

    I think I could probably cut the "Exec   parse_syslog_bsd()" as it was only to extract the facility and severity, which I probably don't need and was only to emulate the output from Kiwi.

  • b0ti's picture
    (NXLog)

    Both should work. The only difference between the two schemes is that for the first you only have YYYY-MM-DD.log files whereas the second scheme rotates foo.log to YYYY-MM-DD.log every day at midnight. It's a matter of preference but some people (and software) prefer foo.log to be always present.

    Also note that none of the above guarantees that you won't have logs generated on a different day in YYYY-MM-DD.log, i.e. a log from the previous day might end up in the file because you are using $EventReceivedTime and now() instead of the actual timestamp in the log. If that's a concern you should use $EventTime as follows:

    File    "/var/log/nxlog/out_" + strftime($EventTime, "%Y%m%d")

    For $EventTime to be present you need parse_syslog().

  • Biggsy's picture

    Thanks again.

     

    I understand that some logs from the beginning or end of one day may end up in the wrong file.  That's something I've lived with for a long time now :-)  

     

    I also understand the reasoning that writing to foo.log and rotating it to YYYY-MM-DD.log would be preferable.  More experimentation required on my part.

     

    Thank you very much for your help and explanations.

  • Biggsy's picture

    Hi b0ti and atmosx,

    Indeed, both of the following methods work:

     

    <Output out>
        Module      om_file
        Exec        $raw_event = $EventReceivedTime + "\t" + $MessageSourceAddress + "\t" + $raw_event;
        File        "%FILENAME%"
        <Schedule>
            When    @daily
            <Exec>
                out->rotate_to("D:\syslog\logs\" + strftime(now() - 86400, "%Y-%m-%d") + ".log");
            </Exec>
        </Schedule>   
    </Output>

     

    <Output out>
        Module      om_file
        Exec        $raw_event = $EventReceivedTime + "\t" + $MessageSourceAddress + "\t" + $raw_event;
        File        ("%OUTDIR%\\" + strftime(now(), "%Y-%m-%d")+ ".log");
    </Output>

     

    I'll go with the first, as recommended.

     

    Thank you both for your help.

     

    BTW, whats the trick to getting text into code boxes in this forum?

Answers (0)