4
responses

I have a im_file to om_tcp route on the community edition.
My applications starts generating logs at 5am and runs every 15 minutes.
I have observed that it essentially skips the first X lines and syncs things only after the x lines starting 5:15 and works fine throughout the day.
There is nothing in logs at 5am.
I have tried routing it to another om_file on the same machine, it works ok.

Can you please help me fix it?

Here is the config:

########################################
# Global directives #
########################################
User nxlog
Group nxlog

LogFile /var/log/nxlog/nxlog.log
LogLevel INFO

########################################
# Modules #
########################################
<Extension _syslog>
Module xm_syslog
</Extension>

<Extension _fileop>
Module xm_fileop
</Extension>

########################################
# Input #
########################################
<Input in-my-important-app>
Module im_file
File "/var/log/tomcat/my-important-app-*.log"
ReadFromLast True
SavePos True
Exec $Filename = "myApp/" +file_basename(file_name());
Exec $Hostname = hostname_fqdn();
</Input>

########################################
# Output #
########################################
# Output to destination
<Output out-destination>
Module om_tcp
Host [destination host]
Port [destination port]
OutputType Binary
</Output>

########################################
# Routes #
########################################
<Route to_destination>
Path in-my-important-app => out-destination
</Route>

AskedJuly 16, 2020 - 4:12am

Answer (1)

Cheers,

Most likely the reason is because of SavePos TRUE and ReadFromLast TRUE.
Those directives are making nxlog to remember, where did it stopped last time, to start from this exact point.
I suggest you to change those directives to FALSE and look what would happen.
Take a note that this will enable nxlog to read all provided files from start to finish every time you will start or restart it so be careful.

Regards, Arch

Comments (3)

  • javid90khan's picture

    I have tried that and the file Syncs fine if that is in place but the sync is not in order. The first X lines are synced after a significant delay. We need the logs synced in order to raise alerts when necessary.

  • javid90khan's picture

    setting that to false is not recommended and our other apps use the same config without any issues.

    coming to what I meant with in order:
    Events:
    1. 5:00 app runs - generates first set of logs
    2. 5:15 more logs generated
    3. 6:00 I checked the destintion server and only find logs from 5:15 <If I wanted to raise issues that happened at 5 I would find nothing>
    4. 8:00 I check the ldestination again and find all the logs correclty in place. 5:00 logs and 5:15 logs

    The destination file eventually became exact replica of source file with logs in correct order.