9
responses

<Input Result_Log>
Module im_file
File "C:\\xxx\\Result*.log"
SavePos TRUE
</Input>

above is my nxlog input config.
Result-3156.log is my log file name and 3156 is pid.
The log file will rotate when it reaches the max size and generate Result-3156.log.1 file.
I find some logs loss at the bottom of Result-123.log.1
And nxlog's log shows "2018-03-01 20:48:02 INFO inode changed for 'C:\xxx\Result-3156.log' (25746->25799): reopening possibly rotated file" at the same time point.

I suspect this is caused by the monitor interval 1 second.

Is it a bug ? or nxlog can not guarantee this scenario ? or something can do to avoid it ?

AskedMarch 2, 2018 - 4:04am

Comments (2)

  • b0ti's picture
    (NXLog)

    The im_file module has been significantly enhanced in the NXLog EE beta that is scheduled for release this month. If you could test the trial and confirm whether the issue is still present that would help us and the CE could be also updated sooner.

  • SamRui's picture

    I tried NXLog EE beta. And problem still exists. I guess the scenario is like this, when log file rotate from file 'xxx.log' to 'xxx.log.1', because the file 'xxx.log.1' does not match im_file module file name pattern, the logs are lost during nxlog monitor interval 1 second. If you have specification about rotate file process, that will help me a lot.

    By the way, when I install "nxlog-3.99.3332-x64_trial.msi" version, the default location is "C:\Program Files (x86)\nxlog\", not "C:\Program Files\nxlog\".

Answer (1)

Log rotation (i.e. rename) should not cause im_file to miss any data while the file is kept open as it also keeps track of the file size but there are a number of parameters such as ActiveFiles, RenameCheck that are also relevant. See the docs for more info.

If you can reproduce this manually (e.g. via a script that fills the file and rotates it) then feel free to provide more information and we will investigate.

Comments (6)

  • SamRui's picture

    Hi, b0ti,

    Thank you for your reply.

    I extract the write log part from my product code. I test it and still miss logs.

    Following is my test program and nxlog config. If you can provide a email address, I will provide my whole test program and test result.

    C# code part:
    EventLogger.cs :

    using System;
    using System.Text;
    using log4net;
    using log4net.Appender;
    using log4net.Core;
    using log4net.Repository.Hierarchy;

    namespace TrendMicro.CloudAtlas.Common.EventLog
    {
    public class EventLogger
    {
    private ILog _log;
    private readonly string _path;
    private readonly string _loggerName;
    private string _appenderName = "CloudAtlasAppender";
    private string _layout = "[%date{yyyy-MM-dd HH:mm:ss}] [%level] %m%n";
    private int _maxSizeRollBackups = 10;
    private string _maxFileSize = "10MB";
    private bool _withBomUtf8 = true;

    public bool WithBomUtf8
    {
    set { _withBomUtf8 = value; }
    get { return _withBomUtf8; }
    }

    public string Layout
    {
    set { _layout = value; }
    get { return _layout; }
    }

    public string AppenderName
    {
    set { _appenderName = value; }
    get { return _appenderName; }
    }

    public string MaxFileSize
    {
    set { _maxFileSize = value; }
    get { return _maxFileSize; }
    }

    public EventLogger(string logFolder, string logFile, string loggerName, string strMaxFileSize, int nMaxSizeRollBackups, bool withBom = true)
    {
    var sb = new StringBuilder(logFolder);
    if (logFolder.EndsWith("\\") == false) sb.Append("\\");
    sb.Append(logFile);

    _path = sb.ToString();
    _loggerName = loggerName;
    _withBomUtf8 = withBom;
    _maxFileSize = (!String.IsNullOrEmpty(strMaxFileSize)) ? strMaxFileSize : _maxFileSize;
    _maxSizeRollBackups = ((nMaxSizeRollBackups > _maxSizeRollBackups) && (nMaxSizeRollBackups <= 100)) ? nMaxSizeRollBackups : _maxSizeRollBackups;

    CreateLogger();
    }

    public void CreateLogger()
    {
    try
    {
    var appender = new RollingFileAppender
    {
    Name = _appenderName,
    File = _path,
    AppendToFile = true,
    RollingStyle = RollingFileAppender.RollingMode.Size,
    MaxSizeRollBackups = _maxSizeRollBackups,
    MaximumFileSize = _maxFileSize,
    StaticLogFileName = false,
    Threshold = Level.All,
    };

    var layout = new log4net.Layout.PatternLayout
    {
    ConversionPattern = _layout,
    };
    layout.ActivateOptions();

    appender.Layout = layout;
    appender.Encoding = new UTF8Encoding(WithBomUtf8);
    appender.ActivateOptions();

    var repository = (Hierarchy) LogManager.GetRepository();

    var logger = (log4net.Repository.Hierarchy.Logger) repository.GetLogger(_loggerName);
    logger.AddAppender(appender);
    logger.Level = Level.All;
    _log = LogManager.GetLogger(_loggerName);

    }
    catch (Exception)
    {
    }
    }

    public void Write(string info)
    {
    if (_log != null)
    {
    _log.Debug(info);

    }

    }
    }
    }

    Program.cs

    namespace hello
    {
    class Program
    {
    static void Main(string[] args)
    {

    BasicConfigurator.Configure();
    ESViolationLogBase m_esViolationLog = new ESViolationLogBase();
    m_esViolationLog.tm_task_id = "hello_task_id";
    m_esViolationLog.tm_scan_type = "hello_scan_type";

    EventLogger loger = null;
    string ScannerLogFileName = "DDAaaSScannerResult";

    var codeBase = Assembly.GetExecutingAssembly().CodeBase;
    var uri = new UriBuilder(codeBase);
    var path = Uri.UnescapeDataString(uri.Path);
    var taskFolder = Path.GetDirectoryName(path);
    string folder = string.Format("{0}\\{1}", taskFolder, ScannerLogFileName);

    string file_name = string.Format("{0}-{1}.log", ScannerLogFileName, Process.GetCurrentProcess().Id);

    loger = new EventLogger(folder, file_name, ScannerLogFileName, "1MB", 25, false);

    for(int i = 0; i < 300000; i++)
    {
    m_esViolationLog.tm_trace_id = "nxlog_" + i.ToString() + "_nxlog";
    string strLog = JsonConvert.SerializeObject(m_esViolationLog, new JsonSerializerSettings
    {
    NullValueHandling = NullValueHandling.Ignore
    });
    loger.Write(strLog);
    }

    Console.ReadKey();

    }
    }
    }

    nxlog part:

    <Input RT_DDAaaS_Result_Log>
    Module im_file
    File "C:\\Program Files\\Trend Micro\\CloudAtlas\\DDAaaSDelegator\\DDAaaSScannerResult\\DDAaaSScannerResult*.log"
    SavePos TRUE
    </Input>

    <Output logServer4Scanner>
    Module om_file
    File "C:\\Program Files\\Trend Micro\\CloudAtlas\\DDAaaSDelegator\\out.log"
    </Output>

    # 10Mb mem buffer
    <Processor memBufScanner>
    Module pm_buffer
    MaxSize 10240
    Type Mem
    WarnLimit 2560
    </Processor>

    # 1Gb disk buffer
    <Processor diskBufScanner>
    Module pm_buffer
    MaxSize 1048576
    Type Disk
    WarnLimit 786432
    </Processor>

    <Route scanner>
    Priority 1
    Path RT_DDAaaS_Result_Log => diskBufScanner => memBufScanner => logServer4Scanner
    </Route>

  • b0ti's picture
    (NXLog)

    Thanks for the details. We were able to confirm this issue and a fix is now on the way into the next build of the NXLog Enterprise Edition. This should also end up in the Community Edition at some point.

  • b0ti's picture
    (NXLog)

    We now have a new version of the NXLog EE beta out (nxlog-3.99.3438) that should address this. It would be great if you could confirm that this helps.

  • smoheiss's picture

    Hello,

    I encounter the exactly same behavior with the community edition ( nxlog-ce-2.10.2150).

    We really need this bug fixed.

    We tried the Enterprise Edition (Trial) which has not this behavior.

    Can you do something for us and fix this behavior in the community edition PLEASE ?

    Thank you

  • Zhengshi's picture
    (NXLog)

    Hello,

    I can understand how this issue is frustrating.

    I verified today that there is a fix coming in NXLog CE for this as it is indeed in the pipeline.

    We will update when there is more information to share.

    Thank you,
    Jesse