4
responses

Hi everybody,
To protect the logs from loss, I turned on the recommended settings in client's config:

PersistLogqueue TRUE
SyncLogqueue TRUE
CacheFlushInterval always
CacheSync TRUE

But now I see that parts of log are duplicated much more often than before these setting was enabled.
Why cache in memory works better? In case with cache on disk I hope to see best result.

Thank you so much!

AskedOctober 26, 2018 - 8:29am

Answer (1)

None of those settings should duplicate events.

PersistLogqueue TRUE Writes the normal log queue to disk instead of RAM. Note that logqueue is written to disk when NXLog shuts down, regardless. SyncLogqueue TRUE Immediately syncs the logqueue to disk after each entry.
CacheFlushInterval always Indicates that the in-memory value will be flushed to cache file after everytime a module sets a value. Default is 5seconds between flush CacheSync TRUE Bypasses filesystem buffering to force sync the cache to disk. Without this setting, the in-memory cache gets flushed to cache file and then waits on the filesystem to sync to disk.

If you are getting duplicate events then there is likely another setting in your config or source that is causing the duplications.
I would reference the rest of your config and see if something else may be causing it. As a troubleshooting step you could also flip the values on each of those settings to see which may be related to the duplicate messages.

Comments (3)

  • hatula's picture

    Thank you Zhengshi!

    I realy see that after enable this configuration my logs come chaotic compared to the buffer in RAM.

    I send via nxlog files from c:\test\YYYY-MM-DD.log. This is my client config:

    PersistLogqueue TRUE
    SyncLogqueue TRUE
    CacheFlushInterval always
    CacheSync TRUE
    
    <Extension multiline>
        Module          xm_multiline
        HeaderLine      /^\[\d\d\d\d.\d\d.\d\d \d\d.\d\d\.\d\d :/
    </Extension>
    
    <Input in>
        Module im_file
        File   'C:\test\????-??-??.log'
        BufferSize  2500000
        SavePos  TRUE
        InputType multiline
        Exec $Message = $raw_event;
        Exec if $Message == '' drop();
        Exec if file_name() =~ /([^\\]+)$/ $FileName = $1;
        Exec $Hostname = TESTCLIENT01";
        Exec $EventTime = $EventReceivedTime;
    </Input>
    
    <Output out>
        BufferSize  9500000
        Module      om_batchcompress
        Host        X.X.X.X
        Port        1514
    </Output>
    
    <Route client>
        Path   in => out
    </Route>
    

    This is my server config:

    <Input in>
        BufferSize  9500000
        Module      im_batchcompress
        ListenAddr  0.0.0.0
        Port        1514
    </Input>
    
    <Output out>
        Module  om_file
        BufferSize  9500000
        File    '/tmp/' + $Hostname + '/' + $FileName
        Exec if $FileName =~ s/-/./g;
        CreateDir TRUE
    </Output>
    
    <Route server>
        Path   in => out
    </Route>
    
    

  • Arkadiy's picture
    (NXLog)

    Hello,

    Don't you mind to drop us some examples of input and output logs?
    Because I can't see anything in your config which could duplicate entries so I would try to reproduce it.

    Regards, Arch