Cache on disk works not good

Post a different question
4
responses

Hi everybody,
To protect the logs from loss, I turned on the recommended settings in client's config:

PersistLogqueue TRUE
SyncLogqueue TRUE
CacheFlushInterval always
CacheSync TRUE

But now I see that parts of log are duplicated much more often than before these setting was enabled.
Why cache in memory works better? In case with cache on disk I hope to see best result.

Thank you so much!

AskedOctober 26, 2018 - 8:29am

Answer (1)

None of those settings should duplicate events.

PersistLogqueue TRUE Writes the normal log queue to disk instead of RAM. Note that logqueue is written to disk when NXLog shuts down, regardless. SyncLogqueue TRUE Immediately syncs the logqueue to disk after each entry.
CacheFlushInterval always Indicates that the in-memory value will be flushed to cache file after everytime a module sets a value. Default is 5seconds between flush CacheSync TRUE Bypasses filesystem buffering to force sync the cache to disk. Without this setting, the in-memory cache gets flushed to cache file and then waits on the filesystem to sync to disk.

If you are getting duplicate events then there is likely another setting in your config or source that is causing the duplications.
I would reference the rest of your config and see if something else may be causing it. As a troubleshooting step you could also flip the values on each of those settings to see which may be related to the duplicate messages.

AnsweredOctober 30, 2018 - 5:27pm

Comments (3)

  • hatula's picture

    Thank you Zhengshi!

    I realy see that after enable this configuration my logs come chaotic compared to the buffer in RAM.

    I send via nxlog files from c:\test\YYYY-MM-DD.log. This is my client config:

    PersistLogqueue TRUE
SyncLogqueue TRUE
CacheFlushInterval always
CacheSync TRUE

<Extension multiline>
    Module          xm_multiline
    HeaderLine      /^\[\d\d\d\d.\d\d.\d\d \d\d.\d\d\.\d\d :/
</Extension>

<Input in>
    Module im_file
    File   'C:\test\????-??-??.log'
    BufferSize  2500000
    SavePos  TRUE
    InputType multiline
    Exec $Message = $raw_event;
    Exec if $Message == '' drop();
    Exec if file_name() =~ /([^\\]+)$/ $FileName = $1;
    Exec $Hostname = TESTCLIENT01";
    Exec $EventTime = $EventReceivedTime;
</Input>

<Output out>
    BufferSize  9500000
    Module      om_batchcompress
    Host        X.X.X.X
    Port        1514
</Output>

<Route client>
    Path   in => out
</Route>

    This is my server config:

    <Input in>
    BufferSize  9500000
    Module      im_batchcompress
    ListenAddr  0.0.0.0
    Port        1514
</Input>

<Output out>
    Module  om_file
    BufferSize  9500000
    File    '/tmp/' + $Hostname + '/' + $FileName
    Exec if $FileName =~ s/-/./g;
    CreateDir TRUE
</Output>

<Route server>
    Path   in => out
</Route>

    October 31, 2018 - 3:53pm
  • Arkadiy's picture

    Hello,

    Don't you mind to drop us some examples of input and output logs?
    Because I can't see anything in your config which could duplicate entries so I would try to reproduce it.

    Regards, Arch

    February 27, 2020 - 12:01am