NXLog client failed to read config cache

Post a different question
20
responses

Hi,

This error is found in the logs nxlog client enterprise from time to time:

2018-12-26 00:00:00 INFO LogFile C:\nxlog\data\nxlog.log reopened
2018-12-26 00:00:00 INFO inode changed for 'C:\nxlog\data\nxlog.log' (79652->79655): reopening possibly rotated file
2018-12-26 00:00:11 ERROR failed to read config cache; string expected for config cache version
2018-12-26 00:00:11 ERROR exiting...

I use recommended configuration for nxlog client:

PersistLogqueue TRUE 
SyncLogqueue TRUE  
CacheFlushInterval always  
CacheSync TRUE 

<Input in>
    Module im_file
    File   'C:\logs\????-??-??.log'
    BufferSize  2500000
    SavePos  TRUE
</Input>

<Output out>
    BufferSize  9500000
    Module      om_batchcompress
    Host        192.168.1.100
    Port        1514 
</Output>

Why is this happening? Thanks for any ideas!

I use

  • client nxlog-3.2.2002
  • server nxlog-4.1.4046
AskedDecember 27, 2018 - 1:47pm

Answers (2)

This issue is normally seen when an older version of NXLog creates the /opt/nxlog/var/spool/nxlog/configcache.dat file, a newer version of NXLog is installed, and then tries to read the old version.
This is due to a format change in configcache.dat between certain versions (adding a version number in your case).

Resolution would be to stop NXLog service, remove /opt/nxlog/var/spool/nxlog/configcache.dat, start NXLog service.

Your 3.2.2002 version is quite old. It would be a good idea to update it when you can. There are a lot of fixes between that version and current.

Thank you,
jesse

AnsweredDecember 28, 2018 - 2:09am

Comments (16)

  • hatula's picture

    Thank you for your reply!

    I updated my nxlog client to version 4.1.4016 and still I have this error:

    ERROR failed to read config cache; config cache version mismatch, expected NX-CC-VERSION 001

    Any ideas? Thanks a lot!

    January 9, 2019 - 7:44am
  • Zhengshi's picture
    (NXLog)

    It sounds like your file is getting corrupted somehow. We don't have any other issues open that are similar.
    You could try changing CacheDir to somewhere else and see if that helps. Another volume, if you have one available.

    January 11, 2019 - 9:15pm
  • hatula's picture

    Thank you for your reply!

    I have this problem on multiple hosts. The status of their disks is good, these hosts have not another problems. It seems that the problem occurs during the rotation of the log.

    Here is what I see in the logs:

    Windows application log

    2019-01-20 00:00:19 ERROR 7031 The nxlog service terminated unexpectedly.  It has done this 7 time(s). 

2019-01-20 00:00:22 ERROR 1000 Faulting application name: nxlog.exe, version: 4.1.4016.0, time stamp: 0x00000000
Faulting module name: libapr-1-0.dll, version: 0.0.0.0, time stamp: 0x00000000
Exception code: 0xc0000005
Fault offset: 0x0000000000008329
Faulting process id: 0x1b9c
Faulting application start time: 0x01d4b039fcbdc7da
Faulting application path: C:\nxlog\nxlog.exe
Faulting module path: C:\nxlog\libapr-1-0.dll
Report Id: 2277bbaa-0709-4976-8568-e93a5e702213
Faulting package full name: 
Faulting package-relative application ID:

    NXLog log

    2019-01-20 00:00:00 INFO LogFile C:\nxlog\data\nxlog.log reopened
2019-01-20 00:00:02 INFO inode changed for 'C:\nxlog\data\nxlog.log' (93394->101410): reopening possibly rotated file
2019-01-20 00:00:08 ERROR failed to read config cache; config cache version mismatch, expected NX-CC-VERSION 001, got in_errors/c:\app\logs\2018-10-13.log
2019-01-20 00:00:08 ERROR exiting...

    January 22, 2019 - 11:03am
  • hatula's picture

    And now I have a problem with the correct exit modules im_exec and im_msvistalog after system shutdown or reboot:

    2019-01-22 15:31:47 WARNING received a system shutdown request
2019-01-22 15:31:47 WARNING stopping nxlog service
2019-01-22 15:31:47 WARNING nxlog received a termination request signal, exiting...
2019-01-22 15:31:47 ERROR EvtNext failed with error 1818: The remote procedure call was cancelled.  
2019-01-22 15:31:47 ERROR EvtNext failed with error 1717: The interface is unknown.  
2019-01-22 15:31:47 ERROR Module process got EOF, process exited?   ** # this is module im_exec**
2019-01-22 15:31:48 ERROR failed to subscribe to msvistalog events using bookmark: The interface is unknown.  
       <QueryList>
         <Query Id='1'>
           <Select Path='System'>*</Select>
         </Query>
       </QueryList>
2019-01-22 15:31:48 ERROR failed to subscribe to msvistalog events [error code: 1717]; The interface is unknown.

    nxlog-4.2.4192 Windows 10

    But Release Notes of this version sayed:

    Do not crash if a module hangs on shutdown.

    Any ideas, please. Thank you!

    January 23, 2019 - 3:07pm
  • Zhengshi's picture
    (NXLog)

    Do not crash if a module hangs on shutdown.
    This does not look to be a module crashing, but the EventLog service shutting down before NXLog. That is why NXLog can not subscribe to the events, no events to subscribe to with the service down.
    By default, Windows does not care much about shutdown order. Please see the following

    Note that during service shutdown, by default, the SCM does not take dependencies into consideration. The SCM enumerates the list of running services and sends the SERVICE_CONTROL_SHUTDOWN command. Therefore, a service may fail because another service it depends on has already stopped. To set the shutdown order of services manually, create a multistring registry value that contains the service names in the order in which they should be shut down and assign it to the Control key's PreshutdownOrder value, as follows: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\PreshutdownOrder="Shutdown Order" To set the shutdown order of dependent services from your application, use the SetProcessShutdownParameters function. The SCM uses this function to give its handler 0x1E0 priority. The SCM sends SERVICE_CONTROL_SHUTDOWN notifications when its control handler is called and waits for the services to exit before returning from its control handler.

    https://docs.microsoft.com/en-us/windows/desktop/Services/service-control-handler-function

    January 27, 2019 - 6:40pm
  • hatula's picture

    Hi b0ti, after update I see this error as before:

    2019-02-04 00:00:00 INFO LogFile C:\nxlog\data\nxlog.log reopened
2019-02-04 00:00:03 INFO inode changed for 'C:\nxlog\data\nxlog.log' (92973->94320): reopening possibly rotated file
2019-02-04 00:00:04 ERROR failed to read config cache; string expected for config cache version
2019-02-04 00:00:04 ERROR exiting...
2019-02-04 00:00:05 INFO nxlog-4.2.4192 started

    Also I see crashed clients nxlog after rotation log file:

    2019-02-03 00:00:00 INFO LogFile C:\nxlog\data\nxlog.log reopened
2019-02-03 00:00:04 INFO inode changed for 'C:\nxlog\data\nxlog.log' (111153->95953): reopening possibly rotated file

    Any ideas, please!

    February 5, 2019 - 4:19pm
  • b0ti's picture
    (NXLog)

    nxlog-4.2.4216 was released a few days ago that should fix the log rotation deadlock.

    We could not reproduce the issue with the configcache.dat corruption.

    February 5, 2019 - 4:26pm
  • hatula's picture

    Hi b0ti,

    after upgrade to version nxlog-4.2.4216 I still have this error with crash service nxlog:

    2019-02-11 00:00:00 INFO LogFile C:\nxlog\data\nxlog.log reopened
2019-02-11 00:00:02 INFO inode changed for 'C:\nxlog\data\nxlog.log' (83660->87988): reopening possibly rotated file
2019-02-11 10:49:56 INFO nxlog-4.2.4216 started

    It is very sad. Can you help me? Thanks a lot!

    February 11, 2019 - 8:53am
  • Zhengshi's picture
    (NXLog)

    This entry is not actually an error message. It is midnight and your log rotate is happening.
    Are you stating that your service crashing at the same time? Could you paste a matching EventLog message showing the service crashing?
    I assume 10:49:56 is when you manually start the service again?

    February 11, 2019 - 9:03pm
  • hatula's picture

    Yes, I know that this is a daily log rotation. Yes, I am absolutely sure that the service has crashed. Yes, I started service manually, and you can see it in the logs. Believe me.

    Problem is actual.

    February 12, 2019 - 11:46am

Meanwhile we have identified a possible issue that might be causing the problem described in this forum question and further comments that results in a crash or errors with configcache.dat.

The workaround would be to set CacheFlushInterval 0 until a fix is available.

AnsweredFebruary 17, 2019 - 6:37pm

Comments (2)

  • hatula's picture

    Thanks, b0ti!

    This case was gone:

    2019-03-19 00:00:00 INFO LogFile C:\nxlog\data\nxlog.log reopened
2019-03-20 13:43:21 ERROR failed to read config cache: Invalid value type detected (0); faulty content is dropped
2019-03-20 13:43:21 INFO nxlog-4.3.4308 started
2019-03-20 13:43:26 INFO connecting to 192.168.1.2:1514
2019-03-20 13:43:26 INFO connected to server OK

    March 20, 2019 - 11:48am