12
responses

Hi,

We have had few instances on several servers in the past few weeks, where the NXlog application crashes when starting the service. Commonality so far is Windows Server 2012 Standard. Reinstall of the application corrects the problem, but doesn't sound like a solution. Anyone else experiencing this issue? Any fixes?

Following is the error from the Application Log:

Faulting application name: nxlog.exe, version: 4.1.4016.0, time stamp: 0x00000000
Faulting module name: ntdll.dll, version: 6.2.9200.22376, time stamp: 0x5a90c271
Exception code: 0xc0000374
Fault offset: 0x00000000000da535
Faulting process id: 0x18a4
Faulting application start time: 0x01d490bb5247014e
Faulting application path: C:\Program Files\nxlog\nxlog.exe
Faulting module path: C:\Windows\SYSTEM32\ntdll.dll
Report Id: 8ffb8b3b-fcae-11e8-944f-005056856b7b
Faulting package full name:
Faulting package-relative application ID:

AskedDecember 10, 2018 - 11:36pm

Answer (1)

Can you provide us with the details needed to reproduce this? The information provided above isn't terribly useful.

Comments (11)

  • sinisa's picture

    It would have also been useful to let me know what details you need, so I can provide them to you, and we can skip this step of me asking you what you need. :)

    So please let me know what kind of detail may I provide you with?

    There is nothing in the NXlog file, since the program doesn't even start. We're running Symantec Antivirus on these servers, and it's our suspect now, because the crash started happening randomly in the middle of the night over the weekend. Not related to windows patching or reboot. We had another Nxlog crashing on Server 2012 R2 last week. Currently trying to fix it by reinstalling, like we fixed the other ones.

    Thanks.

  • b0ti's picture
    (NXLog)

    Version 4.2 has been released. If the issue persists with the new version, please mail us your nxlog.conf along with the steps to reproduce on 2012R2.

    Not sure how reinstalling would help. The only thing I can think of is that it stores the saved positions in configcache.dat and when you reinstall it starts reading from the last position possibly skipping the event record that causes it to crash. You can test this by stopping the service, removing configcache.dat and restarting it.

    If it's caused by the windows eventlog, you can add Exec log_info($RecordNumber); and you'll then see the records that it was still able to process before running into the one that caused it to crash.

  • sinisa's picture

    Hi,

    Thanks for the suggestion - that WORKED! Deleting the configcache.dat file, and then restarting the service corrects the problem! Thank you, thank you, this is slowly starting to spread and it started to worry us.

    So your assumption is that a specific event log(s) causes the crash? I will try to identify what's the last event log sent to syslog and then assumption is that the following log on the Windows server would be the one causing the crash? Would knowing that help you?

    Unfortunately, due to time lines and our large infrastructure, the new version 4.2 won't be deployed before Jan/Feb. We deploy using GPO and have to sequence Dev -> Pre-prod -> Prod. If reboot is required for the update, then it will be even later.

    Thanks again for your continued help!

    Sinisa

  • Zhengshi's picture
    (NXLog)

    Sinisa,
    Knowing the events would help track down the issue.
    Last event processed and the next event, just in the case that something in the processed event caused an issue. Less likely, but good to cover your bases ;)

    ...If reboot is required for the update, then it will be even later.

    Understood. As for the reboot, you will need to restart the service, not reboot the server. Hopefully that helps with your deploy policy.

    Thank you,
    Jesse

  • sinisa's picture

    Hi Jesse,

    Ok thanks. I will provide the logs when it occurs next time. On the last system, the logs already rolled over, so I didn't have them any more.

    Good to hear about no reboot needed. It will expedite deployment for sure. I may be able to do it by the end of Jan then.

    Appreciate everyone's help. Will add logs here when problem occurs and gets discovered again.

    Thanks

    Sinisa

  • Sinisa Trajkovic's picture

    Hi,

    We got another nxlog failure. Fixed with deleting the config file as suggested. Here are some logs that preceded and followed the failure.

    The last log that made it to syslog is:
    2019-01-28T22:19:10.000-08:00 <servername>/<servername> daemon.err Application_Error: EventID[1000] Log[Application] Type[ERROR] Domain[] User[] Faulting application name: nxlog.exe, version: 4.1.4016.0, time stamp: 0x00000000 Faulting module name: ntdll.dll, version: 6.2.9200.22376, time stamp: 0x5a90c271 Exception code: 0xc0000374 Fault offset: 0x00000000000da535 Faulting process id: 0x500 Faulting application start time: 0x01d4a80d0b356c75 Faulting application path: C:\Program Files\nxlog\nxlog.exe Faulting module path: C:\Windows\SYSTEM32\ntdll.dll Report Id: c9d78d12-238d-11e9-9438-005056856702 Faulting package full name: Faulting package-relative application ID:

    The two logs before this one (also made it to syslog - anything before the last one made it as expected):
    2019-01-28T22:17:30.000-08:00 <servername>/<servername> daemon.info vmStatsProvider: EventID[256] Log[Application] Type[INFO] Domain[] User[] The "vmStatsProvider" is successfully initialized for this Virtual Machine. WMI namespace: "root\cimv2".
    2019-01-28T22:17:30.571-08:00 <servername>/<servername> auth.info Microsoft-Windows-Security-Auditing: EventID[4672] Log[Security] Type[AUDIT_SUCCESS] Domain[] User[] SubjectDomain[NT AUTHORITY] SubjectUser[SYSTEM] TargetDomain[] TargetUser[] Special privileges assigned to new logon. Subject: Security ID: <> Account Name: SYSTEM Account Domain: NT AUTHORITY Logon ID: 0x3E7 Privileges: SeAssignPrimaryTokenPrivilege SeTcbPrivilege SeSecurityPrivilege SeTakeOwnershipPrivilege SeLoadDriverPrivilege SeBackupPrivilege SeRestorePrivilege SeDebugPrivilege SeAuditPrivilege SeSystemEnvironmentPrivilege SeImpersonatePrivilege
    2019-01-28T22:14:58.438-08:00 <servername>/<servername> syslog.info nxlog[1280]: --- MARK ---
    2019-01-28T22:14:53.103-08:00 00 <servername>/<servername> kern.info Service_Control_Manager: EventID[7036] Log[System] Type[INFO] Domain[] User[] The WinHTTP Web Proxy Auto-Discovery Service service entered the running state.
    2019-01-28T22:14:52.000-08:00 00 <servername>/<servername> daemon.warning ESENT: EventID[906] Log[Application] Type[WARNING] Domain[] User[] svchost (928) A significant portion of the database buffer cache has been written out to the system paging file. This may result in severe performance degradation. See help link for complete details of possible causes. Resident cache has fallen by 5392 buffers (or 92%) in the last 1 seconds. Current Total Percent Resident: 7% (444 of 5836 buffers)

    And here are some logs that occurred after the last log was delivered to syslog - after NXlog crash:

    Log Name: Application
    Source: Windows Error Reporting
    Date: 1/28/2019 10:19:11 PM
    Event ID: 1001
    Task Category: None
    Level: Information
    Keywords: Classic
    User: N/A
    Computer: <servername>
    Description:
    Fault bucket , type 0
    Event Name: APPCRASH
    Response: Not available
    Cab Id: 0

    Problem signature:
    P1: nxlog.exe
    P2: 4.1.4016.0
    P3: 00000000
    P4: StackHash_83f5
    P5: 6.2.9200.22376
    P6: 5a90c271
    P7: c0000374
    P8: PCH_F0_FROM_ntdll+0x00000000000033FA
    P9:
    P10:

    Attached files:
    C:\Windows\Temp\WER1404.tmp.appcompat.txt
    C:\Windows\Temp\WER14A1.tmp.WERInternalMetadata.xml
    C:\Windows\Temp\WER14A2.tmp.hdmp
    C:\Windows\Temp\WER153F.tmp.dmp

    These files may be available here:
    C:\ProgramData\Microsoft\Windows\WER\ReportQueue\AppCrash_nxlog.exe_146a2f8d8b97d0cf921ee6cb3ca34d9ae933e_cab_68461599

    Analysis symbol:
    Rechecking for solution: 0
    Report Id: c9d78d12-238d-11e9-9438-005056856702
    Report Status: 4
    Hashed bucket:
    Event Xml:
    <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
    <System>
    <Provider Name="Windows Error Reporting" />
    <EventID Qualifiers="0">1001</EventID>
    <Level>4</Level>
    <Task>0</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2019-01-29T06:19:11.000000000Z" />
    <EventRecordID>221268</EventRecordID>
    <Channel>Application</Channel>
    <Computer>servername</Computer>
    <Security />
    </System>
    <EventData>
    <Data>
    </Data>
    <Data>0</Data>
    <Data>APPCRASH</Data>
    <Data>Not available</Data>
    <Data>0</Data>
    <Data>nxlog.exe</Data>
    <Data>4.1.4016.0</Data>
    <Data>00000000</Data>
    <Data>StackHash_83f5</Data>
    <Data>6.2.9200.22376</Data>
    <Data>5a90c271</Data>
    <Data>c0000374</Data>
    <Data>PCH_F0_FROM_ntdll+0x00000000000033FA</Data>
    <Data>
    </Data>
    <Data>
    </Data>
    <Data>
    C:\Windows\Temp\WER1404.tmp.appcompat.txt
    C:\Windows\Temp\WER14A1.tmp.WERInternalMetadata.xml
    C:\Windows\Temp\WER14A2.tmp.hdmp
    C:\Windows\Temp\WER153F.tmp.dmp</Data>
    <Data>C:\ProgramData\Microsoft\Windows\WER\ReportQueue\AppCrash_nxlog.exe_146a2f8d8b97d0cf921ee6cb3ca34d9ae933e_cab_68461599</Data>
    <Data>
    </Data>
    <Data>0</Data>
    <Data>c9d78d12-238d-11e9-9438-005056856702</Data>
    <Data>4</Data>
    <Data>
    </Data>
    </EventData>
    </Event>

    Log Name: System
    Source: Service Control Manager
    Date: 1/28/2019 10:19:11 PM
    Event ID: 7031
    Task Category: None
    Level: Error
    Keywords: Classic
    User: N/A
    Computer: servername
    Description:
    The nxlog service terminated unexpectedly. It has done this 1 time(s). The following corrective action will be taken in 300000 milliseconds: Restart the service.
    Event Xml:
    <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
    <System>
    <Provider Name="Service Control Manager" Guid="{555908d1-a6d7-4695-8e1e-26931d2012f4}" EventSourceName="Service Control Manager" />
    <EventID Qualifiers="49152">7031</EventID>
    <Version>0</Version>
    <Level>2</Level>
    <Task>0</Task>
    <Opcode>0</Opcode>
    <Keywords>0x8080000000000000</Keywords>
    <TimeCreated SystemTime="2019-01-29T06:19:11.286247100Z" />
    <EventRecordID>213270</EventRecordID>
    <Correlation />
    <Execution ProcessID="596" ThreadID="4240" />
    <Channel>System</Channel>
    <Computer>servername</Computer>
    <Security />
    </System>
    <EventData>
    <Data Name="param1">nxlog</Data>
    <Data Name="param2">1</Data>
    <Data Name="param3">300000</Data>
    <Data Name="param4">1</Data>
    <Data Name="param5">Restart the service</Data>
    <Binary>6E0078006C006F0067000000</Binary>
    </EventData>
    </Event>

    Log Name: Application
    Source: Windows Error Reporting
    Date: 1/28/2019 10:19:11 PM
    Event ID: 1001
    Task Category: None
    Level: Information
    Keywords: Classic
    User: N/A
    Computer: servername
    Description:
    Fault bucket , type 0
    Event Name: APPCRASH
    Response: Not available
    Cab Id: 0

    Problem signature:
    P1: nxlog.exe
    P2: 4.1.4016.0
    P3: 00000000
    P4: StackHash_83f5
    P5: 6.2.9200.22376
    P6: 5a90c271
    P7: c0000374
    P8: PCH_F0_FROM_ntdll+0x00000000000033FA
    P9:
    P10:

    Attached files:
    C:\Windows\Temp\WER1404.tmp.appcompat.txt
    C:\Windows\Temp\WER14A1.tmp.WERInternalMetadata.xml
    C:\Windows\Temp\WER14A2.tmp.hdmp
    C:\Windows\Temp\WER153F.tmp.dmp

    These files may be available here:
    C:\ProgramData\Microsoft\Windows\WER\ReportQueue\AppCrash_nxlog.exe_146a2f8d8b97d0cf921ee6cb3ca34d9ae933e_cab_68461599

    Analysis symbol:
    Rechecking for solution: 0
    Report Id: c9d78d12-238d-11e9-9438-005056856702
    Report Status: 0
    Hashed bucket:
    Event Xml:
    <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
    <System>
    <Provider Name="Windows Error Reporting" />
    <EventID Qualifiers="0">1001</EventID>
    <Level>4</Level>
    <Task>0</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2019-01-29T06:19:11.000000000Z" />
    <EventRecordID>221269</EventRecordID>
    <Channel>Application</Channel>
    <Computer>servername</Computer>
    <Security />
    </System>
    <EventData>
    <Data>
    </Data>
    <Data>0</Data>
    <Data>APPCRASH</Data>
    <Data>Not available</Data>
    <Data>0</Data>
    <Data>nxlog.exe</Data>
    <Data>4.1.4016.0</Data>
    <Data>00000000</Data>
    <Data>StackHash_83f5</Data>
    <Data>6.2.9200.22376</Data>
    <Data>5a90c271</Data>
    <Data>c0000374</Data>
    <Data>PCH_F0_FROM_ntdll+0x00000000000033FA</Data>
    <Data>
    </Data>
    <Data>
    </Data>
    <Data>
    C:\Windows\Temp\WER1404.tmp.appcompat.txt
    C:\Windows\Temp\WER14A1.tmp.WERInternalMetadata.xml
    C:\Windows\Temp\WER14A2.tmp.hdmp
    C:\Windows\Temp\WER153F.tmp.dmp</Data>
    <Data>C:\ProgramData\Microsoft\Windows\WER\ReportQueue\AppCrash_nxlog.exe_146a2f8d8b97d0cf921ee6cb3ca34d9ae933e_cab_68461599</Data>
    <Data>
    </Data>
    <Data>0</Data>
    <Data>c9d78d12-238d-11e9-9438-005056856702</Data>
    <Data>0</Data>
    <Data>
    </Data>
    </EventData>
    </Event>

    Log Name: System
    Source: Service Control Manager
    Date: 1/28/2019 10:19:32 PM
    Event ID: 7036
    Task Category: None
    Level: Information
    Keywords: Classic
    User: N/A
    Computer: servername
    Description:
    The WMI Performance Adapter service entered the stopped state.
    Event Xml:
    <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
    <System>
    <Provider Name="Service Control Manager" Guid="{555908d1-a6d7-4695-8e1e-26931d2012f4}" EventSourceName="Service Control Manager" />
    <EventID Qualifiers="16384">7036</EventID>
    <Version>0</Version>
    <Level>4</Level>
    <Task>0</Task>
    <Opcode>0</Opcode>
    <Keywords>0x8080000000000000</Keywords>
    <TimeCreated SystemTime="2019-01-29T06:19:32.034482100Z" />
    <EventRecordID>213271</EventRecordID>
    <Correlation />
    <Execution ProcessID="596" ThreadID="4240" />
    <Channel>System</Channel>
    <Computer>servername</Computer>
    <Security />
    </System>
    <EventData>
    <Data Name="param1">WMI Performance Adapter</Data>
    <Data Name="param2">stopped</Data>
    <Binary>77006D006900410070005300720076002F0031000000</Binary>
    </EventData>
    </Event>

    Log Name: Application
    Source: vmStatsProvider
    Date: 1/28/2019 10:19:32 PM
    Event ID: 258
    Task Category: Guest Library API
    Level: Information
    Keywords: Classic
    User: N/A
    Computer: servername
    Description:
    The "vmGuestLibrary" is successfully initialized for this Virtual Machine.
    Event Xml:
    <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
    <System>
    <Provider Name="vmStatsProvider" />
    <EventID Qualifiers="2">258</EventID>
    <Level>0</Level>
    <Task>2</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2019-01-29T06:19:32.000000000Z" />
    <EventRecordID>221270</EventRecordID>
    <Channel>Application</Channel>
    <Computer>servername/Computer>
    <Security />
    </System>
    <EventData>
    </EventData>
    </Event>

    If you want more logs, I'd be willing to send them, but maybe not here on the public forum.

    Thanks.
    Sinisa

  • b0ti's picture
    (NXLog)

    Hi Sinisa,

    The above shows that it crashes, unfortunately this isn't useful for much more. We would need the .evtx file that can be used to reproduce the crash in order to debug and fix it. If you can send it via email to us that would help.

    Thanks

  • William Scanlon's picture

    I am also experiencing the same issue w/ v4.2.4216.0 on 2012 R2 Standard. Removing configcache.dat allows the service to start in my case as well.