The disappearing Windows DNS debug log

TD;LR

The Windows DNS service may not recreate the debug log file after rollover. If you get hit by the issue, make sure to use the C: drive for the debug log path.

The Windows DNS debug log

The Windows DNS debug log contains information on DNS queries and activity that can be important to monitor and analyze to detect malicious traffic. This requires some configuration changes for the DNS service in order to enable debug logging.  Here is a short description on how to enable debug logging on for the DNS service on windows, this also applies to Windows 2008 and later. It is possible to specify the file and path name of the DNS debug log file as well as the maximum size of the file. 

Note that the DNS server on Windows 2012 can also log audit and analytic information into the Windows Eventlog using the Event Tracing for Windows (ETW) facility. 

Windows DNS debug log rollover

DNS debug log is usually configured with a maximum size. When this limit is reached the log file is cleared to start over. Unfortunately this is not a friendly behavior when you need to monitor this data since the log collector might miss some events when the rollover occurs. Some other services are capable of creating a new log file, the DNS service doesn't. The bigger problem is that the DNS debug log could disappear when it is monitored by log collector tools.

This issue is not NXLog specific and it affects any other tool that opens and reads the log file, for example Splunk users and McAfee SIEM collector users are running into the same issue.

The DNS debug log only disappears if it is monitored, so the conclusion would be to blame the log monitoring tool. The im_file module in NXLog does not delete files and it does not lock log files. Files are opened with READ access only. NXLog and most other log collectors work fine collecting log files being written by most other software. 

Some software requests exclusive locking on a log file that it writes, this of course will prevent the file from being opened and monitored. Locking isn't the problem in this case as the DNS service does not lock the debug log file.

The default behavior of NXLog's im_file module is to keep the monitored file open. The CloseWhenIdle configuration option can be used to instruct it to close the log file after it's done reading the file. Unfortunately this does not solve the disappearing DNS log file issue.

Why does the DNS debug log disappear?

We used Windows Sysinternals Process Monitor to check what's going on. Below is a screenshot showing the operations from both dns.exe and nxlog.exe.

debug log

When the rollover occurs dns.exe creates a backup of the debug log file under C:\Windows\System32\dns\backup\dns.log and then recreates the debug log file by deleting and opening it for read/write. Unfortunately NXLog is still reading data from the file and holds an open handle thus the delete operation does not complete until NXLog is done reading the file and closes it. The DNS service tries to create the new file but receives a DELETE PENDING error and this causes the debug log file to disappear.

Microsoft's recommendation

Unfortunately the only solution at this point looked like to fix the DNS service. The DNS service should tolerate the DELETE PENDING error and wait until this completes. Better yet, it should create a different file as some other services are capable of doing this.

Having contacted Microsoft support about the issue and providing them the procmon traces yielded the following:

Date: Mon, 11 Jan 2016 19:16:57 +0000
From: "xxxxx" <xxxxxx@microsoft.com>
Subject: RE: RE: RE: RE: Re: 115121713506095

Enabling DNS debug logging is designed for temporary troubleshooting.
This  component is not designed to be turned on and left running 24x7.

The server component DNS.exe may be designed and need exclusive access to
its own debug log.  Microsoft is prepared to tell you this as a
resolution.

Also when it comes to monitoring DNS traffic for security threats,
monitoring at your resolver is not a best practice.  We recommend you
purchase a (HSA) Hardware Security Appliance.  Third party content
filters are available from a variety of manufacturers.

Microsoft looked at the threads, stack, handles and DLL's this program
calls this morning.  The processes nxlog.exe maintains an open handle on
the dnslog.txt file and this is causing your issue.

Please see the attached screen shot.

Regards,

xxxxxxxxxx

xxxxxx
Support Professional
xxxxx
Windows Networking T3

Office: 866-425-9899 x2236319
xxxxxxx@microsoft.com<mailto:xxxxxxx@microsoft.com>
Working hours: 9-6 CST

[Twitter]<https://twitter.com/Microsoft>    [Facebook]
<https://www.facebook.com/Microsoft>     [Linkedin]
<https://www.linkedin.com/company/microsoft>     [Youtube]
<https://www.youtube.com/user/Microsoft>

[Microsoft Logo]
MS Premier Support: 1800-936-3100  ~  MS Premier Online:
https://premier.microsoft.com<https://premier.microsoft.com/>

 

Any other solutions?

Purchasing and deploying a Hardware Security Appliance might not be the ideal choice for obvious reasons. Not giving up on this we went on looking for a solution or a workaround. A possible solution was to use API hooking and dll injection. By injecting code into dns.exe we could intercept the CreateFile() system call and redirect it to open a different file. This would solve the PENDING DELETE issue and also provide proper log rotation.

Move vs. Rename

Doing some more testing we discovered that the issue does not manifest if the DNS debug log file is configured to reside on the C: drive. Below is the procmon screenshot showing what dns.exe was doing in this case:

debug log management

Analyzing this closely you will notice that it is different from the procmon trace shown earlier. Here you can see that dns.exe invokes the SetRenameInformationFile operation. The reason for this is that the backup folder resides on the same drive, thus the file is simply renamed (moved) and while nxlog.exe can finish reading from the renamed file, dns.exe will be able to recreate the debug log file.

When the debug log file path is configured to use a different drive than C: (systemroot), the file needs to be copied byte-by-byte to the backup folder.

The recommended solution is to configure the DNS server to use C: for the debug log file path. This also provides a performance benefit since the DNS server does not need to copy the file to the backup folder.

Share this post