KISS: beginner's problems with im_file and om_file


#1 djontra

Hello nxlog world,

Shamed to say, I've spent entire yesterday trying to figure out how to read Windows DHCP log files and ship the events to ElasticSearch.

Problem was with using direct path for folder C:\Windows\System32\dhcp\. Managed to get nxlog to read by sharing the folders (read-only permissions) to the user account used for nxlog service account logon.

As the events were not showing in ES, I'm stuck with trying to write the events into another file, in order to confirm that the source files are being read correctly.

 


OS: Win Srv 2008 R2 Ent

nxlog: v 2.9.1347

Here is the nxlog.conf:

define ROOT C:\Program Files (x86)\nxlog

Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log

# debugging only:
LogLevel DEBUG
NoCache TRUE

  <Input msdhcp>
      Module      im_file
      File        '\\DC5\dhcp\DhcpSrvLog-*.log'
      SavePos     TRUE
      InputType   LineBased
      Exec if $raw_event =~ /^#/ drop();
      Exec $message = $raw_event;
  </Input>

  <Input dns>
      Module      im_file
      File        '\\DC5\dns\dns.log'
      SavePos     TRUE
      InputType   LineBased
      Exec if $raw_event =~ /^#/ drop();
      Exec $message = $raw_event;
  </Input>

  <Output file_test>
      Module      om_file
      File        'C:\Program Files (x86)\nxlog\data\test_file_output.txt'
#      Sync    TRUE
      OutputType   LineBased
  </Output>

<Route test>
     Path  msdhcp,dns => file_test
</Route>

As a result, only DNS events are written in the output file:

21.1.2016. 11:34:00 A6A8 PACKET  0000000003B27E90 UDP Snd 192.168.105.12  3f0d R Q [8085 A DR  NOERROR] A      (8)PLANKING(3)lab(5)rador(0)

21.1.2016. 11:34:00 A6A8 PACKET  0000000003EDA2C0 UDP Rcv 192.168.105.12  3c32   Q [0001   D   NOERROR] A      (8)PLANKING(3)lab(5)rador(0)

21.1.2016. 11:34:00 A6A8 PACKET  0000000003EDA2C0 UDP Snd 192.168.105.12  3c32 R Q [8085 A DR  NOERROR] A      (8)PLANKING(3)lab(5)rador(0)

..but only the new ones, i.e. as the source DNS log file is being appended.

I have tried modifying the SavePos parameter to FALSE of both input modules, but to no avail - same result.

 

Questions:

1. What would be the correct configuration of global NoCache and module specific SavePos parameters, in order to read and output the complete source file, regardless of prior attempts?

2. What is the reason DHCP logs (using wildcard) are not being read (or at least written in the output), as opposed to the same configuration for DNS logs?

 

 

I will provide nxlog debug level log if needed. No visible errors there.

 

Any help greatly appreciated!

#2 adm Nxlog ✓
#1 djontra
Hello nxlog world, Shamed to say, I've spent entire yesterday trying to figure out how to read Windows DHCP log files and ship the events to ElasticSearch. Problem was with using direct path for folder C:\Windows\System32\dhcp\. Managed to get nxlog to read by sharing the folders (read-only permissions) to the user account used for nxlog service account logon. As the events were not showing in ES, I'm stuck with trying to write the events into another file, in order to confirm that the source files are being read correctly.   OS: Win Srv 2008 R2 Ent nxlog: v 2.9.1347 Here is the nxlog.conf: define ROOT C:\Program Files (x86)\nxlog Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data LogFile %ROOT%\data\nxlog.log # debugging only: LogLevel DEBUG NoCache TRUE <Input msdhcp> Module im_file File '\\DC5\dhcp\DhcpSrvLog-*.log' SavePos TRUE InputType LineBased Exec if $raw_event =~ /^#/ drop(); Exec $message = $raw_event; </Input> <Input dns> Module im_file File '\\DC5\dns\dns.log' SavePos TRUE InputType LineBased Exec if $raw_event =~ /^#/ drop(); Exec $message = $raw_event; </Input> <Output file_test> Module om_file File 'C:\Program Files (x86)\nxlog\data\test_file_output.txt' # Sync TRUE OutputType LineBased </Output> <Route test> Path msdhcp,dns => file_test </Route> As a result, only DNS events are written in the output file: 21.1.2016. 11:34:00 A6A8 PACKET 0000000003B27E90 UDP Snd 192.168.105.12 3f0d R Q [8085 A DR NOERROR] A (8)PLANKING(3)lab(5)rador(0) 21.1.2016. 11:34:00 A6A8 PACKET 0000000003EDA2C0 UDP Rcv 192.168.105.12 3c32 Q [0001 D NOERROR] A (8)PLANKING(3)lab(5)rador(0) 21.1.2016. 11:34:00 A6A8 PACKET 0000000003EDA2C0 UDP Snd 192.168.105.12 3c32 R Q [8085 A DR NOERROR] A (8)PLANKING(3)lab(5)rador(0) ..but only the new ones, i.e. as the source DNS log file is being appended. I have tried modifying the SavePos parameter to FALSE of both input modules, but to no avail - same result.   Questions: 1. What would be the correct configuration of global NoCache and module specific SavePos parameters, in order to read and output the complete source file, regardless of prior attempts? 2. What is the reason DHCP logs (using wildcard) are not being read (or at least written in the output), as opposed to the same configuration for DNS logs?     I will provide nxlog debug level log if needed. No visible errors there.   Any help greatly appreciated!

To collect the full log file you need to add this:

ReadFromLast FALSE

There may be cases when the wildcard pattern is treated literally, you will see the following in nxlog.log:

input file does not exist: ....

This may be also printed if the wildcard does not match anything:

Module foo has no input files to read

Otherwise check the debug log, it may have more info on why your dhcp log is not read.