im_file Inputs are not forwarded to logging server


#1 bero.0815 (Last updated )

Hi everyone, I have the following problem.

 

1 The problem:

I trace the performance counters of several Windows clients. For that, CSV files are created and their inputs then are forwarded to our logging system. Each counter type (RAM, CPU, storage, etc) has its own CSV and therefore its own input in NXlog. While it works without any problems on nearly all clients, there is one Workstation where the im_file inputs are not forwardedBesides the im_file module we use the im_msvistalog module for Windows Event entries as well. And the Workstation does forward these events without any problems. It just has problems with the im_file inputs. Log file does not indicate any error and as I said: this configuration (with minor differences) already works flawlessly on the other systems.

 

2 The configuration:

define ROOT     C:\Program Files\nxlog
define CERTDIR	C:\Program Files\nxlog\keys
define CONFDIR  %ROOT%\conf\nxlog.d
define LOGDIR   %ROOT%\data

define LOGFILE  %LOGDIR%\nxlog.log
LogFile %LOGFILE%

Moduledir %ROOT%\modules
CacheDir  %ROOT%\data
Pidfile   %ROOT%\data\nxlog.pid
SpoolDir  %ROOT%\data

<Extension _syslog>
    Module      xm_syslog
</Extension>

<Extension _gelf>
    Module      xm_gelf
</Extension>

<Extension _charconv>
    Module      xm_charconv
    AutodetectCharsets iso8859-2, utf-8, utf-16, utf-32
</Extension>

<Extension _exec>
    Module      xm_exec
</Extension>

<Extension _fileop>
    Module      xm_fileop

    # Check the size of our log file hourly, rotate if larger than 5MB
    <Schedule>
        Every   1 hour
        Exec    if (file_exists('%LOGFILE%') and \
                   (file_size('%LOGFILE%') >= 5M)) \
                    file_cycle('%LOGFILE%', 8);
    </Schedule>

    # Rotate our log file every week on Sunday at midnight
    <Schedule>
        When    @weekly
        Exec    if file_exists('%LOGFILE%') file_cycle('%LOGFILE%', 8);
    </Schedule>
</Extension>


define MonitoredEventIDsSecurity		4624, 4634, 4672

# Collecting event log
  <Input in>
      Module      im_msvistalog
	  SavePos	TRUE
	  ReadFromLast	TRUE
	<QueryXML>
	<QueryList>
	<Query Id="0">
	<Select Path="Security">*</Select>
		</Query>
		</QueryList>
		</QueryXML>
	<Exec>
	if $EventID NOT IN (%MonitoredEventIDsSecurity%) drop();
	</Exec>
  </Input>
 
<Input trace_cpu>
    Module	im_file
	File	'C:\tracing\trace_cpu.csv'
	SavePos	TRUE
	ReadFromLast	TRUE
	Exec	$Message = $raw_event;
</Input>

<Input trace_ram>
    Module	im_file
	File	'C:\tracing\trace_ram.csv'
	SavePos	TRUE
	ReadFromLast	TRUE
	Exec	$Message = $raw_event;
</Input>

<Input trace_networkmain>
    Module	im_file
	File	'C:\tracing\trace_networkmain.csv'
	SavePos	TRUE
	ReadFromLast	TRUE
	Exec	$Message = $raw_event;
</Input>

<Input trace_diskc>
    Module	im_file
	File	'C:\tracing\trace_diskc.csv'
	SavePos	TRUE
	ReadFromLast	TRUE
	Exec	$Message = $raw_event;
</Input>

<Input trace_diskd>
    Module	im_file
	File	'C:\tracing\trace_diskd.csv'
	SavePos	TRUE
	ReadFromLast	TRUE
	Exec	$Message = $raw_event;
</Input>
 
<Input trace_diske>
    Module	im_file
	File	'C:\tracing\trace_diske.csv'
	SavePos	TRUE
	ReadFromLast	TRUE
	Exec	$Message = $raw_event;
</Input>

 <Output x>
     Module      om_ssl
	 Host        %IP_Address%
     Port        %Port%
	 OutputType	 GELF_TCP
	 CAFile      %CERTDIR%\cafile.pem
     CertFile    %CERTDIR%\certfile.pem
     CertKeyFile %CERTDIR%\certkeyfile.pem
	 KeyPass	 %password%
     Exec        to_syslog_snare();
 </Output>
 
#Connect input 'in' to output 'out'
<Route 1>
     Path        in, trace_cpu, trace_ram, trace_networkmain, trace_diskc, trace_diskd, trace_diske => x
</Route>

 

3 The NXLog log:

I think it's irrelevant because it only shows this entries:

2023-05-09 09:13:13 WARNING stopping nxlog service
2023-05-09 09:13:13 WARNING nxlog-ce received a termination request signal, exiting...
2023-05-09 09:13:16 INFO connecting to %IP_Address%:%Port%
2023-05-09 09:13:16 INFO nxlog-ce-3.1.2319 started
2023-05-09 09:13:16 INFO successfully connected to %IP_Address%:%Port%

 

4 Environment info:

The mentioned client runs Windows 10 Pro 22H2, currently installed NXLog Version is ce-3.1.2319 (but also tested it with ce-3.2.2329)

5 Relevant details: 

  • Config works on other clients without problems
  • Only im_file module not working, im_msvistalog entries are being forwarded
  • Until two weeks ago I used one central CSV file for all performance counters and this was forwarded correctly until the separation into individual inputs
  • When deleting the “Exec $Message = $raw_event;” directive from an input, the respective messages get forwarded to logging system but are in a cryptic format and not useable

That would be it for now. Please feel free to ask if you need further information :)

Thanks in advance!

#2 NenadMDeactivated Nxlog ✓
    Did it ever read from the files? Can you find any log in your SIEM from those files after the separation? Are the files static or does something update them. If yes, does the program lock those files?Have you tried re-directing the output to a local test file, just in order to check if the problem is with reading from files (input module) or with the output module?