im_file Inputs are not forwarded to logging server


#1 bero.0815 (Last updated )

Hi everyone, I have the following problem.

 

1 The problem:

I trace the performance counters of several Windows clients. For that, CSV files are created and their inputs then are forwarded to our logging system. Each counter type (RAM, CPU, storage, etc) has its own CSV and therefore its own input in NXlog. While it works without any problems on nearly all clients, there is one Workstation where the im_file inputs are not forwardedBesides the im_file module we use the im_msvistalog module for Windows Event entries as well. And the Workstation does forward these events without any problems. It just has problems with the im_file inputs. Log file does not indicate any error and as I said: this configuration (with minor differences) already works flawlessly on the other systems.

 

2 The configuration:

define ROOT     C:\Program Files\nxlog
define CERTDIR	C:\Program Files\nxlog\keys
define CONFDIR  %ROOT%\conf\nxlog.d
define LOGDIR   %ROOT%\data

define LOGFILE %LOGDIR%\nxlog.log LogFile %LOGFILE%

Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data

<Extension _syslog> Module xm_syslog </Extension>

<Extension _gelf> Module xm_gelf </Extension>

<Extension _charconv> Module xm_charconv AutodetectCharsets iso8859-2, utf-8, utf-16, utf-32 </Extension>

<Extension _exec> Module xm_exec </Extension>

<Extension _fileop> Module xm_fileop

# Check the size of our log file hourly, rotate if larger than 5MB
&lt;Schedule&gt;
    Every   1 hour
    Exec    if (file_exists('%LOGFILE%') and \
               (file_size('%LOGFILE%') &gt;= 5M)) \
                file_cycle('%LOGFILE%', 8);
&lt;/Schedule&gt;

# Rotate our log file every week on Sunday at midnight
&lt;Schedule&gt;
    When    @weekly
    Exec    if file_exists('%LOGFILE%') file_cycle('%LOGFILE%', 8);
&lt;/Schedule&gt;

</Extension>

define MonitoredEventIDsSecurity 4624, 4634, 4672

Collecting event log

<Input in> Module im_msvistalog SavePos TRUE ReadFromLast TRUE <QueryXML> <QueryList> <Query Id="0"> <Select Path="Security">*</Select> </Query> </QueryList> </QueryXML> <Exec> if $EventID NOT IN (%MonitoredEventIDsSecurity%) drop(); </Exec> </Input>

<Input trace_cpu> Module im_file File 'C:\tracing\trace_cpu.csv' SavePos TRUE ReadFromLast TRUE Exec $Message = $raw_event; </Input>

<Input trace_ram> Module im_file File 'C:\tracing\trace_ram.csv' SavePos TRUE ReadFromLast TRUE Exec $Message = $raw_event; </Input>

<Input trace_networkmain> Module im_file File 'C:\tracing\trace_networkmain.csv' SavePos TRUE ReadFromLast TRUE Exec $Message = $raw_event; </Input>

<Input trace_diskc> Module im_file File 'C:\tracing\trace_diskc.csv' SavePos TRUE ReadFromLast TRUE Exec $Message = $raw_event; </Input>

<Input trace_diskd> Module im_file File 'C:\tracing\trace_diskd.csv' SavePos TRUE ReadFromLast TRUE Exec $Message = $raw_event; </Input>

<Input trace_diske> Module im_file File 'C:\tracing\trace_diske.csv' SavePos TRUE ReadFromLast TRUE Exec $Message = $raw_event; </Input>

<Output x> Module om_ssl Host %IP_Address% Port %Port% OutputType GELF_TCP CAFile %CERTDIR%\cafile.pem CertFile %CERTDIR%\certfile.pem CertKeyFile %CERTDIR%\certkeyfile.pem KeyPass %password% Exec to_syslog_snare(); </Output>

#Connect input 'in' to output 'out' <Route 1> Path in, trace_cpu, trace_ram, trace_networkmain, trace_diskc, trace_diskd, trace_diske => x </Route>

 

3 The NXLog log:

I think it's irrelevant because it only shows this entries:

2023-05-09 09:13:13 WARNING stopping nxlog service
2023-05-09 09:13:13 WARNING nxlog-ce received a termination request signal, exiting...
2023-05-09 09:13:16 INFO connecting to %IP_Address%:%Port%
2023-05-09 09:13:16 INFO nxlog-ce-3.1.2319 started
2023-05-09 09:13:16 INFO successfully connected to %IP_Address%:%Port%

 

4 Environment info:

The mentioned client runs Windows 10 Pro 22H2, currently installed NXLog Version is ce-3.1.2319 (but also tested it with ce-3.2.2329)

5 Relevant details: 

  • Config works on other clients without problems
  • Only im_file module not working, im_msvistalog entries are being forwarded
  • Until two weeks ago I used one central CSV file for all performance counters and this was forwarded correctly until the separation into individual inputs
  • When deleting the “Exec $Message = $raw_event;” directive from an input, the respective messages get forwarded to logging system but are in a cryptic format and not useable

That would be it for now. Please feel free to ask if you need further information :)

Thanks in advance!

#2 NenadMDeactivated Nxlog ✓
    Did it ever read from the files? Can you find any log in your SIEM from those files after the separation? Are the files static or does something update them. If yes, does the program lock those files?Have you tried re-directing the output to a local test file, just in order to check if the problem is with reading from files (input module) or with the output module?