im_file Inputs are not forwarded to logging server
Hi everyone, I have the following problem.
1 The problem:
I trace the performance counters of several Windows clients. For that, CSV files are created and their inputs then are forwarded to our logging system. Each counter type (RAM, CPU, storage, etc) has its own CSV and therefore its own input in NXlog. While it works without any problems on nearly all clients, there is one Workstation where the im_file inputs are not forwardedBesides the im_file module we use the im_msvistalog module for Windows Event entries as well. And the Workstation does forward these events without any problems. It just has problems with the im_file inputs. Log file does not indicate any error and as I said: this configuration (with minor differences) already works flawlessly on the other systems.
2 The configuration:
define ROOT C:\Program Files\nxlog
define CERTDIR C:\Program Files\nxlog\keys
define CONFDIR %ROOT%\conf\nxlog.d
define LOGDIR %ROOT%\data
define LOGFILE %LOGDIR%\nxlog.log
LogFile %LOGFILE%
Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
<Extension _syslog>
Module xm_syslog
</Extension>
<Extension _gelf>
Module xm_gelf
</Extension>
<Extension _charconv>
Module xm_charconv
AutodetectCharsets iso8859-2, utf-8, utf-16, utf-32
</Extension>
<Extension _exec>
Module xm_exec
</Extension>
<Extension _fileop>
Module xm_fileop
# Check the size of our log file hourly, rotate if larger than 5MB
<Schedule>
Every 1 hour
Exec if (file_exists('%LOGFILE%') and \
(file_size('%LOGFILE%') >= 5M)) \
file_cycle('%LOGFILE%', 8);
</Schedule>
# Rotate our log file every week on Sunday at midnight
<Schedule>
When @weekly
Exec if file_exists('%LOGFILE%') file_cycle('%LOGFILE%', 8);
</Schedule>
</Extension>
define MonitoredEventIDsSecurity 4624, 4634, 4672
Collecting event log
<Input in>
Module im_msvistalog
SavePos TRUE
ReadFromLast TRUE
<QueryXML>
<QueryList>
<Query Id="0">
<Select Path="Security">*</Select>
</Query>
</QueryList>
</QueryXML>
<Exec>
if $EventID NOT IN (%MonitoredEventIDsSecurity%) drop();
</Exec>
</Input>
<Input trace_cpu>
Module im_file
File 'C:\tracing\trace_cpu.csv'
SavePos TRUE
ReadFromLast TRUE
Exec $Message = $raw_event;
</Input>
<Input trace_ram>
Module im_file
File 'C:\tracing\trace_ram.csv'
SavePos TRUE
ReadFromLast TRUE
Exec $Message = $raw_event;
</Input>
<Input trace_networkmain>
Module im_file
File 'C:\tracing\trace_networkmain.csv'
SavePos TRUE
ReadFromLast TRUE
Exec $Message = $raw_event;
</Input>
<Input trace_diskc>
Module im_file
File 'C:\tracing\trace_diskc.csv'
SavePos TRUE
ReadFromLast TRUE
Exec $Message = $raw_event;
</Input>
<Input trace_diskd>
Module im_file
File 'C:\tracing\trace_diskd.csv'
SavePos TRUE
ReadFromLast TRUE
Exec $Message = $raw_event;
</Input>
<Input trace_diske>
Module im_file
File 'C:\tracing\trace_diske.csv'
SavePos TRUE
ReadFromLast TRUE
Exec $Message = $raw_event;
</Input>
<Output x>
Module om_ssl
Host %IP_Address%
Port %Port%
OutputType GELF_TCP
CAFile %CERTDIR%\cafile.pem
CertFile %CERTDIR%\certfile.pem
CertKeyFile %CERTDIR%\certkeyfile.pem
KeyPass %password%
Exec to_syslog_snare();
</Output>
#Connect input 'in' to output 'out'
<Route 1>
Path in, trace_cpu, trace_ram, trace_networkmain, trace_diskc, trace_diskd, trace_diske => x
</Route>
3 The NXLog log:
I think it's irrelevant because it only shows this entries:
2023-05-09 09:13:13 WARNING stopping nxlog service
2023-05-09 09:13:13 WARNING nxlog-ce received a termination request signal, exiting...
2023-05-09 09:13:16 INFO connecting to %IP_Address%:%Port%
2023-05-09 09:13:16 INFO nxlog-ce-3.1.2319 started
2023-05-09 09:13:16 INFO successfully connected to %IP_Address%:%Port%
4 Environment info:
The mentioned client runs Windows 10 Pro 22H2, currently installed NXLog Version is ce-3.1.2319 (but also tested it with ce-3.2.2329)
5 Relevant details:
- Config works on other clients without problems
- Only im_file module not working, im_msvistalog entries are being forwarded
- Until two weeks ago I used one central CSV file for all performance counters and this was forwarded correctly until the separation into individual inputs
- When deleting the “Exec $Message = $raw_event;” directive from an input, the respective messages get forwarded to logging system but are in a cryptic format and not useable
That would be it for now. Please feel free to ask if you need further information :)
Thanks in advance!
- Did it ever read from the files? Can you find any log in your SIEM from those files after the separation? Are the files static or does something update them. If yes, does the program lock those files?Have you tried re-directing the output to a local test file, just in order to check if the problem is with reading from files (input module) or with the output module?