2
responses

Hi All,

I'm currently trying to ingest some XML files into our SIEM platform (outputting to a local file at the moment for testing), and am having some issues reading files from a large directory.
The application we're ingesting log files from creates a folder structure and a single XML file per log entry.
Example path: "F:\PSP Logs\{4D2D2D9D-5379-4BDF-A331-0AF51BA015ED}\2021\05\06\13\File.xml"

I've written the below config and it works fine if you take an existing XML file and append a new line to it, however if we drop a new XML file into the directory structure, it does not pick up the file.
Any idea's on where we might be going wrong?
Note: We have tried with SavePOS and ReadFromLast on both True and False, and tried both older XML files and newly created ones.

Panic Soft

define INSTALLDIR C:\Program Files\nxlog

#ModuleDir %INSTALLDIR%\modules
#CacheDir %INSTALLDIR%\data
#SpoolDir %INSTALLDIR%\data

define CERTDIR %INSTALLDIR%\cert
define CONFDIR %INSTALLDIR%\conf\nxlog.d

# Note that these two lines define constants only; the log file location
# is ultimately set by the `LogFile` directive (see below). The
# `MYLOGFILE` define is also used to rotate the log file automatically
# (see the `_fileop` block).
define LOGDIR %INSTALLDIR%\data
define MYLOGFILE %LOGDIR%\nxlog.log

# If you are not using NXLog Manager, disable the `include` line
# and enable LogLevel and LogFile.
include %CONFDIR%\*.conf

LogLevel INFO
#LogFile %MYLOGFILE%

<Extension _syslog>
Module xm_syslog
</Extension>

# This block rotates `%MYLOGFILE%` on a schedule. Note that if `LogFile`
# is changed in managed.conf via NXLog Manager, rotation of the new
# file should also be configured there.
<Extension _fileop>
Module xm_fileop

# Check the size of our log file hourly, rotate if larger than 5MB
<Schedule>
Every 1 hour
<Exec>
if ( file_exists('%MYLOGFILE%') and
(file_size('%MYLOGFILE%') >= 5M) )
{
file_cycle('%MYLOGFILE%', 8);
}
</Exec>
</Schedule>

# Rotate our log file every week on Sunday at midnight
<Schedule>
When @weekly
Exec if file_exists('%MYLOGFILE%') file_cycle('%MYLOGFILE%', 8);
</Schedule>
</Extension>

<Extension xm_xml>
Module xm_xml
</Extension>

<Extension xm_json>
module xm_json
</Extension>

<Input MFiles>
Module im_file
File "F:\\PSP Logs\\{4D2D2D9D-5379-4BDF-A331-0AF51BA015ED}\\2021\\05\\*.xml"
##InputType linebased
SavePos FALSE
readfromlast FALSE
Recursive TRUE
<Exec>

parse_xml();

to_json();

log_info('Successful Message Output');

</Exec>
</Input>

<Output local_file>
module om_file
file "C:\\test\\XML Test Output\\Output.txt"
</Output>

<Route local>
Path MFiles => local_file
</Route>

AskedMay 6, 2021 - 5:50pm

Answer (1)

Hello,

I'm not sure if I understood you correctly, so let me ask: does only the file under path F:\\PSP Logs\\{4D2D2D9D-5379-4BDF-A331-0AF51BA015ED}\\2021\\05\\ change, or do you want the whole paths changed?

Best regards,
Rafal

Comments (1)

  • NathanLowey's picture

    Hey Rafal,

    The application outputs it's log files as single XML files and stores them under a year\month\day\hour format, creating new directories where needs be.

    So new folders and files will be created under that path.

    The issue we're having is that when new files are created, they are not being read and ingested.