Hi All,

I'm currently trying to ingest some XML files into our SIEM platform (outputting to a local file at the moment for testing), and am having some issues reading files from a large directory. The application we're ingesting log files from creates a folder structure and a single XML file per log entry. Example path: "F:\PSP Logs{4D2D2D9D-5379-4BDF-A331-0AF51BA015ED}\2021\05\06\13\File.xml"

I've written the below config and it works fine if you take an existing XML file and append a new line to it, however if we drop a new XML file into the directory structure, it does not pick up the file. Any idea's on where we might be going wrong? Note: We have tried with SavePOS and ReadFromLast on both True and False, and tried both older XML files and newly created ones.

Panic Soft

define INSTALLDIR C:\Program Files\nxlog

#ModuleDir %INSTALLDIR%\modules #CacheDir %INSTALLDIR%\data #SpoolDir %INSTALLDIR%\data

define CERTDIR %INSTALLDIR%\cert define CONFDIR %INSTALLDIR%\conf\nxlog.d

Note that these two lines define constants only; the log file location

is ultimately set by the LogFile directive (see below). The

MYLOGFILE define is also used to rotate the log file automatically

(see the _fileop block).

define LOGDIR %INSTALLDIR%\data define MYLOGFILE %LOGDIR%\nxlog.log

If you are not using NXLog Manager, disable the include line

and enable LogLevel and LogFile.

include %CONFDIR%*.conf

LogLevel INFO #LogFile %MYLOGFILE%

<Extension _syslog> Module xm_syslog </Extension>

This block rotates %MYLOGFILE% on a schedule. Note that if LogFile

is changed in managed.conf via NXLog Manager, rotation of the new

file should also be configured there.

<Extension _fileop> Module xm_fileop

# Check the size of our log file hourly, rotate if larger than 5MB
<Schedule>
    Every   1 hour
    <Exec>
        if ( file_exists('%MYLOGFILE%') and
             (file_size('%MYLOGFILE%') >= 5M) )
        {
             file_cycle('%MYLOGFILE%', 8);
        }
    </Exec>
</Schedule>

# Rotate our log file every week on Sunday at midnight
<Schedule>
    When    @weekly
    Exec    if file_exists('%MYLOGFILE%') file_cycle('%MYLOGFILE%', 8);
</Schedule>

</Extension>

<Extension xm_xml> Module xm_xml </Extension>

<Extension xm_json> module xm_json </Extension>

<Input MFiles> Module im_file File "F:\PSP Logs\{4D2D2D9D-5379-4BDF-A331-0AF51BA015ED}\2021\05\*.xml" ##InputType linebased SavePos FALSE readfromlast FALSE Recursive TRUE <Exec>

	parse_xml();
	
	to_json();

	log_info('Successful Message Output');
	
</Exec>

</Input>

<Output local_file> module om_file file "C:\test\XML Test Output\Output.txt" </Output>

<Route local> Path MFiles => local_file </Route>