im_file - parse if file is newer than X days (or fixed date)

#1 kumdabur 3 years ago

Hi,

I'm testing nxlog with IIS servers. It works, however I have following issue: If server has IIS since years and lets assume that logs were stored for 1 year, I have bunch of iis logfiles which all together combined results with around 5GB of Data.

While using nxlog as-it-is, it consumes CPU and also SIEM itself is not entirely happy about receiving sudden "bombarding" of new logs within few minutes timeframe. Unfortunately, I have around 50 servers like that and I really do not need to inject past logs.

I wonder if there is an option to either throttle nxlog a bit or in best case I'd like to send ONLY new iis logs, even starting from exact today. Below you may find part of config

   <Input IIS_Logs_1>
        Module   im_file
        File        'C:\inetpub\logs\LogFiles\W3SVC*\u_ex*.log'
        ReadFromLast FALSE
        Recursive TRUE
        PollInterval  1
        Exec     $FileName = file_name();
        Exec if $raw_event =~ /^#/ drop();\
           else\
           {\
            w3c_1->parse_csv();\
            $SourceName = "IIS";\
           }
    </Input>

I'd appreciate any hints.

Permalink

#2 rafDeactivated Nxlog ✓ 3 years ago

#1 kumdabur

Hi, I'm testing nxlog with IIS servers. It works, however I have following issue: If server has IIS since years and lets assume that logs were stored for 1 year, I have bunch of iis logfiles which all together combined results with around 5GB of Data. While using nxlog as-it-is, it consumes CPU and also SIEM itself is not entirely happy about receiving sudden "bombarding" of new logs within few minutes timeframe. Unfortunately, I have around 50 servers like that and I really do not need to inject past logs. I wonder if there is an option to either throttle nxlog a bit or in best case I'd like to send ONLY new iis logs, even starting from exact today. Below you may find part of config <Input IIS_Logs_1> Module im_file File 'C:\inetpub\logs\LogFiles\W3SVC*\u_ex*.log' ReadFromLast FALSE Recursive TRUE PollInterval 1 Exec $FileName = file_name(); Exec if $raw_event =~ /^#/ drop();\ else\ {\ w3c_1->parse_csv();\ $SourceName = "IIS";\ } </Input> I'd appreciate any hints.

Hello,

Take a look at the docs for `im_file' module https://nxlog.co/docs/nxlog-ce/nxlog-reference-manual.html#im_file:

ReadFromLast

This optional boolean directive instructs the module to only read logs which arrived after NXLog was started if the saved position could not be read (for example on first start). When SavePos is TRUE and a previously saved position value could be read, the module will resume reading from this saved position. If ReadFromLast is FALSE, the module will read all logs from the file. This can result in quite a lot of messages, and is usually not the expected behavior. If this directive is not specified, it defaults to TRUE.

Hope this helps,
Rafal