Schedule log collection. | Log collection solutions

#1 fbrollo 7 years ago

Hi everyone,

Do you know if it is possible to schedule a module execution ?

Or does the Schedule function only takes Exec commands ?

Here is the solution I've come up with to schedule log sending:

<Input im_file_test>
        Module  im_file
        File    '/mnt/test/scheduler/*log'
        InputType       LineBased
        SavePos TRUE
        <Schedule>
                  When 0 12 * * *
                  Exec fileop->file_copy("/mnt/test/test.log", "/mnt/scheduler/test.log");
        </Schedule>
</Input>
<Output om_tcp_siem>
Module  om_tcp
Host    192.168.0.10
Port    514
OutputType      LineBased
</Output>
<Route RouteTestB>
    Path        im_file_test => om_tcp_siem
</Route>

Here is what I would like to do (getting rid of xm_fileop for access privilege reasons:)

<Input im_file_test>
        <Schedule>
          When 0 12 * * *
          Module  im_file
          File    '/mnt/test/test.log'
          InputType       LineBased
          SavePos TRUE
       </Schedule>
</Input>

Thank you for your time :)

Permalink

#2 b0ti Nxlog ✓ 7 years ago (Last updated 5 years ago )

#1 fbrollo

Hi everyone, Do you know if it is possible to schedule a module execution ? Or does the Schedule function only takes Exec commands ? Here is the solution I've come up with to schedule log sending: <Input im_file_test> Module im_file File '/mnt/test/scheduler/*log' InputType LineBased SavePos TRUE <Schedule> When 0 12 * * * Exec fileop->file_copy("/mnt/test/test.log", "/mnt/scheduler/test.log"); </Schedule> </Input> <Output om_tcp_siem> Module om_tcp Host 192.168.0.10 Port 514 OutputType LineBased </Output> <Route RouteTestB> Path im_file_test => om_tcp_siem </Route> Here is what I would like to do (getting rid of xm_fileop for access privilege reasons:) <Input im_file_test> <Schedule> When 0 12 * * * Module im_file File '/mnt/test/test.log' InputType LineBased SavePos TRUE </Schedule> </Input> Thank you for your time :)

Your first option is obviously not very nice since it involves copying the file and the second is not supported. Basically NXLog runs as a daemon/service trying to collect events in real-time but there is also a separate binary called nxlog-processor that exits when it finishes processing the data.

With the above in mind there are 3 options:

Use nxlog-processor and run it from cron with the same crontab specification (0 12 * * *) that you tried to use within the <Schedule> block.
Use nxlog with im_exec that invokes a script reading the required files when you want it. This would be a bit tricky, especially if it needs to follow multiple files.
NXLog has pm_blocker. This can be used to disallow sending data. See this example.

I think the first option would be the most suitable if the requirement is to ship all new data once a day.