collecting duplicate event logs

Post a different question
4
responses

The following config is working for me to send application event logs to server via tcp, but it is sending duplicate messages for every Event Viewer event/entry.

config below as follows.

User nxlog
Group nxlog

LogFile /var/log/nxlog/nxlog.log
LogLevel INFO

<Extension exec>
Module xm_exec
</Extension>

<Extension syslog>
Module xm_syslog
</Extension>

<Input in>
Module im_tcp
Port 1514
Host 0.0.0.0
Exec parse_syslog_bsd();
</Input>

<Output out>
Module om_file
File "/application/logs/" + $Hostname + "_" + "app.log"
<Exec>
if out->file_size() > 1G
{
set_var('newfile', file_name() + strftime(now(), '_%Y%m%d%H%M%S'));
rotate_to(get_var('newfile'));
exec_async('/bin/bzip2', get_var('newfile'));
}
</Exec>
</Output>

<Route 1>
Path in => out
</Route>

AskedJuly 30, 2018 - 9:46am

Comments (4)

  • Zhengshi's picture
    (NXLog)

    I don't see anything in the config that would cause duplication. You mentioned Event Viewer, so I am running under the assumption you have a Windows event log sent over TCP to this server(im_tcp).
    It is possible that the source is sending duplicates for some reason. I would investigate that side. To cut it off on this server, you could use pm_norepeat.

    https://nxlog.co/docs/nxlog-ce/nxlog-reference-manual.html#pm_norepeat

    July 30, 2018 - 11:23pm
  • Suryaprakash's picture

    but still I am facing issues like creating enormous file .while i view the file it shows numbers,PFA for your reference.

    [root@localhost]# cat 1533013973441_kiosk.log
    1533013973441

    [root@localhost]# ls
    1533013964815_kiosk.log 1533013965533_kiosk.log 1533013973442_kiosk.log 1533013985281_kiosk.log 1533013987478_kiosk.log
    1533013964828_kiosk.log 1533013965534_kiosk.log 1533013974444_kiosk.log 1533013985340_kiosk.log 1533013987479_kiosk.log
    1533013964894_kiosk.log 1533013967005_kiosk.log 1533013974645_kiosk.log 1533013985352_kiosk.log 1533013990970_kiosk.log
    1533013964895_kiosk.log 1533013967006_kiosk.log 1533013984568_kiosk.log 1533013985493_kiosk.log 1533013991962_kiosk.log
    1533013964896_kiosk.log 1533013967564_kiosk.log 1533013984588_kiosk.log 1533013985494_kiosk.log 1533013991963_kiosk.log
    1533013964983_kiosk.log 1533013967579_kiosk.log 1533013984589_kiosk.log 1533013986877_kiosk.log 1533013992164_kiosk.log
    1533013965377_kiosk.log 1533013967586_kiosk.log 1533013984673_kiosk.log 1533013986878_kiosk.log 1533013992165_kiosk.log
    1533013965415_kiosk.log 1533013967587_kiosk.log 1533013984675_kiosk.log 1533013987442_kiosk.log
    1533013965423_kiosk.log 1533013973441_kiosk.log 1533013984793_kiosk.log 1533013987465_kiosk.log

    July 31, 2018 - 8:56am
  • Zhengshi's picture
    (NXLog)

    You are parsing the input as syslog. Is it actually syslog? Syslog format looks similar to the below text. myhost is the hostname in this example.

    <38>Nov 21 11:40:27 myhost sshd[8459]: Accepted publickey for john from 192.168.1.22 port 41193 ssh2

    https://nxlog.co/documentation/nxlog-user-guide#fields

    From looking at your post, it seems the file does not have syslog style logging inside of it.

    [root@localhost]# cat 1533013973441_kiosk.log
    1533013973441

    You could try adding Exec log_info($raw_event); to your <Input in> section and running with /bin/nxlog -f to see the events as they come in. If the events coming in are just numbers, then we will not be able to extract hostname info from it.

    July 31, 2018 - 4:29pm

Answers (0)