im_udp dropping syslog udp messages

Tags: syslog

#1 jd01

Hi,

I'm working on a heavy log source which can only send syslog. Now currently i also have filters in the config to remove unwanted logs. I've noticed that some logs are successfully being processed whilst others are lost.

Through packet capture I was able to conclude that from log source to nxlog server udp packets are all being received. Seems like nxlog (config) can’t handle the large amount of syslog UDP messages coming in.

 <Extension _syslog>
    Module      xm_syslog
</Extension>
<Input x_sys_in>
    Module      im_udp
    Port        514
    Host        0.0.0.0
    <Exec>
        parse_syslog_bsd();
        if (($Message =~ /dstip=x\.x\.x\.(?:[0-9]+){1,255} dstport=(?:12|5)3/) #DNS & NTP from x.x.x.x
             or ($Message =~ /dstip=x\.x\.x\.(?:[0-9]+){1,255} dstport=(?:12|5)3/) #DNS & NTP from x.x.x.x
             or ($Message =~ /srcip=[\d.]+ srcport=\d+ srcintf="x" srcintfrole=".+" dstip=[\d.]+ dstport=16[12]/) #x from x Range
             or ($Message =~ /srcip=[\d.]+ srcport=\d+ srcintf=".+" srcintfrole=".+" dstip=[\d.]+ dstport=16[12] dstintf="x"/) #
             #.... more filters
                drop();
    </Exec>
</Input>
<Output x_sys_out>
    Module      om_udp
    Host        x.x.x.x
    Port        514
    Exec        to_syslog_bsd();
</Output>
<Route x>
    Path        x_sys_in => x_sys_out
</Route>

NOTE: I tried already to remove the filter as i initally thought that the filter was mistakenly filtering out unwanted logs, but it wasn't the case.

Is this a license problem? can i increase log capacity intake from nxlog?

#2 ArkadiyDeactivated Nxlog ✓
#1 jd01
Hi, I'm working on a heavy log source which can only send syslog. Now currently i also have filters in the config to remove unwanted logs. I've noticed that some logs are successfully being processed whilst others are lost. Through packet capture I was able to conclude that from log source to nxlog server udp packets are all being received. Seems like nxlog (config) can’t handle the large amount of syslog UDP messages coming in. <Extension _syslog> Module xm_syslog </Extension> <Input x_sys_in> Module im_udp Port 514 Host 0.0.0.0 <Exec> parse_syslog_bsd(); if (($Message =~ /dstip=x\.x\.x\.(?:[0-9]+){1,255} dstport=(?:12|5)3/) #DNS & NTP from x.x.x.x or ($Message =~ /dstip=x\.x\.x\.(?:[0-9]+){1,255} dstport=(?:12|5)3/) #DNS & NTP from x.x.x.x or ($Message =~ /srcip=[\d.]+ srcport=\d+ srcintf="x" srcintfrole=".+" dstip=[\d.]+ dstport=16[12]/) #x from x Range or ($Message =~ /srcip=[\d.]+ srcport=\d+ srcintf=".+" srcintfrole=".+" dstip=[\d.]+ dstport=16[12] dstintf="x"/) # #.... more filters drop(); </Exec> </Input> <Output x_sys_out> Module om_udp Host x.x.x.x Port 514 Exec to_syslog_bsd(); </Output> <Route x> Path x_sys_in => x_sys_out </Route> NOTE: I tried already to remove the filter as i initally thought that the filter was mistakenly filtering out unwanted logs, but it wasn't the case. Is this a license problem? can i increase log capacity intake from nxlog?

Hello.

What is amount of EPS we're talking about?
Also was there any spikes of CPU consumption or RAM running out?

Best regards, Arch