Creating alerts to determine if a data source is no longer sending logs
Hello,
I am collecting from many different types of log sources and many of those are syslog streams where an agent is not installed on the data source. For instance our firewalls send logs over syslog to our nxlog agent for collection. I want to setup an alert that would fire if any of these sources stops sending logs. I read over the documentation at "26.5. Detecting a Dead Agent or Log Source" which would only fire an alert if that input module receives no logs in an hour correct? I would like to have the input module I use to collect syslog detect if any device out of many stops sending in that single feed. For instance the firewalls all have a device name that appears in the log message so I would like to use $raw_event and a grep statement to pull out the device name, count how many events it has sent, and alert if it is less than 1 in 30 minutes. For instance lets say I have several devices with device names that start with "exam1- and can end with any number of letters after that but ends with a closing quote. Currently I filter these messages to send to different outputs like this.
if not ($raw_event =~ /devname="exam1-[^"]+"/) drop();
I would like to have the input module use the same filter message to count the events for each of these devices individually. Not exactly sure how this could be completed and could use some help. Thank you.