Rsyslog Refugee --- some basic questions
Hello:
After spending several months trying to understand rsyslog interstellar configuration -- I am thinking of nxlog (enterprise) for a larger deployment in the coming months. Currently I am messing around with the community edition. while the documentation is nice --- some of example for basic things are a little short on info. Google searches has been a little frustrating as it focuses on Windows alerting which is not what we need.
In our environment we have LTE based devices that connect to our VPN. Essentially these are weather stations for the agricultural industry. The devices are capable of sending itef / bsd style syslog messages to a central server. We are looking for a solution to centrally receive these logs and based on the message content reformat and re forward the messages to something like graylog.
For now I am trying to standardize on a config that would allow me to receive bsd style syslog messages over UDP (plain text) and send the messages to a file. I am running nxlog on Linux -- Centos 7.
nxlog-ce-2.10.2150
usage: nxlog [-h/help] [-c/conf conffile] [-f] [-s/stop] [-v/verify]
[-h] print help
[-f] run in foreground, do not daemonize
[-c conffile] specify an alternate config file
[-r] reload configuration of a running instance
[-s] send stop signal to a running nxlog
[-v] verify configuration file syntax
CentOS Linux release 7.6.1810 (Core)
Requirement:
1) All of the weather stations are in the 10.200.0.0/16 subnets.
2) Dump messages from each weather station into a single file --- in the example below (which doesnt work) I was trying to push the messages into agmon-log
3) If the message contains the words "SENSORFAIL" send only those messages to another file ag-sensor-fail.log
4) add a carriage return / lf after each message so the log is formatted nicely.
Nice to have
- Be able to place the messages from each sensor into an individual file based on some patter of their IP address for example 10.200.16.25 could be agmon-16-25.log
- richer content editing controls --- if the weather station is unable to get a wind reading it sends "NO WINDINFO" or "BAD WINDINFO" message. i would like to kick off a python process if this message is received --- while logging the message to a file.
- If the message contains the words "TEMPDATA=/regexpattern/" would love to log the data to a MARIADB database. Not sure how to look for the regex pattern and if backtick or goup matches apply
Can someone please post some snippets or places to look. Brand new to nxlog. The manual is great but needs better more complete examples. Not sure that nxlog is a fit for this and I would like some help to understand if this is the case.
Im really trying to wrap my head around inputs and routes so that I can direct messages from specific host / based on content to a distinct location. Also trying to determine how vibrant the nxlog community is along with support. Made the mistake of investing in rsyslog --- didnt work out.
########################################
# Global directives #
########################################
User nxlog
Group nxlog
LogFile /var/log/nxlog/nxlog.log
LogLevel INFO
########################################
# Modules #
########################################
<Extension _syslog>
Module xm_syslog
</Extension>
<Input in1>
Module im_udp
Port 514
Exec parse_syslog_bsd();
</Input>
<Input in2>
Module im_tcp
Port 514
</Input>
<Input in5>
Module im_udp
Host 10.200.0.0/16;
Port 514
Exec parse_syslog_bsd();
</Input>
<Output fileout1>
Module om_file
File "/var/log/logmsg.txt"
Exec if $Message =~ /error/ $SeverityValue = syslog_severity_value("error");
Exec to_syslog_bsd();
</Output>
<Output fileout2>
Module om_file
File "/var/log/logmsg2.txt"
</Output>
<Output fileout5>
Module om_file
File "/var/log/agmon-log.txt"
Exec to_syslog_bsd();
</Output>
########################################
# Routes #
########################################
<Route 1>
Path in1 => fileout1
</Route>
<Route tcproute>
Path in2 => fileout2
</Route>
<Route 5>
Path in5 => fileout5
</Route>
Hello, welcome to NXLog :)
I think everything you are looking for is possible, but maybe in different implementation.
Have you checked your log file for any errors?
It may be useful to start off smaller. One input and one file output just to make sure you are getting the data you are expecting, and then move forward.
First issue that you are likely running into is the UDP Host is expecting an IP on the adapter, not a CIDR for in5. This is the interface that NXLog will bind to in order to accept events from remote UDP services. This Host directive also has a semi-colon at the end, this is not required.
The second is you are going to have the same port for two UDP sessions, though in1 is likely not going to do what you are expecting. Without a Host directive, it will default to localhost.
It is probably preferred to consolidate in1 and in5 into one UDP Input and listen on the 'any' ip of 0.0.0.0.
Your outputs may end up having an error as well with Permission denied. The user nxlog will not likely have write access to /var/log in order to write out logmsg.txt, logmsg2.txt, and agmon-log.txt. You may have already resolved this by setting permissions on that directory or the files directly.
For the output fileout5, you shouldn't need the to_syslog_bsd() if it is coming in as syslog BSD already. This would be for a case where you have modified some fields and want to send it back out to BSD Syslog.
Dump messages from each weather station into a single file --- in the example below (which doesnt work) I was trying to push the messages into agmon-log
This config should allow all UDP in into one file.
<Input in1>
Module im_udp
Host 0.0.0.0
Port 514
Exec parse_syslog_bsd();
</Input>
<Output fileout5>
Module om_file
File "/var/log/agmon-log.txt"
</Output>
<Route 5>
Path in1 => fileout5
</Route>
With a slight modification, you can store them all in their own files based on $Hostname field. I suggest adding a directory though so you don't flood your log directory.
<Input in1>
Module im_udp
Host 0.0.0.0
Port 514
Exec parse_syslog_bsd();
</Input>
<Output fileout5>
Module om_file
File "/var/log/nxlogOut/" + $Hostname + ".log"
CreateDir TRUE
</Output>
<Route 5>
Path in1 => fileout5
</Route>
add a carriage return / lf after each message so the log is formatted nicely.
NXLog will place each event on its own line by default. If you are wanting to add an additional line, you could add the following to your output:
`Exec $raw_event = $raw_event + "\n";
If the message contains the words "SENSORFAIL" send only those messages to another file ag-sensor-fail.log
You could use the following in an output Instance that uses om_file to create ag-sensor-fail.log. Any non-matching event would be dropped for this output. This same style can be used for your TEMPDATA example, just use an om_odbc instance. (we do support back reference and grouping as we are PCRE compliant.)
Exec if $raw_event !~ /SENSORFAIL/ drop();
Im really trying to wrap my head around inputs and routes
You don't need to maintain a 1:1 ratio of in to out. You could have both your UDP and TCP going to all outs and filter on the outs to accomplish what you are looking for.
i would like to kick off a python process if this message is received --- while logging the message to a file.
EE can do this, there are several extension modules that can run Python, Perl, Ruby, Go, and Exec for shell/cmd support.
The EE manual has most of these examples. Here is a link to the full guide in HTML. Warning, it is a large document :)
https://nxlog.co/documentation/nxlog-user-guide-full
Hope this helps!
