Hi, I'm working on a heavy log source which can only send syslog. Now currently i also have filters in the config to remove unwanted logs. I've noticed that some logs are successfully being processed whilst others are lost. Through packet capture I was able to conclude that from log source to nxlog server udp packets are all being received. Seems like nxlog (config) can’t handle the large amount of syslog UDP messages coming in. <Extension _syslog> Module xm_syslog </Extension> <Input x_sys_in> Module im_udp Port 514 Host 0.0.0.0 <Exec> parse_syslog_bsd(); if (($Message =~ /dstip=x\.x\.x\.(?:[0-9]+){1,255} dstport=(?:12|5)3/) #DNS & NTP from x.x.x.x or ($Message =~ /dstip=x\.x\.x\.(?:[0-9]+){1,255} dstport=(?:12|5)3/) #DNS & NTP from x.x.x.x or ($Message =~ /srcip=[\d.]+ srcport=\d+ srcintf="x" srcintfrole=".+" dstip=[\d.]+ dstport=16[12]/) #x from x Range or ($Message =~ /srcip=[\d.]+ srcport=\d+ srcintf=".+" srcintfrole=".+" dstip=[\d.]+ dstport=16[12] dstintf="x"/) # #.... more filters drop(); </Exec> </Input> <Output x_sys_out> Module om_udp Host x.x.x.x Port 514 Exec to_syslog_bsd(); </Output> <Route x> Path x_sys_in => x_sys_out </Route> NOTE: I tried already to remove the filter as i initally thought that the filter was mistakenly filtering out unwanted logs, but it wasn't the case. Is this a license problem? can i increase log capacity intake from nxlog?

Hi, I'm currently using nxlog to forward RADIUS messages via syslog to my firewall. However, it has recently started complaining that the packets are too big, and so fragmentation is occurring which it doesn't like. The temporary fix was to force the packets to cut at 1450 bytes, and this is my current config: Panic Soft #NoFreeOnExit TRUE define ROOT C:\Program Files (x86)\nxlog define CERTDIR %ROOT%\cert define CONFDIR %ROOT%\conf define LOGDIR %ROOT%\data define LOGFILE %LOGDIR%\nxlog.log LogFile %LOGFILE% Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data <Extension _xml> Module xm_xml </Extension> <Extension _charconv> Module xm_charconv AutodetectCharsets iso8859-2, utf-8, utf-16, utf-32 </Extension> <Extension _exec> Module xm_exec </Extension> <Extension _fileop> Module xm_fileop # Check the size of our log file hourly, rotate if larger than 5MB &lt;Schedule&gt; Every 1 hour Exec if (file_exists('%LOGFILE%') and \ (file_size('%LOGFILE%') &gt;= 5M)) \ file_cycle('%LOGFILE%', 8); &lt;/Schedule&gt; # Rotate our log file every week on Sunday at midnight &lt;Schedule&gt; When @weekly Exec if file_exists('%LOGFILE%') file_cycle('%LOGFILE%', 8); &lt;/Schedule&gt; </Extension> <Input NPS> Module im_file File "C:\Windows\System32\LogFiles\IN*.log" InputType LineBased SavePos TRUE ReadFromLast TRUE &lt;Exec&gt; # Discard everything that doesn't seem to be an xml event if $raw_event !~ /^&lt;Event&gt;/ drop(); # Filter to only events containing all required data (type, username and ip) if $raw_event !~ /(Type\sdata_type=&quot;0&quot;&gt;)(\d{1,2})(&lt;\/Acct)(.+)(Name\sdata_type=&quot;1&quot;&gt;)([a-zA-Z0-9\$\._-]{3,15})(.*)(&lt;\/User)(.+)(Address\sdata_type=&quot;3&quot;&gt;)([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3})(&lt;\/Framed)/ drop (); # Truncates event to 1400 bytes due to MTU limits $raw_event = substr($raw_event, 0, 1450); # Reduces event string to just required data (type, username and ip) # Parse xml parse_xml(); &lt;/Exec&gt; </Input> <Output Firewall> Module om_udp # Put your Firewal Management interface IP address # Don't change port or protocol (should be UDP 514 or TCP 6514) Host 192.168.1.1 Port 514 </Output> <Output SyslogServer> # Put your Syslog Server IP address and port # Allows monitoring of messages being sent to firewall Module om_udp Host 192.168.1.10 Port 514 </Output> <Route 1> Path NPS => Firewall </Route> <Route 2> Path NPS => SyslogServer </Route> However, I'd prefer a neater solution, rather than just chopping the end off the packet. The only parts of the packet I'm interested in forwarding are: Event Regex: <Acct-Status-Type\sdata_type="0">1</Acct-Status-Type>{1} Username Regex: <User-Name\sdata_type="1">([a-zA-Z0-9\._-]+)</User-Name> Address Regex: <Framed-IP-Address\sdata_type="3">([0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3})</Framed-IP-Address> Is there a way to extract just those bits and parse that to the output? Apologies if it's obvious, but I don't really understand how nxlog works! Give me powershell and I'm happy.... Thanks, Stephen

I'm currently using nxlog to filter and forward syslog: Source => Filter Logs on intermediate server with nxlog installed => forward udp 514 (syslog). Config looks like the following: <Extension _syslog> Module xm_syslog </Extension> #syslog to Nxlog01 <Input _sys_in> Module im_udp Port 514 <Exec> if (($Message =~ /REGEX/) or ($Message =~ /REGEX/)) drop(); </Exec> Exec if $MessageSourceAddress != "1.1.1.1" drop(); </Input> <Processor _sys_norepeat> Module pm_norepeat CheckFields Hostname, Message, SourceName </Processor> #Syslog Output to DST server <Output _sys_out> Module om_udp Host dstserver_ip Port 514 Exec to_syslog_bsd(); </Output> <Route Logs> Path _sys_in => _sys_norepeat => _sys_out </Route> Firstly is this possible with the CE agent? I already performed a tcpdump on the intermediate server where nxlog is installed and i can see the syslog being received but not sent after filtered. There is udp 514 connectivity between nxlog server and destination server. I tried: Removing all regex filtering (since i thought my regex wasn't good enough), no results. Changed im_udp to im_tcp (i thought maybe you can't used the same port in both input and output modules). Checked whether there actually is network connectivity between nxlog server and destination server using nc by sending a udp 514 packet to dst server. Can anyone help with this?

We're using NX Log (CE) as a test to see if it will work for our purposes. The overall idea is to use it as a forwarder of syslog flat files to any brand of SIEM. My config looks like this: Panic Soft #NoFreeOnExit TRUE define ROOT C:\Program Files (x86)\nxlog define CERTDIR %ROOT%\cert define CONFDIR %ROOT%\conf define LOGDIR %ROOT%\data define LOGFILE %LOGDIR%\nxlog.log LogFile %LOGFILE% Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data <Extension syslog> Module xm_syslog </Extension> #<Input Alerts> Module im_file File 'E:\DGQradarExports\ForwarderCust\Alerts\*' ReadFromLast True Exec parse_syslog(); #</Input> <Input Events> Module im_file File 'E:\DGQradarExports\ForwarderCust\Events\*' ReadFromLast True Exec parse_syslog(); </Input> <Input Process> Module im_file File 'E:\DGQradarExports\ForwarderCust\Process\*' ReadFromLast True Exec parse_syslog(); </Input> <Output Customer> Module om_tcp Host 192.168.160.141 Port 514 </Output> <Route customer_siem> Path Events,Process => Customer </Route> <Extension _charconv> Module xm_charconv AutodetectCharsets iso8859-2, utf-8, utf-16, utf-32 </Extension> <Extension _exec> Module xm_exec </Extension> <Extension _fileop> Module xm_fileop # Check the size of our log file hourly, rotate if larger than 5MB &lt;Schedule&gt; Every 1 hour Exec if (file_exists('%LOGFILE%') and \ (file_size('%LOGFILE%') &gt;= 5M)) \ file_cycle('%LOGFILE%', 8); &lt;/Schedule&gt; # Rotate our log file every week on Sunday at midnight &lt;Schedule&gt; When @weekly Exec if file_exists('%LOGFILE%') file_cycle('%LOGFILE%', 8); &lt;/Schedule&gt; </Extension> My log just shows this: 2018-11-19 08:50:43 INFO nxlog-ce-2.10.2102 started 2018-11-19 08:50:43 INFO connecting to 192.168.160.141:514 QRadar shows an information source has registered but no data ever flows. I should see a 'connection was successful' message shouldn't I? Is there any way to up the logging so I can tell if NX Log is even reading the files and attempting to send them? I really can't tell what it's doing currently. Multiple files exist in the input directories, I'm trying to have NX Log work through all of them, send them to SIEM and then wait for more files. Config examples seem straightforward, I just can't tell what it's doing. Any help is appreciated.

Hello everybody, I'm sorry to bother you with another question concerning Windows Eventlog forwarding to graylog. Unfortunately I'm not able to figure this out on my own. used versions: nxlog 2.10.2102 (running on Windows Server 2016) graylog 2.4.6 (running on Debian 9) I have two nxlog setups. One using syslog and another one using GELF. Both do not work as I would expect. 1. Syslog <Extension syslog> Module xm_syslog </Extension> <Input eventlog> Module im_msvistalog Exec delete($Keywords); Exec if ($EventType == "VERBOSE") drop(); </Input> <Output out_graylog> Module om_tcp Host graylog Port 5140 Exec $raw_event = replace($raw_event, "\n", " "); Exec $raw_event = replace($raw_event, "\r", " "); Exec $raw_event = replace($raw_event, "\t", " "); Exec to_syslog_ietf(); </Output> <Route route_eventlog> Path eventlog => out_graylog </Route> The problem is that there are eventlog entries containing line breaks. Unfortunately they are not removed by the replace commands. So in graylog one message is split into many messages with every linebreak. Using wireshark I can observe that the linebreaks consist of LF characters (Unix line endings). 2. Gelf <Extension gelf> Module xm_gelf </Extension> <Input eventlog> Module im_msvistalog Exec delete($Keywords); Exec if ($EventType == "VERBOSE") drop(); </Input> <Output out_graylog> Module om_tcp Host graylog Port 12201 OutputType GELF </Output> <Route route_eventlog> Path eventlog => out_graylog </Route> Unfortunately this setup does not work at all. No messages are showing up in Graylog (of course I've activated the correspnding input). Using wireshark I can observe that a lot of TCP packets are sent to graylog but none of them contain readable messages. Can anybody help me with either setup? Thanks and Regards, Carsten

wiht the community eddition when the nxlog-ce is listening on /dev/log and for some reasons the systemd-journald (debian 9) removes the socket the nxlog blocks the compleate host. (even no login possible) after a restart of nxlog the host recovers.

Here is a sample event when using to_syslog_snare() in the nxlog.conf: <14>Jan 27 10:03:39 event_computer MSWinEventLog        1        Security        32630749        Wed Jan 27 10:03:39 2016        4624        Microsoft-Windows-Security-Auditing        N/A        N/A        Success Audit        event_computer        Logon                An account was successfully logged on.    Subject:   Security ID:  S-1-0-0   Account Name:  -   Account Domain:  -   Logon ID:  0x0    Logon Type:   3    Impersonation Level:  Impersonation    New Logon:   Security ID:  S-1-5-21-2705889813-1605608894-1661845433-43745   Account Name:  account_name   Account Domain:  account_domain   Logon ID:  0x23820B882   Logon GUID:  {00000000-0000-0000-0000-000000000000}    Process Information:   Process ID:  0x0   Process Name:  -    Network Information:   Workstation Name: workstation_name   Source Network Address: source_address   Source Port:  54241    Detailed Authentication Information:   Logon Process:  NtLmSsp    Authentication Package: NTLM   Transited Services: -   Package Name (NTLM only): NTLM V2   Key Length:  0    This event is generated when a logon session is created. It is generated on the computer that was accessed.    The subject fields indicate the account on the local system which requested the logon. This is most commonly a service such as the Server service, or a local process such as Winlogon.exe or Services.exe.    The logon type field indicates the kind of logon that occurred. The most common types are 2 (interactive) and 3 (network).    The New Logon fields indicate the account for whom the new logon was created, i.e. the account that was logged on.    The network fields indicate where a remote logon request originated. Workstation name is not always available and may be left blank in some cases.    The impersonation level field indicates the extent to which a process in the logon session can impersonate.    The authentication information fields provide detailed information about this specific logon request.   - Logon GUID is a unique identifier that can be used to correlate this event with a KDC event.   - Transited services indicate which intermediate services have participated in this logon request.   - Package name indicates which sub-protocol was used among the NTLM protocols.   - Key length indicates the length of the generated session key. This will be 0 if no session key was requested.        35284558 My issue is that I would NOT want to collect the "informational text" representing the event - in this case everything starting from the string "This event is generated---" all the way up until "--was requested." Before I go any deeper into this, let me state that in the logs of this format I call the "<14>Jan 27 10:03:39 event_computer MSWinEventLog        1        Security        32630749        Wed Jan 27 10:03:39 2016        4624        Microsoft-Windows-Security-Auditing        N/A        N/A        Success Audit        event_computer        Logon" portion of the whole log message the HEADER, and the rest is called MESSAGE. Putting it another way, I would like to forward the message using syslog in a format constructed according to the pseudocode below: parse fields from windows event /* e.g. SubjectUserName, LogonType, IpAddress, etc. */ /* print the header "as is" already in the to_syslog_snare() format, i.e. from "<14>---" until and including "---Logon" print HEADER /* e.g. event_time,event_computer,event_type,event_id,... */ for all fields parsed     print "'field_name=field_value'" /* e.g. SubjectUserName=value,LogonType=value,IpAddress=value,... /*      The reason I would like to do this is that the informational text, which gets appended to some Windows events (not all, it seems), takes a lot of space, and we do not really need this information text for anything. Another way to do this would be to statically list all the fields POSSIBLY found in an Windows event and construct the message that way, but this would often leave me with a lot of empty key-value pairs. THUS I would only like to print out those fields that were found in that specific log message while leaving out the informational message. I do acknowledge, though, that especially Application and System events might not contain most or any of the fields that are present in a Security log event. Take for example the following System log event: <14>Jan 27 11:09:21 event_computer MSWinEventLog        1        System        32633951        Wed Jan 27 11:09:21 2016        7036        Service Control Manager        N/A        N/A        Information        event_computer        N/A                The Remote Registry service entered the stopped state.        319889 In the example above, the "header" portion of the whole message only contains the string "The Remote Registry service entered the stopped state." I do hope, though, that the variable where this string is stored is actually the same that hosted the string "An account was successfully logged on.", which would mean that my approach in the pseudocode would still work (i.e. the array or list of fields that is iterated and printed would only contain one field. The HEADER portion of the field is exactly the same in all messages. The description of to_syslog_snare() in the nxlog documentation states: "Create a SNARE Syslog formatted log message in $raw_event. Uses the following fields to construct $raw_event: $EventTime, $Hostname, $SeverityValue, $FileName, $EventID, $SourceName, $AccountName, $AccountType, $EventType, $Category, $Message." Thus when reflecting back to what I said, it seems that what I call the HEADER includes all the fields from $EventTime to (and including) $Category - this I would like to keep as it is. But according to the documentation, the $Message variable actually then holds all the other information in the log, or what I call the MESSAGE portion. So I guess the question is that can the contents of the $Message variable be further filtered, as it obviously is constructed from e.g. EventData's Data fields listed below. I would like to only change the $Message contents so that it would never contain the informational text if there exists such a message in a given log message, and that preferably the Data fields inside $Message would be formatted using key-value pairs instead of the to_syslog_snare format seen in the first example (one or more whitespace as delimiter).