Hi there, I'm having a little trouble trying to filter events with patterndb.xml I'm sending logs to our SIEM but despite the corresponding event ID's missing from patterndb they are still getting pushed. I think my configuration setup is over ruling the patterndb config. Can you please review? Thanks for your time. # # Configuration for converting and sending Windows logs # to AlienVault USM Anywhere. # # Version: 0.1.0 # Last modification: 2019-07-03 # define ROOT C:\Program Files (x86)\nxlog define OUTPUT_DESTINATION_ADDRESS x.x..x.xx define OUTPUT_DESTINATION_PORT 12346789 Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data LogFile %ROOT%\data\nxlog.log Module xm_json Module xm_syslog Module im_internal Module im_msvistalog Query \ \ *\ *\ *\ \ Exec if ($EventID == 5156) OR ($EventID == 5158) drop(); Module om_udp Host %OUTPUT_DESTINATION_ADDRESS% Port %OUTPUT_DESTINATION_PORT% Exec $EventTime = integer($EventTime) / 1000000; Exec $EventReceivedTime = integer($EventReceivedTime) / 1000000; Exec $Message = to_json(); to_syslog_bsd(); Path eventlog, internal => out ############################################################################ #### NXLOG WITH PATTERNDB ##### #### Uncomment the following lines for Windows Events filtered ##### ############################################################################ Module im_internal Module im_msvistalog Query \ \ *\ *\ *\ \ Module pm_pattern PatternFile %ROOT%\conf\patterndb.xml Module om_udp Host %OUTPUT_DESTINATION_ADDRESS% Port %OUTPUT_DESTINATION_PORT% Exec $EventTime = integer($EventTime) / 1000000; Exec if not defined $PatternID or not defined $Message { drop(); } Exec $Message = to_json(); to_syslog_bsd(); Path eventlog_Pattern, internal_Pattern => match_events => out_Pattern ############################################################################ ##### /NXLOG WITH PATTERNDB ##### ############################################################################

Hi, This is my sql_fetch command :

$Retval = sql_fetch("SELECT ServerName, Transmission FROM dbo.SrvAuth WHERE ServerName = ?", $MachineCourt);

This command does find the right record based on ServerName but it is always putting the second field, $Transmission, to the value FALSE.

Here is the MS SQL table definition : Colum Name Data Type Allow Nulls ServerName varchar(50) Unchecked Transmission bit Unchecked

Depending of the record that it is fetch, the DB contains about a third of TRUE and 2 third of FALSE for the Transmission field.

Question : Why do I always fetch FALSE for the Transmission field?

Thanks

When NXLog formats the the Event Log as `om_out` it formats the JSON correctly ``` define Format {if defined($EventTime) $timestamp = strftime($EventTime, '%Y-%m-%dT%H:%M:%SZ');else $timestamp = strftime($EventReceivedTime, '%Y-%m-%dT%H:%M:%SZ');rename_field("service_id", "_service_id");rename_field("timestamp", "_timestamp");rename_field("log_type", "_log_type");$body = $raw_event;$attributes = to_json();if defined($tag) $raw_event = "{" + '"timestamp"' + ':"' + $_timestamp + '",' + '"service_id"' + ':"' + $_service_id + '",' + '"tag"' + ':"' + $tag + '",' + '"log_type"' + ':"' + $_log_type + '",' + '"attributes"' + ':' + $attributes + '}';else $raw_event = "{" + '"timestamp"' + ':"' + $_timestamp + '",' + '"service_id"' + ':"' + $_service_id + '",' + '"log_type"' + ':"' + $_log_type + '",' + '"attributes"' + ':' + $attributes + '}';} ``` This is executed in the `` block which formats it into JSON format. When the `om_http` is called the same way as `om_out` an error is logged as an over sized string. At first the JSON looks normal but as the code goes on you get an excessively long string. Packet capture from Wireshark showing the end of the REST POST request. ``` POST / HTTP/1.1 User-Agent: nxlog-ce Content-Length: 621554 Beginning: {"timestamp":"2020-02-17T14:19:33Z","service_id":"id","tag":"security","log_type":"ea2_test","attributes":{"EventTime":"2020-02-17 14:19:33","Hostname":"hostname","Keywords":-9214364837600034816,"EventType":"AUDIT_SUCCESS","SeverityValue":2,"Severity":"INFO","EventID":4663,"SourceName":"Microsoft-Windows-Security-Auditing", End: Accesses:\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\t\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\tWRITE_DAC\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\r\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\n\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\t\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\t\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\t\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\t\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\r\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\n\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\tAccess Mask:\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\t\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\t0x40000\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"}\\\\\\\\\\\\\\\"}\\\\\\\"}\\\"}\"}"}} ``` Why is it not working when you use the `om_http` module but works with the `om_out` module. Suggestions? Thanks in advance! ***EDIT:*** It looks like NXLog-CE broke itself. I was able to fix this by deleting everything in the nxlog/data folder and then reinstalled the agent. Now, using the exact same config files it appears to be working.

Hi all,

Sorry to come with an other new question about that but I don't understand why the regex didn't match the Message:

regexp /(?x)^\s?[(\d+):(\d+):(\d+)] (.+?) [Classification: (.+?)] [Priority: (\d+)] {(.+?)} (\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3})(:(\d{1,5}))? -> (\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3})(:(\d{1,5}))?\R?/ doesn't match subject string '[129:20:1] TCP session without 3-way handshake [Classification: Potentially Bad Traffic] [Priority: 2] {TCP} 192.168.0.1:8080 -> 192.168.0.2:53590'

If I compare that on Online regex site (PCRE), it works.

Thanks

Hi I got to set up a log forwarding to a syslog-server. I managed everthing to work except one thing.

The newest event at the end of the logfile has no CR, CRLF, LF or something similar. When a event occurs it is processed when the next event occurs ...

How can I make NXLog to read and process to the end of the file (EOF) in case the file has changed? So far I uses the im_file module.

Thank you for you help Daniel

Hi all,

Does Nxlog Enterprise has the possibility to request a table in order to convert some field ?

Like EventID 4624 on Windows and replace LogonType ID to a more readable string:

    "2": "Interactive",
    "3": "Network",
    "4": "Batch",
    "5": "Service",
    "7": "Unlock",
    "8": "NetworkCleartext",
    "9": "NewCredentials",
    "10": "RemoteInteractive",
    "11": "CachedInteractive",

Hello,

I have installed NXLog community edition to collect table data from PostgreSQL database but, the table isn't contain an ID column. As I understand, NXLog required to this field to bookmark but, we don't have. I'm looking for a workaround to solve the issue. On the other hand I can see a workaround from the following link and we can configure the ID with select statements but, the article isn't about the PostgreSQL. Could you please someone help me for PostgreSQL?

https://nxlog.co/documentation/nxlog-user-guide/mssql.html

the second question: Can we define a specific column (such as eventime) for ID (bookmark) with the following sample data?

2020-02-11 15:00:00.0000 2020-02-11 15:00:01.0001 2020-02-11 15:00:02.0002 2020-02-11 15:00:03.0000

Thanks in Advance!

Best Regards

SD

Hi there, a little bit of a novice here. Hope you dont mind pointing me in the right direction. I’m having some difficulty getting the configuration for using patternDB on windows 2003 servers, the configuration works for windows 2008+ The logs I have are as follows: 2020-02-05 13:48:32 ERROR invalid keyword: Query at C:\Program Files\nxlog\conf\nxlog.conf:40 2020-02-05 13:48:32 ERROR invalid keyword: Query at C:\Program Files\nxlog\conf\nxlog.conf:76 2020-02-05 13:48:32 ERROR module 'eventlog' has configuration errors, not adding to route '1' at C:\Program Files\nxlog\conf\nxlog.conf:57 2020-02-05 13:48:32 ERROR module 'eventlog_Pattern' has configuration errors, not adding to route 'route_Pattern' at C:\Program Files\nxlog\conf\nxlog.conf:94 2020-02-05 13:48:32 WARNING not starting unused module eventlog 2020-02-05 13:48:32 WARNING not starting unused module eventlog_Pattern 2020-02-05 13:48:32 INFO nxlog-ce-2.10.2150 started The section of conf is: ############################################################################ #### NXLOG WITH PATTERNDB ##### #### Uncomment the following lines for Windows Events filtered ##### ############################################################################ Module im_internal Module im_msvistalog Query \ \ *\ *\ *\ \ Module pm_pattern PatternFile %ROOT%\conf\patterndb.xml Module om_udp Host %OUTPUT_DESTINATION_ADDRESS% Port %OUTPUT_DESTINATION_PORT% Exec $EventTime = integer($EventTime) / 1000000; Exec if not defined $PatternID or not defined $Message { drop(); } Exec $Message = to_json(); to_syslog_bsd(); Path eventlog_Pattern, internal_Pattern => match_events => out_Pattern ############################################################################ ##### /NXLOG WITH PATTERNDB ##### ############################################################################ Thanks for reading. Please let me know if any more information needs to be included.

Hi all,

According to the documentation found here it indicates that the generic RPM doesn't have all available modules as opposed to the version specific RPM:

The generic RPM above contains all the libraries (such as libpcre and libexpat) 
that are needed by NXLog, the only dependency is libc. 
However, some modules are not available (im_checkpoint, for example). 
The advantage of the generic RPM is that it can be installed on most RPM-based Linux distributions.

Is there documentation for what modules are not available?
Are there any issues for deploying this version that I should know about up front?

Thanks!!

How can we rename nxlog package ? while we are placing both the rpm into spacewalk channel these are updating as “nxlog-ce-2.10.2150-1.x86_64.rpm” .So it's making a duplicate,So i hope renaming the rpm name will help us. Any help will be appreciated on this.

nxlog-ce-2.10.2150-1_rhel6.x86_64.rpm nxlog-ce-2.10.2150-1_rhel7.x86_64.rpm

Thanks! Ela

Our project is planning to reissue certificates for large amount of agents. Do we have API on the certificates so we'll able to reissue on these agents at the same time without doing it manually (one by one)?

Hi,

We are planning to deploy NXLog to thousands of endpoints and need to know when an agent is no longer sending data regularly.

Is there an established method for determining NXLog is working normally at scale?

Thanks!

Hello, I resisted posting here for a while but am finally at a loss to explain what I'm observing. I'm trying to send nginx access logs to graylog, and am mostly using code adapted from the nxlog ce user guide but I haven't been able to get the fields to successfully capture and arrive in graylog. The nginx server in question logs two sorts of traffic: 1. `x.x.x.x - - [04/Feb/2020:03:23:22 +0000] "GET /" 400 271 "-" "-" "-"` - These are status checks from a load balancer which I'm wanting to drop. 2. `x.x.x.x - [04/Feb/2020:03:23:01 +0000] "POST /rest/api/endpoint HTTP/1.1" 201 508 "-" "okhttp/3.3.0" "-"` - This is legitimate traffic to the application behind nginx which I want to parse and capture. I have the following input defined in my config file. Module im_file File '/var/log/nginx/access.log' PollInterval 1 SavePos True ReadFromLast True Recursive False RenameCheck False if $raw_event =~ /(?x)^(\S+)\ \S+\ (\S+)\ \[([^\]]+)\]\ \"(\S+)\ (.+)\ HTTP\/\d\.\d\"\ (\S+)\ (\S+)\ \"([^\"]+)\"\ \"([^\"]+)\"\ \"\S+\"$/ { $Hostname = $1; if $2 != '-' $AccountName = $2; $EventTime = parsedate($3); $HTTPMethod = $4; $HTTPURL = $5; $HTTPResponseStatus = $6; if $7 != '-' $FileSize = $7; if $8 != '-' $HTTPReferer = $8; if $9 != '-' $HTTPUserAgent = $9; delete($Message); } else drop(); If I remove the `else drop();` then this config forwards only the load balancer lines that I don't want, which indicates to me that the lines I do want are matching my regexp. But as shown, this configuration forwards nothing. I also don't think anything is wrong with my outputs or routes since just straight piping this all to graylog in the message field works fine. If anyone can help me understand what is going on here that would be much appreciated. Thanks

Hey y'all. Does anyone know where to find the GPG key used to sign the NXLog Enterprise Edition (trial) RPM packages? Key ID `9354d2051da9e40e`, or just `1da9e40e` for short. If someone at NXLog reads this, maybe [this page][] would be a good site to link it from, if it's not immediately available in the download package. I figured if I post this question here, maybe someone else will find it useful in the future. ([Wisdom of the Ancients, and all that][]) Thanks :) [this page]: https://nxlog.co/documentation/nxlog-user-guide/deploy-rhel.html [Wisdom of the Ancients, and all that]: https://xkcd.com/979/

I am currently running into an issue receiving syslog over ssl/tls. I cannot figure it out for the life of me!

Version: CE-2.10.2150

Error: INFO SSL connection accepted from IP_ADDRESS:PORT ERROR SSL certificate verification failed: unsupported certificate purpose (err: 26) WARNING SSL connection closed from IP_ADDRESS:PORT

Config: <Input in> Module im_ssl Host 0.0.0.0 Port 516 AllowUntrusted TRUE CAFile %CERTDIR%%CA-PEM% CertFile %CERTDIR%%CRT% CertKeyFile %CERTDIR%%KEY% KeyPass %PASSWORD% </Input>

Hi, I am using NXLog to run a script via `im_exec` and the internal `restart true` feature which restarts my script when it ends. The thing I'm noticing is that it's filling up the nxlog.log file with error conditions because the script stopped. `ERROR Module test got EOF, process exited?` The script is restarting deliberately so I don't need to know about these specific messages from this specific module. How can I suppress error logging from a specific module? Have a wonderful day! ***NOTE:*** I was able to get it working a different way. The solution involves executing a script on startup to get the initial useful data and send it to a log file. From there NXLog searches for that file and copies it to `gateway.log`. The input module `im_file` then reads that file which initiates the heartbeat. The way that `im_file` works is that it caches the file and won’t read from it again unless the file changes(modifying last modified doesn’t appear to count as a change). To that end I added logic that would periodically delete and re-copy the file. In this way `im_file` has a new file to read from, and thus send the heartbeat. While NXLog logs when the new files are read, they are logged as WARNING, not ERROR, so you can set the `LogLevel` to `ERROR` ``` Module xm_fileop Every 25 sec if file_exists('%GATEWAY%.log') file_remove('%GATEWAY%.log'); if file_exists('%GATEWAY%') file_copy('%GATEWAY%', '%GATEWAY%.log'); Module im_file PollInterval 60 ReadFromLast false SavePos false File '%GATEWAY%.log' $gateway = $raw_event; ``` This is not meant to be some kind of groundbreaking discovery. I only post it here in case someone else ends up trying to figure out this requirement. Hope this helps!

Attempting to log to a syslog server based upon the specified filtered log ID's. When I use this config, I do not get any errors in the error log when the service starts. However I do not get anything to my syslog server. Not sure if this a problem with the in or output and would love some feedback.

Panic Soft #NoFreeOnExit TRUE

define ROOT C:\Program Files (x86)\nxlog define CERTDIR %ROOT%\cert define CONFDIR %ROOT%\conf define LOGDIR %ROOT%\data define LOGFILE %LOGDIR%\nxlog.log

LogFile %LOGFILE%

Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data

<Extension _syslog> Module xm_syslog </Extension> define HighEventIds 4618, 4649, 4719, 4765, 4766, 4794, 4897, 4964, 5124, 1102

define MediumEventIds 4621, 4675, 4692, 4693, 4706, 4713, 4714, 4715, 4716, 4724,
4727, 4735, 4737, 4739, 4754, 4755, 4764, 4764, 4780, 4816,
4865, 4866, 4867, 4868, 4870, 4882, 4885, 4890, 4892, 4896,
4906, 4907, 4908, 4912, 4960, 4961, 4962, 4963, 4965, 4976,
4977, 4978, 4983, 4984, 5027, 5028, 5029, 5030, 5035, 5037,
5038, 5120, 5121, 5122, 5123, 5376, 5377, 5453, 5480, 5483,
5484, 5485, 6145, 6273, 6274, 6275, 6276, 6277, 6278, 6279,
6280, 24586, 24592, 24593, 24594

define LowEventIds 4608, 4609, 4610, 4611, 4612, 4614, 4615, 4616, 4624, 4625,
4634, 4647, 4648, 4656, 4657, 4658, 4660, 4661, 4662, 4663,
4672, 4673, 4674, 4688, 4689, 4690, 4691, 4696, 4697, 4698,
4699, 4700, 4701, 4702, 4704, 4705, 4707, 4717, 4718, 4720,
4722, 4723, 4725, 4726, 4728, 4729, 4730, 4731, 4732, 4733,
4734, 4738, 4740, 4741, 4742, 4743, 4744, 4745, 4746, 4747,
4748, 4749, 4750, 4751, 4752, 4753, 4756, 4757, 4758, 4759,
4760, 4761, 4762, 4767, 4768, 4769, 4770, 4771, 4772, 4774,
4775, 4776, 4778, 4779, 4781, 4783, 4785, 4786, 4787, 4788,
4789, 4790, 4869, 4871, 4872, 4873, 4874, 4875, 4876, 4877,
4878, 4879, 4880, 4881, 4883, 4884, 4886, 4887, 4888, 4889,
4891, 4893, 4894, 4895, 4898, 5136, 5137

<Input events> Module im_msvistalog <QueryXML> <QueryList> <Query Id="0" Path="Directory Service"> <Select Path="Directory Service">*[System[Provider[ @Name='Microsoft-Windows-ActiveDirectory_DomainService']]] </Select> </Query> </QueryList> </QueryXML> <Exec> if $EventID NOT IN (%HighEventIds%) and $EventID NOT IN (%MediumEventIds%) and $EventID NOT IN (%LowEventIds%) drop(); </Exec> </Input>

<Output udp> Module om_udp Host 172.17.103.13 Port 514 Exec to_syslog_snare(); </Output>

<Route uds_to_udp> Path events => udp </Route>

<Extension _charconv> Module xm_charconv AutodetectCharsets iso8859-2, utf-8, utf-16, utf-32 </Extension>

<Extension _exec> Module xm_exec </Extension>

<Extension _fileop> Module xm_fileop

# Check the size of our log file hourly, rotate if larger than 5MB
<Schedule>
    Every   1 hour
    Exec    if (file_exists('%LOGFILE%') and \
               (file_size('%LOGFILE%') >= 5M)) \
                file_cycle('%LOGFILE%', 8);
</Schedule>

# Rotate our log file every week on Sunday at midnight
<Schedule>
    When    @weekly
    Exec    if file_exists('%LOGFILE%') file_cycle('%LOGFILE%', 8);
</Schedule>

</Extension>

I am looking for the buster Debian package. I already tried stretch version but it complained about unmet dependencies related to libssl.

The following packages have unmet dependencies:
 nxlog-ce : Depends: libperl5.24 (>= 5.24.0) but it is not installable
            Depends: libssl1.0.2 (>= 1.0.2d) but it is not installable

root@debian:~# dpkg --search libssl
libssl1.1:amd64: /usr/share/doc/libssl1.1
libssl1.1:amd64: /usr/share/doc/libssl1.1/changelog.Debian.gz
libssl-dev:amd64: /usr/lib/x86_64-linux-gnu/pkgconfig/libssl.pc
libssl1.1:amd64: /usr/share/doc/libssl1.1/NEWS.Debian.gz
libssl-dev:amd64: /usr/share/doc/libssl-dev/changelog.gz
libssl1.1:amd64: /usr/lib/x86_64-linux-gnu/libssl.so.1.1
libssl-dev:amd64: /usr/share/doc/libssl-dev
libssl-dev:amd64: /usr/lib/x86_64-linux-gnu/libssl.a
libssl-dev:amd64: /usr/share/doc/libssl-dev/copyright
libssl1.1:amd64: /usr/share/doc/libssl1.1/copyright
libssl-dev:amd64: /usr/lib/x86_64-linux-gnu/libssl.so
libssl-dev:amd64: /usr/share/doc/libssl-dev/changelog.Debian.gz
android-libboringssl: /usr/lib/x86_64-linux-gnu/android/libssl.so.0
libssl1.1:amd64: /usr/share/doc/libssl1.1/changelog.gz
root@debian:~# dpkg --search libperl
libperl5.28:amd64: /usr/share/doc/libperl5.28/changelog.Debian.gz
libperl5.28:amd64: /usr/lib/x86_64-linux-gnu/libperl.so.5.28
libperl5.28:amd64: /usr/share/doc/libperl5.28
libperl5.28:amd64: /usr/lib/x86_64-linux-gnu/libperl.so.5.28.1
libperl5.28:amd64: /usr/share/doc/libperl5.28/copyright

I think it's because of compile, so I tried to compiled it myself on a Debian buster but I stuck on ./configure which can not find libcrypto.(libssl-dev installed, libraries exist in lib path, ... )

I appreciate if anyone can share the binary package for buster release.

Thank you.

Hi, I'm trying to execute the following bash command using NXLog's im_exec input module: `/usr/sbin/route -n | grep 'UG[ \t]' | awk '{print $2}'` From the Linux terminal that command will give the local gateway. When I attempt to run the command in NXLog I get the following: ``` 2020-01-24 13:43:09 WARNING im_exec process /usr/sbin/route exited 2020-01-24 13:43:10 ERROR subprocess '8114' returned a non-zero exit value of 3 ``` Input module I'm using to execute the command: ``` Module im_exec Restart true Command /usr/sbin/route Arg -n | grep 'UG[ \t]' | awk '{print $2}' Arg grep 'UG[ \t]' Arg awk '{print $2}' Arg sleep 30 $gateway = $raw_event; ``` I've tried various combinations of configuring the `Arg`'s but so far no luck. If I try putting everything in the `Command` field I get this error: `/usr/sbin/route -n |grep 'UG[ \t]'| awk '{print $2}';sleep 30;No such file or directory` Can someone help me with the correct syntax? ***NOTE: It seems like the problem is with using the `|` character in the command. Anyone know if NXLog will accept this argument or what syntax I need to use?*** Thanks in advance!!

I am following the nxlog to splunk guide here: https://nxlog.co/documentation/nxlog-user-guide/splunk.html. Specifically, section '93.3. Sending Specific Log Types for Splunk to Parse'. When testing, even using the config from the page, I am still getting an error (see further below)

<Input eventxml> Module im_msvistalog Channel Security CaptureEventXML TRUE Exec $raw_event = $EventXML; </Input>

<Output splunk_hec> Module om_http URL https://127.0.0.1:8088/services/collector/raw AddHeader Authorization: Splunk c6580856-29e8-4abf-8bcb-ee07f06c80b3 </Output>

This generates this error: ERROR invalid keyword: CaptureEventXML at C:\Program Files (x86)\nxlog\conf\nxlog.conf

Any ideas? thanks