We are happy to announce NXLog Enterprise Edition v5.0.
This is a new major release that lays the foundation of improvements for the near future. We worked on hundreds of issues, features and changes.
Some of the highlights follow:
- Core event processing changes enable us to reach up to 40% higher event throughput
- Support for directly collecting systemd journal
- Support for collecting from named pipes
- Support for passive network monitoring
- Improved and simplified flow control implementation
- Improved IP version 6 support
- Support for resolving SID and GUID values on Windows
- Support for resolving numeric ids in Linux audit logs
If you want to read the summary of the features check this article.
See the changelog below.
* 2020-06-17 5.0.XX  Updated Windows installer to properly migrate config files on upgrade  Fixed an error causing statistical counters showing undef before the end of the first interval  Fixed an error in im_wseventing causing repeated tls handshakes  Unified network module config syntax  Implemented file rotation on open for xm_zlib  fixed an error where getfile() failed when xm_admin was loaded via xm_soapadmin symlink  Added redirection for changed config file names to xm_admin  Fixed *m_pipe pipe permission issues  Updated librdkafka dependency to 1.4.2 in generic builds  Added modbus protocol parser to im_pcap  Added function to chain multiple types in OutputType or InputType directives  implemented file_hash() in xm_fileop  Fixed segfault on shutdown  Added support for TLSv3  Fixed an issue limiting om_kafka performance  Fixed im_pcap to properly handle multiple protocol capture  Added logic to xm_rewrite for detecting and handling infinite recursion  Implemented logic to create registry entries on first start on Windows Nano  Re-added Reconnect and ReconnectInterval directives  Fixed error causing nxlog refusing to stop on Windows  Fixed handling of path beginning with \ for LogFile directive on Windows  Re-added LogConnections directive to im_wseventing  Fixed im_ssl segmentation fault on Windows  Added parsing for quoted string values in im pcap's Filter directive  Implemented encryption and decryption support in xm_crypto  Implemented compression and decompression module xm_zlib  Fixed an issue in im_msvistalog causing EventData/ContextInfo field to be ignored  Improved parsing for double quoted strings in im_linuxaudit  Fixed null characters showing up in internal log during high load  Moved log4ensics.conf to managed.conf  Refactored SSL/TLS common code  Implemented ReusePort for im_tcp and im_udp  Implemented is_scanning() function in im_fim  Implemented parsing for quoted value in PatternFile directive  Fixed incorrect EventTime field in im_pcap  Fixed incorrect escape sequence error in om_raijin  Fixed a runtime fault during loading xm_leef on Windows 2016 Datacenter  Fixed assersion error at line 797 in syslog.c/logdata_linebreaks_replace() in xm_syslog  Implemented capability handling in im_pcap  Fixed assertion error at line 68 in coremodule.c/nx_coremodule_dropped_records_log()  Updated networking code to support new libapr function apr_sockaddr_info_copy()  Fixed an issue where parse_syslog() was adding bogus EventTime field to invalid events  Fixed error causing im_pcap to return empty raw_event field  Fixed runaway CPU usage issue with om_http  Moved from xm_soapadmin to xm_admin  Fixed bogus warning about thread count  Added logging for dropped events when flow control is false  Fixed crash bug in TCP and UDP modules  Added max queue size reporting ability to xm_admin  Fixed a segfault in Java modules caused by trying to add non-existent file to ClassPath  Implemented increasing reconnect delay in xm_admin  Fixed error causing the configuration parser to refuse / as path separator on Windows  Added logging of IP address in addition to DNS names to network modules  Fixed an error causing im tcp refusing IPv6 any host address "::" as invalid.  Fixed an error in om_tcp causing constant repeated reconnects  Fixed memory leak in failover code  Changed networking modules to log client address in error messages  Changed default configuration to use etc/nxlog.d as config include directory  Fixed error causing config parser to ignore empty lines when calculating position in config files  Unified nxlog_exit() in main-unix.c and main-win32.c  Implemented id resolution in im_linuxaudit  Refactored network stack  Fixed om_batchcompress LocalPort parsing error when combined with Host destination:port format  Fixed an error causing the first execution of a Schedule block to occur 4:25m runtime and showing 0 counter value  Removed support for kafka modules on AIX as librdkafka lost upstream support on that platform  Fixed an error causing config validation to throw an error instead of a warning when no routes are defined  Fixed a potential stack corruption issue in nx_module_pollset_poll  Fix error causing empty $raw_message field with im_bsm  Disabled python modules on AIX  Refactored the per TCP connection pool usage in modules  Fixed an error causing panic on shutdown  Fixed an error in im_msvistalog causing "[error code: 0] no error" being reported  Updated snmp library in xm_snmp  Refactored connect/reconnect code  Fixed escaping with regards to Windows paths  Renamed nx_logdata_t to nx_record_t to align with move to internal batch processing  Fixed a segmentation fault in xm_admin triggered by expired server certificate  Implemented support for per URI path batching in om_http  Fixed error snappy compression not available in Windows package  Implemented multiline batch mode in om_http and im_http  Fixed memory leak in pm_norepeat  Refactored netflow code  Fixed an error in parse_syslog causing Hostname and EventTime fields to remain empty when hostname contains numbers  FlowControl now drops oldest record first  Fixed an error sometimes causing messages to be logged with the wrong context when SuppressRepeatingLogs TRUE  Fixed an error where ASCII NULL characters showed up in nxlog.log with SuppressRepeatingLogs TRUE  Disabled im_pcap module on FreeBSD  Fixed memory leak in om_http  Fixed a parsing error in im_bsm producing empty event records  Added support for all uppercase module names like IM_NULL in addition to literal name im_null  Fixed im_aixaudit hang  Fixed an error where \ at a comment line's end turned the following uncommented line into a comment  Fixed memory leak in nx_module_stop_self()  Fixed an error causing LocalPort to become ineffective in om_udpspoof  Added error message when LocalPort is used in Listen mode for om_tcp  Implemented retry logic with backoff for apr_file_open() errorsin im_kernel variants  Cleaned up leftover reconnect code in om_http  Fixed memory leak in xm_filelist  Fixed memory leak in xm_asl  Fixed various inconsistencies in the FlowControl implementation  Fixed an error in xm_leef (LEEFHeader directive) causing processing to stop  Added support for redis pub/sub communication  Added support for read-only system volume to macOS installer  Added support for retrieving certificates using their thumbprints from Windows certificate store  Fixed an error causing the Windows executable to refuse config check (-c) without running in foreground (-f)  Fixed an error causing add_http_header() to fail after xm_rewrite call  Fixed a memory leak in the config cache code  Fixed consistency problems when handling duplicated audit rules in im_linuxaudit  Added internal queue for im_internal  Implemented common parser function for SSL configuration options  Fixed an error causing nxlog configuration check to accept configuration with only an output module  Fixed segmentation fault in nxlog_version()  Fixed an error causing im_odbc to lose last read position in table  Fixed an SSL related memory leak in im_http  Fixed a memory leak in xm_kvp  Fixed a memory leak in xm_fileop  Fixed memory leak in the failover code  Added field prefix support to avoid field name collisions to parse_kvp() to xm_kvp  Fixed an error in om_udp causing high CPU usage  Fixed a segmentation fault in xm_json's escape_json() function  Fixed an error causing om_udp not to fail over despite receiving port unreachable  Fixed an error causing om_http to fail with empty path (Eg. "URL http://server:8080")  Fixed an error preventing nxlog from starting in docker if im_internal is used  Fixed nxlog startup to make sure event processing does not start before all modules are initialized  Fixed memory leak in im_ssl with low open file limit  Added the ability to preserve original SID and GUID values when resolving to im_msvistalog  Fixed an error in xm_leef resulting in sporadic parsing issues under high event load  Fixed debug log parsing error in xm_msdns  Fixed memory leaks found by valgrind in pm_pattern  Fixed memory leaks found by valgrind in xm_pattern  Fixed an issue causing nxlog.log being removed, but not recreated on rotation  Fixed issue where NULL characters truncated the response to getLog getFile or serverInfo requests  Added option TCPNoDelay to om_ssl and om_tcp  Added ReadTimeout for nxlog-processor to exit the process when its inputs have no more data  Fixed inconsistencies in xm_leef leading to parsing errors when delimiter is not TAB  Fixed a memory leak found by valgrind in xm_charconv  Fixed a bug causing crashes in nxlog-processor when ActiveFiles > 1300 and LogLevel debug  Fixed a memory leak found by valgrind in pm_evcorr  Fixed various thread safety issues discovered by valgrind  Modified the default value of IncludeHiddenFields to TRUE in all applicable extension modules  Fixed an error causing slow TLS negotiation in im_batchcompress  Fixed an error causing paused modules to reject connection attempts  Deprecated obsolete im_wmi module  Deprecated experimental xm_stdinpw module  Fixed uninitialized bytes error found by valgrind  Fixed an error mapping from "$SeverityValue" to "sev" in xm_leef/to_leef()  Fixed an error preventing failover in case of name resolution errors  Added support for Amazon Linux on ARM64 (packaging)  Added the ability to detect LEEF events with missing fields to parse_leef()  Added the ability to detect missing timestamp or hostname LEEF events to parse_leef()  Added feature to return value from xm_exec  Fixed an error preventing xm_msdns from parsing flag codes from PACKET events  Fixed malformed SSL error log when PEM file is missing on SLES15  Added BatchFlushInterval directive  Implemented batch processing architecture  Fixed librdkafka compilation error in librdkafka with OpenSSL 1.0.2s on Windows  Fixed an issue where "Include nxlog.d/*.conf" was not loading files in alphabetical order  Fixed missing separator in xm_leef output  Fixed error causing upgrades from nxsec package to nxlog package to ignore existing agent configuration  Added IncludeHiddenFields to enable to_json() in xm_json to handle field names starting with . or _  Added support for multiple File directives to im_msvistalog  Added better support for PersistLogqueue to om_kafka  Added support for librdkafka 1.1.0 on Windows  Fixed error handling for "resource temporarily unavailable" errors thrown by the OS  Fixed om_kafka to handle the lack of support for security.protocol in librdkafka 0.8.x  Added AddHeader directive to om_elasticsearch for sending various http headers like Authorization  Added parse_windows_eventlog_xml() to xm_xml for parsing Window XML Eventdata  Disabled im_pcap on OpenBSD  Fixed segmentation fault in om_elasticsearch caused by the introduction of failover functionality  Added im_pcap for capturing network traffic  Package nxlog for RHEL 8  Fixed om_kafka error causing the last queued event to be duplicated on restart  Packaged nxlog for Debian 10  Added support for kerberos/sasl to om_kafka in Windows and generic packages  Fixed an error causing neither om_http or im_http starting SSL handshake and waiting indefinitely after connecting  Added TCP_NODELAY socket option to om_ssl and om_tcp  Fixed regression causing nxlog started message being omitted from im_internal's log  Fixed an error causing a segmentation fault in CTRL-C handler when im_internal is in use  Modified xm_cef to validate CEFSeverity field, extension field keys according to current specification  Fixed SSL modules to conform to documented SSLProtocol behaviour  Added command line switch to suppress logging to standard output  Added functionality to nx_value_from_string() for detecting int64 overflow and converting data to string  Removed deprecated im_oci and om_oci modules  Refactored widetoutf8() from individual modules to common core  Added separate packaging of Java modules to OS specific packages  Added AllowUntrusted to SSL modules to allow connections with expired certificates  Fixed use-after-free error in im_msvistalog causing crashes  Fixed error in im_dbi that caused the raw_event field to remain empty  Fixed a buffer handling error causing im_batchcompress to get stuck in a loop  Changed default SSL Protocol version value to TLSv1.2 only  Fixed xm_cef to follow up the upstream type change of externalID field from integer to string  Added functions for selectively resolving SID and GUID values to xm_resolver  Added support for Windows Certificate store to all SSL enabled modules  Fixed multiple race conditions in xm_grok  Added function get_registryvalue() for querying registry entries to nxlog language on Widnows  Fixed an type detection and conversion error in to_json() in xm_json  Added DetectNumericValues to parse_kvp() to identify integers  Changed log4ensics.conf location to conf.d  Added multipart batch mode to http modules  Added support for output module failover  Fixed an issue causing nxlog to stop shipping logs with PersistLogqueue  Moved json related code into common code  Fixed startup crash in chroot environment  Add CreateDir directive to pm_buffer  Fixed error in im_msvistalog causing failed authentications for the service user used by nxlog  Added support for signed binary macOS packages  Fixed an error causing message "not enough data to decode serialized binary buffer" to be printed  Fixed logging issue causing xm_soapadmin and xm_admin to log spurious errors and warnings  Fixed error causing assertion failure while loading invalid python script  Added module om_raijin for sending data to Raijin the schemaless database  Added custom labels to xm_soapadmin and xm_admin to support storing arbitrary strings  Removed libnxfilepath  Fixed an error SpoolDir and CacheDir directive handling that was causing relative paths to fail  Fixed error causing xm_admin to log sever_info calls only in debug logs  Added support for storing resolved SID/GUID values in separate fields to im_msvistalog  Added support for specifying LogLevel at the level of modules  Improved startup time with large number of queue files  Added INSTALLDIR variable to default nxlog.conf  Fixed error causing om_kafka to randomly stop polling for new events  Fixed parsing error with empty fields in parse_cef() in xm_cef  Added sha1sum, md5sum, sha512sum, base64encode and base64decode to nxlog's internal language  Added java input ouput and extension modules  Added support for Severity string to parse_cef() in xm_cef  Added support for millisecond resolution parsing of "start" field to xm_cef  Fixed error causing om_kafka to connect even if it is not used by any route  Added go input ouput and extension modules  Fixed segmentation fault with Threads set to 2  Fixed several errors in xm_cef  Fixed crash while accessing file via UNC path in im_file  Fixed compatibility issue with librdkafka 1.0.0 in om_kafka  Fixed compatibility issue with librdkafka 0.8.3 in om_kafka  Fixed segmentation fault om nx_module_input_func_linereader_clean  Added support for RenderingInfo element to im_msvistalog  Added STATIC_ASSERT() to enable compile time assertion checks  Added Level, MatchAnyKeyword and MatchAllKeyword directive to im_etw replacing hardcoded values  Added INFO message reporting successful reconnect to om_udp  Added support for "Flags" field to im_etw  Added IPADDR data type replacing and unifying IP4ADDR and IP6ADDR data types  Fixed an issue where xm_soapadmin gets stuck in a loop  Fixed data corruption in parse_cef() when multiple module instances are using it  Fixed memory leak in im_dbi with PostgreSQL  Fixed segmentation fault in om_kafka seen when the process is interrupted with CTRL-C just after startup  Fixed error in im_wseventing where raw_event field was left empty  Fixed im_internal crash caused by running division by 0 in Exec  Fixed om_kafka issue where the module was reading data from route even when it was not connected to Kafka  Fixed error handling in xm_soapadmin where it did not send a SOAP fault for local configuration issues  Added AllowInvalidCounters for im_winperfcount to enable module startup when invalid counters are referenced  Fixed om_kafka error printing duplicate error messages for incorrect properties in Options  Fixed om_kafka related crash caused by librdkafka 0.9.4  Fixed "unknown publisher" error in signed Windows MSI installers  Fixed duplicate debug message in xm_msdns  Fixed parsing error of 12:00:00 PM in xm_msdns  Fixed multiple issues with event type and severity assignment in im_wseventing  Fixed issue in xm_multiline where "/s" regez modifier in HeaderLine was causing a syntax error  Added module and instance names to internal log entries  Added missing LOG::NXLog perl module to Windows packages  Fixed im_bsm parsing issues on macOS  Added compression support to openssl on Windows  Added logic to pm_buffer clean up queue files after events have been sent  Fixed a memory leak in om_elasticsearch  Fixed race condition in configcache triggered by multiple instances of im_msvistalog  Fixed a segmentation fault caused by sending malformed json in request to "getLog" in xm_admin  Fixed test failures caused by pcre2 update  Added im_systemd to collect from systemd journal  Added Call directive to *m_perl and *m_python modules  Set default value for SpoolDir on Windows  Added support for "Microsoft-Windows-IIS-FTP" event provider  Fixed im_acct to use CamelCase field names  Updated Windows packages to OpenSSL 1.1.1a  Added support for dynamic fields names to Windows XML event parser  Added CreateDir support to im_uds  Cleaned out deprecated GetProcAddress usage from various modules on Windows  Added om_pipe and im_pipe writing logs to and reading logs from named pipes on UNIX like systems  Updated Perl version shipped on Windows  Added support for parsing second, millisecond, microsecond resolution timestamps  Added support for seconds and milliseconds to datetime()  Added strcasestr() for use on platforms where it is not provided  Fixed im_linuxaudit parsing issue causing valid rules failing to load  Fixed packaging scripts so alternative names of library files would be symlinked in generic deb packages  Fixed xm_bsm issue caused by replacing getauevnum() with getauevnum_r() on Solaris  Fixed an issue causing delayed event collection in im_msvistalog  Added support for TLS compression to SSL enabled modules  Added support for event grouping to pm_evcorr  Fixed a hang in file_cycle during file rotation  Fixed kafka modules disappeared from generic packages  Modified im_msvistalog to show channel name in error messages  Fixed issue where subscription errors would throw an ERROR despite TolerateQueryErrors being true  Refactored xm_syslog's xm_syslog_input_func_rfc5425  Fix FlowControl error  Added support for parsing XML in UserData and EventData fields to im_msvistalog  Added support for creating prefixed copies of EventData and UserData fields to im_msvistalog  Updated AIX packages to openSSL 1.1.x, pcre2  Migrated from pcre to pcre2 on Debian, Ubuntu, FreeBSD, OpenBSD, MacOS, Solaris  Fixed xm_bsm breakage on macOS 10.14 (Mojave)  Added ResolveGUID directive for im_msvistalog  Fixed "xm_soapadmin_free_input" error in xm_soapadmin  Added support for verbose audit output to xm_aixaudit on AIX  Updated SLES12, SLES15, FreeBSD, OpenBSD, MacOS, Solaris packages for OpenSSL 1.1  Fixed libapr dependency issue in generic RPM packages [MR1136] Started using libssl package instead of libssl1.0 for building deb packages  Fixed an error causing General Protection Failure on shutdown  Fixed memory leak and misuse of log_info() for debug output  Fixed segmentation fault on exit on Windows  Fixed memory leak in im_fim  Added the ability to pass arguments to perl module function calls  Added support for different timestamp formats supported by xm_msdns  Fixed regression where the module om_http did not call om_http_erase_hdrflds on module stop [MR1111] Fixed NXLog Manager address handling in Docker containers  Refactored pointer usage  Fixed encoding error when loading Ruby gems  Updated FreeBSD and OpenBSD installers to deploy nxlog.conf instead of nxlog.conf.sample  Added IPv6 support  Fixed an error where Exec after RubyCode would lost events  Fixed issues found during fuzz testing of veriour parser functions  Changed the handling of the Hostname field to accept IPAddress in addition to host name as a string  Fixed RubyCode relative path parsing  om_ruby now requires RubyCode directive  Fixed memory leak in im_file  Deprecated xm_stdinpw  Deprecated *m_oci  Fixed crash when running multiple im_perl instances  Added support for collecting raw XML to im_msvistalog  Set the default configuration file location to INSTALLDIR on Windows  Fixed om_python crash on NULL value  Fixed im_wseventing to collect events produced while nxlog was not running  Fixed xm_charconv assertion errors on malformed UTF-16LE file  Added support for Severity and SeverityValue fields to im_acct  Disabled invalid methods for ruby modules  Added locking to xm_fileop to prevent race condition when multiple directives refer to the same file  Fixed errors found during fuzz testing parse_nps  Dropped Module instance from required om_ruby arguments as it was unused  Fixed errors found during fuzz testing parse_leef  Fixed errors found during fuzz testing parse_xml and parse_multiline  Fixed om_webhdfs timeout and x509  Added wildcard support for the File directive to im_msvistalog  im_msvistalog now detects file changes and reopens them when it is set to read from File  Added support for resolving SID values in XML Userdata in im_msvistalog  Added the ability to set "_id" in om_elasticsearch  Make im_fim insensitive to case-only file name changes on Windows  Log "NXLog started" event through im_internal  Fixed errors found while fuzz testing xm_bsm  Fixed error handling to prevent division by zero from crashing NXLog  Suppressed DNS lookup failures in xm_resolver  Fix xm_gelf interoperability with im_file  Added proxy support to om_http and om_elasticsearch  Clean up connection code in networked modules  Fix xm_netflow error "No template definition ... cannot parse v9 packet until template definitions are refreshed"  Follow up ProcessID change to ExecutionProcessID in to_syslog_  xm_rewrite's empty Keep directive now throws error  Fix busy loop in im_linuxaudit  Fix segmentation fault on loading configuration with partial default route  Introduce PatternFile and fix related error handling in xm_grok  Fix om_ruby hang on exit in Valgrind  Set default SpoolDir  Fixed error where im_wseventing ignored HTTPSCAFile  Fix im_wseventing bookmark handling error  Fixed spelling error in log message  Fix xm_perl assertion failure  Fixed im_wseventing stall  Fix connection cleanup in batchcompress  Fixed an error causing im_batchcompress to not receive full packet  Fixed an issue causing om_elasticsearch to stop shipping logs after a while  Added public call() procedure to xm_python  Fixed hang when im_python calls to xm_python  Changed im_etw Channel string to ChannelId integer  Restrict im_bsm to reading device file  Fixed library file location on MacOS  Fixed failed assertion on exit in im_udp  Fixed run user change does not work on some operating systems  Fixed nx_date_fix_year should not set time in the future  Fixed kerberos handling in im_wseventing  Fixed xm_charconv BOM handling  Fixed PdhAddEnglishCounterA() failure resulting in xm_soapadmin disconnecting  Disabled invalid methods for python modules  Added man pages to Unix/Linux installers  Added backup script to Solaris package to ease "upgrade"  Fixed xm_admin crashes in Listen mode  Improved exit handling of im_checkpoint to prevent it becoming a zombie  Added TCP 2514 as default port form om_batchcompress