Hi Is there a garbage collector service, when using the pm_buffer to disk, so that the buffer file on disk is emptied? If yes, how often is this run, and can it be configured? /Johan

I'm trying to pass only Warning / Error / Critical level Application Logs through NXLog to my ELK stack. When I have this configuration <Input EventLog_In> Module im_msvistalog <QueryList>\ <Query Id="0">\ <Select Path="Application">*</Select>\ </Query>\ </QueryList> Exec to_json(); </Input> everything works fine, and I'm collecting all levels of Application logs. I tried putting in a parameter on the <Select Path> line like this <Select Path="Application">*[Application/Level=1]</Select>\ And it craps itself and I get nothing. NXLog isn't reporting any issue, and I'm not seeing anything on the logstash side of things. I got the information about Event Viewer querying from this thread and adapted it to my use case: https://serverfault.com/questions/543494/query-specific-logs-from-event-log-using-nxlog

we need to separate and aggregate events per IP address during a period of time, such that, a single email is sent conteining multiple messages where the same IP is present, is this something that can be done with pm_evcorr? i hace tried and not yet able to get this functionality, if possible pls provide a quick example. thanks.

hi , 2 days ago i started getting this error : ERROR ### ASSERTION FAILED at line 33 in xm_gelf.c/xm_gelf_writer_udp(): "deflateInit(&strm, Z_DEFAULT_COMPRESSION) == Z_OK" ### INFO reconnecting in 1 seconds   anu idea ? tks .

Dear community, I'm currently working on parsing MS Exchange logs and sending them via GELF to my graylog instance. I'd like to convert the sender- and recipient-address field to lowercase. Sounds pretty easy, in fact, I need help :( my current config looks like this (below). Any help is appreciated. I've tried to work with "Exec       $sender-address = lc($sender-address);" within the input as well as Output backet - neither did work. define BASEDIR C:\Program Files\Microsoft\Exchange Server\V14\TransportRoles\Logs\MessageTracking <Extension csv> Module xm_csv Fields $date-time, $client-ip, $client-hostname, $server-ip, $server-hostname, $source-context, $connector-id, $exchange_source, $event-id, $internal-message-id, $message-id, $recipient-address, $recipient-status, $total-bytes, $recipient-count, $related-recipient-address, $reference, $message-subject, $sender-address, $return-path, $message-info, $directionality, $tenant-id, $original-client-ip, $original-server-ip, $custom-data FieldTypes string, string, string, string, string, string, string, string, string, integer, string, string, string, integer, integer, string, string, string, string, string, string, string, string, string, string, string Delimiter , </Extension> <Input in_exchange> Module im_file File '%BASEDIR%\MSGTRK????????*-*.LOG' SavePos TRUE Exec if $raw_event =~ /HealthMailbox/ drop(); Exec if $raw_event =~ /^#/ drop(); Exec csv->parse_csv(); </Input> <Output out_exchange> Module om_udp Host graylog.local Port 12203 OutputType GELF Exec $SourceName = 'exchange_msgtrk_log'; </Output> <Route exchange> Path in_exchange => out_exchange </Route>

I am trying to use NXlog to ship Event Logs and IIS logs to Graylog. No matter what I do in NXlog the IIS log timestamp in Graylog is the same as the EventReceivedTime. However, the Event Log timestamps are correct. Here is a link to my config on pastebin. All the configs I found on Google set the EventTime the same way. See anything wrong? $EventTime = parsedate($date + " " + $time); \ I tried doing a WireShark capture to see what the difference was. NXlog compresses the data before sending it. I haven't taken the time to learn how to uncompressit. I also tried setting the timestamp manually. That didn't work either... $timestamp = integer(parsedate($date + " " + $time))/1000; \

I am trying to figure out how many events are coming in per hour on a given a input module named win. I have searched around and haven't found any definitive solution. Most of what I have seen implements the create_stat function. But from there, I am lost. Here is my current config for the input, output, and route. How would I implement this feature into what I currently have? My end goal is to calculate EPS and write it out to log_info every hour with a message saying something like: EPS calulcated: 3,019 <Input win>     module          im_tcp         host            0.0.0.0         port            524         Exec            parse_syslog();         Exec            log_info("Severity Windows Collector: " + $SyslogSeverity + ", Hostname: " + $Hostname); </Input> <Output winout>         Module          om_file         CreateDir    true         File            '%WINLOG%'         <Schedule>                 Every           60 sec                 Exec            if (file_size('%WINLOG%') >= 2G) \                                 { \                                        file_cycle('%WINLOG%', 500); \                                         winout->reopen(); \                                 }         </Schedule> </Output> <Route 5>     Path            win => winout, Grid </Route>

NxLog Community Edition 2.8.1248 sometimes requires manual service restart,strangely,even reboot of OS doesnot help for: ERROR fatal connection error, reconnection will not be attempted (statuscode: 731004); apr_sockaddr_info failed for ____; The requested name is valid, but no data of the requested type was found. Can you please check,confirm, provide some fix &/or release updated version ...

On a Window 2012 server I'm collecting system events and then sending them to a remote server using OutputType GELF. This works fine on my serverrs behind a firewall however I have a AWS server that I would like to log and send logs over a TLS connection. Here's what my working Output looks like: <Output out>     Module      om_udp     Host        XXX.XXX.XXX.XXX     Port        12201     OutputType  GELF </Output> <Route 1>     Path      insql, in => out </Route> I have tested configs like below without success, the SSL connection is made but no logs are sent and the machine just repeats connections over and over to my graylog server. <Output out>     Module      om_ssl     Host        XXX.XXX.XXX.XXX     Port        12201     CAFile     %ROOT%\cert\ca.pem     CertFile    %ROOT%\cert\client-cert.pem     CertKeyFile    %ROOT%\cert\client-key.pem     OutputType  GELF </Output> <Route 1>     Path      insql, in => out </Route> Any ideas on how to proxy a "OutputType GELF" within a TLS/SSL connection? Thanks, Chipmunk  

Hi, I've started playing with NXlog and have found strange behaviour. This is my configuration: [code] <Input b-logs>     Module      im_file     File "d:\\Temp\\Logs\\test.txt"          SavePos TRUE     InputType LineBased     Exec if ( $raw_event =~ /^#/ )    \     {    \         $raw_event="ok";    \     }    \     else    \     {    \         $raw_event="bad";    \     } </Input>   <Output b-logs-out>     Module      om_tcp     Host        192.168.0.8     Port        8888 </Output> <Route 1>     Path        b-logs => b-logs-out </Route> [/code] I wrote simple python server that listens on TCP port 8888 on machine 192.168.0.8. What I want to achieve is to display "OK" when line in my test log file, begins with "#" and "BAD" when it is not. It works at the beginning. Output looks like following example: OK OK BAD OK OK Then I'm trying to change character "#" to any other and it stop working. Then I revert changes, put "#" into nxlog.conf again (restart Windows service) and instead of BAD or OK messages, NXLog sends entire line from log file. I'm getting following results: #line1 #Line2 TestLine3 #Comment at line4 I can't find reason of this behaviour.  Could you please give me any hint what may be the reason? My exampe of configuration is not useful, but I just want to understand how nxlog works to be able to use it to more sophisticated tasks in future.

I cannot compile nxlog on Solaris. On configure i got: checking for apr_socket_create in -lapr-1... no configure: error: libapr-1 not found If I have APR in /usr/apr etc, I am trying: ./configure CFLAGS="-I/usr/apr/1.3/include -I/usr/apr-util/1.3/include" LDFLAGS="-L/usr/apr/1.3/lib/sparcv9 -L/usr/apr-util/1.3/lib/sparcv9" --with-apr=/usr/apr/1.3/ --with-included-apr --prefix=/export/home/user/compiled And there is the same.

i have looked on the avail;able docs but have not yet seen such an example, is there a way to convert IPs to dns names, given the message below how to replace 192.168.225.2 with its dns name, host.name.com? <132>Sep 22 20:24:01 qare RouteAnalyzer[21700]: Prefix 192.168.42.64/32 (192.168.42.64/32) from router 192.168.225.2 in BGP/AS64512 went down.Configured thanks.

Is there a command or switch to prevent nxlog from deleting log files that have already been consumed and forward to their destination? I am want to forward the Exchange 2013 Message Tracking logs to a Graylog Server but need to leave the tracking logs in place. Thanks, Dan

Hi, Below patch enables NXLog to correctly skip UTF-8 BOMs in RFC5424 syslog messages. Should I also log a support ticket for this? Ron --- syslog.c.orig 2014-07-19 23:52:06.000000000 +1000 +++ syslog.c 2015-09-22 11:24:39.834615100 +1000 @@ -1091,7 +1091,7 @@ if ( *ptr == ' ' ) ptr++; // skip space // MESSAGE if ( (ptr[0] == 0xEF) && (ptr[0] == 0xBB) && (ptr[0] == 0xBF) ) if ( (ptr[0] == 0xEF) && (ptr[1] == 0xBB) && (ptr[2] == 0xBF) ) { //Skip UTF8 BOM ptr += 3; }

xm_perl.so is missing from the nxlog-ce-2.8.1248.tar.gz. this is causing nxlog to file. Is there any package with this missing file? I appreciate your help. Thank You.

I am trying to use the multlog module in order to start ingesting a custom log: I have the following regex: \^(\d{2}|\d).(\d{2}|\d).(\d{4})\s(\d\d|\d):(\d\d|\d):(\d\d|\d)\s(AM|PM).\[(.*)\](.*) This works in a regex test; however I cannot get it to work with the log file that looks something like this 9/10/2015 11:29:16 AM [0-3-1-SecondaryPortStatus.cs-17] GetStatus for IP: 192.168.0.231 on port: 5016 9/10/2015 11:29:16 AM [0-3-1-SecondaryPortStatus.cs-47] <TRANSACTION> <FUNCTION_TYPE>SECONDARYPORT</FUNCTION_TYPE> <COMMAND>STATUS</COMMAND> <MAC_LABEL>P_061</MAC_LABEL> <MAC>az4FMuLbvrPz720bBeKWz3c+zBh6MsKVo4nJEW96B04=</MAC> <COUNTER>217</COUNTER> </TRANSACTION> 9/10/2015 11:29:16 AM [0-3-1-SecondaryPortStatus.cs-57] <RESPONSE> <RESPONSE_TEXT>Operation SUCCESSFUL</RESPONSE_TEXT> <RESULT>OK</RESULT> <RESULT_CODE>-1</RESULT_CODE> <TERMINATION_STATUS>SUCCESS</TERMINATION_STATUS> <COUNTER>217</COUNTER> <SECONDARY_DATA>10</SECONDARY_DATA> <SERIAL_NUMBER>285498613</SERIAL_NUMBER> </RESPONSE> 9/10/2015 11:29:16 AM [0-1-1-LandingPage.xaml.cs-49] POS opened However when running the nxlog.conf for this I am getting the following error  2015-09-15 08:00:43 ERROR couldn't parse expression at line 12, character 13 in C:\Program Files (x86)\nxlog\conf\nxlog.conf; invalid character: '\' I am unsure what i need to do in order to get this correct; does anyone have any insight or resources I should further explore. Is there a REGEX specific doc for NXLOG? 

I need to create new extension module but I cannot deduce some base code structure for such thing. Is there available some piece of code for new module?

during installation on ubuntu i followed these steps.   ubuntu@nagios-2015:~$ uname -a Linux nagios-2015 3.13.0-48-generic #80-Ubuntu SMP Thu Mar 12 11:16:15 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux   sudo wget http://downloads.sourceforge.net/project/nxlog-ce/nxlog-ce-2.8.1248.tar.gz tar zxvf nxlog-ce-2.8.1248.tar.gz cd nxlog-ce-2.8.1248/ aptitude install libpcre3-dev libapr1-dev libssl-dev libexpat-dev make ./configure make make install mkdir -p /usr/local/var/run/nxlog/ mkdir /var/log/nxlog/ mkdir -p /usr/local/var/spool/nxlog/ mkdir /usr/local/etc/nxlog useradd nxlog cp /root/nxlog-ce-2.8.1248/packaging/debian/nxlog.init /etc/init.d/nxlog sed -i 's/\/usr\/bin\/nxlog/\/usr\/local\/bin\/nxlog/g' /etc/init.d/nxlog vim /usr/local/etc/nxlog/nxlog.conf bash -x /etc/init.d/nxlog start now i am seeing this error   2015-09-14 19:09:35 ERROR Failed to load module from /usr/local/libexec/nxlog/modules/extension/xm_perl.so, /usr/local/libexec/nxlog/modules/extension/xm_perl.so: cannot open shared object file: No such file or directory;DSO load failed 2015-09-14 19:09:35 ERROR Couldn't parse Exec block at /usr/local/etc/nxlog/nxlog.conf:88;couldn't parse statement at line 88, character 28 in /usr/local/etc/nxlog/nxlog.conf;module perl not found How do i install that extension individually?

I'm attempting to build nxlog with some updated libraries: Latest APR (1.5.2) Non-Heartbleed vulnerable OpenSSL sources PCRE 8.37 Zlib 1.2.8 After building all the dependencies I'm a little stuck on getting nxlogs to build, specifically I'm stuck on the step where I run ./configure At first it couldn't find apr-1-config, so I added /local/apr/bin to the path. Then it couldn't fine libapr-1 so I added /local/apr/lib to the path, this is where the problems started. When APR built there wasn't a "libapr-1" file in /local/apr/lib, only libapr-1.a, libapr-1.la, libapr-1.dll.a. Did I build APR incorrectly? I'm trying to build this on windows List of steps to get where I am: 1. Install MINGW using MinGW Installation Manager Add packages: mingw-developer-toolkit mingw-base mingw-expat bin mingw32-libexpat dev msys-libopenssl dev msys-automake msys-autoconf Setup msys fstab (c:/mingw     /mingw) 2. Install Python (2.5) 3. Add Python and mingw to system path (C:\Python25;C:\MinGW\bin;C:\MinGW\msys\1.0\bin) 3. Get and build APR source (I could not get APR iconv to compile) Download: http://ftp.gnu.org/pub/gnu/libiconv/libiconv-1.11.tar.gz http://mirror.nexcess.net/apache//apr/apr-1.5.2-win32-src.zip http://mirror.nexcess.net/apache//apr/apr-util-1.5.4-win32-src.zip http://sourceforge.net/projects/pcre/files/pcre/8.37/pcre-8.37.zip/download http://zlib.net/zlib128.zip Build: Extract all files to c:\mingw\msys\1.0\src Compile libiconv cd libiconv-1.11 ./configure CFLAGS="-O2 -s -mms-bitfields -march=i686" CXXFLAGS="-O2 -s -mms-bitfields -march=i686" make && make install Compile APR cd apr ./buildconf   ./configure CFLAGS="-O0 -s -mms-bitfields -march=i686" CXXFLAGS="-O0 -s -mms-bitfields -march=i686" make && make install cd .. Compile APR-UTIL cd apr-util-1.5.4 ./buildconf --with-apr=/usr/src/apr-1.5.2 ./configure CFLAGS="-O2 -s -mms-bitfields -march=i686" CXXFLAGS="-O2 -s -mms-bitfields -march=i686" --with-apr=/usr/src/apr-1.5.2 make && make install cd .. Compile PCRE cd pcre-.37 ./configure make && make install (make threw an error corrected with make clean, autoconf -i --force, started back at step 1) cd .. Compile ZLIB cd zlib-1.2.8 make -f win32/Makefile.gcc Compile nxlog cd nxlog-ce-2.8.1248 ./configure This is where the problems began. First it couldn't find apr-1-config. Fixed by adding /local/apr/bin to path. Now it can't find libapr-1, addint /local/apr/lib to the path doesn't help. There is no libapr-1 file in the MinGW directory tree. Ideas?   -pacmanwa  

Hi, When the nxlog is already started and the logfile is yet to be created by the application, I see "WARNING input file does not exist" message in nxlog.log file. How often does nxlog service retry\check for missing files which are created after starting nxlog service? Thanks & Regards, Mohan Guttikonda