Hi, Below patch enables NXLog to correctly skip UTF-8 BOMs in RFC5424 syslog messages. Should I also log a support ticket for this? Ron --- syslog.c.orig 2014-07-19 23:52:06.000000000 +1000 +++ syslog.c 2015-09-22 11:24:39.834615100 +1000 @@ -1091,7 +1091,7 @@ if ( *ptr == ' ' ) ptr++; // skip space // MESSAGE if ( (ptr[0] == 0xEF) && (ptr[0] == 0xBB) && (ptr[0] == 0xBF) ) if ( (ptr[0] == 0xEF) && (ptr[1] == 0xBB) && (ptr[2] == 0xBF) ) { //Skip UTF8 BOM ptr += 3; }

xm_perl.so is missing from the nxlog-ce-2.8.1248.tar.gz. this is causing nxlog to file. Is there any package with this missing file? I appreciate your help. Thank You.

I am trying to use the multlog module in order to start ingesting a custom log: I have the following regex: \^(\d{2}|\d).(\d{2}|\d).(\d{4})\s(\d\d|\d):(\d\d|\d):(\d\d|\d)\s(AM|PM).\[(.*)\](.*) This works in a regex test; however I cannot get it to work with the log file that looks something like this 9/10/2015 11:29:16 AM [0-3-1-SecondaryPortStatus.cs-17] GetStatus for IP: 192.168.0.231 on port: 5016 9/10/2015 11:29:16 AM [0-3-1-SecondaryPortStatus.cs-47] <TRANSACTION> <FUNCTION_TYPE>SECONDARYPORT</FUNCTION_TYPE> <COMMAND>STATUS</COMMAND> <MAC_LABEL>P_061</MAC_LABEL> <MAC>az4FMuLbvrPz720bBeKWz3c+zBh6MsKVo4nJEW96B04=</MAC> <COUNTER>217</COUNTER> </TRANSACTION> 9/10/2015 11:29:16 AM [0-3-1-SecondaryPortStatus.cs-57] <RESPONSE> <RESPONSE_TEXT>Operation SUCCESSFUL</RESPONSE_TEXT> <RESULT>OK</RESULT> <RESULT_CODE>-1</RESULT_CODE> <TERMINATION_STATUS>SUCCESS</TERMINATION_STATUS> <COUNTER>217</COUNTER> <SECONDARY_DATA>10</SECONDARY_DATA> <SERIAL_NUMBER>285498613</SERIAL_NUMBER> </RESPONSE> 9/10/2015 11:29:16 AM [0-1-1-LandingPage.xaml.cs-49] POS opened However when running the nxlog.conf for this I am getting the following error  2015-09-15 08:00:43 ERROR couldn't parse expression at line 12, character 13 in C:\Program Files (x86)\nxlog\conf\nxlog.conf; invalid character: '\' I am unsure what i need to do in order to get this correct; does anyone have any insight or resources I should further explore. Is there a REGEX specific doc for NXLOG? 

I need to create new extension module but I cannot deduce some base code structure for such thing. Is there available some piece of code for new module?

during installation on ubuntu i followed these steps.   ubuntu@nagios-2015:~$ uname -a Linux nagios-2015 3.13.0-48-generic #80-Ubuntu SMP Thu Mar 12 11:16:15 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux   sudo wget http://downloads.sourceforge.net/project/nxlog-ce/nxlog-ce-2.8.1248.tar.gz tar zxvf nxlog-ce-2.8.1248.tar.gz cd nxlog-ce-2.8.1248/ aptitude install libpcre3-dev libapr1-dev libssl-dev libexpat-dev make ./configure make make install mkdir -p /usr/local/var/run/nxlog/ mkdir /var/log/nxlog/ mkdir -p /usr/local/var/spool/nxlog/ mkdir /usr/local/etc/nxlog useradd nxlog cp /root/nxlog-ce-2.8.1248/packaging/debian/nxlog.init /etc/init.d/nxlog sed -i 's/\/usr\/bin\/nxlog/\/usr\/local\/bin\/nxlog/g' /etc/init.d/nxlog vim /usr/local/etc/nxlog/nxlog.conf bash -x /etc/init.d/nxlog start now i am seeing this error   2015-09-14 19:09:35 ERROR Failed to load module from /usr/local/libexec/nxlog/modules/extension/xm_perl.so, /usr/local/libexec/nxlog/modules/extension/xm_perl.so: cannot open shared object file: No such file or directory;DSO load failed 2015-09-14 19:09:35 ERROR Couldn't parse Exec block at /usr/local/etc/nxlog/nxlog.conf:88;couldn't parse statement at line 88, character 28 in /usr/local/etc/nxlog/nxlog.conf;module perl not found How do i install that extension individually?

I'm attempting to build nxlog with some updated libraries: Latest APR (1.5.2) Non-Heartbleed vulnerable OpenSSL sources PCRE 8.37 Zlib 1.2.8 After building all the dependencies I'm a little stuck on getting nxlogs to build, specifically I'm stuck on the step where I run ./configure At first it couldn't find apr-1-config, so I added /local/apr/bin to the path. Then it couldn't fine libapr-1 so I added /local/apr/lib to the path, this is where the problems started. When APR built there wasn't a "libapr-1" file in /local/apr/lib, only libapr-1.a, libapr-1.la, libapr-1.dll.a. Did I build APR incorrectly? I'm trying to build this on windows List of steps to get where I am: 1. Install MINGW using MinGW Installation Manager Add packages: mingw-developer-toolkit mingw-base mingw-expat bin mingw32-libexpat dev msys-libopenssl dev msys-automake msys-autoconf Setup msys fstab (c:/mingw     /mingw) 2. Install Python (2.5) 3. Add Python and mingw to system path (C:\Python25;C:\MinGW\bin;C:\MinGW\msys\1.0\bin) 3. Get and build APR source (I could not get APR iconv to compile) Download: http://ftp.gnu.org/pub/gnu/libiconv/libiconv-1.11.tar.gz http://mirror.nexcess.net/apache//apr/apr-1.5.2-win32-src.zip http://mirror.nexcess.net/apache//apr/apr-util-1.5.4-win32-src.zip http://sourceforge.net/projects/pcre/files/pcre/8.37/pcre-8.37.zip/download http://zlib.net/zlib128.zip Build: Extract all files to c:\mingw\msys\1.0\src Compile libiconv cd libiconv-1.11 ./configure CFLAGS="-O2 -s -mms-bitfields -march=i686" CXXFLAGS="-O2 -s -mms-bitfields -march=i686" make && make install Compile APR cd apr ./buildconf   ./configure CFLAGS="-O0 -s -mms-bitfields -march=i686" CXXFLAGS="-O0 -s -mms-bitfields -march=i686" make && make install cd .. Compile APR-UTIL cd apr-util-1.5.4 ./buildconf --with-apr=/usr/src/apr-1.5.2 ./configure CFLAGS="-O2 -s -mms-bitfields -march=i686" CXXFLAGS="-O2 -s -mms-bitfields -march=i686" --with-apr=/usr/src/apr-1.5.2 make && make install cd .. Compile PCRE cd pcre-.37 ./configure make && make install (make threw an error corrected with make clean, autoconf -i --force, started back at step 1) cd .. Compile ZLIB cd zlib-1.2.8 make -f win32/Makefile.gcc Compile nxlog cd nxlog-ce-2.8.1248 ./configure This is where the problems began. First it couldn't find apr-1-config. Fixed by adding /local/apr/bin to path. Now it can't find libapr-1, addint /local/apr/lib to the path doesn't help. There is no libapr-1 file in the MinGW directory tree. Ideas?   -pacmanwa  

Hi, When the nxlog is already started and the logfile is yet to be created by the application, I see "WARNING input file does not exist" message in nxlog.log file. How often does nxlog service retry\check for missing files which are created after starting nxlog service? Thanks & Regards, Mohan Guttikonda

Hi. How can I add 3 hours to the time that looks like this 2015-09-10 10:21:11. Something like this? Exec             $EventTime = parsedate($1) + 3h;  

Hi Team,   We are looking for nxlog installation in solaris machine. All our servers are running in  SunOS sparc sun4v and we wanted to install nxlog in these servers. Can you please let us know is it possible to install nxlog on these servers?.. If yes please provide some stetps how to achive this?.   Regards, Mohan.

Hi, I'm using nxlog to send logs from Windows eventlog to elasticsearch, and using Kibana view. I'm getting all the message as it is in the 'Message' column, I want to re-order it so the hostname parameter will be the windows server (and not the elasticsearch server), add 'Type' to the messages, etc. this is the configuration file of nxlog: * 55.2.110.4=elasticsearch server <Extension json>  Module xm_json </Extension> # Nxlog internal logs <Input internal>    Module im_internal    Exec $EventReceivedTime = integer($EventReceivedTime) / 1000000; </Input>   # Windows Event Log <Input eventlog> # Uncomment im_msvistalog for Windows Vista/2008 and later    Module im_msvistalog   # Uncomment im_mseventlog for Windows XP/2000/2003 #   Module im_mseventlog      Exec $EventReceivedTime = integer($EventReceivedTime) / 1000000; </Input>   <Output out>    Module om_http    URL  https://55.2.110.4:443    HTTPSAllowUntrusted    TRUE </Output>   <Route 1>    Path internal, eventlog => out </Route>   this is an example message from kibana:   Field Action Value @timestamp   2015-09-08T07:35:47.064Z @version   1 _id   AU-r4dtqVULqkki94YkZ _index   logstash-2015.09.08 _type   logs host   55.2.110.4 http_port   5005 message   2015-09-08 07:35:43 dc-prod-zone-a.organization.com AUDIT_SUCCESS 4634 An account was logged off. Subject: Security ID: S-1-5-21-1595779987-1987268195-2987234418-1104 Account Name: DC-PROD-ZONE-C$ Account Domain: ORGANIZATION Logon ID: 0x679381 Logon Type: 3 This event is generated when a logon session is destroyed. It may be positively correlated with a logon event using the Logon ID value. Logon IDs are only unique between reboots on the same computer.     Thanks a lot

Hi,   I am running NXlog CE version 2.9.1347 on Windows 2012 R2.  The service keeps crashing with the following;   Log Name:      Application Source:        Application Error Date:          07/09/2015 09:26:42 Event ID:      1000 Task Category: (100) Level:         Error Keywords:      Classic User:          N/A Computer:      myServer Description: Faulting application name: nxlog.exe, version: 0.0.0.0, time stamp: 0x54fedd1a Faulting module name: ntdll.dll, version: 6.3.9600.17668, time stamp: 0x54c846bb Exception code: 0xc0000005 Fault offset: 0x000195da Faulting process id: 0x3504 Faulting application start time: 0x01d0e94f3d2521e0 Faulting application path: C:\Program Files (x86)\nxlog\nxlog.exe Faulting module path: C:\Windows\SYSTEM32\ntdll.dll Report Id: 8c5cf6ce-5542-11e5-80c1-005056bc12a5 Faulting package full name:  Faulting package-relative application ID:  Event Xml: <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">   <System>     <Provider Name="Application Error" />     <EventID Qualifiers="0">1000</EventID>     <Level>2</Level>     <Task>100</Task>     <Keywords>0x80000000000000</Keywords>     <TimeCreated SystemTime="2015-09-07T09:26:42.000000000Z" />     <EventRecordID>123651</EventRecordID>     <Channel>Application</Channel>     <Computer>myServer</Computer>     <Security />   </System>   <EventData>     <Data>nxlog.exe</Data>     <Data>0.0.0.0</Data>     <Data>54fedd1a</Data>     <Data>ntdll.dll</Data>     <Data>6.3.9600.17668</Data>     <Data>54c846bb</Data>     <Data>c0000005</Data>     <Data>000195da</Data>     <Data>3504</Data>     <Data>01d0e94f3d2521e0</Data>     <Data>C:\Program Files (x86)\nxlog\nxlog.exe</Data>     <Data>C:\Windows\SYSTEM32\ntdll.dll</Data>     <Data>8c5cf6ce-5542-11e5-80c1-005056bc12a5</Data>     <Data>     </Data>     <Data>     </Data>   </EventData> </Event>   I have seen other posts about a hotfix that fixes another app crashing issue.  But does it also fix this one? Thanks, Darren.

Hello  I have created a regex expression to extract values from my logs, I have difficulties in writing these values in the nxlog output The issue is that the output file is identical to the input log, not sure what I'm doing wrong here, any help is welcome My nxlog.conf is as follow <Input in>         Module  im_file         File    "C:\\tmp\example-log.txt"         SavePos  TRUE         Recursive TRUE         InputType       multiline         exec if $raw_event =~ /(?:\{"([0-9]+?), ([0-9]+?), "(.+?)", "(.+?)", "(?:.+?)", "(.+?)\n(?:[a-zA-Z0-9:,"\n\r\s\f\t\-\{\}\.\(\)]+?)New\sLogon:\s*(?:.+?)\n\s*Account\sName:\s*(.+?)\n(?:[a-zA-Z0-9:,"\n\r\s\f\t\-\{\}\.\(\)]+?)Source\sNetwork\sAddress:\s*([0-9\.]{7,15})\n(?:[a-zA-Z0-9:,"\n\r\s\f\t\-\{\}\.\(\)]+?)"\})/ \     { \     $timestamp = $1; \     $event = $2; \     $status = $3; \     $type = $4; \     $short = $5; \     $user = $6; \     $source = $7; \     } </Input> #<Output out> #    Module      om_tcp #    Host        192.168.1.15 #    Port        12201 #    OutputType  GELF_TCP #</Output> <Processor one>     Module      pm_null     Exec        $raw_event = $timestamp + $event + $status + $type + $short + $user + $source; </Processor> <Output out>     Module    om_file     File    'C:\\tmp\output' </Output> <Route 1>     Path        in => one => out </Route>    Thanks heaps Mark  

Hy, I need some help in configuring my nxlog in windows XP. System: S.O. -> windows XP NXLOG Version -> 2.8.1248 LOGS: We use a log file for each event and these log files are stored in an hierarchical directory as: f:\year\month\day\accxxxx.txt (e.g. f:\2015\09\03\event000001.txt) ​ nxlog.conf excerpts: <Extension multiline>     Module xm_multiline     HeaderLine '<?xml version="1.0" encoding="UTF-8" standalone="yes"?>' </Extension> <Input in>     Module       im_file     File         'E:\\hl7\\' + strftime(now(),"%Y\\\\%m\\\\%d") + '\\AccEvent*.txt'     SavePos      TRUE     ReadFromLast TRUE     PollInterval 15     Recursive    TRUE     InputType    multiline </Input> I tested these cases: 1. (service nxlog active), add new file in  f:\2015\09\03\-> nxlog find the new file and analyze [OK] 2. Stop service nxlog, add new file in f:\2015\09\03\, start service -> nxlog don't find the new file. [X] 3. (service nxlog active), rename a file  in f:\2015\09\03\ already analyzed -> nxlog find the file renamed and analyze (in the log warning input file deleted) [OK] 4. (service nxlog active), update an existing file (same name, change only datetime last modified) -> nxlog don't analyze [OK] I need to resolve step 2. If I set ReadFromLast =FALSE when I restart service all fiels are all analyzed and that is not the desiderata.   Thanks in advance for your help. Dario  

Hello, I think this is an easy answer, but I am having some issues.  I am trying to read in a log file and send out a syslog message.  If the line in the file contains the word error, I want a syslog message to be sent.  If the file does not contain that word, just drop the message.  What would be the proper way to do this?  Below is what I currently have.   <Input watchfile_%service%>   Module im_file   File '%servicepath%'   Exec $Message = $raw_event;   Exec $SyslogSeverityValue = 6;   Exec if file_name() =~ /.*\\(.*)/ $SourceName = $1;    Exec if $raw_event =~ /ERROR/ $SyslogSeverityValue = 3;    SavePos TRUE     Recursive TRUE   PollInterval 10 </Input> Thank You

Hi, I am trying to read multiline xml files (results of ELMAH logs in ASP.NET), but for some of them I am unable to do so, as they contain lots of information. This results in errors like this: 2015-09-02 08:50:50 ERROR data size (119671) is over the limit (65000), will be truncated I tried googling for a resolution to this problem, but I couldn't find any information anywhere about a data size limit in nxlog. Could some please advise what I could do to fix this?

Hello, Recently, i ran into a problem with file locking. One application here is generating some log file, one per client, when nxlog try to read file, i got error : ERROR failed to open LOGFILEPATH; Le processus ne peut pas accéder au fichier car ce fichier est utilisé par un autre processus. Initaly my configuration was : <Input Orbis>     Module im_file     File 'LogDirectory\\*'     SavePos TRUE     ReadFromLast FALSE     CloseWhenIdle TRUE     InputType orbisLog     Exec if $raw_event !~ /^L:.*/  drop();     Exec $ClientNAme = replace(file_basename(file_name()),'.LOG',''); </Input> I have changed the PollIntervall to 60 and 3600, no effect... Except that at PollInterval 3600, nxlog grab 40% of CPU. Is there a way to insctruct nxlog to retry to read the file ?

Hi , I am trying to compile nxlog on solairs 11 sparc platform. I am encoutering the following errors while compiling. configure: error: libapr-1 not found gcc: error: unrecognized command line option '-mt' I used the following parameters :: ./configure APRCONFIG=/usr/apr/1.3/bin/apr-1-config Please let me know if any one of you have successfully compiled it on Solaris 11. Thanking you with regards.    

Hello, I'm quite new to nxlog, so forgive me if my question is  trivial but I'm having hard time to get the values I extract from my  logs using exec and a regex. I have a very large stash of old windows logs in text file (in multiline format), what I want to do is use nxlog to load them in graylog, but I want to format the log in a different way. It works perfectly when I do not use the the exec and the regexp, but it fails with the message: 2015-08-31 12:12:42 ERROR invalid keyword: $timestamp at C:\Program Files (x86)\nxlog\conf\nxlog.conf:36 The regex works when I test it using http://www.regexr.com/ The error seems in the way I'm trying to assign/write the variables matched by the regex   My nxlog.conf is like the below: ## This is a sample configuration file. See the nxlog reference manual about the ## configuration options. It should be installed locally and is also available ## online at http://nxlog.org/nxlog-docs/en/nxlog-reference-manual.html ## Please set the ROOT to the folder your nxlog was installed into, ## otherwise it will not start. #define ROOT C:\Program Files\nxlog define ROOT C:\Program Files (x86)\nxlog Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data LogFile %ROOT%\data\nxlog.log <Extension gelf>     Module       xm_gelf </Extension> <Extension multiline>     Module      xm_multiline     HeaderLine  /^{/     EndLine     /^}/ </Extension> <Input in>         Module  im_file         File    "C:\\tmp\\\\example-log.txt"         SavePos  TRUE         Recursive TRUE         InputType       multiline         exec if $raw_event =~ /(?:\{"([0-9]+?), ([0-9]+?), "(.+?)", "(.+?)", "(?:.+?)","(.+?)\n(?:[a-zA-Z0-9:,"\n\r\s\f\t\-\{\}\.\(\)]+?)New\sLogon:\s*(?:.+?)\n\s*Account\sName:\s*(.+?)\n(?:[a-zA-Z0-9:,"\n\r\s\f\t\-\{\}\.\(\)]+?)Source\sNetwork\sAddress:\s*([0-9\.]{7,15})\n(?:[a-zA-Z0-9:,"\n\r\s\f\t\-\{\}\.\(\)]+?)"\})/g;            $timestamp = $1;      $event = $2;      $status = $3;      $type = $4;      $short = $5;      $user = $6;      $source = $7;       </Input> #<Output out> #    Module      om_tcp #    Host        192.168.1.15 #    Port        12201 #    OutputType  GELF_TCP #</Output> <Output out>     Module    om_file     File    'C:\\tmp\\output' </Output> <Route 1>     Path        in => out </Route>    Could someone kindly help me out on this? I tried to read the nxlog manual but I need to confess that I could not understand what I'm doing wrong  Basically I just would the output to be composed by the raw message and the few fields I match with the regex   Thanks very much!   Mark

I've got some data that comes in with a somewhat unusual format.  It's a set of fixed fields, followed by a variable length set of keys, followed by a set of values.  It looks something like this (but with more fields): col1, col2, col3, description(key1; key2; ...;keyn), val1, val2, ..., valn I'm trying to transform this into something more like: a=col1, b=col2, c=col3, key1=val1, key2=val2, ..., keyn=valn I've actually got this working by using Exec and a bit of perl that I wrote that tears apart $raw_event and writes the modified logline to a domain socket, where a second instance Route is listening and sends the log over the network to its destination.  My problem is that this is not terribly performant, since it starts a perl process per log line.  I've had trouble figuring out another way to do this, mostly because the number of keys/values is variable. Any suggestions on ways this might be done that are likely to have better performance?

Hello there! I am using nxlog for tailing specific files (im_file) and output them via GELF_TCP to Graylog which goes good except 1 thing > short_messages get truncated to 64 chars which, if I understand right, is related to this directive (or I might be wrong): ShortMessageLength >This optional directive can be used to specify the length of the short_message field. This defaults to 64 if the directive is not explicitly specified. If the field short_message or ShortMessage is present, it will not be truncated. The question might be fairly simple (to someone who knows): how to use this directive? could someone please write an example how to use it so short_messages do not get truncated? Many thanks in advance! P.S. Same problem here > https://groups.google.com/forum/#!topic/graylog2/wUQIaFdUlZs