I am trying to figure out how many events are coming in per hour on a given a input module named win.

I have searched around and haven't found any definitive solution. Most of what I have seen implements the create_stat function. But from there, I am lost. Here is my current config for the input, output, and route. How would I implement this feature into what I currently have?

My end goal is to calculate EPS and write it out to log_info every hour with a message saying something like: EPS calulcated: 3,019

NxLog Community Edition 2.8.1248 sometimes requires manual service restart,strangely,even reboot of OS doesnot help for:

ERROR fatal connection error, reconnection will not be attempted (statuscode: 731004); apr_sockaddr_info failed for ____; The requested name is valid, but no data of the requested type was found.

Can you please check,confirm, provide some fix &/or release updated version ...

On a Window 2012 server I'm collecting system events and then sending them to a remote server using OutputType GELF. This works fine on my serverrs behind a firewall however I have a AWS server that I would like to log and send logs over a TLS connection.

Here's what my working Output looks like:

<Output out>
    Module      om_udp
    Host        XXX.XXX.XXX.XXX
    Port        12201
    OutputType  GELF
</Output>

<Route 1>
    Path      insql, in => out
</Route>

I have tested configs like below without success, the SSL connection is made but no logs are sent and the machine just repeats connections over and over to my graylog server.

<Output out>
    Module      om_ssl
    Host        XXX.XXX.XXX.XXX
    Port        12201
    CAFile     %ROOT%\cert\ca.pem
    CertFile    %ROOT%\cert\client-cert.pem
    CertKeyFile    %ROOT%\cert\client-key.pem
    OutputType  GELF
</Output>

<Route 1>
    Path      insql, in => out
</Route>

Any ideas on how to proxy a "OutputType GELF" within a TLS/SSL connection?

Thanks,

Chipmunk

Hi,

I've started playing with NXlog and have found strange behaviour.

This is my configuration:

[code]

<Input b-logs>
    Module      im_file

    File "d:\\Temp\\Logs\\test.txt"
    
    SavePos TRUE
    InputType LineBased

    Exec if ( $raw_event =~ /^#/ )    \
    {    \
        $raw_event="ok";    \
    }    \
    else    \
    {    \
        $raw_event="bad";    \
    }

</Input>
 
<Output b-logs-out>
    Module      om_tcp
    Host        192.168.0.8
    Port        8888
</Output>

<Route 1>
    Path        b-logs => b-logs-out
</Route>

[/code]

I wrote simple python server that listens on TCP port 8888 on machine 192.168.0.8.
What I want to achieve is to display "OK" when line in my test log file, begins with "#" and "BAD" when it is not.

It works at the beginning. Output looks like following example:

OK
OK
BAD
OK
OK

Then I'm trying to change character "#" to any other and it stop working. Then I revert changes, put "#" into nxlog.conf again (restart Windows service) and instead of BAD or OK messages, NXLog sends entire line from log file.

I'm getting following results:
#line1
#Line2
TestLine3
#Comment at line4

I can't find reason of this behaviour. 

Could you please give me any hint what may be the reason?
My exampe of configuration is not useful, but I just want to understand how nxlog works to be able to use it to more sophisticated tasks in future.

I cannot compile nxlog on Solaris. On configure i got:

checking for apr_socket_create in -lapr-1... no
configure: error: libapr-1 not found

If I have APR in /usr/apr etc, I am trying:

./configure CFLAGS="-I/usr/apr/1.3/include -I/usr/apr-util/1.3/include" LDFLAGS="-L/usr/apr/1.3/lib/sparcv9 -L/usr/apr-util/1.3/lib/sparcv9" --with-apr=/usr/apr/1.3/ --with-included-apr --prefix=/export/home/user/compiled

And there is the same.

i have looked on the avail;able docs but have not yet seen such an example, is there a way to convert IPs to dns names, given the message below how to replace 192.168.225.2 with its dns name, host.name.com?

<132>Sep 22 20:24:01 qare RouteAnalyzer[21700]: Prefix 192.168.42.64/32 (192.168.42.64/32) from router 192.168.225.2 in BGP/AS64512 went down.Configured

thanks.

Is there a command or switch to prevent nxlog from deleting log files that have already been consumed and forward to their destination?

I am want to forward the Exchange 2013 Message Tracking logs to a Graylog Server but need to leave the tracking logs in place.

Thanks,

Dan

Hi,

Below patch enables NXLog to correctly skip UTF-8 BOMs in RFC5424 syslog messages.

Should I also log a support ticket for this?

Ron

--- syslog.c.orig 2014-07-19 23:52:06.000000000 +1000 +++ syslog.c 2015-09-22 11:24:39.834615100 +1000 @@ -1091,7 +1091,7 @@ if ( *ptr == ' ' ) ptr++; // skip space

 // MESSAGE

• if ( (ptr[0] == 0xEF) && (ptr[0] == 0xBB) && (ptr[0] == 0xBF) )

• if ( (ptr[0] == 0xEF) && (ptr[1] == 0xBB) && (ptr[2] == 0xBF) ) { //Skip UTF8 BOM ptr += 3; }

xm_perl.so is missing from the nxlog-ce-2.8.1248.tar.gz.

this is causing nxlog to file.

Is there any package with this missing file?

I appreciate your help.

Thank You.

I am trying to use the multlog module in order to start ingesting a custom log:

I have the following regex: \^(\d{2}|\d).(\d{2}|\d).(\d{4})\s(\d\d|\d):(\d\d|\d):(\d\d|\d)\s(AM|PM).\[(.*)\](.*) 

This works in a regex test; however I cannot get it to work with the log file that looks something like this

9/10/2015 11:29:16 AM [0-3-1-SecondaryPortStatus.cs-17] GetStatus for IP: 192.168.0.231 on port: 5016

9/10/2015 11:29:16 AM [0-3-1-SecondaryPortStatus.cs-47] <TRANSACTION>
  <FUNCTION_TYPE>SECONDARYPORT</FUNCTION_TYPE>
  <COMMAND>STATUS</COMMAND>
  <MAC_LABEL>P_061</MAC_LABEL>
  <MAC>az4FMuLbvrPz720bBeKWz3c+zBh6MsKVo4nJEW96B04=</MAC>
  <COUNTER>217</COUNTER>
</TRANSACTION>

9/10/2015 11:29:16 AM [0-3-1-SecondaryPortStatus.cs-57] <RESPONSE>
  <RESPONSE_TEXT>Operation SUCCESSFUL</RESPONSE_TEXT>
  <RESULT>OK</RESULT>
  <RESULT_CODE>-1</RESULT_CODE>
  <TERMINATION_STATUS>SUCCESS</TERMINATION_STATUS>
  <COUNTER>217</COUNTER>
  <SECONDARY_DATA>10</SECONDARY_DATA>
  <SERIAL_NUMBER>285498613</SERIAL_NUMBER>
</RESPONSE>

9/10/2015 11:29:16 AM [0-1-1-LandingPage.xaml.cs-49] POS opened

However when running the nxlog.conf for this I am getting the following error 

2015-09-15 08:00:43 ERROR couldn't parse expression at line 12, character 13 in C:\Program Files (x86)\nxlog\conf\nxlog.conf; invalid character: '\'

I am unsure what i need to do in order to get this correct; does anyone have any insight or resources I should further explore. Is there a REGEX specific doc for NXLOG? 

I need to create new extension module but I cannot deduce some base code structure for such thing. Is there available some piece of code for new module?

during installation on ubuntu i followed these steps.

ubuntu@nagios-2015:~$ uname -a
Linux nagios-2015 3.13.0-48-generic #80-Ubuntu SMP Thu Mar 12 11:16:15 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

sudo wget http://downloads.sourceforge.net/project/nxlog-ce/nxlog-ce-2.8.1248.tar.gz
tar zxvf nxlog-ce-2.8.1248.tar.gz
cd nxlog-ce-2.8.1248/
aptitude install libpcre3-dev libapr1-dev libssl-dev libexpat-dev make
./configure
make
make install
mkdir -p /usr/local/var/run/nxlog/
mkdir /var/log/nxlog/
mkdir -p /usr/local/var/spool/nxlog/
mkdir /usr/local/etc/nxlog
useradd nxlog
cp /root/nxlog-ce-2.8.1248/packaging/debian/nxlog.init /etc/init.d/nxlog
sed -i 's/\/usr\/bin\/nxlog/\/usr\/local\/bin\/nxlog/g' /etc/init.d/nxlog
vim /usr/local/etc/nxlog/nxlog.conf
bash -x /etc/init.d/nxlog start

now i am seeing this error

2015-09-14 19:09:35 ERROR Failed to load module from /usr/local/libexec/nxlog/modules/extension/xm_perl.so, /usr/local/libexec/nxlog/modules/extension/xm_perl.so: cannot open shared object file: No such file or directory;DSO load failed
2015-09-14 19:09:35 ERROR Couldn't parse Exec block at /usr/local/etc/nxlog/nxlog.conf:88;couldn't parse statement at line 88, character 28 in /usr/local/etc/nxlog/nxlog.conf;module perl not found

How do i install that extension individually?

I'm attempting to build nxlog with some updated libraries:

• Latest APR (1.5.2)
• Non-Heartbleed vulnerable OpenSSL sources
• PCRE 8.37
• Zlib 1.2.8

After building all the dependencies I'm a little stuck on getting nxlogs to build, specifically I'm stuck on the step where I run ./configure

At first it couldn't find apr-1-config, so I added /local/apr/bin to the path.

Then it couldn't fine libapr-1 so I added /local/apr/lib to the path, this is where the problems started. When APR built there wasn't a "libapr-1" file in /local/apr/lib, only libapr-1.a, libapr-1.la, libapr-1.dll.a.

Did I build APR incorrectly?

I'm trying to build this on windows

List of steps to get where I am:

1. Install MINGW using MinGW Installation Manager

Add packages:

• mingw-developer-toolkit
• mingw-base
• mingw-expat bin
• mingw32-libexpat dev
• msys-libopenssl dev
• msys-automake
• msys-autoconf

Setup msys fstab (c:/mingw     /mingw)

2. Install Python (2.5)

3. Add Python and mingw to system path (C:\Python25;C:\MinGW\bin;C:\MinGW\msys\1.0\bin)

3. Get and build APR source (I could not get APR iconv to compile)

Download:

• http://ftp.gnu.org/pub/gnu/libiconv/libiconv-1.11.tar.gz
• http://mirror.nexcess.net/apache//apr/apr-1.5.2-win32-src.zip
• http://mirror.nexcess.net/apache//apr/apr-util-1.5.4-win32-src.zip
• http://sourceforge.net/projects/pcre/files/pcre/8.37/pcre-8.37.zip/download
• http://zlib.net/zlib128.zip

Build:

1. Extract all files to c:\mingw\msys\1.0\src
2. Compile libiconv
   1. cd libiconv-1.11

   2. ./configure CFLAGS="-O2 -s -mms-bitfields -march=i686" CXXFLAGS="-O2 -s -mms-bitfields -march=i686"

   3. make && make install
3. Compile APR
   1. cd apr
   2. ./buildconf
   3.  

      ./configure CFLAGS="-O0 -s -mms-bitfields -march=i686" CXXFLAGS="-O0 -s -mms-bitfields -march=i686"

   4. make && make install
   5. cd ..
4. Compile APR-UTIL
   1. cd apr-util-1.5.4
   2. ./buildconf --with-apr=/usr/src/apr-1.5.2

   3. ./configure CFLAGS="-O2 -s -mms-bitfields -march=i686" CXXFLAGS="-O2 -s -mms-bitfields -march=i686" --with-apr=/usr/src/apr-1.5.2

   4. make && make install
   5. cd ..
5. Compile PCRE
   1. cd pcre-.37
   2. ./configure
   3. make && make install
   4. (make threw an error corrected with make clean, autoconf -i --force, started back at step 1)
   5. cd ..
6. Compile ZLIB
   1. cd zlib-1.2.8
   2. make -f win32/Makefile.gcc
7. Compile nxlog
   1. cd nxlog-ce-2.8.1248
   2. ./configure

This is where the problems began. First it couldn't find apr-1-config.

Fixed by adding /local/apr/bin to path.

Now it can't find libapr-1, addint /local/apr/lib to the path doesn't help. There is no libapr-1 file in the MinGW directory tree. Ideas?

-pacmanwa

Hi,

When the nxlog is already started and the logfile is yet to be created by the application, I see "WARNING input file does not exist" message in nxlog.log file.
How often does nxlog service retry\check for missing files which are created after starting nxlog service?

Thanks & Regards,
Mohan Guttikonda

Hi. How can I add 3 hours to the time that looks like this 2015-09-10 10:21:11. Something like this?

Exec             $EventTime = parsedate($1) + 3h;
 

Hi Team,

Hi,

I'm using nxlog to send logs from Windows eventlog to elasticsearch, and using Kibana view.

I'm getting all the message as it is in the 'Message' column, I want to re-order it so the hostname parameter will be the windows server (and not the elasticsearch server), add 'Type' to the messages, etc.

this is the configuration file of nxlog:

* 55.2.110.4=elasticsearch server

<Extension json>
 Module xm_json
</Extension>

# Nxlog internal logs
<Input internal>
   Module im_internal
   Exec $EventReceivedTime = integer($EventReceivedTime) / 1000000;
</Input>
 
# Windows Event Log
<Input eventlog>
# Uncomment im_msvistalog for Windows Vista/2008 and later
   Module im_msvistalog
 
# Uncomment im_mseventlog for Windows XP/2000/2003
#   Module im_mseventlog
 
   Exec $EventReceivedTime = integer($EventReceivedTime) / 1000000;
</Input>
 
<Output out>
   Module om_http
   URL  https://55.2.110.4:443
   HTTPSAllowUntrusted    TRUE
</Output>
 
<Route 1>
   Path internal, eventlog => out
</Route>

this is an example message from kibana:

 

Field Action Value @timestamp   2015-09-08T07:35:47.064Z @version   1 _id   AU-r4dtqVULqkki94YkZ _index   logstash-2015.09.08 _type   logs host   55.2.110.4 http_port   5005 message  

2015-09-08 07:35:43 dc-prod-zone-a.organization.com AUDIT_SUCCESS 4634 An account was logged off. Subject: Security ID: S-1-5-21-1595779987-1987268195-2987234418-1104 Account Name: DC-PROD-ZONE-C$ Account Domain: ORGANIZATION Logon ID: 0x679381 Logon Type: 3 This event is generated when a logon session is destroyed. It may be positively correlated with a logon event using the Logon ID value. Logon IDs are only unique between reboots on the same computer.

 

Thanks a lot

Hi,

I am running NXlog CE version 2.9.1347 on Windows 2012 R2.  The service keeps crashing with the following;

Log Name:      Application
Source:        Application Error
Date:          07/09/2015 09:26:42
Event ID:      1000
Task Category: (100)
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      myServer
Description:
Faulting application name: nxlog.exe, version: 0.0.0.0, time stamp: 0x54fedd1a
Faulting module name: ntdll.dll, version: 6.3.9600.17668, time stamp: 0x54c846bb
Exception code: 0xc0000005
Fault offset: 0x000195da
Faulting process id: 0x3504
Faulting application start time: 0x01d0e94f3d2521e0
Faulting application path: C:\Program Files (x86)\nxlog\nxlog.exe
Faulting module path: C:\Windows\SYSTEM32\ntdll.dll
Report Id: 8c5cf6ce-5542-11e5-80c1-005056bc12a5
Faulting package full name: 
Faulting package-relative application ID: 
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Application Error" />
    <EventID Qualifiers="0">1000</EventID>
    <Level>2</Level>
    <Task>100</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2015-09-07T09:26:42.000000000Z" />
    <EventRecordID>123651</EventRecordID>
    <Channel>Application</Channel>
    <Computer>myServer</Computer>
    <Security />
  </System>
  <EventData>
    <Data>nxlog.exe</Data>
    <Data>0.0.0.0</Data>
    <Data>54fedd1a</Data>
    <Data>ntdll.dll</Data>
    <Data>6.3.9600.17668</Data>
    <Data>54c846bb</Data>
    <Data>c0000005</Data>
    <Data>000195da</Data>
    <Data>3504</Data>
    <Data>01d0e94f3d2521e0</Data>
    <Data>C:\Program Files (x86)\nxlog\nxlog.exe</Data>
    <Data>C:\Windows\SYSTEM32\ntdll.dll</Data>
    <Data>8c5cf6ce-5542-11e5-80c1-005056bc12a5</Data>
    <Data>
    </Data>
    <Data>
    </Data>
  </EventData>
</Event>

I have seen other posts about a hotfix that fixes another app crashing issue.  But does it also fix this one?

Thanks,

Darren.

Hello 

I have created a regex expression to extract values from my logs, I have difficulties in writing these values in the nxlog output

The issue is that the output file is identical to the input log, not sure what I'm doing wrong here, any help is welcome

My nxlog.conf is as follow

<Input in>
        Module  im_file
        File    "C:\\tmp\example-log.txt"
        SavePos  TRUE
        Recursive TRUE
        InputType       multiline
        exec if $raw_event =~ /(?:\{"([0-9]+?), ([0-9]+?), "(.+?)", "(.+?)", "(?:.+?)", "(.+?)\n(?:[a-zA-Z0-9:,"\n\r\s\f\t\-\{\}\.\(\)]+?)New\sLogon:\s*(?:.+?)\n\s*Account\sName:\s*(.+?)\n(?:[a-zA-Z0-9:,"\n\r\s\f\t\-\{\}\.\(\)]+?)Source\sNetwork\sAddress:\s*([0-9\.]{7,15})\n(?:[a-zA-Z0-9:,"\n\r\s\f\t\-\{\}\.\(\)]+?)"\})/ \
    { \
    $timestamp = $1; \
    $event = $2; \
    $status = $3; \
    $type = $4; \
    $short = $5; \
    $user = $6; \
    $source = $7; \
    }
</Input>

#<Output out>
#    Module      om_tcp
#    Host        192.168.1.15
#    Port        12201
#    OutputType  GELF_TCP
#</Output>

<Processor one>
    Module      pm_null
    Exec        $raw_event = $timestamp + $event + $status + $type + $short + $user + $source;
</Processor>

<Output out>
    Module    om_file
    File    'C:\\tmp\output'
</Output>

<Route 1>
    Path        in => one => out
</Route>

Thanks heaps

Mark

Hy, I need some help in configuring my nxlog in windows XP.

System:
S.O. -> windows XP
NXLOG Version -> 2.8.1248

LOGS:
We use a log file for each event and these log files are stored in an hierarchical directory as: f:\year\month\day\accxxxx.txt (e.g. f:\2015\09\03\event000001.txt)
​

nxlog.conf excerpts:

<Extension multiline>
    Module xm_multiline
    HeaderLine '<?xml version="1.0" encoding="UTF-8" standalone="yes"?>'
</Extension>

<Input in>
    Module       im_file
    File         'E:\\hl7\\' + strftime(now(),"%Y\\\\%m\\\\%d") + '\\AccEvent*.txt'

    SavePos      TRUE
    ReadFromLast TRUE
    PollInterval 15
    Recursive    TRUE

    InputType    multiline

</Input>

I tested these cases:
1. (service nxlog active), add new file in  f:\2015\09\03\-> nxlog find the new file and analyze [OK]
2. Stop service nxlog, add new file in f:\2015\09\03\, start service -> nxlog don't find the new file. [X]
3. (service nxlog active), rename a file  in f:\2015\09\03\ already analyzed -> nxlog find the file renamed and analyze (in the log warning input file deleted) [OK]
4. (service nxlog active), update an existing file (same name, change only datetime last modified) -> nxlog don't analyze [OK]

I need to resolve step 2.
If I set ReadFromLast =FALSE when I restart service all fiels are all analyzed and that is not the desiderata.

Thanks in advance for your help.

Dario