Dear community,

I'm currently working on parsing MS Exchange logs and sending them via GELF to my graylog instance.

I'd like to convert the sender- and recipient-address field to lowercase. Sounds pretty easy, in fact, I need help :(

my current config looks like this (below). Any help is appreciated.

I've tried to work with "Exec       $sender-address = lc($sender-address);" within the input as well as Output backet - neither did work.

define BASEDIR C:\Program Files\Microsoft\Exchange Server\V14\TransportRoles\Logs\MessageTracking

<Extension csv>
   Module      xm_csv
   Fields      $date-time, $client-ip, $client-hostname, $server-ip, $server-hostname, $source-context, $connector-id, $exchange_source, $event-id, $internal-message-id, $message-id, $recipient-address, $recipient-status, $total-bytes, $recipient-count, $related-recipient-address, $reference, $message-subject, $sender-address, $return-path, $message-info, $directionality, $tenant-id, $original-client-ip, $original-server-ip, $custom-data
   FieldTypes  string, string, string, string, string, string, string, string, string, integer, string, string, string, integer, integer, string, string, string, string, string, string, string, string, string, string, string
   Delimiter   ,
</Extension>

<Input in_exchange>  
   Module     im_file
   File       '%BASEDIR%\MSGTRK????????*-*.LOG'
   SavePos    TRUE
   Exec       if $raw_event =~ /HealthMailbox/ drop();
   Exec       if $raw_event =~ /^#/ drop();
   Exec       csv->parse_csv();
</Input>

<Output out_exchange>  
   Module     om_udp
   Host       graylog.local
   Port       12203
   OutputType GELF
   Exec       $SourceName = 'exchange_msgtrk_log';
</Output>

<Route exchange>  
    Path      in_exchange => out_exchange
</Route> 

I am trying to figure out how many events are coming in per hour on a given a input module named win.

I have searched around and haven't found any definitive solution. Most of what I have seen implements the create_stat function. But from there, I am lost. Here is my current config for the input, output, and route. How would I implement this feature into what I currently have?

My end goal is to calculate EPS and write it out to log_info every hour with a message saying something like: EPS calulcated: 3,019

NxLog Community Edition 2.8.1248 sometimes requires manual service restart,strangely,even reboot of OS doesnot help for:

ERROR fatal connection error, reconnection will not be attempted (statuscode: 731004); apr_sockaddr_info failed for ____; The requested name is valid, but no data of the requested type was found.

Can you please check,confirm, provide some fix &/or release updated version ...

On a Window 2012 server I'm collecting system events and then sending them to a remote server using OutputType GELF. This works fine on my serverrs behind a firewall however I have a AWS server that I would like to log and send logs over a TLS connection.

Here's what my working Output looks like:

<Output out>
    Module      om_udp
    Host        XXX.XXX.XXX.XXX
    Port        12201
    OutputType  GELF
</Output>

<Route 1>
    Path      insql, in => out
</Route>

I have tested configs like below without success, the SSL connection is made but no logs are sent and the machine just repeats connections over and over to my graylog server.

<Output out>
    Module      om_ssl
    Host        XXX.XXX.XXX.XXX
    Port        12201
    CAFile     %ROOT%\cert\ca.pem
    CertFile    %ROOT%\cert\client-cert.pem
    CertKeyFile    %ROOT%\cert\client-key.pem
    OutputType  GELF
</Output>

<Route 1>
    Path      insql, in => out
</Route>

Any ideas on how to proxy a "OutputType GELF" within a TLS/SSL connection?

Thanks,

Chipmunk

Hi,

I've started playing with NXlog and have found strange behaviour.

This is my configuration:

[code]

<Input b-logs>
    Module      im_file

    File "d:\\Temp\\Logs\\test.txt"
    
    SavePos TRUE
    InputType LineBased

    Exec if ( $raw_event =~ /^#/ )    \
    {    \
        $raw_event="ok";    \
    }    \
    else    \
    {    \
        $raw_event="bad";    \
    }

</Input>
 
<Output b-logs-out>
    Module      om_tcp
    Host        192.168.0.8
    Port        8888
</Output>

<Route 1>
    Path        b-logs => b-logs-out
</Route>

[/code]

I wrote simple python server that listens on TCP port 8888 on machine 192.168.0.8.
What I want to achieve is to display "OK" when line in my test log file, begins with "#" and "BAD" when it is not.

It works at the beginning. Output looks like following example:

OK
OK
BAD
OK
OK

Then I'm trying to change character "#" to any other and it stop working. Then I revert changes, put "#" into nxlog.conf again (restart Windows service) and instead of BAD or OK messages, NXLog sends entire line from log file.

I'm getting following results:
#line1
#Line2
TestLine3
#Comment at line4

I can't find reason of this behaviour. 

Could you please give me any hint what may be the reason?
My exampe of configuration is not useful, but I just want to understand how nxlog works to be able to use it to more sophisticated tasks in future.

I cannot compile nxlog on Solaris. On configure i got:

checking for apr_socket_create in -lapr-1... no
configure: error: libapr-1 not found

If I have APR in /usr/apr etc, I am trying:

./configure CFLAGS="-I/usr/apr/1.3/include -I/usr/apr-util/1.3/include" LDFLAGS="-L/usr/apr/1.3/lib/sparcv9 -L/usr/apr-util/1.3/lib/sparcv9" --with-apr=/usr/apr/1.3/ --with-included-apr --prefix=/export/home/user/compiled

And there is the same.

i have looked on the avail;able docs but have not yet seen such an example, is there a way to convert IPs to dns names, given the message below how to replace 192.168.225.2 with its dns name, host.name.com?

<132>Sep 22 20:24:01 qare RouteAnalyzer[21700]: Prefix 192.168.42.64/32 (192.168.42.64/32) from router 192.168.225.2 in BGP/AS64512 went down.Configured

thanks.

Is there a command or switch to prevent nxlog from deleting log files that have already been consumed and forward to their destination?

I am want to forward the Exchange 2013 Message Tracking logs to a Graylog Server but need to leave the tracking logs in place.

Thanks,

Dan

Hi,

Below patch enables NXLog to correctly skip UTF-8 BOMs in RFC5424 syslog messages.

Should I also log a support ticket for this?

Ron

--- syslog.c.orig 2014-07-19 23:52:06.000000000 +1000 +++ syslog.c 2015-09-22 11:24:39.834615100 +1000 @@ -1091,7 +1091,7 @@ if ( *ptr == ' ' ) ptr++; // skip space

 // MESSAGE

• if ( (ptr[0] == 0xEF) && (ptr[0] == 0xBB) && (ptr[0] == 0xBF) )

• if ( (ptr[0] == 0xEF) && (ptr[1] == 0xBB) && (ptr[2] == 0xBF) ) { //Skip UTF8 BOM ptr += 3; }

xm_perl.so is missing from the nxlog-ce-2.8.1248.tar.gz.

this is causing nxlog to file.

Is there any package with this missing file?

I appreciate your help.

Thank You.

I am trying to use the multlog module in order to start ingesting a custom log:

I have the following regex: \^(\d{2}|\d).(\d{2}|\d).(\d{4})\s(\d\d|\d):(\d\d|\d):(\d\d|\d)\s(AM|PM).\[(.*)\](.*) 

This works in a regex test; however I cannot get it to work with the log file that looks something like this

9/10/2015 11:29:16 AM [0-3-1-SecondaryPortStatus.cs-17] GetStatus for IP: 192.168.0.231 on port: 5016

9/10/2015 11:29:16 AM [0-3-1-SecondaryPortStatus.cs-47] <TRANSACTION>
  <FUNCTION_TYPE>SECONDARYPORT</FUNCTION_TYPE>
  <COMMAND>STATUS</COMMAND>
  <MAC_LABEL>P_061</MAC_LABEL>
  <MAC>az4FMuLbvrPz720bBeKWz3c+zBh6MsKVo4nJEW96B04=</MAC>
  <COUNTER>217</COUNTER>
</TRANSACTION>

9/10/2015 11:29:16 AM [0-3-1-SecondaryPortStatus.cs-57] <RESPONSE>
  <RESPONSE_TEXT>Operation SUCCESSFUL</RESPONSE_TEXT>
  <RESULT>OK</RESULT>
  <RESULT_CODE>-1</RESULT_CODE>
  <TERMINATION_STATUS>SUCCESS</TERMINATION_STATUS>
  <COUNTER>217</COUNTER>
  <SECONDARY_DATA>10</SECONDARY_DATA>
  <SERIAL_NUMBER>285498613</SERIAL_NUMBER>
</RESPONSE>

9/10/2015 11:29:16 AM [0-1-1-LandingPage.xaml.cs-49] POS opened

However when running the nxlog.conf for this I am getting the following error 

2015-09-15 08:00:43 ERROR couldn't parse expression at line 12, character 13 in C:\Program Files (x86)\nxlog\conf\nxlog.conf; invalid character: '\'

I am unsure what i need to do in order to get this correct; does anyone have any insight or resources I should further explore. Is there a REGEX specific doc for NXLOG? 

I need to create new extension module but I cannot deduce some base code structure for such thing. Is there available some piece of code for new module?

during installation on ubuntu i followed these steps.

ubuntu@nagios-2015:~$ uname -a
Linux nagios-2015 3.13.0-48-generic #80-Ubuntu SMP Thu Mar 12 11:16:15 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

sudo wget http://downloads.sourceforge.net/project/nxlog-ce/nxlog-ce-2.8.1248.tar.gz
tar zxvf nxlog-ce-2.8.1248.tar.gz
cd nxlog-ce-2.8.1248/
aptitude install libpcre3-dev libapr1-dev libssl-dev libexpat-dev make
./configure
make
make install
mkdir -p /usr/local/var/run/nxlog/
mkdir /var/log/nxlog/
mkdir -p /usr/local/var/spool/nxlog/
mkdir /usr/local/etc/nxlog
useradd nxlog
cp /root/nxlog-ce-2.8.1248/packaging/debian/nxlog.init /etc/init.d/nxlog
sed -i 's/\/usr\/bin\/nxlog/\/usr\/local\/bin\/nxlog/g' /etc/init.d/nxlog
vim /usr/local/etc/nxlog/nxlog.conf
bash -x /etc/init.d/nxlog start

now i am seeing this error

2015-09-14 19:09:35 ERROR Failed to load module from /usr/local/libexec/nxlog/modules/extension/xm_perl.so, /usr/local/libexec/nxlog/modules/extension/xm_perl.so: cannot open shared object file: No such file or directory;DSO load failed
2015-09-14 19:09:35 ERROR Couldn't parse Exec block at /usr/local/etc/nxlog/nxlog.conf:88;couldn't parse statement at line 88, character 28 in /usr/local/etc/nxlog/nxlog.conf;module perl not found

How do i install that extension individually?

I'm attempting to build nxlog with some updated libraries:

• Latest APR (1.5.2)
• Non-Heartbleed vulnerable OpenSSL sources
• PCRE 8.37
• Zlib 1.2.8

After building all the dependencies I'm a little stuck on getting nxlogs to build, specifically I'm stuck on the step where I run ./configure

At first it couldn't find apr-1-config, so I added /local/apr/bin to the path.

Then it couldn't fine libapr-1 so I added /local/apr/lib to the path, this is where the problems started. When APR built there wasn't a "libapr-1" file in /local/apr/lib, only libapr-1.a, libapr-1.la, libapr-1.dll.a.

Did I build APR incorrectly?

I'm trying to build this on windows

List of steps to get where I am:

1. Install MINGW using MinGW Installation Manager

Add packages:

• mingw-developer-toolkit
• mingw-base
• mingw-expat bin
• mingw32-libexpat dev
• msys-libopenssl dev
• msys-automake
• msys-autoconf

Setup msys fstab (c:/mingw     /mingw)

2. Install Python (2.5)

3. Add Python and mingw to system path (C:\Python25;C:\MinGW\bin;C:\MinGW\msys\1.0\bin)

3. Get and build APR source (I could not get APR iconv to compile)

Download:

• http://ftp.gnu.org/pub/gnu/libiconv/libiconv-1.11.tar.gz
• http://mirror.nexcess.net/apache//apr/apr-1.5.2-win32-src.zip
• http://mirror.nexcess.net/apache//apr/apr-util-1.5.4-win32-src.zip
• http://sourceforge.net/projects/pcre/files/pcre/8.37/pcre-8.37.zip/download
• http://zlib.net/zlib128.zip

Build:

1. Extract all files to c:\mingw\msys\1.0\src
2. Compile libiconv
   1. cd libiconv-1.11

   2. ./configure CFLAGS="-O2 -s -mms-bitfields -march=i686" CXXFLAGS="-O2 -s -mms-bitfields -march=i686"

   3. make && make install
3. Compile APR
   1. cd apr
   2. ./buildconf
   3.  

      ./configure CFLAGS="-O0 -s -mms-bitfields -march=i686" CXXFLAGS="-O0 -s -mms-bitfields -march=i686"

   4. make && make install
   5. cd ..
4. Compile APR-UTIL
   1. cd apr-util-1.5.4
   2. ./buildconf --with-apr=/usr/src/apr-1.5.2

   3. ./configure CFLAGS="-O2 -s -mms-bitfields -march=i686" CXXFLAGS="-O2 -s -mms-bitfields -march=i686" --with-apr=/usr/src/apr-1.5.2

   4. make && make install
   5. cd ..
5. Compile PCRE
   1. cd pcre-.37
   2. ./configure
   3. make && make install
   4. (make threw an error corrected with make clean, autoconf -i --force, started back at step 1)
   5. cd ..
6. Compile ZLIB
   1. cd zlib-1.2.8
   2. make -f win32/Makefile.gcc
7. Compile nxlog
   1. cd nxlog-ce-2.8.1248
   2. ./configure

This is where the problems began. First it couldn't find apr-1-config.

Fixed by adding /local/apr/bin to path.

Now it can't find libapr-1, addint /local/apr/lib to the path doesn't help. There is no libapr-1 file in the MinGW directory tree. Ideas?

-pacmanwa

Hi,

When the nxlog is already started and the logfile is yet to be created by the application, I see "WARNING input file does not exist" message in nxlog.log file.
How often does nxlog service retry\check for missing files which are created after starting nxlog service?

Thanks & Regards,
Mohan Guttikonda

Hi. How can I add 3 hours to the time that looks like this 2015-09-10 10:21:11. Something like this?

Exec             $EventTime = parsedate($1) + 3h;
 

Hi Team,

Hi,

I'm using nxlog to send logs from Windows eventlog to elasticsearch, and using Kibana view.

I'm getting all the message as it is in the 'Message' column, I want to re-order it so the hostname parameter will be the windows server (and not the elasticsearch server), add 'Type' to the messages, etc.

this is the configuration file of nxlog:

* 55.2.110.4=elasticsearch server

<Extension json>
 Module xm_json
</Extension>

# Nxlog internal logs
<Input internal>
   Module im_internal
   Exec $EventReceivedTime = integer($EventReceivedTime) / 1000000;
</Input>
 
# Windows Event Log
<Input eventlog>
# Uncomment im_msvistalog for Windows Vista/2008 and later
   Module im_msvistalog
 
# Uncomment im_mseventlog for Windows XP/2000/2003
#   Module im_mseventlog
 
   Exec $EventReceivedTime = integer($EventReceivedTime) / 1000000;
</Input>
 
<Output out>
   Module om_http
   URL  https://55.2.110.4:443
   HTTPSAllowUntrusted    TRUE
</Output>
 
<Route 1>
   Path internal, eventlog => out
</Route>

this is an example message from kibana:

 

Field Action Value @timestamp   2015-09-08T07:35:47.064Z @version   1 _id   AU-r4dtqVULqkki94YkZ _index   logstash-2015.09.08 _type   logs host   55.2.110.4 http_port   5005 message  

2015-09-08 07:35:43 dc-prod-zone-a.organization.com AUDIT_SUCCESS 4634 An account was logged off. Subject: Security ID: S-1-5-21-1595779987-1987268195-2987234418-1104 Account Name: DC-PROD-ZONE-C$ Account Domain: ORGANIZATION Logon ID: 0x679381 Logon Type: 3 This event is generated when a logon session is destroyed. It may be positively correlated with a logon event using the Logon ID value. Logon IDs are only unique between reboots on the same computer.

 

Thanks a lot

Hi,

I am running NXlog CE version 2.9.1347 on Windows 2012 R2.  The service keeps crashing with the following;

Log Name:      Application
Source:        Application Error
Date:          07/09/2015 09:26:42
Event ID:      1000
Task Category: (100)
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      myServer
Description:
Faulting application name: nxlog.exe, version: 0.0.0.0, time stamp: 0x54fedd1a
Faulting module name: ntdll.dll, version: 6.3.9600.17668, time stamp: 0x54c846bb
Exception code: 0xc0000005
Fault offset: 0x000195da
Faulting process id: 0x3504
Faulting application start time: 0x01d0e94f3d2521e0
Faulting application path: C:\Program Files (x86)\nxlog\nxlog.exe
Faulting module path: C:\Windows\SYSTEM32\ntdll.dll
Report Id: 8c5cf6ce-5542-11e5-80c1-005056bc12a5
Faulting package full name: 
Faulting package-relative application ID: 
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Application Error" />
    <EventID Qualifiers="0">1000</EventID>
    <Level>2</Level>
    <Task>100</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2015-09-07T09:26:42.000000000Z" />
    <EventRecordID>123651</EventRecordID>
    <Channel>Application</Channel>
    <Computer>myServer</Computer>
    <Security />
  </System>
  <EventData>
    <Data>nxlog.exe</Data>
    <Data>0.0.0.0</Data>
    <Data>54fedd1a</Data>
    <Data>ntdll.dll</Data>
    <Data>6.3.9600.17668</Data>
    <Data>54c846bb</Data>
    <Data>c0000005</Data>
    <Data>000195da</Data>
    <Data>3504</Data>
    <Data>01d0e94f3d2521e0</Data>
    <Data>C:\Program Files (x86)\nxlog\nxlog.exe</Data>
    <Data>C:\Windows\SYSTEM32\ntdll.dll</Data>
    <Data>8c5cf6ce-5542-11e5-80c1-005056bc12a5</Data>
    <Data>
    </Data>
    <Data>
    </Data>
  </EventData>
</Event>

I have seen other posts about a hotfix that fixes another app crashing issue.  But does it also fix this one?

Thanks,

Darren.