Community Forum | Log collection solutions

PM_buffer module

Hi

Is there a garbage collector service, when using the pm_buffer to disk, so that the buffer file on disk is emptied? If yes, how often is this run, and can it be configured?

/Johan

j_aagaard created 10 years ago

Replies: 1

View post »
last updated 10 years ago

Issue selecting specific levels of windows application logs in NXLog

I'm trying to pass only Warning / Error / Critical level Application Logs through NXLog to my ELK stack. When I have this configuration

<Input EventLog_In> Module im_msvistalog

<QueryList>\ <Query Id="0">\

<Select Path="Application">*</Select>\

</Query>\ </QueryList>

Exec to_json(); </Input>

everything works fine, and I'm collecting all levels of Application logs. I tried putting in a parameter on the <Select Path> line like this

<Select Path="Application">*[Application/Level=1]</Select>\

And it craps itself and I get nothing. NXLog isn't reporting any issue, and I'm not seeing anything on the logstash side of things.

I got the information about Event Viewer querying from this thread and adapted it to my use case: https://serverfault.com/questions/543494/query-specific-logs-from-event-log-using-nxlog

pcort42 created 10 years ago

Replies: 1

View post »
last updated 10 years ago

windows | nxlog | eventviewer

Is there a way to aggregate multiple messages into one email?

we need to separate and aggregate events per IP address during a period of time, such that, a single email is sent conteining multiple messages where the same IP is present, is this something that can be done with pm_evcorr?

i hace tried and not yet able to get this functionality, if possible pls provide a quick example.

thanks.

nxlogdesonim created 10 years ago

Replies: 1

View post »
last updated 10 years ago

ASSERTION FAILED at line 33 in xm_gelf.c/xm_gelf_writer_udp()

hi , 2 days ago i started getting this error :

ERROR ### ASSERTION FAILED at line 33 in xm_gelf.c/xm_gelf_writer_udp(): "deflateInit(&strm, Z_DEFAULT_COMPRESSION) == Z_OK" ###
INFO reconnecting in 1 seconds

anu idea ? tks .

logstarter created 10 years ago

Replies: 5

View post »
last updated 10 years ago

CSV-input: converting specific field(s) to lowercase

Dear community,

I'm currently working on parsing MS Exchange logs and sending them via GELF to my graylog instance.

I'd like to convert the sender- and recipient-address field to lowercase. Sounds pretty easy, in fact, I need help :(

my current config looks like this (below). Any help is appreciated.

I've tried to work with "Exec $sender-address = lc($sender-address);" within the input as well as Output backet - neither did work.

define BASEDIR C:\Program Files\Microsoft\Exchange Server\V14\TransportRoles\Logs\MessageTracking

<Extension csv>
   Module      xm_csv
   Fields      $date-time, $client-ip, $client-hostname, $server-ip, $server-hostname, $source-context, $connector-id, $exchange_source, $event-id, $internal-message-id, $message-id, $recipient-address, $recipient-status, $total-bytes, $recipient-count, $related-recipient-address, $reference, $message-subject, $sender-address, $return-path, $message-info, $directionality, $tenant-id, $original-client-ip, $original-server-ip, $custom-data
   FieldTypes  string, string, string, string, string, string, string, string, string, integer, string, string, string, integer, integer, string, string, string, string, string, string, string, string, string, string, string
   Delimiter   ,
</Extension>

<Input in_exchange>  
   Module     im_file
   File       '%BASEDIR%\MSGTRK????????*-*.LOG'
   SavePos    TRUE
   Exec       if $raw_event =~ /HealthMailbox/ drop();
   Exec       if $raw_event =~ /^#/ drop();
   Exec       csv->parse_csv();
</Input>

<Output out_exchange>  
   Module     om_udp
   Host       graylog.local
   Port       12203
   OutputType GELF
   Exec       $SourceName = 'exchange_msgtrk_log';
</Output>

<Route exchange>  
    Path      in_exchange => out_exchange
</Route>

nomoresecrets created 10 years ago

Replies: 1

View post »
last updated 7 years ago

graylog csv exchange

NXlog IIS log Shipping issues

I am trying to use NXlog to ship Event Logs and IIS logs to Graylog. No matter what I do in NXlog the IIS log timestamp in Graylog is the same as the EventReceivedTime. However, the Event Log timestamps are correct. Here is a link to my config on pastebin. All the configs I found on Google set the EventTime the same way. See anything wrong?


$EventTime = parsedate($date + " " + $time); \

I tried doing a WireShark capture to see what the difference was. NXlog compresses the data before sending it. I haven't taken the time to learn how to uncompressit. I also tried setting the timestamp manually. That didn't work either...


$timestamp = integer(parsedate($date + " " + $time))/1000; \

remedy73 created 10 years ago

Replies: 1

View post »
last updated 10 years ago

graylog iis

Calculating the events per second (EPS)

I am trying to figure out how many events are coming in per hour on a given a input module named win.

I have searched around and haven't found any definitive solution. Most of what I have seen implements the create_stat function. But from there, I am lost. Here is my current config for the input, output, and route. How would I implement this feature into what I currently have?

My end goal is to calculate EPS and write it out to log_info every hour with a message saying something like: EPS calulcated: 3,019


<Input win>

    module          im_tcp

        host            0.0.0.0

        port            524

        Exec            parse_syslog();

        Exec            log_info("Severity Windows Collector: " + $SyslogSeverity + ", Hostname: " + $Hostname);

</Input>

<Output winout>

        Module          om_file

        CreateDir    true

        File            '%WINLOG%'

        <Schedule>

                Every           60 sec

                Exec            if (file_size('%WINLOG%') >= 2G) \

                                { \

                                       file_cycle('%WINLOG%', 500); \

                                        winout->reopen(); \

                                }

        </Schedule>

</Output>

<Route 5>

    Path            win => winout, Grid

</Route>

chrisc created 10 years ago

Replies: 1

View post »
last updated 5 years ago

eps nxlog scheduler

ERROR fatal connection error, reconnection will not be attempted (statuscode: 731004); apr_sockaddr_info failed for _; The requested name is valid, but no data of the requested type was found.

NxLog Community Edition 2.8.1248 sometimes requires manual service restart,strangely,even reboot of OS doesnot help for:

ERROR fatal connection error, reconnection will not be attempted (statuscode: 731004); apr_sockaddr_info failed for ____; The requested name is valid, but no data of the requested type was found.

Can you please check,confirm, provide some fix &/or release updated version ...

siddharth created 10 years ago

Replies: 3

View post »
last updated 10 years ago

how to proxy a "OutputType GELF" within a TLS/SSL connection?

On a Window 2012 server I'm collecting system events and then sending them to a remote server using OutputType GELF. This works fine on my serverrs behind a firewall however I have a AWS server that I would like to log and send logs over a TLS connection.

Here's what my working Output looks like:

<Output out>
Module om_udp
Host XXX.XXX.XXX.XXX
Port 12201
OutputType GELF
</Output>

<Route 1>
Path insql, in => out
</Route>

I have tested configs like below without success, the SSL connection is made but no logs are sent and the machine just repeats connections over and over to my graylog server.

<Output out>
Module om_ssl
Host XXX.XXX.XXX.XXX
Port 12201
CAFile    %ROOT%\cert\ca.pem
   CertFile   %ROOT%\cert\client-cert.pem
   CertKeyFile   %ROOT%\cert\client-key.pem
   OutputType GELF
</Output>

<Route 1>
Path insql, in => out
</Route>

Any ideas on how to proxy a "OutputType GELF" within a TLS/SSL connection?

Thanks,

Chipmunk

chipmunk created 10 years ago

Replies: 1

View post »
last updated 10 years ago

Strange behaviour of NXLog for Windows - configuration is valid random times

Hi,

I've started playing with NXlog and have found strange behaviour.

This is my configuration:

[code]

<Input b-logs>
Module im_file

   File "d:\\Temp\\Logs\\test.txt"

   SavePos TRUE
   InputType LineBased

Exec if ( $raw_event =~ /^#/ )   \
   {   \
       $raw_event="ok";   \
   }   \
   else   \
   {   \
       $raw_event="bad";   \
   }

</Input>

<Output b-logs-out>
   Module om_tcp
   Host 192.168.0.8
   Port 8888
</Output>

[/code]

I wrote simple python server that listens on TCP port 8888 on machine 192.168.0.8.
What I want to achieve is to display "OK" when line in my test log file, begins with "#" and "BAD" when it is not.

It works at the beginning. Output looks like following example:

OK
OK
BAD
OK
OK

Then I'm trying to change character "#" to any other and it stop working. Then I revert changes, put "#" into nxlog.conf again (restart Windows service) and instead of BAD or OK messages, NXLog sends entire line from log file.

I'm getting following results:
#line1
#Line2
TestLine3
#Comment at line4

I can't find reason of this behaviour.

Could you please give me any hint what may be the reason?
My exampe of configuration is not useful, but I just want to understand how nxlog works to be able to use it to more sophisticated tasks in future.

reeaver created 10 years ago

Replies: 1

View post »
last updated 10 years ago

Solaris: configure: error: libapr-1 not found

I cannot compile nxlog on Solaris. On configure i got:

checking for apr_socket_create in -lapr-1... no
configure: error: libapr-1 not found

If I have APR in /usr/apr etc, I am trying:

./configure CFLAGS="-I/usr/apr/1.3/include -I/usr/apr-util/1.3/include" LDFLAGS="-L/usr/apr/1.3/lib/sparcv9 -L/usr/apr-util/1.3/lib/sparcv9" --with-apr=/usr/apr/1.3/ --with-included-apr --prefix=/export/home/user/compiled

And there is the same.

Tuxizm created 10 years ago

Replies: 1

View post »
last updated 10 years ago

configure compilation Solaris

a way for nxlog to replace syslog message IPs with hostnames

i have looked on the avail;able docs but have not yet seen such an example, is there a way to convert IPs to dns names, given the message below how to replace 192.168.225.2 with its dns name, host.name.com?

<132>Sep 22 20:24:01 qare RouteAnalyzer[21700]: Prefix 192.168.42.64/32 (192.168.42.64/32) from router 192.168.225.2 in BGP/AS64512 went down.Configured

thanks.

nxlogdesonim created 10 years ago

Replies: 1

View post »
last updated 10 years ago

Preventing nxlog from deleting log files

Is there a command or switch to prevent nxlog from deleting log files that have already been consumed and forward to their destination?

I am want to forward the Exchange 2013 Message Tracking logs to a Graylog Server but need to leave the tracking logs in place.

Thanks,

Dan

dmcfadden created 10 years ago

Replies: 1

View post »
last updated 10 years ago

[patch] Correctly skip UTF-8 BOM in nx_syslog_parse_rfc5424()

Hi,

Below patch enables NXLog to correctly skip UTF-8 BOMs in RFC5424 syslog messages.

Should I also log a support ticket for this?

Ron

--- syslog.c.orig 2014-07-19 23:52:06.000000000 +1000 +++ syslog.c 2015-09-22 11:24:39.834615100 +1000 @@ -1091,7 +1091,7 @@ if ( *ptr == ' ' ) ptr++; // skip space

 // MESSAGE

if ( (ptr[0] == 0xEF) && (ptr[0] == 0xBB) && (ptr[0] == 0xBF) )

if ( (ptr[0] == 0xEF) && (ptr[1] == 0xBB) && (ptr[2] == 0xBF) ) { //Skip UTF8 BOM ptr += 3; }

ron-macneil-ice created 10 years ago

Replies: 1

View post »
last updated 10 years ago

xm_perl.so is missing from the package?

xm_perl.so is missing from the nxlog-ce-2.8.1248.tar.gz.

this is causing nxlog to file.

Is there any package with this missing file?

I appreciate your help.

Thank You.

sinkak created 10 years ago

Replies: 1

View post »
last updated 10 years ago

Multiline Headerline Regex Error

I am trying to use the multlog module in order to start ingesting a custom log:

I have the following regex: \^(\d{2}|\d).(\d{2}|\d).(\d{4})\s(\d\d|\d):(\d\d|\d):(\d\d|\d)\s(AM|PM).\[(.*)\](.*)

This works in a regex test; however I cannot get it to work with the log file that looks something like this


9/10/2015 11:29:16 AM [0-3-1-SecondaryPortStatus.cs-17] GetStatus for IP: 192.168.0.231 on port: 5016

9/10/2015 11:29:16 AM [0-3-1-SecondaryPortStatus.cs-47] <TRANSACTION>
  <FUNCTION_TYPE>SECONDARYPORT</FUNCTION_TYPE>
  <COMMAND>STATUS</COMMAND>
  <MAC_LABEL>P_061</MAC_LABEL>
  <MAC>az4FMuLbvrPz720bBeKWz3c+zBh6MsKVo4nJEW96B04=</MAC>
  <COUNTER>217</COUNTER>
</TRANSACTION>

9/10/2015 11:29:16 AM [0-3-1-SecondaryPortStatus.cs-57] <RESPONSE>
  <RESPONSE_TEXT>Operation SUCCESSFUL</RESPONSE_TEXT>
  <RESULT>OK</RESULT>
  <RESULT_CODE>-1</RESULT_CODE>
  <TERMINATION_STATUS>SUCCESS</TERMINATION_STATUS>
  <COUNTER>217</COUNTER>
  <SECONDARY_DATA>10</SECONDARY_DATA>
  <SERIAL_NUMBER>285498613</SERIAL_NUMBER>
</RESPONSE>

9/10/2015 11:29:16 AM [0-1-1-LandingPage.xaml.cs-49] POS opened

However when running the nxlog.conf for this I am getting the following error

2015-09-15 08:00:43 ERROR couldn't parse expression at line 12, character 13 in C:\Program Files (x86)\nxlog\conf\nxlog.conf; invalid character: '\'

I am unsure what i need to do in order to get this correct; does anyone have any insight or resources I should further explore. Is there a REGEX specific doc for NXLOG?

chris.ried created 10 years ago

Replies: 1

View post »
last updated 10 years ago

xm_multiline | multiline | regex | regular expressions

Extension module structure

I need to create new extension module but I cannot deduce some base code structure for such thing. Is there available some piece of code for new module?

Tuxizm created 10 years ago

extension module

ERROR Failed to load module from /usr/local/libexec/nxlog/modules/extension/ xm_perl.so

during installation on ubuntu i followed these steps.

ubuntu@nagios-2015:~$ uname -a
Linux nagios-2015 3.13.0-48-generic #80-Ubuntu SMP Thu Mar 12 11:16:15 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

sudo wget http://downloads.sourceforge.net/project/nxlog-ce/nxlog-ce-2.8.1248.tar.gz
tar zxvf nxlog-ce-2.8.1248.tar.gz
cd nxlog-ce-2.8.1248/
aptitude install libpcre3-dev libapr1-dev libssl-dev libexpat-dev make
./configure
make
make install
mkdir -p /usr/local/var/run/nxlog/
mkdir /var/log/nxlog/
mkdir -p /usr/local/var/spool/nxlog/
mkdir /usr/local/etc/nxlog
useradd nxlog
cp /root/nxlog-ce-2.8.1248/packaging/debian/nxlog.init /etc/init.d/nxlog
sed -i 's/\/usr\/bin\/nxlog/\/usr\/local\/bin\/nxlog/g' /etc/init.d/nxlog
vim /usr/local/etc/nxlog/nxlog.conf
bash -x /etc/init.d/nxlog start

now i am seeing this error

2015-09-14 19:09:35 ERROR Failed to load module from /usr/local/libexec/nxlog/modules/extension/xm_perl.so, /usr/local/libexec/nxlog/modules/extension/xm_perl.so: cannot open shared object file: No such file or directory;DSO load failed
2015-09-14 19:09:35 ERROR Couldn't parse Exec block at /usr/local/etc/nxlog/nxlog.conf:88;couldn't parse statement at line 88, character 28 in /usr/local/etc/nxlog/nxlog.conf;module perl not found

How do i install that extension individually?

sinkak created 10 years ago

Replies: 1

View post »
last updated 10 years ago

nxlog

Attempting to build nxlog with updated libraries, stuck at libapr-1 running ./configure

I'm attempting to build nxlog with some updated libraries:

Latest APR (1.5.2)
Non-Heartbleed vulnerable OpenSSL sources
PCRE 8.37
Zlib 1.2.8

After building all the dependencies I'm a little stuck on getting nxlogs to build, specifically I'm stuck on the step where I run ./configure

At first it couldn't find apr-1-config, so I added /local/apr/bin to the path.

Then it couldn't fine libapr-1 so I added /local/apr/lib to the path, this is where the problems started. When APR built there wasn't a "libapr-1" file in /local/apr/lib, only libapr-1.a, libapr-1.la, libapr-1.dll.a.

Did I build APR incorrectly?

I'm trying to build this on windows

List of steps to get where I am:

1. Install MINGW using MinGW Installation Manager

Add packages:

mingw-developer-toolkit
mingw-base
mingw-expat bin
mingw32-libexpat dev
msys-libopenssl dev
msys-automake
msys-autoconf

Setup msys fstab (c:/mingw /mingw)

2. Install Python (2.5)

3. Add Python and mingw to system path (C:\Python25;C:\MinGW\bin;C:\MinGW\msys\1.0\bin)

3. Get and build APR source (I could not get APR iconv to compile)

Download:

http://ftp.gnu.org/pub/gnu/libiconv/libiconv-1.11.tar.gz
http://mirror.nexcess.net/apache//apr/apr-1.5.2-win32-src.zip
http://mirror.nexcess.net/apache//apr/apr-util-1.5.4-win32-src.zip
http://sourceforge.net/projects/pcre/files/pcre/8.37/pcre-8.37.zip/download
http://zlib.net/zlib128.zip

Build:

Extract all files to c:\mingw\msys\1.0\src

Compile libiconv

cd libiconv-1.11


./configure CFLAGS="-O2 -s -mms-bitfields -march=i686" CXXFLAGS="-O2 -s -mms-bitfields -march=i686"

make && make install

Compile APR

cd apr
./buildconf


./configure CFLAGS="-O0 -s -mms-bitfields -march=i686" CXXFLAGS="-O0 -s -mms-bitfields -march=i686"

make && make install
cd ..

Compile APR-UTIL

cd apr-util-1.5.4
./buildconf --with-apr=/usr/src/apr-1.5.2


./configure CFLAGS="-O2 -s -mms-bitfields -march=i686" CXXFLAGS="-O2 -s -mms-bitfields -march=i686" --with-apr=/usr/src/apr-1.5.2

make && make install
cd ..

Compile PCRE
1. cd pcre-.37
2. ./configure
3. make && make install
4. (make threw an error corrected with make clean, autoconf -i --force, started back at step 1)
5. cd ..
Compile ZLIB
1. cd zlib-1.2.8
2. make -f win32/Makefile.gcc
Compile nxlog
1. cd nxlog-ce-2.8.1248
2. ./configure

This is where the problems began. First it couldn't find apr-1-config.

Fixed by adding /local/apr/bin to path.

Now it can't find libapr-1, addint /local/apr/lib to the path doesn't help. There is no libapr-1 file in the MinGW directory tree. Ideas?

-pacmanwa

pacmanwa created 10 years ago

Replies: 1

View post »
last updated 10 years ago

windows | mingw | nxlogs

WARNING input file does not exist

Hi,

When the nxlog is already started and the logfile is yet to be created by the application, I see "WARNING input file does not exist" message in nxlog.log file.
How often does nxlog service retry\check for missing files which are created after starting nxlog service?

Thanks & Regards,
Mohan Guttikonda

MohanGuttikonda created 10 years ago

Replies: 1

View post »
last updated 10 years ago