Ask questions. Get answers. Find technical product solutions from passionate experts in the NXLog community.
NXlog IIS log Shipping issues
remedy73 created
I am trying to use NXlog to ship Event Logs and IIS logs to Graylog. No matter what I do in NXlog the IIS log timestamp in Graylog is the same as the EventReceivedTime. However, the Event Log timestamps are correct. Here is a link to my config on pastebin. All the configs I found on Google set the EventTime the same way. See anything wrong?
$EventTime = parsedate($date + " " + $time); \
I tried doing a WireShark capture to see what the difference was. NXlog compresses the data before sending it. I haven't taken the time to learn how to uncompressit. I also tried setting the timestamp manually. That didn't work either...
$timestamp = integer(parsedate($date + " " + $time))/1000; \
remedy73 created
Calculating the events per second (EPS)
chrisc created
I am trying to figure out how many events are coming in per hour on a given a input module named win.
I have searched around and haven't found any definitive solution. Most of what I have seen implements the create_stat function. But from there, I am lost. Here is my current config for the input, output, and route. How would I implement this feature into what I currently have?
My end goal is to calculate EPS and write it out to log_info every hour with a message saying something like: EPS calulcated: 3,019
<Input win>
module im_tcp
host 0.0.0.0
port 524
Exec parse_syslog();
Exec log_info("Severity Windows Collector: " + $SyslogSeverity + ", Hostname: " + $Hostname);
</Input>
<Output winout>
Module om_file
CreateDir true
File '%WINLOG%'
<Schedule>
Every 60 sec
Exec if (file_size('%WINLOG%') >= 2G) \
{ \
file_cycle('%WINLOG%', 500); \
winout->reopen(); \
}
</Schedule>
</Output>
<Route 5>
Path win => winout, Grid
</Route>
chrisc created
ERROR fatal connection error, reconnection will not be attempted (statuscode: 731004); apr_sockaddr_info failed for _; The requested name is valid, but no data of the requested type was found.
siddharth created
NxLog Community Edition 2.8.1248 sometimes requires manual service restart,strangely,even reboot of OS doesnot help for:
ERROR fatal connection error, reconnection will not be attempted (statuscode: 731004); apr_sockaddr_info failed for ____; The requested name is valid, but no data of the requested type was found.
Can you please check,confirm, provide some fix &/or release updated version ...
siddharth created
how to proxy a "OutputType GELF" within a TLS/SSL connection?
chipmunk created
On a Window 2012 server I'm collecting system events and then sending them to a remote server using OutputType GELF. This works fine on my serverrs behind a firewall however I have a AWS server that I would like to log and send logs over a TLS connection.
Here's what my working Output looks like:
<Output out>
Module om_udp
Host XXX.XXX.XXX.XXX
Port 12201
OutputType GELF
</Output>
<Route 1>
Path insql, in => out
</Route>
I have tested configs like below without success, the SSL connection is made but no logs are sent and the machine just repeats connections over and over to my graylog server.
<Output out>
Module om_ssl
Host XXX.XXX.XXX.XXX
Port 12201
CAFile %ROOT%\cert\ca.pem
CertFile %ROOT%\cert\client-cert.pem
CertKeyFile %ROOT%\cert\client-key.pem
OutputType GELF
</Output>
<Route 1>
Path insql, in => out
</Route>
Any ideas on how to proxy a "OutputType GELF" within a TLS/SSL connection?
Thanks,
Chipmunk
chipmunk created
Strange behaviour of NXLog for Windows - configuration is valid random times
reeaver created
Hi,
I've started playing with NXlog and have found strange behaviour.
This is my configuration:
[code]
<Input b-logs>
Module im_file
File "d:\\Temp\\Logs\\test.txt"
SavePos TRUE
InputType LineBased
Exec if ( $raw_event =~ /^#/ ) \
{ \
$raw_event="ok"; \
} \
else \
{ \
$raw_event="bad"; \
}
</Input>
<Output b-logs-out>
Module om_tcp
Host 192.168.0.8
Port 8888
</Output>
<Route 1>
Path b-logs => b-logs-out
</Route>
[/code]
I wrote simple python server that listens on TCP port 8888 on machine 192.168.0.8.
What I want to achieve is to display "OK" when line in my test log file, begins with "#" and "BAD" when it is not.
It works at the beginning. Output looks like following example:
OK
OK
BAD
OK
OK
Then I'm trying to change character "#" to any other and it stop working. Then I revert changes, put "#" into nxlog.conf again (restart Windows service) and instead of BAD or OK messages, NXLog sends entire line from log file.
I'm getting following results:
#line1
#Line2
TestLine3
#Comment at line4
I can't find reason of this behaviour.
Could you please give me any hint what may be the reason?
My exampe of configuration is not useful, but I just want to understand how nxlog works to be able to use it to more sophisticated tasks in future.
reeaver created
Solaris: configure: error: libapr-1 not found
Tuxizm created
I cannot compile nxlog on Solaris. On configure i got:
checking for apr_socket_create in -lapr-1... no
configure: error: libapr-1 not found
If I have APR in /usr/apr etc, I am trying:
./configure CFLAGS="-I/usr/apr/1.3/include -I/usr/apr-util/1.3/include" LDFLAGS="-L/usr/apr/1.3/lib/sparcv9 -L/usr/apr-util/1.3/lib/sparcv9" --with-apr=/usr/apr/1.3/ --with-included-apr --prefix=/export/home/user/compiled
And there is the same.
Tuxizm created
a way for nxlog to replace syslog message IPs with hostnames
nxlogdesonim created
i have looked on the avail;able docs but have not yet seen such an example, is there a way to convert IPs to dns names, given the message below how to replace 192.168.225.2 with its dns name, host.name.com?
<132>Sep 22 20:24:01 qare RouteAnalyzer[21700]: Prefix 192.168.42.64/32 (192.168.42.64/32) from router 192.168.225.2 in BGP/AS64512 went down.Configured
thanks.
nxlogdesonim created
Preventing nxlog from deleting log files
dmcfadden created
Is there a command or switch to prevent nxlog from deleting log files that have already been consumed and forward to their destination?
I am want to forward the Exchange 2013 Message Tracking logs to a Graylog Server but need to leave the tracking logs in place.
Thanks,
Dan
dmcfadden created
[patch] Correctly skip UTF-8 BOM in nx_syslog_parse_rfc5424()
ron-macneil-ice created
Hi,
Below patch enables NXLog to correctly skip UTF-8 BOMs in RFC5424 syslog messages.
Should I also log a support ticket for this?
Ron
--- syslog.c.orig 2014-07-19 23:52:06.000000000 +1000
+++ syslog.c 2015-09-22 11:24:39.834615100 +1000
@@ -1091,7 +1091,7 @@
if ( *ptr == ' ' ) ptr++; // skip space
// MESSAGE
if ( (ptr[0] == 0xEF) && (ptr[0] == 0xBB) && (ptr[0] == 0xBF) )
if ( (ptr[0] == 0xEF) && (ptr[1] == 0xBB) && (ptr[2] == 0xBF) )
{ //Skip UTF8 BOM
ptr += 3;
}
ron-macneil-ice created
xm_perl.so is missing from the package?
sinkak created
xm_perl.so is missing from the nxlog-ce-2.8.1248.tar.gz.
this is causing nxlog to file.
Is there any package with this missing file?
I appreciate your help.
Thank You.
sinkak created
Multiline Headerline Regex Error
chris.ried created
I am trying to use the multlog module in order to start ingesting a custom log:
I have the following regex: \^(\d{2}|\d).(\d{2}|\d).(\d{4})\s(\d\d|\d):(\d\d|\d):(\d\d|\d)\s(AM|PM).\[(.*)\](.*)
This works in a regex test; however I cannot get it to work with the log file that looks something like this
9/10/2015 11:29:16 AM [0-3-1-SecondaryPortStatus.cs-17] GetStatus for IP: 192.168.0.231 on port: 5016
9/10/2015 11:29:16 AM [0-3-1-SecondaryPortStatus.cs-47] <TRANSACTION>
<FUNCTION_TYPE>SECONDARYPORT</FUNCTION_TYPE>
<COMMAND>STATUS</COMMAND>
<MAC_LABEL>P_061</MAC_LABEL>
<MAC>az4FMuLbvrPz720bBeKWz3c+zBh6MsKVo4nJEW96B04=</MAC>
<COUNTER>217</COUNTER>
</TRANSACTION>
9/10/2015 11:29:16 AM [0-3-1-SecondaryPortStatus.cs-57] <RESPONSE>
<RESPONSE_TEXT>Operation SUCCESSFUL</RESPONSE_TEXT>
<RESULT>OK</RESULT>
<RESULT_CODE>-1</RESULT_CODE>
<TERMINATION_STATUS>SUCCESS</TERMINATION_STATUS>
<COUNTER>217</COUNTER>
<SECONDARY_DATA>10</SECONDARY_DATA>
<SERIAL_NUMBER>285498613</SERIAL_NUMBER>
</RESPONSE>
9/10/2015 11:29:16 AM [0-1-1-LandingPage.xaml.cs-49] POS opened
However when running the nxlog.conf for this I am getting the following error
2015-09-15 08:00:43 ERROR couldn't parse expression at line 12, character 13 in C:\Program Files (x86)\nxlog\conf\nxlog.conf; invalid character: '\'
I am unsure what i need to do in order to get this correct; does anyone have any insight or resources I should further explore. Is there a REGEX specific doc for NXLOG?
chris.ried created
Extension module structure
Tuxizm created
I need to create new extension module but I cannot deduce some base code structure for such thing. Is there available some piece of code for new module?
Tuxizm created
ERROR Failed to load module from /usr/local/libexec/nxlog/modules/extension/ xm_perl.so
sinkak created
during installation on ubuntu i followed these steps.
ubuntu@nagios-2015:~$ uname -a
Linux nagios-2015 3.13.0-48-generic #80-Ubuntu SMP Thu Mar 12 11:16:15 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
sudo wget http://downloads.sourceforge.net/project/nxlog-ce/nxlog-ce-2.8.1248.tar.gz
tar zxvf nxlog-ce-2.8.1248.tar.gz
cd nxlog-ce-2.8.1248/
aptitude install libpcre3-dev libapr1-dev libssl-dev libexpat-dev make
./configure
make
make install
mkdir -p /usr/local/var/run/nxlog/
mkdir /var/log/nxlog/
mkdir -p /usr/local/var/spool/nxlog/
mkdir /usr/local/etc/nxlog
useradd nxlog
cp /root/nxlog-ce-2.8.1248/packaging/debian/nxlog.init /etc/init.d/nxlog
sed -i 's/\/usr\/bin\/nxlog/\/usr\/local\/bin\/nxlog/g' /etc/init.d/nxlog
vim /usr/local/etc/nxlog/nxlog.conf
bash -x /etc/init.d/nxlog start
now i am seeing this error
2015-09-14 19:09:35 ERROR Failed to load module from /usr/local/libexec/nxlog/modules/extension/xm_perl.so, /usr/local/libexec/nxlog/modules/extension/xm_perl.so: cannot open shared object file: No such file or directory;DSO load failed
2015-09-14 19:09:35 ERROR Couldn't parse Exec block at /usr/local/etc/nxlog/nxlog.conf:88;couldn't parse statement at line 88, character 28 in /usr/local/etc/nxlog/nxlog.conf;module perl not found
How do i install that extension individually?
sinkak created
Attempting to build nxlog with updated libraries, stuck at libapr-1 running ./configure
pacmanwa created
I'm attempting to build nxlog with some updated libraries:
Latest APR (1.5.2)
Non-Heartbleed vulnerable OpenSSL sources
PCRE 8.37
Zlib 1.2.8
After building all the dependencies I'm a little stuck on getting nxlogs to build, specifically I'm stuck on the step where I run ./configure
At first it couldn't find apr-1-config, so I added /local/apr/bin to the path.
Then it couldn't fine libapr-1 so I added /local/apr/lib to the path, this is where the problems started. When APR built there wasn't a "libapr-1" file in /local/apr/lib, only libapr-1.a, libapr-1.la, libapr-1.dll.a.
Did I build APR incorrectly?
I'm trying to build this on windows
List of steps to get where I am:
1. Install MINGW using MinGW Installation Manager
Add packages:
mingw-developer-toolkit
mingw-base
mingw-expat bin
mingw32-libexpat dev
msys-libopenssl dev
msys-automake
msys-autoconf
Setup msys fstab (c:/mingw /mingw)
2. Install Python (2.5)
3. Add Python and mingw to system path (C:\Python25;C:\MinGW\bin;C:\MinGW\msys\1.0\bin)
3. Get and build APR source (I could not get APR iconv to compile)
Download:
http://ftp.gnu.org/pub/gnu/libiconv/libiconv-1.11.tar.gz
http://mirror.nexcess.net/apache//apr/apr-1.5.2-win32-src.zip
http://mirror.nexcess.net/apache//apr/apr-util-1.5.4-win32-src.zip
http://sourceforge.net/projects/pcre/files/pcre/8.37/pcre-8.37.zip/download
http://zlib.net/zlib128.zip
Build:
Extract all files to c:\mingw\msys\1.0\src
Compile libiconv
cd libiconv-1.11
./configure CFLAGS="-O2 -s -mms-bitfields -march=i686" CXXFLAGS="-O2 -s -mms-bitfields -march=i686"
make && make install
Compile APR
cd apr
./buildconf
./configure CFLAGS="-O0 -s -mms-bitfields -march=i686" CXXFLAGS="-O0 -s -mms-bitfields -march=i686"
make && make install
cd ..
Compile APR-UTIL
cd apr-util-1.5.4
./buildconf --with-apr=/usr/src/apr-1.5.2
./configure CFLAGS="-O2 -s -mms-bitfields -march=i686" CXXFLAGS="-O2 -s -mms-bitfields -march=i686" --with-apr=/usr/src/apr-1.5.2
make && make install
cd ..
Compile PCRE
cd pcre-.37
./configure
make && make install
(make threw an error corrected with make clean, autoconf -i --force, started back at step 1)
cd ..
Compile ZLIB
cd zlib-1.2.8
make -f win32/Makefile.gcc
Compile nxlog
cd nxlog-ce-2.8.1248
./configure
This is where the problems began. First it couldn't find apr-1-config.
Fixed by adding /local/apr/bin to path.
Now it can't find libapr-1, addint /local/apr/lib to the path doesn't help. There is no libapr-1 file in the MinGW directory tree. Ideas?
-pacmanwa
pacmanwa created
WARNING input file does not exist
MohanGuttikonda created
Hi,
When the nxlog is already started and the logfile is yet to be created by the application, I see "WARNING input file does not exist" message in nxlog.log file.
How often does nxlog service retry\check for missing files which are created after starting nxlog service?
Thanks & Regards,
Mohan Guttikonda
MohanGuttikonda created
Adding 3 hours to the time
bigfoot created
Hi. How can I add 3 hours to the time that looks like this 2015-09-10 10:21:11. Something like this?
Exec $EventTime = parsedate($1) + 3h;
bigfoot created
Nxlog Installation in Solaris ( SunOS sparc sun4v)
nmohanraj.be@gmail.com created
Hi Team,
We are looking for nxlog installation in solaris machine. All our servers are running in SunOS sparc sun4v and we wanted to install nxlog in these servers. Can you please let us know is it possible to install nxlog on these servers?.. If yes please provide some stetps how to achive this?.
Regards,
Mohan.
nmohanraj.be@gmail.com created
Order messages from Windows Event Log with nxlog-elasticsearch-Kibana
moses created
Hi,
I'm using nxlog to send logs from Windows eventlog to elasticsearch, and using Kibana view.
I'm getting all the message as it is in the 'Message' column, I want to re-order it so the hostname parameter will be the windows server (and not the elasticsearch server), add 'Type' to the messages, etc.
this is the configuration file of nxlog:
* 55.2.110.4=elasticsearch server
<Extension json>
Module xm_json
</Extension>
# Nxlog internal logs
<Input internal>
Module im_internal
Exec $EventReceivedTime = integer($EventReceivedTime) / 1000000;
</Input>
# Windows Event Log
<Input eventlog>
# Uncomment im_msvistalog for Windows Vista/2008 and later
Module im_msvistalog
# Uncomment im_mseventlog for Windows XP/2000/2003
# Module im_mseventlog
Exec $EventReceivedTime = integer($EventReceivedTime) / 1000000;
</Input>
<Output out>
Module om_http
URL https://55.2.110.4:443
HTTPSAllowUntrusted TRUE
</Output>
<Route 1>
Path internal, eventlog => out
</Route>
this is an example message from kibana:
Field
Action
Value
@timestamp
2015-09-08T07:35:47.064Z
@version
1
_id
AU-r4dtqVULqkki94YkZ
_index
logstash-2015.09.08
_type
logs
host
55.2.110.4
http_port
5005
message
2015-09-08 07:35:43 dc-prod-zone-a.organization.com AUDIT_SUCCESS 4634 An account was logged off. Subject: Security ID: S-1-5-21-1595779987-1987268195-2987234418-1104 Account Name: DC-PROD-ZONE-C$ Account Domain: ORGANIZATION Logon ID: 0x679381 Logon Type: 3 This event is generated when a logon session is destroyed. It may be positively correlated with a logon event using the Logon ID value. Logon IDs are only unique between reboots on the same computer.
Thanks a lot
moses created
NXlog crashes continously.
Appsupport created
Hi,
I am running NXlog CE version 2.9.1347 on Windows 2012 R2. The service keeps crashing with the following;
Log Name: Application
Source: Application Error
Date: 07/09/2015 09:26:42
Event ID: 1000
Task Category: (100)
Level: Error
Keywords: Classic
User: N/A
Computer: myServer
Description:
Faulting application name: nxlog.exe, version: 0.0.0.0, time stamp: 0x54fedd1a
Faulting module name: ntdll.dll, version: 6.3.9600.17668, time stamp: 0x54c846bb
Exception code: 0xc0000005
Fault offset: 0x000195da
Faulting process id: 0x3504
Faulting application start time: 0x01d0e94f3d2521e0
Faulting application path: C:\Program Files (x86)\nxlog\nxlog.exe
Faulting module path: C:\Windows\SYSTEM32\ntdll.dll
Report Id: 8c5cf6ce-5542-11e5-80c1-005056bc12a5
Faulting package full name:
Faulting package-relative application ID:
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
<Provider Name="Application Error" />
<EventID Qualifiers="0">1000</EventID>
<Level>2</Level>
<Task>100</Task>
<Keywords>0x80000000000000</Keywords>
<TimeCreated SystemTime="2015-09-07T09:26:42.000000000Z" />
<EventRecordID>123651</EventRecordID>
<Channel>Application</Channel>
<Computer>myServer</Computer>
<Security />
</System>
<EventData>
<Data>nxlog.exe</Data>
<Data>0.0.0.0</Data>
<Data>54fedd1a</Data>
<Data>ntdll.dll</Data>
<Data>6.3.9600.17668</Data>
<Data>54c846bb</Data>
<Data>c0000005</Data>
<Data>000195da</Data>
<Data>3504</Data>
<Data>01d0e94f3d2521e0</Data>
<Data>C:\Program Files (x86)\nxlog\nxlog.exe</Data>
<Data>C:\Windows\SYSTEM32\ntdll.dll</Data>
<Data>8c5cf6ce-5542-11e5-80c1-005056bc12a5</Data>
<Data>
</Data>
<Data>
</Data>
</EventData>
</Event>
I have seen other posts about a hotfix that fixes another app crashing issue. But does it also fix this one?
Thanks,
Darren.
Appsupport created
process log and rewrite output
mark created
Hello
I have created a regex expression to extract values from my logs, I have difficulties in writing these values in the nxlog output
The issue is that the output file is identical to the input log, not sure what I'm doing wrong here, any help is welcome
My nxlog.conf is as follow
<Input in>
Module im_file
File "C:\\tmp\example-log.txt"
SavePos TRUE
Recursive TRUE
InputType multiline
exec if $raw_event =~ /(?:\{"([0-9]+?), ([0-9]+?), "(.+?)", "(.+?)", "(?:.+?)", "(.+?)\n(?:[a-zA-Z0-9:,"\n\r\s\f\t\-\{\}\.\(\)]+?)New\sLogon:\s*(?:.+?)\n\s*Account\sName:\s*(.+?)\n(?:[a-zA-Z0-9:,"\n\r\s\f\t\-\{\}\.\(\)]+?)Source\sNetwork\sAddress:\s*([0-9\.]{7,15})\n(?:[a-zA-Z0-9:,"\n\r\s\f\t\-\{\}\.\(\)]+?)"\})/ \
{ \
$timestamp = $1; \
$event = $2; \
$status = $3; \
$type = $4; \
$short = $5; \
$user = $6; \
$source = $7; \
}
</Input>
#<Output out>
# Module om_tcp
# Host 192.168.1.15
# Port 12201
# OutputType GELF_TCP
#</Output>
<Processor one>
Module pm_null
Exec $raw_event = $timestamp + $event + $status + $type + $short + $user + $source;
</Processor>
<Output out>
Module om_file
File 'C:\\tmp\output'
</Output>
<Route 1>
Path in => one => out
</Route>
Thanks heaps
Mark
mark created