Ask questions. Get answers. Find technical product solutions from passionate experts in the NXLog community.
GELF messages from Ubuntu are being truncated
dtilly created
My Setup:
Graylog2 server to collect logs
Ubuntu machine running Zimbra sending logs from various Zimbra logfiles in GELF format
My problem:
Messaged received by Graylog are truncated.
Here is an actual message as it appeared on my Ubuntu server's "mailbox.log" file (please note that I have X'd out the email address):
2016-06-12 08:51:17,832 INFO [ImapSSLServer-95] [name=XXXXXXX@XXX.org;ip=10.10.48.74;ua=iPod touch Mail/13C75;] imap - ID elapsed=0
Here is the log as received by Graylog:
2016-06-12 08:51:17,832 INFO [ImapSSLServer-95] [name=XXXXXXX
All message seem to be truncated after exactly the same number of characters. I cannot seem to figure this out and would love some help. Below I have pasted my nxlog.conf file:
## This is a sample configuration file. See the nxlog reference manual about the
## configuration options. It should be installed locally under
## /usr/share/doc/nxlog-ce/ and is also available online at
## http://nxlog.org/docs
########################################
# Global directives #
########################################
User nxlog
Group nxlog
LogFile /var/log/nxlog/nxlog.log
LogLevel INFO
########################################
# Modules #
########################################
<Extension gelf>
Module xm_gelf
</Extension>
<Extension syslog2>
Module xm_syslog
</Extension>
<Input mailbox.log>
Module im_file
File "/opt/zimbra/log/mailbox.log"
InputType LineBased
SavePos TRUE
</Input>
<Input access_log>
Module im_file
File "/opt/zimbra/log/access_log*"
SavePos True
</Input>
<Input audit.log>
Module im_file
File "/opt/zimbra/log/audit.log"
SavePos TRUE
</Input>
<Input clamd.log>
Module im_file
File "/opt/zimbra/log/clamd.log"
SavePos TRUE
</Input>
<Input freshclam.log>
Module im_file
File "/opt/zimbra/log/freshclam.log"
SavePos TRUE
</Input>
<Input mysql_error.log>
Module im_file
File "/opt/zimbra/log/mysql_error.log"
SavePos TRUE
</Input>
<Input mail.log>
Module im_file
File "/var/log/mail.log"
SavePos TRUE
</Input>
<Input zimbra.log>
Module im_file
File "/var/log/zimbra.log"
SavePos TRUE
</Input>
<Input syslog>
Module im_file
File "/var/log/syslog"
SavePos TRUE
</Input>
<Input zimbra-stats.log>
Module im_file
File "/var/log/zimbra-stats.log"
SavePos TRUE
</Input>
<Output out>
Module om_udp
Host 10.10.90.45
Port 5407
Exec to_syslog_snare();
</Output>
<Output out2>
Module om_udp
Host 10.10.90.45
Port 5413
OutputType GELF
</Output>
########################################
# Routes #
########################################
<Route 1>
Path mailbox.log => out
</Route>
<Route 2>
Path access_log => out
</Route>
<Route 3>
Path audit.log => out
</Route>
<Route 4>
Path clamd.log => out
</Route>
<Route 5>
Path freshclam.log => out
</Route>
<Route 6>
Path mysql_error.log => out
</Route>
<Route 7
Path mail.log => out
</Route>
<Route 8>
Path zimbra.log => out
</Route>
<Route 9>
Path syslog => out2
</Route>
<Route 10>
Path zimbra-stats.log => out
</Route>
dtilly created
Binary distribution for AIX?
rochbu created
I've seen the blog posting for AIX support following the download source and compile route.
Is a binary distribution of AIX in the roadmap to anyone's knowledge?
Thanks,
Rob
rochbu created
xm_multiline module for Oracle alert.log on AIX System
gfouere created
Hello,
I would like to obtain information in alert.log multiple lines for an event but without success. Below, my configuration :
<Extension multiline>
Module xm_multiline
HeaderLine /^\w\w\w\s\w\w\w\s\d\d\s\d\d:\d\d:\d\d\s\d\d\d\d/
</Extension>
<Extension syslog>
Module xm_syslog
</Extension>
<Input aix_syslog>
Module im_file
File "/var/log/syslog.out*"
Exec to_syslog_bsd();
</Input>
<Input oracle_log_DBA102XX>
Module im_file
InputType multiline
File "/oradata/DBA102XX/admin/trace/bdump/alert_DBA102XX.log"
Exec to_syslog_bsd();
Exec $SourceName = "DBA102XX";
</Input>
<Output fluent_aix>
Module om_udp
Host 172.31.12.104
Port 5139
</Output>
<Output fluent_oracle>
Module om_udp
Host 172.31.12.104
Port 5127
</Output>
########################################
# Routes #
########################################
<Route 1>
Path aix_syslog => fluent_aix
</Route>
<Route 2>
Path oracle_log_DBA102XX => fluent_oracle
</Route>
Result Fluentd side :
7 Jun 2016 23:21:21 oracle.user.notice {"host":"xmgpoc01","ident":"Tue","message":"21:21 DFT 2016"}
7 Jun 2016 23:21:21 oracle.user.notice {"host":"xmgpoc01","ident":"Thread","message":"1 advanced to log sequence 66 (LGWR switch)"}
7 Jun 2016 23:21:21 oracle.user.notice {"host":"xmgpoc01","ident":"","message":"/oradata/DBA102XX/oraredo01/LOG3A_DBA102XX.ORA"}
7 Jun 2016 23:21:21 oracle.user.notice {"host":"xmgpoc01","ident":"","message":"/oradata/DBA102XX/oraredo02/LOG3B_DBA102XX.ORA"}
This line "Exec $SourceName = "DBA102XX";" doesn't works, i dont know why.
Thank you for your help.
gfouere created
Configuring Options for sending Selective events
karthikaravind created
Hi,
I am a newbie and trying to configure NXLog on a windows machine to forward logs to a syslog server. With the below configuration, all events are being forwarded to the server. we need only critical and error level events and only Security and System related events to be forwarded to the server. Below is my configuration:
<Extension _syslog>
Module xm_syslog
</Extension>
<Input in>
Module im_msvistalog
</Input>
<Output out>
Module om_udp
Host 192.168.1.25
Port 514
Exec to_syslog_snare();
</Output>
<Route 1>
Path in => out
</Route>
Could some one help me what changes I should make so that only Critical and Error levels should be forwarded and only Security and System events should be forwarded ?
karthikaravind created
Can we get an Ubuntu 16.04 build or recent sources for ce 2.9?
wizhippo created
I would like to use nxlog on the recent Ubuntu server edition and the current package does not work on it. A new deb for Ubuntu 16.04 would be great or recent sources for the current 2.9 version.
wizhippo created
How to Append (concatenate) additional data on the end of syslog messages?
bluelotus created
How would I change the syslog event message on an output module so that every message gets an additional field? I want to add another value called "Project X" after the message portion of all syslogs events/messages as they are forwarded to another server? Would I use $raw_event as I show in my example?
Current config:
<Output out>
Module om_udp
Host 192.168.1.10
Port 514
</Output>
Would I do this?
<Output out>
Module om_udp
Host 192.168.1.10
Port 514
Exec $raw_event = $raw_event + "Project X";
</Output>
OR is it more like this...?
<Output Out>
Module om_udp
Host 192.168.1.10
Port 514
$Message = $Message + "Project X"
Exec to_syslog_bsd();
</Output out>
bluelotus created
om_http authentication?
Michael.Seto II created
I am sending data to Elasticsearch via port 9200 and I am also using security for Elasticsearch. Any access to the rest API requiries authentication.
Is there a way to configure the om_http output to provide headers to authenticate when sending data to Elasticsearch?
Thank you
Michael.Seto II created
ASSERTION FAILED nx_module_output_fill_buffer
bourazaniss created
What this error means that leads some output modules to stop sending logs without crashing?
ASSERTION FAILED at line 21 in writerfuncs.c/nx_module_output_fill_buffer(): "output->buflen == 0"
bourazaniss created
Windows eventlog transfert
sophie created
HI all, I am trying to transfer Windows eventlog (Securty, Application and System) from a server A to a server B. It is almost working if I send all the events of server A in a flat file on server B.
But my goal is a little bit different : I need to have the serverA-Events loggued in the eventviewer of the server B.
Do yo know if it is possible to achieve this ?
Thanks in advance !
sophie created
NXLog for Performance Monitoring
bourazaniss created
There is a tone of infrastructure and application monitoring tools out there ( uberagent, vmturbo etc., powershell scirpts) to collect proccess details as their main task.
Allthough some could be integrated with NXLog ( lets say by using im_exec, or xm_exec) they have limited filtering and output capabilities compared to NXLog.
If NXLog could implement one day one let's say im_perfmon module to collect performance metrics then it could be easily transformed to a top application to this category.
bourazaniss created
Windows .evt files to graylog
mathieurv created
Has anyone succeed in sending .evt file content to graylog ?
Actually, I found that:
- Using im_file module I can parse .evt file and send its content outside, but logs are bad formatted
- Using im_vistalog module I can't parse .evt files only the Windows Event log, but logs are well formatted
Any advice someone ?
Maybe it is possible to send the ouput of im_file to im_vistalog ?
Thanks,
--
Mathieu
mathieurv created
Random nxlog crashes every now and then
Michael.Seto II created
Every now and then I get reports of logs not reporting. I investigate and 99.9% of the time, it is due to a loss of connectivity to the log server due to an nxlog crash. Typically, it is due to a faulting module, per Windows Event Viewer.
OS - Windows Server 2012 R2 Datacenter
NXLOG Version - How do I check?
Event Viewer ::
Faulting application name: nxlog.exe, version: 0.0.0.0, time stamp: 0x54fedd1a
Faulting module name: libapr-1-0.dll, version: 0.0.0.0, time stamp: 0x54fedd1a
Exception code: 0xc0000005
Fault offset: 0x00015190
Faulting process id: 0x160
Faulting application start time: 0x01d1b804aaa52028
Faulting application path: D:\Program Files (x86)\nxlog\nxlog.exe
Faulting module path: D:\Program Files (x86)\nxlog\libapr-1-0.dll
Report Id: 79778f7a-2701-11e6-80c2-00155d590419
Faulting package full name:
Faulting package-relative application ID:
Is this a known issue? Are there ways to prevent this from happening?
Thank you!
Michael.Seto II created
Compile failure on FreeBSD - SSL error
root created
I'm attempting to compile the latest nxlog on FreeBSD and it fails with the following error. Any thoughts as to a fix or workaround would be appreciated. My environment is as follows:
FreeBSD 10.1-STABLE
OpenSSL 1.0.2h 3 May 2016
[/usr/local/src/nxlog-ce-2.8.1248]#make
Making all in src
Making all in common
make all-am
/bin/sh ../../libtool --tag=CC --mode=compile cc -DHAVE_CONFIG_H -I. -I/usr/local/include/apr-1 -I/usr/local/include -pipe -W -Wall -Wshadow -Wno-address -Wcast-qual -Wbad-function-cast -Wsign-compare -Wconversion -funsigned-char -Werror-implicit-function-declaration -Wswitch-default -fstrict-aliasing -Wpointer-arith -Wcast-align -Wwrite-strings -Wmissing-noreturn -Wmissing-format-attribute -Wformat -Wextra -Werror=format-security -rdynamic -ggdb3 -MT ssl.lo -MD -MP -MF .deps/ssl.Tpo -c -o ssl.lo ssl.c
libtool: compile: cc -DHAVE_CONFIG_H -I. -I/usr/local/include/apr-1 -I/usr/local/include -pipe -W -Wall -Wshadow -Wno-address -Wcast-qual -Wbad-function-cast -Wsign-compare -Wconversion -funsigned-char -Werror-implicit-function-declaration -Wswitch-default -fstrict-aliasing -Wpointer-arith -Wcast-align -Wwrite-strings -Wmissing-noreturn -Wmissing-format-attribute -Wformat -Wextra -Werror=format-security -rdynamic -ggdb3 -MT ssl.lo -MD -MP -MF .deps/ssl.Tpo -c ssl.c -o ssl.o
cc: warning: argument unused during compilation: '-rdynamic'
ssl.c:46:27: warning: cast from function call of type 'apr_os_thread_t'
(aka 'struct pthread *') to non-matching type 'unsigned long'
[-Wbad-function-cast]
ret = (unsigned long) apr_os_thread_current();
^~~~~~~~~~~~~~~~~~~~~~~
ssl.c:343:52: warning: cast to 'void *' from smaller integer type 'int'
[-Wint-to-void-pointer-cast]
SSL_set_ex_data(ssl, nx_ssl_verify_result_idx, (void *) verify_result);
^
ssl.c:372:12: error: implicit declaration of function 'SSLv3_method' is invalid
in C99 [-Werror,-Wimplicit-function-declaration]
meth = SSLv3_method();
^
ssl.c:372:10: warning: incompatible integer to pointer conversion assigning to
'const SSL_METHOD *' (aka 'const struct ssl_method_st *') from 'int'
[-Wint-conversion]
meth = SSLv3_method();
^ ~~~~~~~~~~~~~~
3 warnings and 1 error generated.
*** Error code 1
Stop.
make[3]: stopped in /usr/local/src/nxlog-ce-2.8.1248/src/common
*** Error code 1
Stop.
make[2]: stopped in /usr/local/src/nxlog-ce-2.8.1248/src/common
*** Error code 1
Stop.
make[1]: stopped in /usr/local/src/nxlog-ce-2.8.1248/src
*** Error code 1
Stop.
make: stopped in /usr/local/src/nxlog-ce-2.8.1248
root created
How to set File encoding to type Unicode to process MSSQLServer Error logs
mahibabar created
Hello Team,
I am sending MS SQL Server Error logs from NXLog to our TCP server for processing.
I need to select File encoding to Unicode, I tried but it is goving error if I set encoding to Unicode/unicode.
If I set encoding to utf-8/utf-16 it is adding dot(.) after every single character in a source log line.
Any suggestions/help would be really appriciated.
Thanks,
Mahesh
mahibabar@gmail.com
+91-9923622938
mahibabar created
Redis Module LPUSH
bourazaniss created
NXLog Redis output module sends data to a Redis server with the RPUSH command allowing only to change the Redis key ( = database).
RPUSH inserts all the specified values at the tail of the list stored at key
I suggest that the Redis command has to be implemented as a parameter and to support at least the LPUSH.
LPUSH inserts all the specified values at the head of the list stored at key
There are clients that support only retreiving data from the head. The most usual case is to send data to Redis with LPUSH and then have Logstash poll record from Redis with LPOP.
Thank you.
bourazaniss created
How to insert any string at end of multiline log file using nxlog?
sandeep.sigdel created
I have a multiline log sample as follow:
2/22/2016 4:19:30 PM 09F8 EVENT The DNS server has finished the background loading of zones. All zones are now available for DNS updates and zone transfers, as allowed by their individual zone configuration.
2/22/2016 4:19:38 PM 082C EVENT The DNS server has started.
2/22/2016 4:19:38 PM 08E0 PACKET 0000000001564000 UDP Rcv 192.168.2.213 526c Q [0001 D NOERROR] SOA (5)_ldap(4)_tcp(4)snpl(3)com(2)np(0)
UDP question info at 0000000001564000
Socket = 324
Remote addr 192.168.2.213, port 57611
Time Query=104249, Queued=0, Expire=0
Buf length = 0x0fa0 (4000)
Msg length = 0x0028 (40)
Message:
XID 0x526c
Flags 0x0100
QR 0 (QUESTION)
OPCODE 0 (QUERY)
AA 0
TC 0
RD 1
RA 0
Z 0
CD 0
AD 0
RCODE 0 (NOERROR)
QCOUNT 1
ACOUNT 0
NSCOUNT 0
ARCOUNT 0
QUESTION SECTION:
Offset = 0x000c, RR count = 0
Name "(5)_ldap(4)_tcp(4)snpl(3)com(2)np(0)"
QTYPE SOA (6)
QCLASS 1
ANSWER SECTION:
empty
AUTHORITY SECTION:
empty
ADDITIONAL SECTION:
empty
I wish to add the some string at end of this log say (ENDOFLOG) to the end of logfile. How can i achieve this?
sandeep.sigdel created
2016-05-20 09:41:31 ERROR om_tcp detected a connection error; An existing connection was forcibly closed by the remote host.
dmm3369 created
Has anyone seen this with NXlog when you are able to telnet to the host. When I set this up in a test lab it works fine. I assume a firewall but I can telnet fine to the host IP and port.
2016-05-20 09:41:31 ERROR om_tcp detected a connection error; An existing connection was forcibly closed by the remote host.
2016-05-20 09:41:32 INFO connecting to seclog.lowes.com:514
Very basic config.
#define ROOT C:\Program Files\nxlog
define ROOT C:\Program Files (x86)\nxlog
Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log
<Extension _syslog>
Module xm_syslog
</Extension>
<Input in>
#Module im_msvistalog
# For windows 2003 and earlier use the following:
Module im_mseventlog
Exec if $EventID NOT IN (528, 529, 567, 592, 601, 602, 608, 612, 636, 7034, 7035, 7036, 7040, 4097, 64004, 2, 3005) drop();
</Input>
<Output out>
Module om_tcp
Host 192.168.x.x
Port 514
Exec to_syslog_snare();
</Output>
<Route 1>
Path in => out
</Route>
dmm3369 created
how to add a field depending on different judgment statement
tuankun created
Hi All,
I use nxlog-ce-2.9.1504 to read log and parse it,the format of log is like:
20160523 10:58:22 sda 0 0 0 0 14 2 2 9 9 1 1 0 0
20160523 10:58:22 sdb 0 3 2 0 20 4 8 4 2 5 1 7 2
I want get the purpose:
if the third field is "sda",nxlog automatically add a string value "disk1" at the tail of the same row,if "sdb",add "disk2",like
20160523 10:58:22 sda 0 0 0 0 14 2 2 9 9 1 1 0 0 disk1
20160523 10:58:22 sdb 0 3 2 0 20 4 8 4 2 5 1 7 2 disk2
I configure the nxlog.conf as following,but it can not add any fields,I don't know where the mistake,anybody could help me? thanks so so so much~
<Extension kvp>
Module xm_kvp
</Extension>
<Input perf>
Module im_file
File "/collectl/LKNODE01*.dsk"
SavePos FALSE
ReadFromLast FALSE
Exec if $raw_event =~ /^(\S+) (\S+) (\S+) (\S+) (\S+) (\S+) (\S+) (\S+) (\S+) (\S+) (\S+) (\S+) (\S+) (\S+) (\S+) (\S+)/\
{\
kvp->parse_kvp();\
if $3 == "sda" $raw_event = $raw_event + " disk1";\
else if $3 == "sdb" $raw_event = $raw_event + " disk2";\
}
</Input>
<Output file>
Module om_file
file '/tmp/2.txt'
</Output>
<Route 1>
Path perf=> file
</Route>
tuankun created
Basic question about NXLOG file transfert configuration
sophie created
Hi all,
I am new with NXLOG and I have a very basic question to ask you : is it possible to simply copy a file from server A (source file name : c:\osit\log\df_mon.log) to server B (target file name : c:\osit\log15\df_mon.log). I spent already hours on NXLOG config file but I was not able to make it worked because the instruction "File" seeems not to be compatible wit the "om_tcp" output modeule and Host/Port settings.
Thanks a lot in advance, Sophie
sophie created
Tomcat Catalina, today date on filename
Pierrick.Prost created
Hi everyboy,
I have a log rotation folder on catalina application with a filename based on date like that "catalina.2016-04-15.log". Catalina create a log file with the date of the day, only if he need to write log (not automatically a file per day).
Some one has a exemple for configure NX log ?
Thanks in advance !
Pierrick.Prost created