Hello All
Happy New Year!!!
I am trying to delete IIS logs older than 7 days.
I tried using the following:
<Extension _fileop> Module xm_fileop <Schedule> Every 1 day Exec file_remove('E:\IIS Logs\W3SVC1\*.log', (now() - 6048000 )); </Schedule> </Extension>
But the above did not work - the file(s) were not deleted and no entries in the nxlog logfile.
Note: the nxlog-ce documentation states the following regarding wildcard usage:
file_remove(string file); Remove file. It is possible to specify a wildcard in the filename (but not in the path). The backslash () must be escaped if used as the directory separator with wildcards (for example, C:\test\*.log). This procedure will reopen the LogFile if it is removed. An error is logged if the operation fails.
For Debugging purposes, I tried using the following:
<Extension _fileop> Module xm_fileop
<Schedule> Every 1 day Exec if (file_ctime("E:\IIS Logs\W3SVC1\*.log") <= now()) log_info("File is older"); </Schedule> </Extension>
The following entry is in the nxlog logfile:
2019-01-02 18:10:19 ERROR failed to read file creation time on 'E:\IIS Logs\W3SVC1*.log': The filename, directory name, or volume label syntax is incorrect.
Note: Maybe file_cname cannot except wildcards in the filename because if I use the following:
<Schedule> Every 10 sec Exec if (file_ctime("E:\IIS Logs\W3SVC1\u_ex181224.log") <= now()) log_info("File is older"); </Schedule>
Then I receive the following in the nxlog logfile:
2019-01-02 18:23:08 INFO File is older
If I try the following:
<Schedule> Every 10 sec Exec if (file_ctime("E:\IIS Logs\W3SVC1\u_ex181224.log") <= now()) log_info("File is older"); Exec file_remove("E:\IIS Logs\W3SVC1\u_ex181224.log", (now() - 6048000 )); </Schedule>
The following appears in nxlog logfile:
2019-01-02 18:26:26 INFO File is older 2019-01-02 18:26:28 WARNING input file was deleted: E:\IIS Logs\W3SVC1\u_ex181224.log
All of the above is leading me to believe the file_remove procedure does not support wildcards in the name.
Does anyone have any experience with this?
Anyone have a suggestion on how I can implement the above? Ideally, I would like to pass in the IIS Logs parent folder and recursively delete files older than 7 days in all sub-folders.
Using nxlog-ce-2.10.2150
Many thanks
Stephen
stephen created
I am having this issue on both Ubuntu and FreeBSD. While trying to compile the pattern parser, it is failing because xcc doesn't exist:
Ubuntu:
xcc -b -i patterndb_parser.xcc -o patterndb_parser.c
/bin/bash: xcc: command not found
make[5]: [Makefile:738: patterndb_parser.c] Error 127 (ignored)
FreeBSD:
Making all in pattern
xcc -b -i patterndb_parser.xcc -o patterndb_parser.c
make[4]: exec(xcc) failed (No such file or directory)
*** Error code 1 (ignored)
Both of these lead to the build failing because patterndb_parser.c was not generated:
Ubuntu:
gcc: error: patterndb_parser.c: No such file or directory
gcc: fatal error: no input files
compilation terminated.
make[5]: *** [Makefile:480: patterndb_parser.lo] Error 1
FreeBSD:
cc: error: no such file or directory: 'patterndb_parser.c'
cc: error: no input files
*** Error code 1
Stop.
Where does xcc come from? Google is not helping me at all, nor find, nor looking though the git repository.
squigley created
Hi everybody,
I use nxlog with module om_batchcompress + SSL (self-signed certificates).
In nxlog client's log many errors sometimes:
ERROR remote ssl socket was reset? (SSL_ERROR_SYSCALL with errno=9); End of file found
What is means? Thank you so much.
versions:
- client nxlog-3.2.2002
- server nxlog-4.1.4046
hatula created
hatula created
Hello,
I have an application that logs some API requests and responses. Each request is logged in a different file, as a single line. In the system there are thousands of files, and Nxlog seems to have issues sending the logs to Elasticsearch. It reads the file, I can see im_file_add_file command in logs, but it takes a long time to actually send the message.
How does Nxlog process multiple files in a single directory? Thanks
kullboys created
Hi, I am very new to this and trying to configure the nxLog as a syslog server on windows 2012. I added the following to the default config file:
<Extension _syslog>
Module xm_syslog
</Extension>
<Input ssl>
Module im_ssl
Host localhost
Port 6393
CertFile %CERTDIR%/ukfff6292.pem
CertKeyFile %CERTDIR%/private.pfx
InputType Syslog_TLS
Exec parse_syslog_ietf();
</Input>
Then on cmd, I start the nxLog with -f option. I get the following error
2018-12-24 11:32:55 ERROR SSL error, couldn't read cert, no start line, 2018-12-24 11:32:55 WARNING no functional input modules! 2018-12-24 11:32:55 INFO nxlog-ce-2.10.2150 started 2018-12-24 11:32:55 ERROR SSL error, couldn't read cert, no start line,
Please help me regarding this.
infogatherer created
I'm trying to install/compile the Community Edition on FreeBSD 11. I'm having the same issues as were addressed in here: https://nxlog.co/question/2570/freebsd-port
ie "The port currently needs a little bit more work since it does not properly pick up PCRE from when not in /usr (as on FreeBSD) and I don't really like to hack use to work around that ;-)."
and also here: https://nxlog-ce-users.narkive.com/CyxivMgF/nxlog-2-7-1191-fails-to-configure-on-vanilla-freebsd-10-0
"checking pcre.h usability... no checking pcre.h presence... no checking for pcre.h... no configure: error: please install pcre-dev ***@logger:~/nxlog-ce-2.7.1191 #
pcre.h is installed from ports:
***@logger:~/nxlog-ce-2.7.1191 # find / -name pcre.h /usr/ports/devel/pcre/work/pcre-8.33/pcre.h /usr/local/include/pcre.h"
I have a similar configuration and error:
checking pcre.h usability... no checking pcre.h presence... no checking for pcre.h... no configure: error: please install pcre-dev
[simonq@freebsd11 /usr/local]$ find . -name pcre.h ./include/pcre.h
I tried the same fix from there, modifying the configure.sh script to set CFLAGS="-I/usr/local/include", and even adding --includedir=/usr/local/include to the ./configure line, and oldincludedir, since that is apparently used by clang, vs gcc/includedir, but with no change.
The config.log looks like it should be looking in the right place:
Invocation command line was $ ./configure --libexecdir=/usr/local/lib --bindir=/usr/local/bin --sysconfdir=/usr/local/etc --libdir=/usr/local/lib --localstatedir=/var --includedir=/usr/local/include --oldincludedir=/usr/local/include
PCRE_CFLAGS='-I/usr/local/include' PCRE_LIBS='-L/usr/local/lib -Wl,-R/usr/local/lib -lpcre' includedir='/usr/local/include' oldincludedir='/usr/local/include'
configure:13250: checking pcre.h usability configure:13250: cc -c conftest.c >&5 conftest.c:64:10: fatal error: 'pcre.h' file not found #include <pcre.h> ^~~~~~~~ 1 error generated.
| #include <pcre.h> configure:13250: result: no configure:13250: checking pcre.h presence configure:13250: cc -E conftest.c conftest.c:31:10: fatal error: 'pcre.h' file not found #include <pcre.h> ^~~~~~~~ 1 error generated.
| #include <pcre.h> configure:13250: result: no configure:13250: checking for pcre.h configure:13250: result: no configure:13257: error: please install pcre-dev
I'm just wondering if anyone else has worked this out? There is a package for the Enterprise Edition for FreeBSD, so there must be a way to get it to compile..
squigley created
We are facing these error messages in NXLog log files:
2018-12-09 23:45:16 ERROR om_tcp detected a connection error;Connection timed out 2018-12-09 23:45:16 INFO reconnecting in 2 seconds 2018-12-09 23:45:17 INFO connecting to 192.168.1.2:1468 2018-12-10 01:56:53 INFO reconnecting in 1 seconds 2018-12-10 01:56:53 ERROR om_tcp detected a connection error;Connection timed out 2018-12-10 01:56:53 INFO reconnecting in 2 seconds 2018-12-10 01:56:54 INFO connecting to 192.168.1.2:1468 2018-12-10 04:08:30 INFO reconnecting in 1 seconds 2018-12-10 04:08:30 ERROR om_tcp detected a connection error;Connection timed out 2018-12-10 04:08:30 INFO reconnecting in 2 seconds 2018-12-10 04:08:31 INFO connecting to 192.168.1.2:1468
I suspect that our firewall is blocking those connections eventually as i can't see any TCP Keep-Alive packets in tcpdump from the server where the NXLog agent is installed. On 2015-04-01 2.9.1361 changelog is this line:
All tcp based network modules are now using TCP_KEEPALIVE to detect dead peers.
Any ideas on how to fix this?
JaVa created
Hi,
We have had few instances on several servers in the past few weeks, where the NXlog application crashes when starting the service. Commonality so far is Windows Server 2012 Standard. Reinstall of the application corrects the problem, but doesn't sound like a solution. Anyone else experiencing this issue? Any fixes?
Following is the error from the Application Log:
Faulting application name: nxlog.exe, version: 4.1.4016.0, time stamp: 0x00000000 Faulting module name: ntdll.dll, version: 6.2.9200.22376, time stamp: 0x5a90c271 Exception code: 0xc0000374 Fault offset: 0x00000000000da535 Faulting process id: 0x18a4 Faulting application start time: 0x01d490bb5247014e Faulting application path: C:\Program Files\nxlog\nxlog.exe Faulting module path: C:\Windows\SYSTEM32\ntdll.dll Report Id: 8ffb8b3b-fcae-11e8-944f-005056856b7b Faulting package full name: Faulting package-relative application ID:
sinisa created
Hello, I have two questions.
First, a latest nxlog-ce formally supports Windows Server 2016 ? I saw the following url, and now it seems to support the Windows Server 2016.
https://nxlog.co/documentation/nxlog-user-guide/platforms.html
Second, the eventlog source limitation on Windows Server 2016 (the OS have over 256 sources, but the nxlog could treat up to 256 sources) is solved ? I saw the following forum questions.
https://nxlog.co/question/2594/256-sources-limit https://nxlog.co/question/3200/eventlog-source-limitation-server-2016
Deleted user created
I have the following NXLog config below that reads a file from disk and forwards it to another host (syslog). By default, it appears NXLog only forwards the data in the log file if it has changed.
How can you configure 'nxlog.conf' so that it always forwards the file, even if the data hasn't changed?
<Input in> Module im_file File "C:\myfile.txt" </Input>
<Output out> Module om_tcp Host 127.0.0.1 Port 12345 </Output>
<Route 1> Path in => out </Route>
demetri created
Austin.Downing created
Olistra created
Dear All!
We are trying to read the listener log of an Oracle database with Nxlog.
Sample: <msg time='2018-12-01T23:50:08.409+01:00' org_id='oracle' comp_id='tnslsnr' type='UNKNOWN' level='16' host_id='server11' host_addr='10.10.10.10'> <txt>01-DEC-2018 23:50:08 * (CONNECT_DATA=(SID=oracle)(CID=(PROGRAM=JDBC Thin Client)(HOST=jdbc)(USER=user1$))) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.10.10.10)(PORT=54468)) * establish * db1 * 0 </txt> </msg>
It is clear that this is a multiline message which can be read by xm-multiline, then with xm_xml it is possible to convert it to XML entries.
After the 'txt' part is available for processing I would like to capture the following info from it: DATE=01-DEC-2018 23:50:08 (this is simple, just cut the first 20 characters. Action=establish (this is simple, splitting the text by the '') Target=db1 (this is simple, splitting the text by the '') Result=0 (this is simple, splitting the text by the '*') SID=oracle PROGRAM=JDBC Thin Client USER=user1$ HOST=10.10.10.10 PORT=54468
In my opinion the SID/PROGRAM/USER/HOST/PORT cannot be captured by the xm_kvp module as it is not separated by the same symbol. What would you recommend to collect the proper info from that part of the txt?
I have not found any solution for reading the Oracle listener log - that's why it would be a big help to get your opinions!
Thank you!
Pdudas created
Hi,
is it possible to generate event log entries to Windows Event Log at the start/stop of Nxlog? At the moment start/stop events are in nxlog.log - for safety reasons it would make sense to add entries to Windows Event log if possible.
Thank you!
Pdudas created
I'm new to NXlog, and I'm trying to get NXlog working on a Windows Server 2012 R2 server.
I downloaded NXlog Community Edition, and amended the nxlog.conf file as I required. When trying to start the 'nxlog' service I get the following error:
"Windows could not start the nxlog service on Local Computer. Error 2: The system cannot find the file specified."
After doing some troubleshooting I noticed the file 'C:\Program Files (x86)\nxlog\nxlog.exe' is missing. It doesn't seem to have got installed during installation.
I've had the same behaviour on two of my test servers, any ideas?
gizmo83 created
Problem is that NXlog doesn't start and I get following errors:
status nxlog.service
● nxlog.service - NXLog daemon
Loaded: loaded (/usr/lib/systemd/system/nxlog.service; enabled; vendor preset: disabled)
Active: activating (auto-restart) (Result: exit-code) since Sat 2018-11-24 02:44:31 EET; 2s ago
Process: 10620 ExecStartPre=/opt/nxlog/bin/nxlog -v (code=exited, status=1/FAILURE)
Nov 24 02:44:31 fetcher nxlog[10620]: 2018-11-24 02:44:31 DEBUG stopping EXTENSION modules
Nov 24 02:44:31 fetcher nxlog[10620]: 2018-11-24 02:44:31 DEBUG stopping module csv
Nov 24 02:44:31 fetcher nxlog[10620]: 2018-11-24 02:44:31 DEBUG shutdown_modules: INPUT
Nov 24 02:44:31 -fetcher nxlog[10620]: 2018-11-24 02:44:31 DEBUG shutdown_modules: PROCESSOR
Nov 24 02:44:31 -fetcher nxlog[10620]: 2018-11-24 02:44:31 DEBUG shutdown_modules: OUTPUT
Nov 24 02:44:31 fetcher nxlog[10620]: 2018-11-24 02:44:31 DEBUG shutdown_modules: EXTENSION
Nov 24 02:44:31 fetcher nxlog[10620]: 2018-11-24 02:44:31 DEBUG no entries found, not writing configcache.dat
Nov 24 02:44:31 fetcher nxlog[10620]: 2018-11-24 02:44:31 DEBUG nxlog_shutdown() leave
Nov 24 02:44:31 fetcher systemd[1]: Unit nxlog.service entered failed state.
Nov 24 02:44:31 fetcher systemd[1]: nxlog.service failed.
2018-11-24 02:48:15 DEBUG Error in Exec block: [to_syslog_bsd();]
2018-11-24 02:48:15 ERROR [modules.c:489/nx_ctx_config_modules()] -;[module.c:1567/nx_module_parse_exec_block()] Couldn't parse Exec block at /opt/nxlog/etc/nxlog.conf:37;[expr-grammar.y:381/parser_do()] couldn't parse statement at line 37, character 21 in /opt/nxlog/etc/nxlog.conf;[expr.c:3359/nx_expr_statement_new_procedure()] procedure 'to_syslog_bsd()' does not exist or takes different arguments
2018-11-24 02:48:15 DEBUG nxlog_shutdown() enter
2018-11-24 02:48:15 DEBUG stopping INPUT modules
2018-11-24 02:48:15 DEBUG stopping module in
2018-11-24 02:48:15 DEBUG stopping PROCESSOR modules
2018-11-24 02:48:15 DEBUG stopping OUTPUT modules
2018-11-24 02:48:15 DEBUG stopping module out
2018-11-24 02:48:15 DEBUG stopping EXTENSION modules
2018-11-24 02:48:15 DEBUG stopping module csv
2018-11-24 02:48:15 DEBUG shutdown_modules: INPUT
2018-11-24 02:48:15 DEBUG shutdown_modules: PROCESSOR
2018-11-24 02:48:15 DEBUG shutdown_modules: OUTPUT
2018-11-24 02:48:15 DEBUG shutdown_modules: EXTENSION
2018-11-24 02:48:15 DEBUG no entries found, not writing configcache.dat
2018-11-24 02:48:15 DEBUG nxlog_shutdown() leave
########################################
# Global directives #
########################################
User nxlog
Group nxlog
LogFile /opt/nxlog/var/log/nxlog/nxlog.log
LogLevel debug
Moduledir /opt/nxlog/libexec/nxlog/modules/
define ROOT /opt/nxlog
CacheDir /opt/nxlog/var/log/nxlog/
Pidfile /opt/nxlog/var/log/nxlog/nxlog.pid
SpoolDir /opt/nxlog/var/log/nxlog/
########################################
# Modules #
########################################
<Extension csv>
Module xm_csv
</Extension>
<Input in>
Module im_file
File "/home/cust/*.csv"
ReadFromLast FALSE
SavePos FALSE
</Input>
<Output out>
Module om_udp
Host 10.10.10.10
Port 514
Exec to_syslog_bsd();
#Exec to_syslog_ietf();
</Output>
########################################
# Routes #
########################################
<Route 1>
Path in => out
</Route>
joniro created
Hi Team, trying to capture some log files, and get them to the right facility or local, currently I have this
<Input messages> Module im_file File "D:\dhcp\logs\*.log" </Input>
Which does work but I need to get them to facility 21 and local5, any idea how to format my entry so this works?
Thanks,
zamarac created
Hello,
I'm currently facing an issue with missing log when log file rotate. Same problem as described here: https://nxlog.co/question/3495/missing-log-when-log-file-rotate I'm using NXLog CE 2.10.2102. I tried to upgrade to last version (2.10.2150) but still same problem. Could you tell me if this issue has been fixed in CE edition please? If not do you have any idea when it will be fixed? Thank you very much.
Nicolas
Chewb created
We're using NX Log (CE) as a test to see if it will work for our purposes. The overall idea is to use it as a forwarder of syslog flat files to any brand of SIEM.
My config looks like this:
Panic Soft #NoFreeOnExit TRUE
define ROOT C:\Program Files (x86)\nxlog define CERTDIR %ROOT%\cert define CONFDIR %ROOT%\conf define LOGDIR %ROOT%\data define LOGFILE %LOGDIR%\nxlog.log LogFile %LOGFILE%
Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data
<Extension syslog> Module xm_syslog </Extension>
#<Input Alerts>
Module im_file
File 'E:\DGQradarExports\ForwarderCust\Alerts\*'
ReadFromLast True
Exec parse_syslog();
#</Input>
<Input Events> Module im_file File 'E:\DGQradarExports\ForwarderCust\Events\*' ReadFromLast True Exec parse_syslog(); </Input>
<Input Process> Module im_file File 'E:\DGQradarExports\ForwarderCust\Process\*' ReadFromLast True Exec parse_syslog(); </Input>
<Output Customer> Module om_tcp Host 192.168.160.141 Port 514 </Output>
<Route customer_siem> Path Events,Process => Customer </Route>
<Extension _charconv> Module xm_charconv AutodetectCharsets iso8859-2, utf-8, utf-16, utf-32 </Extension>
<Extension _exec> Module xm_exec </Extension>
<Extension _fileop> Module xm_fileop
# Check the size of our log file hourly, rotate if larger than 5MB
<Schedule>
Every 1 hour
Exec if (file_exists('%LOGFILE%') and \
(file_size('%LOGFILE%') >= 5M)) \
file_cycle('%LOGFILE%', 8);
</Schedule>
# Rotate our log file every week on Sunday at midnight
<Schedule>
When @weekly
Exec if file_exists('%LOGFILE%') file_cycle('%LOGFILE%', 8);
</Schedule>
</Extension>
My log just shows this: 2018-11-19 08:50:43 INFO nxlog-ce-2.10.2102 started 2018-11-19 08:50:43 INFO connecting to 192.168.160.141:514
QRadar shows an information source has registered but no data ever flows.
I should see a 'connection was successful' message shouldn't I? Is there any way to up the logging so I can tell if NX Log is even reading the files and attempting to send them? I really can't tell what it's doing currently.
Multiple files exist in the input directories, I'm trying to have NX Log work through all of them, send them to SIEM and then wait for more files. Config examples seem straightforward, I just can't tell what it's doing.
Any help is appreciated.
nhart created