Ask questions. Get answers. Find technical product solutions from passionate experts in the NXLog community.
Change Syslog Facility when reading/forwarding windows events.
sswager created
Just installed nxlog to begin forwarding events to AlienVault, everything seems to be working so far with reading and forwarding events from the windows log using the im_msvistalog module. One thing we would like to change to set the Syslog Facility before forwarding it AlienVault. Have been unable to locate how to do so.
sswager created
nxlog community edition issue on Windows 8/8.1 operating systems.
MarcoFranceschini1971 created
Hi, i used your nxlog-CE but only my Windows 7 hosts don't exhibit this issue.
"Nome dell'applicazione che ha generato l'errore: nxlog.exe, versione: 0.0.0.0, timestamp: 0x53ca79be
Nome del modulo che ha generato l'errore: im_msvistalog.dll, versione: 0.0.0.0, timestamp: 0x53ca79bd
Codice eccezione: 0xc0000005
Offset errore 0x00001cca
ID processo che ha generato l'errore: 0xbfc
Ora di avvio dell'applicazione che ha generato l'errore: 0x01d02f68896fae85
Percorso dell'applicazione che ha generato l'errore: C:\Program Files (x86)\nxlog\nxlog.exe
Percorso del modulo che ha generato l'errore: C:\Program Files (x86)\nxlog\modules\input\im_msvistalog.dll"
nxlog community edition version 2.8.1248
Many thanks.
MarcoFranceschini1971 created
Windows: Auto Start of nxlog service after the install
pmjanvre created
Hi,
I noticed the NXlog service is not started at the end of the install process. I would the service to start automatically at the end of the install of the MSI.
Our goal is:
- modify the conf file in the MSI file to have it ready out of the box.
- Deploy with GPO on all servers.
For this second step, we need to ensure it starts automatically during the install process. The best would be to edit the install file to make a "net start nxlog" at the end.
Thanks
PM
pmjanvre created
Inconsistent log sending from windows to graylog2
avner created
Hi,
We are using the community edition of nxlog 2.8.1248 on windows 2008 R2 server. We are having forwarding event log and IIS logs to graylog2.
This is the conf file is pasted below.
If we just have the IIS udp forwarding, it *sometimes* works. We think its not working and then a few hours later we see data coming through, then it might stop again.
This IIS issue is also incosistent across machines. Some machines send data, while others never do
There are no errors in the nxlog.log file
The event log forwarding worked when we used om_udp and GELF format, but when its turned on in combination with IIS (as per conf below) it sends nothing.
Graylog2 server is up and running with the respective inputs.
I've tested UDP packets can get through to the Graylog2 server
I've checked the IIS csv parsing is correct, and as mentioned I don't see errors in the log.
Would appreciate ideas on what be going on, and how we might troubleshoot this issue?
Thanks,
Av
## This is a sample configuration file. See the nxlog reference manual about the
## configuration options. It should be installed locally and is also available
## online at http://nxlog.org/nxlog-docs/en/nxlog-reference-manual.html
## Please set the ROOT to the folder your nxlog was installed into,
## otherwise it will not start.
#define ROOT C:\Program Files\nxlog
define ROOT C:\Program Files (x86)\nxlog
Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log
LogLevel INFO
<Extension syslog>
Module xm_syslog
</Extension>
<Extension gelf>
Module xm_gelf
</Extension>
# Create the parse rule for IIS logs. You can copy these from the header of the IIS log file.
<Extension w3c>
Module xm_csv
Fields $date, $time, $s-sitename, $s-computername, $s-ip, $cs-method, $cs-uri-stem, $cs-uri-query, $s-port, $cs-username, $c-ip, $cs-version, $cs(User-Agent), $cs(Cookie), $cs(Referer), $cs-host, $sc-status, $sc-substatus, $sc-win32-status, $sc-bytes, $cs-bytes $time-taken
FieldTypes string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, integer, integer, integer, string, string, string
Delimiter ' '
QuoteChar '"'
EscapeControl FALSE
UndefValue -
</Extension>
# Enable json extension
<Extension json>
Module xm_json
</Extension>
# Convert the IIS logs to JSON and use the original event time
# Uncomment IIS_IN section if logging for IIS logging
<Input eventlog>
# Use 'im_mseventlog' for Windows XP, 2000 and 2003
Module im_msvistalog
# Uncomment the following to collect specific event logs only
Query <QueryList>\
<Query Id="0">\
<Select Path="Application">*</Select>\
<Select Path="System">*</Select>\
</Query>\
</QueryList>
</Input>
<Input IIS_Site1>
Module im_file
File "C:\\inetpub\\logs\\LogFiles\\W3SVC6\\u_ex*.log"
SavePos TRUE
Exec if $raw_event =~ /^#/ drop(); \
else \
{ \
w3c->parse_csv(); \
$EventTime = parsedate($date + " " + $time); \
$SourceName = "IIS"; \
$Message = to_json(); \
}
</Input>
<Output Event_Out>
Module om_udp
Host 10.85.105.215
Port 12201
OutputType GELF
</Output>
<Output IIS_Out>
Module om_udp
Host 10.85.105.215
Port 514
</Output>
<Route IIS>
Path IIS_Site1 => IIS_Out
</Route>
<Route Events>
Path eventlog => Event_Out
</Route>
avner created
Unnecessary syslog header was recorded from 0:00 to 9:00 on Jan 1, 2015 of JST(UTC+9:00)
kaiedak created
Unnecessary syslog header was recorded from 0:00 to 9:00 on Jan 1, 2015 of JST(UTC+9:00).
Example
-----
<133>Dec 31 23:55:04 OTSS0101 OTxx01xx: warning
<133>Jan 1 00:20:12 10.70.0.32 Jan 01 00:21:51 OTSS0101 OTxx01xx: critical
--
<133>Jan 1 08:00:15 10.70.0.32 Jan 01 08:01:52 OTSS0101 OTxx01xx: critical
<133>Jan 1 09:01:51 OTSS0101 OTxx01xx: critical
-----
I use the following input and output.
-----
input
Module im_udp
Exec parse_syslog_bsd();
--
output
Module om_file
Exec to_syslog_bsd();
-----
Is this known behavior?
kaiedak created
Trouble with IPv6
awood created
I am trying to setup a listener on IPv6 (nxlog community edition) and am receiving errors on startup. When using IPv4, it starts without error:
Platform CentOS 6.6 x64_86
Configured with both short and long IPv6 notation:
2015-01-04 06:07:27 ERROR apr_sockaddr_info failed for 2604:234::1234:5678:41514;Address family for hostname not supported
2015-01-04 04:58:34 ERROR apr_sockaddr_info failed for 2604:0234:0000:0000:0000:0000:1234:5678:41514;Address family for hostname not supported
Configured with DNS (Long IPv6 notation in DNS record, though resolves short)
Using a AAAA hostname:
Name: ramscan02-6.test.org
Address: 2604:234::1234:5678
2015-01-04 05:53:19 ERROR couldn't bind ssl socket to ramscan02-6.test.org:41514;Cannot assign requested address
Any help would be appreciated,
Andy
awood created
Feature request: om_email
akumar created
Botond,
Would you consider creating an output module for email? It would be quite useful for generating alerts. While one can use exec_async to send email, it is rather cludgy.
Cheers
Ash
akumar created
NXlog agent do not send events from Windows Security log
Barns2 created
Well...
NXlog (last vrsion from this site) installed on windows server 2012R2
Configured to get win-logs:
SavePos TRUE
Module im_msvistalog
Query <QueryList> \
<Query Id="0" Path="Security"> \
<Select Path="Application">*</Select> \
<Select Path="Security">*</Select> \
<Select Path="System">*</Select> \
<Suppress Path="Security">*[System[(EventID=5156 or EventID=4663 or EventID=5158 or EventID=5440 or EventID=5444)]]</Suppress> \
<Suppress Path="System">*[System[(EventID=5156 or EventID=4663 or EventID=5158 or EventID=5440 or EventID=5444)]]</Suppress> \
</Query> \
</QueryList>
As a result I see events only from System and Application... Nothing from Security
Any Idea why it can happen?
Logs are captured by windows - I can see it with eventvwr.msc, but nothing with Nxlog
NXlog have no information, looks like everything is ok:
...INFO nxlog-ce-2.8.1248 started... - no errors, no warnings... nothing else
Barns2 created
Counter tracking assistance
akumar created
I have multiple windows hosts sending events in binary to a single tcp listener.<Input windows>
Module im_tcp
Port 9999
Host 0.0.0.0
InputType Binary
I am trying to track the rate of logs from the servers and create email alerts when the rate either drops or crosses a high watermark per hour.
To do that I need to create a stat / variable appending the hostname and hourstamp such as
create_stat("Rate-" + '$Hostname' + 'stroftime($EventTimeStamp, something something)') or
create_var("Rate-" + '$Hostname' + 'stroftime($EventTimeStamp, something something)')
Next I use the schedule code to detect a low watermark
<Schedule>
Every 3600 sec
Exec create_stat("rate" + '$Hostname' + 'stroftime($EventTimeStamp, something something)'', "RATE", 10); add_stat("rate" + '$Hostname'' + 'stroftime($EventTimeStamp, something something)', 0);
Exec log_info("Current Counts " + ":" + get_stat("rate" + '$Hostname'));
Exec if defined get_stat("rate" + '$Hostname') and get_stat("rate" + '$Hostname') <= 1 \
{ \
log_warning("No messages received from Host" ); \
exec_async("/bin/sh", "-c", 'echo "' + $Hostname + \
'"|/usr/bin/mail -a "Content-Type: text/plain; charset=UTF-8" -s "ALERT" ' \
+ 'analyst@company.com' ); \
}
</Schedule>
Two problems: How do I insert the variable/statistic name and value in the log message and how do I extract the hour stamp from the Event Time?
Thanks
Ash
PS: I could not get the deployment tool to work. have you had more success with it?
akumar created
im_file configuration
guyl created
i defined an input for im_file that is being sent as syslog
i have multiple files in a folder what i want to do is read each file and on EOF copy to another folder.
didn`t find the option to identify EOF
guyl created
multiline bug?
pgs created
Hi,
I'm trying to use the xm_multiline module with nxlog to forward content of a logfile to logstash The log contains different xml elements which are properly indented (opening and closing elements are located at the start of the line) . E.g.
<data
version="x"
xmlns:bla="http://www.example.com/bla">
<val:InfoSet>
...
...
...
</val:InfoSet>
</data>
<message ...>
<ns>bla</ns>
...
...
</message>
Because the elements have different names, I can only use < and </ to find the start and end line. I was hoping a filter like this should be enough to select the correct lines:
HeaderLine /^</
EndLine /^<//
But somehow nxlog gets confused with the / in the regex pattern. I also tried escaping which dindn't help. More testing showed that it needs at least one letter. I tried to specify all letters via regex but that didn't work:
HeaderLine /^<[a-z]/
Only way that seems to work is to specify all letters in the square braket (with the exception of the lettern, which breaks).
HeaderLine /^<[abcdefghijklmopqrstuvwxyz]/ (left out n)
Here all my test results.
These lines worked:
HeaderLine /^<m/
EndLine /^</m/
HeaderLine /^<m/
EndLine /^<\/m/
HeaderLine /^<[abcdefghijklm]/
EndLine /^<\/[abcdefghijklm]/
HeaderLine /^<[abcdefghijklmo]/
EndLine /^<\/[abcdefghijklmo]/
HeaderLine /^<[abcdefghijklmopqrstuvwxyz]/ (left out n)
EndLine /^<\/[abcdefghijklmopqrstuvwxyz]/
HeaderLine /^<[abcdefghijklmopqrstuvwxyz]/ (left out n + not escaped
EndLine /^</[abcdefghijklmopqrstuvwxyz]/
These lines didn't work:
HeaderLine /^</
EndLine /^</m/
HeaderLine /^<[a-z]/
EndLine /^</m/
HeaderLine /^<\w/
EndLine /^</m/
HeaderLine /^<[abcdefghijklmn]/
EndLine /^<\/[abcdefghijklmn]/
HeaderLine /^<[bcdefghijklmn]/
EndLine /^<\/[bcdefghijklmn]/
HeaderLine /^<[abcdefghijklmopqrstuvwxyzn]/
EndLine /^<\/[abcdefghijklmopqrstuvwxyzn]/
HeaderLine /^<[abcdefghijklmnopqrstuvwxyz]/
EndLine /^</[abcdefghijklmnopqrstuvwxyz]/
Right now I still have a problem because many of my bessages start with <n. I think this is a bug in the module. Can you confirm so I can open a ticket? Thanks
Fyi, this is a duplicate of http://stackoverflow.com/questions/27429234/which-headerline-and-endline-for-multiline-xml-with-different-elements
pgs created
im_dbi & MS SQL
ghostcat created
Hello Team,
I'm looking to use nxlog to retrieve a table entries from MS SQL , To be clear I do not require any of the database server logging only the info contained within a few tables within a database on the server.
I have followed the setup and battled first under windows and then linux versions of nxlog. I am now trying to use libdbi-freetds without success after having out of memory upon using the mysql hooks.
Here is my im_dbi without the user and pass for security.
<Input DB01-elog>
Module im_dbi
SavePos TRUE
SQL SELECT AllXml FROM db_table
Driver freetds
Option host 192.168.148.227
Option port 49000 (non standard port changed from 1433)
Option username domain\username
Option password password
Option dbname database
</Input>
The SQL server im attempting to connect to is MS SQL Server 2014, I also attempted this using mysql however recieved out of memory from the libdbi during execution of the query.
The OS machine has 8GB of mem available currently upped from 4GB when the out of memory initially occured.
I'm secretly hoping someone has run into this before and has been able to successfully retrieve table data and log it from MS SQL Server 2014.
If not im happy to work with the team to resolution on this.
Many Thanks,
GC
ghostcat created
Customize windows version
Barns2 created
Hello!
I need to make some changes in sources of modules. Ok - I found necessary code, changed it.
But how to compile windows version? No documentation found :(
Barns2 created
Exclude event logs for a particular application.
logsec created
Hi
Please help to exclude/ignore logs for a particular application
Thanks
Virender
logsec created
Large eventlog entries makes nxlog "hang"
MagnusBjarnlid created
We are using nxlog to collect eventlog information. Some entries can be large, in fact some message are split over several entries as a workaround for the maximum eventlog entry size. However, these large entries seem to hang nxlog so that it stops processing new entries. Typical error messages are:
---------------------------------------
2014-10-27 17:10:32 ERROR EvtNext failed with error 1734: The array bounds are invalid.
2014-10-27 17:10:33 ERROR EvtUpdateBookmark failed: The handle is invalid.
----------------------------------------
Why is this? Is there any workaround?
MagnusBjarnlid created
IIS log & bad characters causes NXlog crash causing random event dump
chris.trotter.sci created
We are using NXlog to send Windows EventLogs and IIS logs to Logstash (ELK). It works very well indeed with only one problem - at 2am (we suspect IIS log rotation, or maybe weird Netscaler packet) every day the NXlog service crashes on ALL servers that are sending IIS logs. We have some other servers only sending EventLogs - no crashing occurs. I have set the Windows NXlog service to restart on a crash, so service is only interrupted for a minute, but here's the weird part - NXlog re-sends a (random?) huge chunk of messages. I say 'random', because check out this list of time received vs. event time: (these are all from the same server)
2014-10-28T06:00:54.074Z & 2014-10-25 04:18:08
2014-10-28T06:00:54.074Z & 2014-10-25 03:58:08
2014-10-28T06:00:54.072Z & 2014-10-25 03:20:02
2014-10-28T06:00:54.058Z & 2014-10-25 03:10:33
2014-10-28T06:00:54.058Z & 2014-10-25 03:18:10
The weird time received jumping forward/backward in time continues (perhaps it's an ELK stack processing oddity)...keeping in mind we're talking hundreds of entries over a 1-2 minute period. On our UAT servers I observed that this large chunk of logs contains logs from the last week or more - but it's not consistent enough to say 'it's re-sending everything from the last week'.
Anyways - what causes the crash is bad input, like weird bad characters...I am sure the following copy/paste won't do it justice.
If anyone has ideas, I'm open!! Happy to either fix the weird log dumps, or fix the bad input data. We're continuing to investigate, but I thought it important (and not Google-friendly) enough to post here. A number of Google hits for questions like mine with no answers (or 'no bad input filtering, is a bug, sorry').
(disregard the timestamp)
2014-10-22 02:51:39 Not enough fields in CSV input, expected 22, got 1 in input 'PK
'
2014-10-22 02:51:39 Not enough fields in CSV input, expected 22, got 1 in input 'U@¦Ä-*¸H¶
ÜH—ÙùPÜÙ¡žŒÛŠ”¹‘ðè²Þè'
2014-10-22 02:51:39 Not enough fields in CSV input, expected 22, got 2 in input '¹'O‚¨ˆÌ(ˆÌÈF&È”™™ØÌDEéA3)B3éA3±¡é#eEff63AO‚È,ŠÈ,‚È,zÈ,ŠÈ,ld¢Ùé1³g3³&ŸI™½ 2:2KBõÔPD2£F‚³ËäL¡¢PÜÙÁ¢'
2014-10-22 02:51:39 cannot parse integer, invalid modifier: '+'
2014-10-22 02:51:39 Not enough fields in CSV input, expected 22, got 1 in input '%6»Ô^È=ùÙý™y¨(wv°(#2Ñ¡³%ÍdlhúH±ýŽ• .ê13'
2014-10-22 02:51:39 Not enough fields in CSV input, expected 22, got 2 in input '2³Éòì'
2014-10-22 02:51:39 cannot parse integer, invalid modifier: '³'
2014-10-22 02:51:39 Not enough fields in CSV input, expected 22, got 2 in input 'zÈ
‘©¨€Ê ÈÌÍLž'
2014-10-22 02:51:39 Not enough fields in CSV input, expected 22, got 1 in input 'PY°‘ zb'ETè[1]ÈÏNÏÿºÿA£Ó@….€þ‹Ô~WO÷?G÷7]'
2014-10-22 02:51:39 Not enough fields in CSV input, expected 22, got 1 in input 'ß;º'
2014-10-22 02:51:39 Not enough fields in CSV input, expected 22, got 1 in input 'ŠÌlº…;;X”q‹'
2014-10-22 02:51:39 Not enough fields in CSV input, expected 22, got 1 in input '>›™5LEptzÌ\(2sÁf&*Jš•'
2014-10-22 02:51:39 Not enough fields in CSV input, expected 22, got 1 in input 'M°xU™•ÍL”‰'
2014-10-22 02:51:39 Not enough fields in CSV input, expected 22, got 5 in input 'M´xUš•MMwóÀ†fM>“‑3-è1ÓL™fzÌ4Óc¦™ 3Íô˜i¦ÇÌÈf&š]dCô$ÈÌ$ÈÌ$ÈÌ$È̤ÈÌ$ÈÌÄf&È$h,ë13
2SQ[1]YaC3øH‚̤K °v…L?:º
chris.trotter.sci created
Load Balanced Output Config
FormerSplunk created
Trying to configure the NXLog forwarder to load balance it's output stream. I tried creating 2 output stanzas out1 and out2 with the appropriate IPs and then routed as Path eventlog => out1, out2.
This seemed to clone the output stream as logs were going to both receivers. What would be the proper way to tell NXLog to use one IP if available, else another IP?
FormerSplunk created
Routing messages based on type and source in a client server configuration
imperimus created
Currently process and transform the windows event/iis logs on the client, however as I have more servers I am wondering about routing everything to a central point using the binary format and then processing them into the relevant tables in to a mysql db. I am struggling with at which stage this filtering and tansforming is done and what the route should look like. Do I use the patern filter in a process stage and then use and if statement in the route based on the patern id?
Clients
im_msvistalog => om_tcp (binary)
w3c extension(im_file) => om_tcp (binary)
Server
im_tcp => ?????????? => ?????? (om_dbi but based on source message type evntlog table |syslog table |iis log table|apache log table|security log table)
Can you point me in the right direction?
imperimus created
xm_perl.dll is not present in extension directory
MagnusBjarnlid created
I am trying to use the perl extension in Windows, but nxlog complains that it cannot find the xm_perl.dll. The complaint is correct, the file is not there. My question is why? Does the windows version not support using perl?
MagnusBjarnlid created
Windows Nxlog creating multiple dyn$ folders
lucaspro created
Hi,
I really would like some assistance in the forum -
Here is the scenario:
Installed nxlog.c.e in Windows 2008 R2
Used the query list -to get security logs
Added that in the module - started the server - everything is fine.
I tested this in my Windows mahcine - to my Redhat server.
Works.
The issue happens when the Windows machine is registered to the domain.
Once the service is started in windows, the syslog server creates host folders which include the Windows client name as well as a bunch of other stuff.
Such as Authlite, Process, 0 00 1 etc etc.. list goes on.
Has anyone encountered this issue? Kindly advise.
Additional information:
This issue only occurs with windows client - registered in the domain.
When the nxlog forwards information to my rsyslog server - (which uses the template dyn$ to create host folders) I get folders with 0 00 Authlite ... etc etc.
Is there anyway to stop this of fix it. Again it only happens with Windows client registered in the domain.
lucaspro created