Ask questions. Get answers. Find technical product solutions from passionate experts in the NXLog community.

Windows 8.1 and corrupted event log ?
I am getting an error message to nxlog.log. 2015-02-17 08:16:23 INFO nxlog-ce-2.8.1248 started 2015-02-17 08:16:35 ERROR Couldn't read next event, corrupted eventlog?; The data is invalid. And after this error no more events or log messages are generated. However I can read event log with event viewer and I can see new events. These are messages in my graylog2: 2015-02-17 08:16:35.000 wintoosa Couldn't read next event, corrupted eventlog? The data is inval 2015-02-17 08:16:23.000 wintoosa nxlog-ce-2.8.1248 started Seems that nxlog is running but it can't handle events after this error. How can I fix this ? My very basic nxlog.conf: define ROOT C:\Program Files (x86)\nxlog Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data LogFile %ROOT%\data\nxlog.log #LogLevel DEBUG <Extension syslog> Module xm_syslog </Extension> <Extension gelf> Module xm_gelf </Extension> <Input internal> Module im_internal </Input> <Input eventlog> Module im_msvistalog For windows 2003 and earlier use the following: Module im_mseventlog </Input> <Output out> Module om_udp Host 10.0.0.103 Port 12900 outputType GELF </Output> <Route 1> Path internal, eventlog => out </Route>

nautilus created
Replies: 3
View post »
last updated
Write to message queues
Botond, I would like to write logs out to a message queue such as kafka or redis or rabbitmq etc. Is there a road map for that? Alternately would a workaround such as writing out in JSON format to a om_tcp destination work? Can I also track the state of a tcp connection and send output to an alternate destination? I could write an external shell that would track the nxlog.log for "connection lost/made" and decision off that but I suspect that would be very expensive. Thanks Ash 

akumar created
Replies: 1
View post »
last updated
Log detail being dropped
Hi everyone, We have an internal application on a windows box which logs in a way which is slightly mangled. All other windows logs come through nxlog but on these messages the entire message field gets dropped. Is there a way to parse using ifs? eg: if $service == "Homegrown app" { } So that I can either change the output format or parse them untouched into a file etc... Does anyone know if there are any characters that don't get escaped properly that might mangle the json output? Sorry for not giving a huge amount of information, I have to keep this as agnostic as I can.

keefbaker created
Replies: 2
View post »
last updated
Log to MS SQL Server
Is there a way to use the Community Edition to log to MS SQL Server? I have tried writing a Perl script to dump the text log files into the database, but have had no luck. I understand that you can use im_odbc and om_odbc in the Enterprise version, but not the Community Edition (CE). Has anyone had any luck doing this in the CE?

nberlanga created
multiline extension not getting the endline regex condition
Hi, I am trying to parse a log4net file into json. Here's my sample log4net: ---------------- 2015-01-27 01:06:18,859 [7] ERROR Web.Cms.Content.Base.Taxonomy.TaxonomyDetectionProvider [(null)] - Get taxonomy Type Failed for Tools 2015-01-27 06:34:31,051 [26] ERROR www.Status404 [(null)] - ErrorId: 20150127_102b01c6-3208-48c5-8c8b-ae4f92cf2b20     UserAgent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.99 Safari/537.36     HostAddress: 192.168.10.2     RequestUrl: /ErrorPages/404.aspx     MachineName: QA01     Raw Url:/undefined/     Referrer: http://qa1.www.something.com/toolset.aspx 2015-01-27 06:34:33,270 [26] DEBUG Web.Caching.Core.CacheManagerBase [(null)] - Custom CacheProvider:Web.Caching.Core.AppFabricCacheManager,Web.Caching.Core Disabled   Now I am using xm_multiline to capture each log entries. ---------------- <Extension multiline>     Module        xm_multiline     HeaderLine    /^\d{4}\-\d{2}\-\d{2} \d{2}\:\d{2}\:\d{2},\d{3}/     EndLine        /\r?\n\r?\n^\d{4}\-\d{2}\-\d{2} \d{2}\:\d{2}\:\d{2},\d{3}/ </Extension> I use a regex to capture the timestamp as the header then I use a regex to capture twice newline then the next timestamp as endline. However it still treat the second and last entry as ONE log entry. Here's the output: ---------------- {     "EventReceivedTime":"2015-01-27 01:06:35",   "SourceModuleName":"log4net",   "SourceModuleType":"im_file",   "time":"2015-01-27 01:06:18,859",   "thread":"7",   "level":"ERROR",   "logger":"Web.Cms.Content.Base.Taxonomy.TaxonomyDetectionProvider",   "ndc":"(null)",   "message":"Get taxonomy Type Failed for Tools"}{     "EventReceivedTime":"2015-01-27 06:34:35",   "SourceModuleName":"log4net",   "SourceModuleType":"im_file",   "time":"2015-01-27 06:34:31,051",   "thread":"26",   "level":"ERROR",   "logger":"www.Status404",   "ndc":"(null)",   "message":"  ErrorId: 20150127_102b01c6-3208-48c5-8c8b-ae4f92cf2b20\r\n  UserAgent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.99  Safari/537.36\r\n  HostAddress: 192.168.10.2\r\n  RequestUrl: /ErrorPages/404.aspx\r\n  MachineName: QA01\r\n   Raw Url:/undefined/\r\n  Referrer: http://qa1.www.something.com/toolset.aspx\r\n\r\n2015-01-27 06:34:33,270 [26] DEBUG Web.Caching.Core.CacheManagerBase [(null)] - Custom CacheProvider:Web.Caching.Core.AppFabricCacheManager,Web.Caching.Core Disabled"} I used this to produce that output: ---------------- Exec        if $raw_event =~ /^(\d{4}\-\d{2}\-\d{2} \d{2}\:\d{2}\:\d{2},\d{3}) \[(\S+)\] (\S+) (\S+) \[(\S+)\] \- (.*)/s \                 { \                     $time = $1; \                     $thread = $2; \                     $level = $3; \                     $logger = $4; \                     $ndc = $5; \                     $message = $6; \                     to_json(); \                 } \                 else \                 { \                     drop(); \                 }     I've also tried to tweak it by using this to avoid the combining the last two entries as one. However I am not able to get the last entry anymore. ---------------- Exec        if $raw_event =~ /^(\d{4}\-\d{2}\-\d{2} \d{2}\:\d{2}\:\d{2},\d{3}) \[(\S+)\] (\S+) (\S+) \[(\S+)\] \- ([\s\S]*?)(\r?\n\r?\n|$)/ \                 { \                     $time = $1; \                     $thread = $2; \                     $level = $3; \                     $logger = $4; \                     $ndc = $5; \                     $message = $6; \                     to_json(); \                 } \                 else \                 { \                     drop(); \                 }​

mvf.right created
Replies: 1
View post »
last updated
Nxlog for Centos7
Will there be a Nxlog for Centos 7. If so are there an expected date. BTW has anybody succeded in compiling the tar file. I simply cannot figure out how to build the RPM files.   Regards Jesper

JesperH created
Replies: 2
View post »
last updated
Windows DNS Log 2008/2012
I am having an issue where I am outputing a DNS log from Windows Server and having NXLOG read it and ship it to a Logstash environment. My issue however is that after a period of time the log file disappears and is not created. I am assuming it is an issue with NXLog having a lock on the file and when windows rotates it, it fails and doesnt get created. The DNS Event View Log shows: The DNS server was unable to open file E:\Log Files\DNS\dns.log for write.  Most likely the file is a zone file that is already open.  Close the zone file and re-initiate zone write. Stopping and restarting the service does not help unless I completely stop NXLOG first. How can I change it so NXLog does not completely lock the log file so that windows can't rotate it? This seems to occur on various versions of windows from 2008 to 2012 R2. Appreciate any help you can provide regarding this issue.

rrspyder created
Replies: 1
View post »
last updated
Change Syslog Facility when reading/forwarding windows events.
Just installed nxlog to begin forwarding events to AlienVault, everything seems to be working so far with reading and forwarding events from the windows log using the im_msvistalog module.  One thing we would like to change to set the Syslog Facility before forwarding it AlienVault.  Have been unable to locate how to do so.

sswager created
Replies: 1
View post »
last updated
nxlog community edition issue on Windows 8/8.1 operating systems.
Hi, i used your nxlog-CE but only my Windows 7 hosts don't exhibit this issue. "Nome dell'applicazione che ha generato l'errore: nxlog.exe, versione: 0.0.0.0, timestamp: 0x53ca79be Nome del modulo che ha generato l'errore: im_msvistalog.dll, versione: 0.0.0.0, timestamp: 0x53ca79bd Codice eccezione: 0xc0000005 Offset errore 0x00001cca ID processo che ha generato l'errore: 0xbfc Ora di avvio dell'applicazione che ha generato l'errore: 0x01d02f68896fae85 Percorso dell'applicazione che ha generato l'errore: C:\Program Files (x86)\nxlog\nxlog.exe Percorso del modulo che ha generato l'errore: C:\Program Files (x86)\nxlog\modules\input\im_msvistalog.dll" nxlog community edition version 2.8.1248 Many thanks.

MarcoFranceschini1971 created
Replies: 1
View post »
last updated
Windows: Auto Start of nxlog service after the install
Hi,   I noticed the NXlog service is not started at the end of the install process. I would the service to start automatically at the end of the install of the MSI. Our goal is: - modify the conf file in the MSI file to have it ready out of the box. - Deploy with GPO on all servers. For this second step, we need to ensure it starts automatically during the install process. The best would be to edit the install file to make a "net start nxlog" at the end. Thanks   PM

pmjanvre created
Replies: 1
View post »
last updated
Inconsistent log sending from windows to graylog2
Hi,   We are using the community edition of nxlog 2.8.1248 on windows 2008 R2 server. We are having forwarding event log and IIS logs to graylog2. This is the conf file is pasted below. If we just have the IIS udp forwarding, it *sometimes* works. We think its not working and then a few hours later we see data coming through, then it might stop again. This IIS issue is also incosistent across machines. Some machines send data, while others never do There are no errors in the nxlog.log file The event log forwarding worked when we used om_udp and GELF format, but when its turned on in combination with IIS (as per conf below) it sends nothing. Graylog2 server is up and running with the respective inputs. I've tested UDP packets can get through to the Graylog2 server I've checked the IIS csv parsing is correct, and as mentioned I don't see errors in the log. Would appreciate ideas on what be going on, and how we might troubleshoot this issue? Thanks, Av   ## This is a sample configuration file. See the nxlog reference manual about the ## configuration options. It should be installed locally and is also available ## online at http://nxlog.org/nxlog-docs/en/nxlog-reference-manual.html ## Please set the ROOT to the folder your nxlog was installed into, ## otherwise it will not start. #define ROOT C:\Program Files\nxlog define ROOT C:\Program Files (x86)\nxlog Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data LogFile %ROOT%\data\nxlog.log LogLevel INFO <Extension syslog> Module xm_syslog </Extension> <Extension gelf> Module xm_gelf </Extension> # Create the parse rule for IIS logs. You can copy these from the header of the IIS log file. <Extension w3c> Module xm_csv Fields $date, $time, $s-sitename, $s-computername, $s-ip, $cs-method, $cs-uri-stem, $cs-uri-query, $s-port, $cs-username, $c-ip, $cs-version, $cs(User-Agent), $cs(Cookie), $cs(Referer), $cs-host, $sc-status, $sc-substatus, $sc-win32-status, $sc-bytes, $cs-bytes $time-taken FieldTypes string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, integer, integer, integer, string, string, string Delimiter ' ' QuoteChar '"' EscapeControl FALSE UndefValue - </Extension> # Enable json extension <Extension json> Module xm_json </Extension> # Convert the IIS logs to JSON and use the original event time # Uncomment IIS_IN section if logging for IIS logging <Input eventlog> # Use 'im_mseventlog' for Windows XP, 2000 and 2003 Module im_msvistalog # Uncomment the following to collect specific event logs only Query <QueryList>\ <Query Id="0">\ <Select Path="Application">*</Select>\ <Select Path="System">*</Select>\ </Query>\ </QueryList> </Input> <Input IIS_Site1> Module im_file File "C:\\inetpub\\logs\\LogFiles\\W3SVC6\\u_ex*.log" SavePos TRUE Exec if $raw_event =~ /^#/ drop(); \ else \ { \ w3c->parse_csv(); \ $EventTime = parsedate($date + " " + $time); \ $SourceName = "IIS"; \ $Message = to_json(); \ } </Input> <Output Event_Out> Module om_udp Host 10.85.105.215 Port 12201 OutputType GELF </Output> <Output IIS_Out> Module om_udp Host 10.85.105.215 Port 514 </Output> <Route IIS> Path IIS_Site1 => IIS_Out </Route> <Route Events> Path eventlog => Event_Out </Route>  

avner created
Replies: 1
View post »
last updated
Unnecessary syslog header was recorded from 0:00 to 9:00 on Jan 1, 2015 of JST(UTC+9:00)
Unnecessary syslog header was recorded from 0:00 to 9:00 on Jan 1, 2015 of JST(UTC+9:00). Example ----- <133>Dec 31 23:55:04 OTSS0101 OTxx01xx: warning <133>Jan  1 00:20:12 10.70.0.32 Jan 01 00:21:51 OTSS0101 OTxx01xx: critical -- <133>Jan  1 08:00:15 10.70.0.32 Jan 01 08:01:52 OTSS0101 OTxx01xx: critical <133>Jan  1 09:01:51 OTSS0101 OTxx01xx: critical ----- I use the following input and output. ----- input Module im_udp Exec parse_syslog_bsd(); -- output Module om_file Exec to_syslog_bsd(); ----- Is this known behavior?  

kaiedak created
Replies: 1
View post »
last updated
Trouble with IPv6
I am trying to setup a listener on IPv6 (nxlog community edition) and am receiving errors on startup.  When using IPv4, it starts without error: Platform CentOS 6.6 x64_86 Configured with both short and long IPv6 notation: 2015-01-04 06:07:27 ERROR apr_sockaddr_info failed for 2604:234::1234:5678:41514;Address family for hostname not supported 2015-01-04 04:58:34 ERROR apr_sockaddr_info failed for 2604:0234:0000:0000:0000:0000:1234:5678:41514;Address family for hostname not supported Configured with DNS (Long IPv6 notation in DNS record, though resolves short) Using a AAAA hostname: Name:    ramscan02-6.test.org Address:  2604:234::1234:5678 2015-01-04 05:53:19 ERROR couldn't bind ssl socket to ramscan02-6.test.org:41514;Cannot assign requested address Any help would be appreciated, Andy    

awood created
Replies: 1
View post »
last updated
Feature request: om_email
Botond, Would you consider creating an output module for email? It would be quite useful for generating alerts. While one can use exec_async to send email, it is rather cludgy.  Cheers   Ash

akumar created
Replies: 1
View post »
last updated
NXlog agent do not send events from Windows Security log
Well... NXlog (last vrsion from this site) installed on windows server 2012R2 Configured to get win-logs: SavePos TRUE     Module      im_msvistalog     Query     <QueryList>                        \               <Query Id="0" Path="Security">            \                 <Select Path="Application">*</Select>    \                     <Select Path="Security">*</Select>    \                     <Select Path="System">*</Select>    \                     <Suppress Path="Security">*[System[(EventID=5156 or EventID=4663 or EventID=5158 or EventID=5440 or EventID=5444)]]</Suppress>    \                     <Suppress Path="System">*[System[(EventID=5156 or EventID=4663 or EventID=5158 or EventID=5440 or EventID=5444)]]</Suppress>    \                   </Query>                \         </QueryList> As a result I see events only from System and Application... Nothing from Security Any Idea why it can happen? Logs are captured by windows - I can see it with eventvwr.msc, but nothing with Nxlog   NXlog have no information, looks like everything is ok: ...INFO nxlog-ce-2.8.1248 started... - no errors, no warnings... nothing else

Barns2 created
Replies: 2
View post »
last updated
Counter tracking assistance
I have multiple windows hosts sending events in binary to a single tcp listener.<Input windows>     Module     im_tcp     Port       9999     Host       0.0.0.0     InputType  Binary I am trying to track the rate of logs from the servers and create email alerts when the rate either drops or crosses a high watermark per hour.  To do that I need to create a stat / variable appending the hostname and hourstamp such as create_stat("Rate-" + '$Hostname' + 'stroftime($EventTimeStamp, something something)') or create_var("Rate-" + '$Hostname' + 'stroftime($EventTimeStamp, something something)')  Next I use the schedule code to detect a low watermark  <Schedule>         Every   3600 sec         Exec    create_stat("rate" + '$Hostname' + 'stroftime($EventTimeStamp, something something)'', "RATE", 10); add_stat("rate" + '$Hostname'' + 'stroftime($EventTimeStamp, something something)', 0);         Exec    log_info("Current Counts " + ":" + get_stat("rate" + '$Hostname'));         Exec    if defined get_stat("rate" + '$Hostname') and get_stat("rate" + '$Hostname') <= 1 \                 { \                     log_warning("No messages received from Host" ); \                     exec_async("/bin/sh", "-c", 'echo "' + $Hostname + \                            '"|/usr/bin/mail -a "Content-Type: text/plain; charset=UTF-8" -s "ALERT" '  \                            + 'analyst@company.com' );                                                      \                 }   </Schedule>   Two problems: How do I insert the variable/statistic name and value in the log message and how do I extract the hour stamp from the Event Time?   Thanks  Ash  PS: I could not get the deployment tool to work. have you had more success with it?

akumar created
Replies: 1
View post »
last updated
im_file configuration
i defined an input for im_file that is being sent as syslog i have multiple files in a folder what i want to do is read each file and on EOF copy to another folder. didn`t find the option to identify EOF

guyl created
Replies: 1
View post »
last updated
multiline bug?
Hi, I'm trying to use the xm_multiline module with nxlog to forward content of a logfile to logstash The log contains different xml elements which are properly indented (opening and closing elements are located at the start of the line) . E.g. <data version="x" xmlns:bla="http://www.example.com/bla"> <val:InfoSet> ... ... ... </val:InfoSet> </data> <message ...> <ns>bla</ns> ... ... </message> Because the elements have different names, I can only use < and </ to find the start and end line. I was hoping a filter like this should be enough to select the correct lines: HeaderLine /^</ EndLine /^<// But somehow nxlog gets confused with the / in the regex pattern. I also tried escaping which dindn't help. More testing showed that it needs at least one letter. I tried to specify all letters via regex but that didn't work: HeaderLine /^<[a-z]/ Only way that seems to work is to specify all letters in the square braket (with the exception of the lettern, which breaks).  HeaderLine /^<[abcdefghijklmopqrstuvwxyz]/ (left out n) Here all my test results. These lines worked: HeaderLine /^<m/ EndLine /^</m/ HeaderLine /^<m/ EndLine /^<\/m/ HeaderLine /^<[abcdefghijklm]/ EndLine /^<\/[abcdefghijklm]/ HeaderLine /^<[abcdefghijklmo]/ EndLine /^<\/[abcdefghijklmo]/ HeaderLine /^<[abcdefghijklmopqrstuvwxyz]/ (left out n) EndLine /^<\/[abcdefghijklmopqrstuvwxyz]/ HeaderLine /^<[abcdefghijklmopqrstuvwxyz]/ (left out n + not escaped EndLine /^</[abcdefghijklmopqrstuvwxyz]/ These lines didn't work: HeaderLine /^</ EndLine /^</m/ HeaderLine /^<[a-z]/ EndLine /^</m/ HeaderLine /^<\w/ EndLine /^</m/ HeaderLine /^<[abcdefghijklmn]/ EndLine /^<\/[abcdefghijklmn]/ HeaderLine /^<[bcdefghijklmn]/ EndLine /^<\/[bcdefghijklmn]/ HeaderLine /^<[abcdefghijklmopqrstuvwxyzn]/ EndLine /^<\/[abcdefghijklmopqrstuvwxyzn]/ HeaderLine /^<[abcdefghijklmnopqrstuvwxyz]/ EndLine /^</[abcdefghijklmnopqrstuvwxyz]/ Right now I still have a problem because many of my bessages start with <n. I think this is a bug in the module. Can you confirm so I can open a ticket? Thanks   Fyi, this is a duplicate of http://stackoverflow.com/questions/27429234/which-headerline-and-endline-for-multiline-xml-with-different-elements  

pgs created
Replies: 1
View post »
last updated
im_dbi & MS SQL
Hello Team, I'm looking to use nxlog to retrieve a table entries from MS SQL , To be clear I do not require any of the database server logging only the info contained within a few tables within a database on the server. I have followed the setup and battled first under windows and then linux versions of nxlog. I am now trying to use libdbi-freetds without success after having out of memory upon using the mysql hooks. Here is my im_dbi without the user and pass for security. <Input DB01-elog>     Module      im_dbi     SavePos     TRUE     SQL         SELECT AllXml FROM db_table     Driver      freetds     Option      host 192.168.148.227     Option      port 49000 (non standard port changed from 1433)     Option      username domain\username     Option      password password     Option      dbname database </Input>   The SQL server im attempting to connect to is MS SQL Server 2014, I also attempted this using mysql however recieved out of memory from the libdbi during execution of the query. The OS machine has 8GB of mem available currently upped from 4GB when the out of memory initially occured. I'm secretly hoping someone has run into this before and has been able to successfully retrieve table data and log it from MS SQL Server 2014. If not im happy to work with the team to resolution on this. Many Thanks, GC

ghostcat created
Replies: 1
View post »
last updated
Customize windows version
Hello! I need to make some changes in sources of modules. Ok - I found necessary code, changed it. But how to compile windows version? No documentation found :(

Barns2 created
Replies: 1
View post »
last updated