Is there a way to use the Community Edition to log to MS SQL Server? I have tried writing a Perl script to dump the text log files into the database, but have had no luck. I understand that you can use im_odbc and om_odbc in the Enterprise version, but not the Community Edition (CE). Has anyone had any luck doing this in the CE?

Hi, I am trying to parse a log4net file into json. Here's my sample log4net: ---------------- 2015-01-27 01:06:18,859 [7] ERROR Web.Cms.Content.Base.Taxonomy.TaxonomyDetectionProvider [(null)] - Get taxonomy Type Failed for Tools 2015-01-27 06:34:31,051 [26] ERROR www.Status404 [(null)] - ErrorId: 20150127_102b01c6-3208-48c5-8c8b-ae4f92cf2b20     UserAgent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.99 Safari/537.36     HostAddress: 192.168.10.2     RequestUrl: /ErrorPages/404.aspx     MachineName: QA01     Raw Url:/undefined/     Referrer: http://qa1.www.something.com/toolset.aspx 2015-01-27 06:34:33,270 [26] DEBUG Web.Caching.Core.CacheManagerBase [(null)] - Custom CacheProvider:Web.Caching.Core.AppFabricCacheManager,Web.Caching.Core Disabled   Now I am using xm_multiline to capture each log entries. ---------------- <Extension multiline>     Module        xm_multiline     HeaderLine    /^\d{4}\-\d{2}\-\d{2} \d{2}\:\d{2}\:\d{2},\d{3}/     EndLine        /\r?\n\r?\n^\d{4}\-\d{2}\-\d{2} \d{2}\:\d{2}\:\d{2},\d{3}/ </Extension> I use a regex to capture the timestamp as the header then I use a regex to capture twice newline then the next timestamp as endline. However it still treat the second and last entry as ONE log entry. Here's the output: ---------------- {     "EventReceivedTime":"2015-01-27 01:06:35",   "SourceModuleName":"log4net",   "SourceModuleType":"im_file",   "time":"2015-01-27 01:06:18,859",   "thread":"7",   "level":"ERROR",   "logger":"Web.Cms.Content.Base.Taxonomy.TaxonomyDetectionProvider",   "ndc":"(null)",   "message":"Get taxonomy Type Failed for Tools"}{     "EventReceivedTime":"2015-01-27 06:34:35",   "SourceModuleName":"log4net",   "SourceModuleType":"im_file",   "time":"2015-01-27 06:34:31,051",   "thread":"26",   "level":"ERROR",   "logger":"www.Status404",   "ndc":"(null)",   "message":"  ErrorId: 20150127_102b01c6-3208-48c5-8c8b-ae4f92cf2b20\r\n  UserAgent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.99  Safari/537.36\r\n  HostAddress: 192.168.10.2\r\n  RequestUrl: /ErrorPages/404.aspx\r\n  MachineName: QA01\r\n   Raw Url:/undefined/\r\n  Referrer: http://qa1.www.something.com/toolset.aspx\r\n\r\n2015-01-27 06:34:33,270 [26] DEBUG Web.Caching.Core.CacheManagerBase [(null)] - Custom CacheProvider:Web.Caching.Core.AppFabricCacheManager,Web.Caching.Core Disabled"} I used this to produce that output: ---------------- Exec        if $raw_event =~ /^(\d{4}\-\d{2}\-\d{2} \d{2}\:\d{2}\:\d{2},\d{3}) \[(\S+)\] (\S+) (\S+) \[(\S+)\] \- (.*)/s \                 { \                     $time = $1; \                     $thread = $2; \                     $level = $3; \                     $logger = $4; \                     $ndc = $5; \                     $message = $6; \                     to_json(); \                 } \                 else \                 { \                     drop(); \                 }     I've also tried to tweak it by using this to avoid the combining the last two entries as one. However I am not able to get the last entry anymore. ---------------- Exec        if $raw_event =~ /^(\d{4}\-\d{2}\-\d{2} \d{2}\:\d{2}\:\d{2},\d{3}) \[(\S+)\] (\S+) (\S+) \[(\S+)\] \- ([\s\S]*?)(\r?\n\r?\n|$)/ \                 { \                     $time = $1; \                     $thread = $2; \                     $level = $3; \                     $logger = $4; \                     $ndc = $5; \                     $message = $6; \                     to_json(); \                 } \                 else \                 { \                     drop(); \                 }​

Will there be a Nxlog for Centos 7. If so are there an expected date. BTW has anybody succeded in compiling the tar file. I simply cannot figure out how to build the RPM files.   Regards Jesper

I am having an issue where I am outputing a DNS log from Windows Server and having NXLOG read it and ship it to a Logstash environment. My issue however is that after a period of time the log file disappears and is not created. I am assuming it is an issue with NXLog having a lock on the file and when windows rotates it, it fails and doesnt get created. The DNS Event View Log shows: The DNS server was unable to open file E:\Log Files\DNS\dns.log for write.  Most likely the file is a zone file that is already open.  Close the zone file and re-initiate zone write. Stopping and restarting the service does not help unless I completely stop NXLOG first. How can I change it so NXLog does not completely lock the log file so that windows can't rotate it? This seems to occur on various versions of windows from 2008 to 2012 R2. Appreciate any help you can provide regarding this issue.

Just installed nxlog to begin forwarding events to AlienVault, everything seems to be working so far with reading and forwarding events from the windows log using the im_msvistalog module.  One thing we would like to change to set the Syslog Facility before forwarding it AlienVault.  Have been unable to locate how to do so.

Hi, i used your nxlog-CE but only my Windows 7 hosts don't exhibit this issue. "Nome dell'applicazione che ha generato l'errore: nxlog.exe, versione: 0.0.0.0, timestamp: 0x53ca79be Nome del modulo che ha generato l'errore: im_msvistalog.dll, versione: 0.0.0.0, timestamp: 0x53ca79bd Codice eccezione: 0xc0000005 Offset errore 0x00001cca ID processo che ha generato l'errore: 0xbfc Ora di avvio dell'applicazione che ha generato l'errore: 0x01d02f68896fae85 Percorso dell'applicazione che ha generato l'errore: C:\Program Files (x86)\nxlog\nxlog.exe Percorso del modulo che ha generato l'errore: C:\Program Files (x86)\nxlog\modules\input\im_msvistalog.dll" nxlog community edition version 2.8.1248 Many thanks.

Hi,   I noticed the NXlog service is not started at the end of the install process. I would the service to start automatically at the end of the install of the MSI. Our goal is: - modify the conf file in the MSI file to have it ready out of the box. - Deploy with GPO on all servers. For this second step, we need to ensure it starts automatically during the install process. The best would be to edit the install file to make a "net start nxlog" at the end. Thanks   PM

Hi,   We are using the community edition of nxlog 2.8.1248 on windows 2008 R2 server. We are having forwarding event log and IIS logs to graylog2. This is the conf file is pasted below. If we just have the IIS udp forwarding, it *sometimes* works. We think its not working and then a few hours later we see data coming through, then it might stop again. This IIS issue is also incosistent across machines. Some machines send data, while others never do There are no errors in the nxlog.log file The event log forwarding worked when we used om_udp and GELF format, but when its turned on in combination with IIS (as per conf below) it sends nothing. Graylog2 server is up and running with the respective inputs. I've tested UDP packets can get through to the Graylog2 server I've checked the IIS csv parsing is correct, and as mentioned I don't see errors in the log. Would appreciate ideas on what be going on, and how we might troubleshoot this issue? Thanks, Av   ## This is a sample configuration file. See the nxlog reference manual about the ## configuration options. It should be installed locally and is also available ## online at http://nxlog.org/nxlog-docs/en/nxlog-reference-manual.html ## Please set the ROOT to the folder your nxlog was installed into, ## otherwise it will not start. #define ROOT C:\Program Files\nxlog define ROOT C:\Program Files (x86)\nxlog Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data LogFile %ROOT%\data\nxlog.log LogLevel INFO <Extension syslog> Module xm_syslog </Extension> <Extension gelf> Module xm_gelf </Extension> # Create the parse rule for IIS logs. You can copy these from the header of the IIS log file. <Extension w3c> Module xm_csv Fields $date, $time, $s-sitename, $s-computername, $s-ip, $cs-method, $cs-uri-stem, $cs-uri-query, $s-port, $cs-username, $c-ip, $cs-version, $cs(User-Agent), $cs(Cookie), $cs(Referer), $cs-host, $sc-status, $sc-substatus, $sc-win32-status, $sc-bytes, $cs-bytes $time-taken FieldTypes string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, integer, integer, integer, string, string, string Delimiter ' ' QuoteChar '"' EscapeControl FALSE UndefValue - </Extension> # Enable json extension <Extension json> Module xm_json </Extension> # Convert the IIS logs to JSON and use the original event time # Uncomment IIS_IN section if logging for IIS logging <Input eventlog> # Use 'im_mseventlog' for Windows XP, 2000 and 2003 Module im_msvistalog # Uncomment the following to collect specific event logs only Query <QueryList>\ <Query Id="0">\ <Select Path="Application">*</Select>\ <Select Path="System">*</Select>\ </Query>\ </QueryList> </Input> <Input IIS_Site1> Module im_file File "C:\\inetpub\\logs\\LogFiles\\W3SVC6\\u_ex*.log" SavePos TRUE Exec if $raw_event =~ /^#/ drop(); \ else \ { \ w3c->parse_csv(); \ $EventTime = parsedate($date + " " + $time); \ $SourceName = "IIS"; \ $Message = to_json(); \ } </Input> <Output Event_Out> Module om_udp Host 10.85.105.215 Port 12201 OutputType GELF </Output> <Output IIS_Out> Module om_udp Host 10.85.105.215 Port 514 </Output> <Route IIS> Path IIS_Site1 => IIS_Out </Route> <Route Events> Path eventlog => Event_Out </Route>  

Unnecessary syslog header was recorded from 0:00 to 9:00 on Jan 1, 2015 of JST(UTC+9:00). Example ----- <133>Dec 31 23:55:04 OTSS0101 OTxx01xx: warning <133>Jan  1 00:20:12 10.70.0.32 Jan 01 00:21:51 OTSS0101 OTxx01xx: critical -- <133>Jan  1 08:00:15 10.70.0.32 Jan 01 08:01:52 OTSS0101 OTxx01xx: critical <133>Jan  1 09:01:51 OTSS0101 OTxx01xx: critical ----- I use the following input and output. ----- input Module im_udp Exec parse_syslog_bsd(); -- output Module om_file Exec to_syslog_bsd(); ----- Is this known behavior?  

I am trying to setup a listener on IPv6 (nxlog community edition) and am receiving errors on startup.  When using IPv4, it starts without error: Platform CentOS 6.6 x64_86 Configured with both short and long IPv6 notation: 2015-01-04 06:07:27 ERROR apr_sockaddr_info failed for 2604:234::1234:5678:41514;Address family for hostname not supported 2015-01-04 04:58:34 ERROR apr_sockaddr_info failed for 2604:0234:0000:0000:0000:0000:1234:5678:41514;Address family for hostname not supported Configured with DNS (Long IPv6 notation in DNS record, though resolves short) Using a AAAA hostname: Name:    ramscan02-6.test.org Address:  2604:234::1234:5678 2015-01-04 05:53:19 ERROR couldn't bind ssl socket to ramscan02-6.test.org:41514;Cannot assign requested address Any help would be appreciated, Andy    

Botond, Would you consider creating an output module for email? It would be quite useful for generating alerts. While one can use exec_async to send email, it is rather cludgy.  Cheers   Ash

Well... NXlog (last vrsion from this site) installed on windows server 2012R2 Configured to get win-logs: SavePos TRUE     Module      im_msvistalog     Query     <QueryList>                        \               <Query Id="0" Path="Security">            \                 <Select Path="Application">*</Select>    \                     <Select Path="Security">*</Select>    \                     <Select Path="System">*</Select>    \                     <Suppress Path="Security">*[System[(EventID=5156 or EventID=4663 or EventID=5158 or EventID=5440 or EventID=5444)]]</Suppress>    \                     <Suppress Path="System">*[System[(EventID=5156 or EventID=4663 or EventID=5158 or EventID=5440 or EventID=5444)]]</Suppress>    \                   </Query>                \         </QueryList> As a result I see events only from System and Application... Nothing from Security Any Idea why it can happen? Logs are captured by windows - I can see it with eventvwr.msc, but nothing with Nxlog   NXlog have no information, looks like everything is ok: ...INFO nxlog-ce-2.8.1248 started... - no errors, no warnings... nothing else

I have multiple windows hosts sending events in binary to a single tcp listener.<Input windows>     Module     im_tcp     Port       9999     Host       0.0.0.0     InputType  Binary I am trying to track the rate of logs from the servers and create email alerts when the rate either drops or crosses a high watermark per hour.  To do that I need to create a stat / variable appending the hostname and hourstamp such as create_stat("Rate-" + '$Hostname' + 'stroftime($EventTimeStamp, something something)') or create_var("Rate-" + '$Hostname' + 'stroftime($EventTimeStamp, something something)')  Next I use the schedule code to detect a low watermark  <Schedule>         Every   3600 sec         Exec    create_stat("rate" + '$Hostname' + 'stroftime($EventTimeStamp, something something)'', "RATE", 10); add_stat("rate" + '$Hostname'' + 'stroftime($EventTimeStamp, something something)', 0);         Exec    log_info("Current Counts " + ":" + get_stat("rate" + '$Hostname'));         Exec    if defined get_stat("rate" + '$Hostname') and get_stat("rate" + '$Hostname') <= 1 \                 { \                     log_warning("No messages received from Host" ); \                     exec_async("/bin/sh", "-c", 'echo "' + $Hostname + \                            '"|/usr/bin/mail -a "Content-Type: text/plain; charset=UTF-8" -s "ALERT" '  \                            + 'analyst@company.com' );                                                      \                 }   </Schedule>   Two problems: How do I insert the variable/statistic name and value in the log message and how do I extract the hour stamp from the Event Time?   Thanks  Ash  PS: I could not get the deployment tool to work. have you had more success with it?

i defined an input for im_file that is being sent as syslog i have multiple files in a folder what i want to do is read each file and on EOF copy to another folder. didn`t find the option to identify EOF

Hi, I'm trying to use the xm_multiline module with nxlog to forward content of a logfile to logstash The log contains different xml elements which are properly indented (opening and closing elements are located at the start of the line) . E.g. <data version="x" xmlns:bla="http://www.example.com/bla"> <val:InfoSet> ... ... ... </val:InfoSet> </data> <message ...> <ns>bla</ns> ... ... </message> Because the elements have different names, I can only use < and </ to find the start and end line. I was hoping a filter like this should be enough to select the correct lines: HeaderLine /^</ EndLine /^<// But somehow nxlog gets confused with the / in the regex pattern. I also tried escaping which dindn't help. More testing showed that it needs at least one letter. I tried to specify all letters via regex but that didn't work: HeaderLine /^<[a-z]/ Only way that seems to work is to specify all letters in the square braket (with the exception of the lettern, which breaks).  HeaderLine /^<[abcdefghijklmopqrstuvwxyz]/ (left out n) Here all my test results. These lines worked: HeaderLine /^<m/ EndLine /^</m/ HeaderLine /^<m/ EndLine /^<\/m/ HeaderLine /^<[abcdefghijklm]/ EndLine /^<\/[abcdefghijklm]/ HeaderLine /^<[abcdefghijklmo]/ EndLine /^<\/[abcdefghijklmo]/ HeaderLine /^<[abcdefghijklmopqrstuvwxyz]/ (left out n) EndLine /^<\/[abcdefghijklmopqrstuvwxyz]/ HeaderLine /^<[abcdefghijklmopqrstuvwxyz]/ (left out n + not escaped EndLine /^</[abcdefghijklmopqrstuvwxyz]/ These lines didn't work: HeaderLine /^</ EndLine /^</m/ HeaderLine /^<[a-z]/ EndLine /^</m/ HeaderLine /^<\w/ EndLine /^</m/ HeaderLine /^<[abcdefghijklmn]/ EndLine /^<\/[abcdefghijklmn]/ HeaderLine /^<[bcdefghijklmn]/ EndLine /^<\/[bcdefghijklmn]/ HeaderLine /^<[abcdefghijklmopqrstuvwxyzn]/ EndLine /^<\/[abcdefghijklmopqrstuvwxyzn]/ HeaderLine /^<[abcdefghijklmnopqrstuvwxyz]/ EndLine /^</[abcdefghijklmnopqrstuvwxyz]/ Right now I still have a problem because many of my bessages start with <n. I think this is a bug in the module. Can you confirm so I can open a ticket? Thanks   Fyi, this is a duplicate of http://stackoverflow.com/questions/27429234/which-headerline-and-endline-for-multiline-xml-with-different-elements  

Hello Team, I'm looking to use nxlog to retrieve a table entries from MS SQL , To be clear I do not require any of the database server logging only the info contained within a few tables within a database on the server. I have followed the setup and battled first under windows and then linux versions of nxlog. I am now trying to use libdbi-freetds without success after having out of memory upon using the mysql hooks. Here is my im_dbi without the user and pass for security. <Input DB01-elog>     Module      im_dbi     SavePos     TRUE     SQL         SELECT AllXml FROM db_table     Driver      freetds     Option      host 192.168.148.227     Option      port 49000 (non standard port changed from 1433)     Option      username domain\username     Option      password password     Option      dbname database </Input>   The SQL server im attempting to connect to is MS SQL Server 2014, I also attempted this using mysql however recieved out of memory from the libdbi during execution of the query. The OS machine has 8GB of mem available currently upped from 4GB when the out of memory initially occured. I'm secretly hoping someone has run into this before and has been able to successfully retrieve table data and log it from MS SQL Server 2014. If not im happy to work with the team to resolution on this. Many Thanks, GC

Hello! I need to make some changes in sources of modules. Ok - I found necessary code, changed it. But how to compile windows version? No documentation found :(

Hi Please help to exclude/ignore logs for a particular application   Thanks Virender

We are using nxlog to collect eventlog information. Some entries can be large, in fact some message are split over several entries as a workaround for the maximum eventlog entry size. However, these large entries seem to hang nxlog so that it stops processing new entries. Typical error messages are: --------------------------------------- 2014-10-27 17:10:32 ERROR EvtNext failed with error 1734: The array bounds are invalid.   2014-10-27 17:10:33 ERROR EvtUpdateBookmark failed: The handle is invalid. ---------------------------------------- Why is this? Is there any workaround?  

We are using NXlog to send Windows EventLogs and IIS logs to Logstash (ELK).  It works very well indeed with only one problem - at 2am (we suspect IIS log rotation, or maybe weird Netscaler packet) every day the NXlog service crashes on ALL servers that are sending IIS logs.  We have some other servers only sending EventLogs - no crashing occurs.  I have set the Windows NXlog service to restart on a crash, so service is only interrupted for a minute, but here's the weird part - NXlog re-sends a (random?) huge chunk of messages.  I say 'random', because check out this list of time received vs. event time: (these are all from the same server) 2014-10-28T06:00:54.074Z & 2014-10-25 04:18:08 2014-10-28T06:00:54.074Z & 2014-10-25 03:58:08 2014-10-28T06:00:54.072Z & 2014-10-25 03:20:02 2014-10-28T06:00:54.058Z & 2014-10-25 03:10:33 2014-10-28T06:00:54.058Z & 2014-10-25 03:18:10 The weird time received jumping forward/backward in time continues (perhaps it's an ELK stack processing oddity)...keeping in mind we're talking hundreds of entries over a 1-2 minute period.  On our UAT servers I observed that this large chunk of logs contains logs from the last week or more - but it's not consistent enough to say 'it's re-sending everything from the last week'. Anyways - what causes the crash is bad input, like weird bad characters...I am sure the following copy/paste won't do it justice.   If anyone has ideas, I'm open!!  Happy to either fix the weird log dumps, or fix the bad input data.  We're continuing to investigate, but I thought it important (and not Google-friendly) enough to post here.  A number of Google hits for questions like mine with no answers (or 'no bad input filtering, is a bug, sorry'). (disregard the timestamp) 2014-10-22 02:51:39 Not enough fields in CSV input, expected 22, got 1 in input 'PK ' 2014-10-22 02:51:39 Not enough fields in CSV input, expected 22, got 1 in input 'U@¦Ä-*¸H¶ ÜH—ÙùPÜÙ¡žŒÛŠ”¹‘ðè²Þè' 2014-10-22 02:51:39 Not enough fields in CSV input, expected 22, got 2 in input '¹'O‚¨ˆÌ(ˆÌÈF&È”™™ØÌDEéA3)B3éA3±¡é#eEff63AO‚È,ŠÈ,‚È,zÈ,ŠÈ,ld¢Ùé1³g3³&ŸI™½ 2:2KBõÔPD2£F‚³ËäL¡¢PÜÙÁ¢' 2014-10-22 02:51:39 cannot parse integer, invalid modifier: '+' 2014-10-22 02:51:39 Not enough fields in CSV input, expected 22, got 1 in input '%6»Ô^È=ùÙý™y¨(wv°(#2Ñ¡³%ÍdlhúH±ýŽ• .ê13' 2014-10-22 02:51:39 Not enough fields in CSV input, expected 22, got 2 in input '2³Éòì' 2014-10-22 02:51:39 cannot parse integer, invalid modifier: '³' 2014-10-22 02:51:39 Not enough fields in CSV input, expected 22, got 2 in input 'zÈ ‘©¨€Ê È́ÍLž' 2014-10-22 02:51:39 Not enough fields in CSV input, expected 22, got 1 in input 'PY°‘ zb'ETè[1]ÈÏNÏÿºÿA£Ó@….€þ‹Ô~WO÷?G÷7]' 2014-10-22 02:51:39 Not enough fields in CSV input, expected 22, got 1 in input 'ß;º' 2014-10-22 02:51:39 Not enough fields in CSV input, expected 22, got 1 in input 'ŠÌlº…;;X”q‹' 2014-10-22 02:51:39 Not enough fields in CSV input, expected 22, got 1 in input '>›™5LEptzÌ\(2sÁf&*Jš•' 2014-10-22 02:51:39 Not enough fields in CSV input, expected 22, got 1 in input 'M°xU™•ÍL”‰' 2014-10-22 02:51:39 Not enough fields in CSV input, expected 22, got 5 in input 'M´xUš•MMwóÀ†fM>“‑3-è1ÓL™fzÌ4Óc¦™ 3Íô˜i¦ÇÌÈf&š]dCô$ÈÌ$ÈÌ$ÈÌ$È̤ÈÌ$ÈÌÄf&È$h,ë13 2SQ[1]YaC3øH‚̤K °v…L?:º