Hello, Having an issue with using om_ssl from a Windows NXlog client to a Linux ELK server. The issue lies with the encryption part, the connection is established on the associated destination SSL port I set but no SSL traffic can be see/captured. For information here is the logstash config and the NXlog config, I believe the issue lies with the certs. input {   tcp {     port => 5000     type => syslog   }   udp {     port => 5000     type => syslog   }   tcp {     port => 5001     type => syslog     ssl_cacert => "/etc/pki/tls/certs/rootCA.pem"     ssl_cert => "/etc/pki/tls/certs/logstash.crt"     ssl_key => "/etc/pki/tls/private/logstash.key"     ssl_enable => true nxlog.conf: <Output syslogout> #This is for TCP non-SSL traffic   Module om_tcp   Host 192.168.0.20   Port 5000 </Output> <Output omsslout> #This is for SSL traffic only, omit all "#"   Module          om_ssl   Host            192.168.0.20   Port            5001   CAFile          %CERTDIR%\rootCA.pem   OutputType LineBased   AllowUntrusted FALSE </Output> <Route 1>   Path eventlog => eventlog_transformer => omsslout </Route> Only included relative SSL parts as everything else works over TCP and UDP. Many thanks

Hello, is there any chance that nxlog supports Output om_ssl with OutputType GELF? As descriped here https://www.mail-archive.com/nxlog-ce-users@lists.sourceforge.net/msg00397.html currently NXLog only supports GELF via om_udp . It would be great, because Graylog now (19. February 2015) supports TLS for all standard TCP and AMQP inputs.

I am getting an error message to nxlog.log. 2015-02-17 08:16:23 INFO nxlog-ce-2.8.1248 started 2015-02-17 08:16:35 ERROR Couldn't read next event, corrupted eventlog?; The data is invalid. And after this error no more events or log messages are generated. However I can read event log with event viewer and I can see new events. These are messages in my graylog2: 2015-02-17 08:16:35.000 wintoosa Couldn't read next event, corrupted eventlog? The data is inval 2015-02-17 08:16:23.000 wintoosa nxlog-ce-2.8.1248 started Seems that nxlog is running but it can't handle events after this error. How can I fix this ? My very basic nxlog.conf: define ROOT C:\Program Files (x86)\nxlog Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data LogFile %ROOT%\data\nxlog.log #LogLevel DEBUG <Extension syslog> Module xm_syslog </Extension> <Extension gelf> Module xm_gelf </Extension> <Input internal> Module im_internal </Input> <Input eventlog> Module im_msvistalog For windows 2003 and earlier use the following: Module im_mseventlog </Input> <Output out> Module om_udp Host 10.0.0.103 Port 12900 outputType GELF </Output> <Route 1> Path internal, eventlog => out </Route>

Botond, I would like to write logs out to a message queue such as kafka or redis or rabbitmq etc. Is there a road map for that? Alternately would a workaround such as writing out in JSON format to a om_tcp destination work? Can I also track the state of a tcp connection and send output to an alternate destination? I could write an external shell that would track the nxlog.log for "connection lost/made" and decision off that but I suspect that would be very expensive. Thanks Ash 

Hi everyone, We have an internal application on a windows box which logs in a way which is slightly mangled. All other windows logs come through nxlog but on these messages the entire message field gets dropped. Is there a way to parse using ifs? eg: if $service == "Homegrown app" { } So that I can either change the output format or parse them untouched into a file etc... Does anyone know if there are any characters that don't get escaped properly that might mangle the json output? Sorry for not giving a huge amount of information, I have to keep this as agnostic as I can.

Is there a way to use the Community Edition to log to MS SQL Server? I have tried writing a Perl script to dump the text log files into the database, but have had no luck. I understand that you can use im_odbc and om_odbc in the Enterprise version, but not the Community Edition (CE). Has anyone had any luck doing this in the CE?

Hi, I am trying to parse a log4net file into json. Here's my sample log4net: ---------------- 2015-01-27 01:06:18,859 [7] ERROR Web.Cms.Content.Base.Taxonomy.TaxonomyDetectionProvider [(null)] - Get taxonomy Type Failed for Tools 2015-01-27 06:34:31,051 [26] ERROR www.Status404 [(null)] - ErrorId: 20150127_102b01c6-3208-48c5-8c8b-ae4f92cf2b20     UserAgent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.99 Safari/537.36     HostAddress: 192.168.10.2     RequestUrl: /ErrorPages/404.aspx     MachineName: QA01     Raw Url:/undefined/     Referrer: http://qa1.www.something.com/toolset.aspx 2015-01-27 06:34:33,270 [26] DEBUG Web.Caching.Core.CacheManagerBase [(null)] - Custom CacheProvider:Web.Caching.Core.AppFabricCacheManager,Web.Caching.Core Disabled   Now I am using xm_multiline to capture each log entries. ---------------- <Extension multiline>     Module        xm_multiline     HeaderLine    /^\d{4}\-\d{2}\-\d{2} \d{2}\:\d{2}\:\d{2},\d{3}/     EndLine        /\r?\n\r?\n^\d{4}\-\d{2}\-\d{2} \d{2}\:\d{2}\:\d{2},\d{3}/ </Extension> I use a regex to capture the timestamp as the header then I use a regex to capture twice newline then the next timestamp as endline. However it still treat the second and last entry as ONE log entry. Here's the output: ---------------- {     "EventReceivedTime":"2015-01-27 01:06:35",   "SourceModuleName":"log4net",   "SourceModuleType":"im_file",   "time":"2015-01-27 01:06:18,859",   "thread":"7",   "level":"ERROR",   "logger":"Web.Cms.Content.Base.Taxonomy.TaxonomyDetectionProvider",   "ndc":"(null)",   "message":"Get taxonomy Type Failed for Tools"}{     "EventReceivedTime":"2015-01-27 06:34:35",   "SourceModuleName":"log4net",   "SourceModuleType":"im_file",   "time":"2015-01-27 06:34:31,051",   "thread":"26",   "level":"ERROR",   "logger":"www.Status404",   "ndc":"(null)",   "message":"  ErrorId: 20150127_102b01c6-3208-48c5-8c8b-ae4f92cf2b20\r\n  UserAgent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.99  Safari/537.36\r\n  HostAddress: 192.168.10.2\r\n  RequestUrl: /ErrorPages/404.aspx\r\n  MachineName: QA01\r\n   Raw Url:/undefined/\r\n  Referrer: http://qa1.www.something.com/toolset.aspx\r\n\r\n2015-01-27 06:34:33,270 [26] DEBUG Web.Caching.Core.CacheManagerBase [(null)] - Custom CacheProvider:Web.Caching.Core.AppFabricCacheManager,Web.Caching.Core Disabled"} I used this to produce that output: ---------------- Exec        if $raw_event =~ /^(\d{4}\-\d{2}\-\d{2} \d{2}\:\d{2}\:\d{2},\d{3}) \[(\S+)\] (\S+) (\S+) \[(\S+)\] \- (.*)/s \                 { \                     $time = $1; \                     $thread = $2; \                     $level = $3; \                     $logger = $4; \                     $ndc = $5; \                     $message = $6; \                     to_json(); \                 } \                 else \                 { \                     drop(); \                 }     I've also tried to tweak it by using this to avoid the combining the last two entries as one. However I am not able to get the last entry anymore. ---------------- Exec        if $raw_event =~ /^(\d{4}\-\d{2}\-\d{2} \d{2}\:\d{2}\:\d{2},\d{3}) \[(\S+)\] (\S+) (\S+) \[(\S+)\] \- ([\s\S]*?)(\r?\n\r?\n|$)/ \                 { \                     $time = $1; \                     $thread = $2; \                     $level = $3; \                     $logger = $4; \                     $ndc = $5; \                     $message = $6; \                     to_json(); \                 } \                 else \                 { \                     drop(); \                 }​

Will there be a Nxlog for Centos 7. If so are there an expected date. BTW has anybody succeded in compiling the tar file. I simply cannot figure out how to build the RPM files.   Regards Jesper

I am having an issue where I am outputing a DNS log from Windows Server and having NXLOG read it and ship it to a Logstash environment. My issue however is that after a period of time the log file disappears and is not created. I am assuming it is an issue with NXLog having a lock on the file and when windows rotates it, it fails and doesnt get created. The DNS Event View Log shows: The DNS server was unable to open file E:\Log Files\DNS\dns.log for write.  Most likely the file is a zone file that is already open.  Close the zone file and re-initiate zone write. Stopping and restarting the service does not help unless I completely stop NXLOG first. How can I change it so NXLog does not completely lock the log file so that windows can't rotate it? This seems to occur on various versions of windows from 2008 to 2012 R2. Appreciate any help you can provide regarding this issue.

Just installed nxlog to begin forwarding events to AlienVault, everything seems to be working so far with reading and forwarding events from the windows log using the im_msvistalog module.  One thing we would like to change to set the Syslog Facility before forwarding it AlienVault.  Have been unable to locate how to do so.

Hi, i used your nxlog-CE but only my Windows 7 hosts don't exhibit this issue. "Nome dell'applicazione che ha generato l'errore: nxlog.exe, versione: 0.0.0.0, timestamp: 0x53ca79be Nome del modulo che ha generato l'errore: im_msvistalog.dll, versione: 0.0.0.0, timestamp: 0x53ca79bd Codice eccezione: 0xc0000005 Offset errore 0x00001cca ID processo che ha generato l'errore: 0xbfc Ora di avvio dell'applicazione che ha generato l'errore: 0x01d02f68896fae85 Percorso dell'applicazione che ha generato l'errore: C:\Program Files (x86)\nxlog\nxlog.exe Percorso del modulo che ha generato l'errore: C:\Program Files (x86)\nxlog\modules\input\im_msvistalog.dll" nxlog community edition version 2.8.1248 Many thanks.

Hi,   I noticed the NXlog service is not started at the end of the install process. I would the service to start automatically at the end of the install of the MSI. Our goal is: - modify the conf file in the MSI file to have it ready out of the box. - Deploy with GPO on all servers. For this second step, we need to ensure it starts automatically during the install process. The best would be to edit the install file to make a "net start nxlog" at the end. Thanks   PM

Hi,   We are using the community edition of nxlog 2.8.1248 on windows 2008 R2 server. We are having forwarding event log and IIS logs to graylog2. This is the conf file is pasted below. If we just have the IIS udp forwarding, it *sometimes* works. We think its not working and then a few hours later we see data coming through, then it might stop again. This IIS issue is also incosistent across machines. Some machines send data, while others never do There are no errors in the nxlog.log file The event log forwarding worked when we used om_udp and GELF format, but when its turned on in combination with IIS (as per conf below) it sends nothing. Graylog2 server is up and running with the respective inputs. I've tested UDP packets can get through to the Graylog2 server I've checked the IIS csv parsing is correct, and as mentioned I don't see errors in the log. Would appreciate ideas on what be going on, and how we might troubleshoot this issue? Thanks, Av   ## This is a sample configuration file. See the nxlog reference manual about the ## configuration options. It should be installed locally and is also available ## online at http://nxlog.org/nxlog-docs/en/nxlog-reference-manual.html ## Please set the ROOT to the folder your nxlog was installed into, ## otherwise it will not start. #define ROOT C:\Program Files\nxlog define ROOT C:\Program Files (x86)\nxlog Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data LogFile %ROOT%\data\nxlog.log LogLevel INFO <Extension syslog> Module xm_syslog </Extension> <Extension gelf> Module xm_gelf </Extension> # Create the parse rule for IIS logs. You can copy these from the header of the IIS log file. <Extension w3c> Module xm_csv Fields $date, $time, $s-sitename, $s-computername, $s-ip, $cs-method, $cs-uri-stem, $cs-uri-query, $s-port, $cs-username, $c-ip, $cs-version, $cs(User-Agent), $cs(Cookie), $cs(Referer), $cs-host, $sc-status, $sc-substatus, $sc-win32-status, $sc-bytes, $cs-bytes $time-taken FieldTypes string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, integer, integer, integer, string, string, string Delimiter ' ' QuoteChar '"' EscapeControl FALSE UndefValue - </Extension> # Enable json extension <Extension json> Module xm_json </Extension> # Convert the IIS logs to JSON and use the original event time # Uncomment IIS_IN section if logging for IIS logging <Input eventlog> # Use 'im_mseventlog' for Windows XP, 2000 and 2003 Module im_msvistalog # Uncomment the following to collect specific event logs only Query <QueryList>\ <Query Id="0">\ <Select Path="Application">*</Select>\ <Select Path="System">*</Select>\ </Query>\ </QueryList> </Input> <Input IIS_Site1> Module im_file File "C:\\inetpub\\logs\\LogFiles\\W3SVC6\\u_ex*.log" SavePos TRUE Exec if $raw_event =~ /^#/ drop(); \ else \ { \ w3c->parse_csv(); \ $EventTime = parsedate($date + " " + $time); \ $SourceName = "IIS"; \ $Message = to_json(); \ } </Input> <Output Event_Out> Module om_udp Host 10.85.105.215 Port 12201 OutputType GELF </Output> <Output IIS_Out> Module om_udp Host 10.85.105.215 Port 514 </Output> <Route IIS> Path IIS_Site1 => IIS_Out </Route> <Route Events> Path eventlog => Event_Out </Route>  

Unnecessary syslog header was recorded from 0:00 to 9:00 on Jan 1, 2015 of JST(UTC+9:00). Example ----- <133>Dec 31 23:55:04 OTSS0101 OTxx01xx: warning <133>Jan  1 00:20:12 10.70.0.32 Jan 01 00:21:51 OTSS0101 OTxx01xx: critical -- <133>Jan  1 08:00:15 10.70.0.32 Jan 01 08:01:52 OTSS0101 OTxx01xx: critical <133>Jan  1 09:01:51 OTSS0101 OTxx01xx: critical ----- I use the following input and output. ----- input Module im_udp Exec parse_syslog_bsd(); -- output Module om_file Exec to_syslog_bsd(); ----- Is this known behavior?  

I am trying to setup a listener on IPv6 (nxlog community edition) and am receiving errors on startup.  When using IPv4, it starts without error: Platform CentOS 6.6 x64_86 Configured with both short and long IPv6 notation: 2015-01-04 06:07:27 ERROR apr_sockaddr_info failed for 2604:234::1234:5678:41514;Address family for hostname not supported 2015-01-04 04:58:34 ERROR apr_sockaddr_info failed for 2604:0234:0000:0000:0000:0000:1234:5678:41514;Address family for hostname not supported Configured with DNS (Long IPv6 notation in DNS record, though resolves short) Using a AAAA hostname: Name:    ramscan02-6.test.org Address:  2604:234::1234:5678 2015-01-04 05:53:19 ERROR couldn't bind ssl socket to ramscan02-6.test.org:41514;Cannot assign requested address Any help would be appreciated, Andy    

Botond, Would you consider creating an output module for email? It would be quite useful for generating alerts. While one can use exec_async to send email, it is rather cludgy.  Cheers   Ash

Well... NXlog (last vrsion from this site) installed on windows server 2012R2 Configured to get win-logs: SavePos TRUE     Module      im_msvistalog     Query     <QueryList>                        \               <Query Id="0" Path="Security">            \                 <Select Path="Application">*</Select>    \                     <Select Path="Security">*</Select>    \                     <Select Path="System">*</Select>    \                     <Suppress Path="Security">*[System[(EventID=5156 or EventID=4663 or EventID=5158 or EventID=5440 or EventID=5444)]]</Suppress>    \                     <Suppress Path="System">*[System[(EventID=5156 or EventID=4663 or EventID=5158 or EventID=5440 or EventID=5444)]]</Suppress>    \                   </Query>                \         </QueryList> As a result I see events only from System and Application... Nothing from Security Any Idea why it can happen? Logs are captured by windows - I can see it with eventvwr.msc, but nothing with Nxlog   NXlog have no information, looks like everything is ok: ...INFO nxlog-ce-2.8.1248 started... - no errors, no warnings... nothing else

I have multiple windows hosts sending events in binary to a single tcp listener.<Input windows>     Module     im_tcp     Port       9999     Host       0.0.0.0     InputType  Binary I am trying to track the rate of logs from the servers and create email alerts when the rate either drops or crosses a high watermark per hour.  To do that I need to create a stat / variable appending the hostname and hourstamp such as create_stat("Rate-" + '$Hostname' + 'stroftime($EventTimeStamp, something something)') or create_var("Rate-" + '$Hostname' + 'stroftime($EventTimeStamp, something something)')  Next I use the schedule code to detect a low watermark  <Schedule>         Every   3600 sec         Exec    create_stat("rate" + '$Hostname' + 'stroftime($EventTimeStamp, something something)'', "RATE", 10); add_stat("rate" + '$Hostname'' + 'stroftime($EventTimeStamp, something something)', 0);         Exec    log_info("Current Counts " + ":" + get_stat("rate" + '$Hostname'));         Exec    if defined get_stat("rate" + '$Hostname') and get_stat("rate" + '$Hostname') <= 1 \                 { \                     log_warning("No messages received from Host" ); \                     exec_async("/bin/sh", "-c", 'echo "' + $Hostname + \                            '"|/usr/bin/mail -a "Content-Type: text/plain; charset=UTF-8" -s "ALERT" '  \                            + 'analyst@company.com' );                                                      \                 }   </Schedule>   Two problems: How do I insert the variable/statistic name and value in the log message and how do I extract the hour stamp from the Event Time?   Thanks  Ash  PS: I could not get the deployment tool to work. have you had more success with it?

i defined an input for im_file that is being sent as syslog i have multiple files in a folder what i want to do is read each file and on EOF copy to another folder. didn`t find the option to identify EOF

Hi, I'm trying to use the xm_multiline module with nxlog to forward content of a logfile to logstash The log contains different xml elements which are properly indented (opening and closing elements are located at the start of the line) . E.g. <data version="x" xmlns:bla="http://www.example.com/bla"> <val:InfoSet> ... ... ... </val:InfoSet> </data> <message ...> <ns>bla</ns> ... ... </message> Because the elements have different names, I can only use < and </ to find the start and end line. I was hoping a filter like this should be enough to select the correct lines: HeaderLine /^</ EndLine /^<// But somehow nxlog gets confused with the / in the regex pattern. I also tried escaping which dindn't help. More testing showed that it needs at least one letter. I tried to specify all letters via regex but that didn't work: HeaderLine /^<[a-z]/ Only way that seems to work is to specify all letters in the square braket (with the exception of the lettern, which breaks).  HeaderLine /^<[abcdefghijklmopqrstuvwxyz]/ (left out n) Here all my test results. These lines worked: HeaderLine /^<m/ EndLine /^</m/ HeaderLine /^<m/ EndLine /^<\/m/ HeaderLine /^<[abcdefghijklm]/ EndLine /^<\/[abcdefghijklm]/ HeaderLine /^<[abcdefghijklmo]/ EndLine /^<\/[abcdefghijklmo]/ HeaderLine /^<[abcdefghijklmopqrstuvwxyz]/ (left out n) EndLine /^<\/[abcdefghijklmopqrstuvwxyz]/ HeaderLine /^<[abcdefghijklmopqrstuvwxyz]/ (left out n + not escaped EndLine /^</[abcdefghijklmopqrstuvwxyz]/ These lines didn't work: HeaderLine /^</ EndLine /^</m/ HeaderLine /^<[a-z]/ EndLine /^</m/ HeaderLine /^<\w/ EndLine /^</m/ HeaderLine /^<[abcdefghijklmn]/ EndLine /^<\/[abcdefghijklmn]/ HeaderLine /^<[bcdefghijklmn]/ EndLine /^<\/[bcdefghijklmn]/ HeaderLine /^<[abcdefghijklmopqrstuvwxyzn]/ EndLine /^<\/[abcdefghijklmopqrstuvwxyzn]/ HeaderLine /^<[abcdefghijklmnopqrstuvwxyz]/ EndLine /^</[abcdefghijklmnopqrstuvwxyz]/ Right now I still have a problem because many of my bessages start with <n. I think this is a bug in the module. Can you confirm so I can open a ticket? Thanks   Fyi, this is a duplicate of http://stackoverflow.com/questions/27429234/which-headerline-and-endline-for-multiline-xml-with-different-elements