when i manually execute python script it works fine (cat /var/log/nxlog/pktdesign-alerts.log1|./sec-parse.py) , but whe nxlog calls keep on seeing the error messages below and scripot does not execute,    ===> when nxlog config was whats listed below produced error "2015-11-23 20:52:41 ERROR subprocess '4843' returned a non-zero exit value of 127" <Output alertout-pktdesign>     Module      om_file     File        "/var/log/nxlog/pktdesign-alerts.log1"     Exec        exec_async("/bin/sh", "-c", 'echo "' + $raw_event +  '"|./sec-parse.py' ); </Output>     ===> when nxlog config was what's listed below produced error  "2015-11-23 21:25:41 ERROR subprocess '8319' returned a non-zero exit value of 1" <Output alertout-pktdesign>     Module      om_file     File        "/var/log/nxlog/pktdesign-alerts.log1"     Exec        exec_async("/bin/sh", "-c", 'echo "' + $raw_event +  '"|/etc/nxlog/sec-parse.py' ); </Output>   any help is appreciated. thanks  

Hi, I'm setting up nxlog on a window box to read files to send to elasticsearch, and I have some questions: 1) is there a limit on the number of im_file inputs that a output (om_tcp) can handle? And is nxlog reading those monitored files concurrently or in serial one after each other? 2) we can setup multiple paths in nxlog, different path means parallel processing, e.g. each path is handled by different threads?? 3) can we install mutiple nxlog services in one box? Thanks a lot!

Hi there, I have several hosts where lots of logs are generated. On every of this hosts logs are placed into one directory and have different names with *.log extension. My point is to use nxlog on that hosts to forward logs into one central storage but I have problem with populating files names. I would like that file source/first.log to be populated to destination/first.log. Similarly source/second.log -> destination/second.log etc. Below is my config file. Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data LogFile %ROOT%\data\nxlog.log define LOG_SERVER \\\\192.168.199.10 <Extension _syslog> Module xm_syslog </Extension> <Input app-logs> Module im_file File "d:\\Logs\\\\*.log" InputType LineBased <Schedule> Every 30 sec Exec log_info("scheduled execution at " + now()); </Schedule> Exec $fileName = file_name(); </Input> <Output out> Module om_file CreateDir TRUE File "%LOG_SERVER%\\Shared\\Logs\\" + $fileName </Output> <Route 1> Path app-logs => out </Route> Logs are not populating and I have in nxlog logs something like that 2015-11-20 15:11:00 ERROR CreateDir is TRUE but couldn't create directory: \\192.168.199.10\Shared\Logs\d:\Logs\; The specified path is invalid. Will be gratefull for help with my issue.  

Hi everyone, I've got a box running Kibana and Elasticsearch, with the information being handed across by fluentd.  It's all working pretty well, and I've moved on to pushing Windows events through - which is where I'm struggling. I'm using nxlog on a windows machine, and it's pushing the data directly to elasticsearch - unfortunately I cannot for love nor money get the date to be handled as a date, rather than a string; the nxlog output is:     URL    http://192.0.0.10:9200     ContentType application/json     Exec   set_http_request_path(strftime($EventTime, "/windowsevents-%Y.%m.%d/" + $SourceModuleName)); delete($EventReceivedTime); rename_field("timestamp","@timestamp"); to_json(); I create a new index with a pattern of *windowsevents*\*, but the *EventTime* field is stated as a string, not a date - so I can't sort data by age.  *@timestamp* clearly isn't getting populated, as the *timestamp* field isn't being used.  Has anyone used a similar setup?  Is there a way to push the data on as a date? Apologies if I've missed out pertinent information - it's all a bit new to me.

Hello, I am trying to update the nxlog software via gpo startscript on our clients. Powershell: (Get-WmiObject -Class Win32_Product | where {$_.Name -like "nxlog"} | where {$_.Version -ne "2.9.1427"}).Uninstall() Start-Process \\server0001\NETLOGON\SoftwarePackets\nxlog-2.9.msi /quiet   Batch: WMIC /interactive:off product where 'name like "%%nxlog%%" and not version like "%%2.9.1427%%"' call uninstall msiexec.exe /package \\server0001\NETLOGON\SoftwarePackets\NXLOG\nxlog-2.9.msi /quiet   The Scripts are working fine, if manually started on the client. Only in the group policy the scripts run trought, but doesn´t install the software. (Other Software like Java, PDF24 etc. is working fine this way)   Kind regards

Hi! What about support for MS SQL Profiler trace (*.trc) files? Please, can nxlog to read from these files? Thanks.

I cannot seem to get NXLog to ship IIS Logs to LogAnalyzer.  It is collecting Event logs perfectly.  I have commented out all event logs in an attempt to isolate just IIS flow and there is nothing. define ROOT C:\\Program Files (x86)\\nxlog define ROOT_STRING C:\\Program Files (x86)\\nxlog define CERTDIR %ROOT%\\cert   Moduledir %ROOT%\\modules CacheDir %ROOT%\\data Pidfile %ROOT%\\data\\nxlog.pid SpoolDir %ROOT%\\data LogFile %ROOT%\\data\\nxlog.log   # Include fileop while debugging, also enable in the output module below #<Extension fileop> # Module xm_fileop #</Extension>   <Extension json>  Module xm_json </Extension>   #<Extension syslog> # Module xm_syslog #</Extension>   <Input internal>  Module im_internal  Exec $Message = to_json(); </Input>   <Input IIS>    Module im_file  File C:\inetpub\logs\LogFiles\W3SVC1\*  SavePos True  InputType LineBased </Input>     # Windows Event Log #<Input eventlog> # Uncomment im_msvistalog for Windows Vista/2008 and later # Module im_msvistalog   #Uncomment im_mseventlog for Windows XP/2000/2003 #Module im_mseventlog   # Exec $Message = to_json(); # Exec if ($EventID == 5156) drop(); # Exec if ($EventID == 4656) drop(); # Exec if ($EventID == 4658) drop(); #</Input>   <Output out>  Module om_tcp  Host x.x.x.x  Port 514 </Output>   <Route 1>  Path internal, eventlog => out </Route>   I am sure I am missing something simple.  I have tried file path quotes (single and double), several different Input configs with varying levels of detail, I have tried variations of the wildcard to pull the log file, I even tried changing to double slashes in the file path (grasping at straws with that one).....   Any ideas would be appreicated...

Hello guys, i have http input in logstash configuration:   http {         port => "60114"         type => "PaaS"         user => "logs_account"         password => "password"   } And i wanna send logs from NXLOG to logstash via this http endpoint. I think i can login/password in this format: <Output elasticsearch>     Module      om_http     URL         "http://logs_account:password@logsserver:60111/" </Output> But NXLOG can't parse this URL and i have errors in log file 2015-10-30 16:25:55 ERROR invalid url: "http://logs_account:password@logsserver:60111/" 2015-10-30 16:25:55 ERROR Failed to parse url "logs_account:password@logsserver:60111/" at C:\Program Files (x86)\nxlog\conf\nxlog.conf:43  

can event coorelation be used with a trigger such that when an event matches it collects subsequent if all come in within specifc time frame (say within 30second from first event) and write those or send via email? we currently have such functionality out of per based "SEC" but are trying to migrate to NXLOG.   thanks.

Hi Is there a garbage collector service, when using the pm_buffer to disk, so that the buffer file on disk is emptied? If yes, how often is this run, and can it be configured? /Johan

I'm trying to pass only Warning / Error / Critical level Application Logs through NXLog to my ELK stack. When I have this configuration <Input EventLog_In> Module im_msvistalog <QueryList>\ <Query Id="0">\ <Select Path="Application">*</Select>\ </Query>\ </QueryList> Exec to_json(); </Input> everything works fine, and I'm collecting all levels of Application logs. I tried putting in a parameter on the <Select Path> line like this <Select Path="Application">*[Application/Level=1]</Select>\ And it craps itself and I get nothing. NXLog isn't reporting any issue, and I'm not seeing anything on the logstash side of things. I got the information about Event Viewer querying from this thread and adapted it to my use case: https://serverfault.com/questions/543494/query-specific-logs-from-event-log-using-nxlog

we need to separate and aggregate events per IP address during a period of time, such that, a single email is sent conteining multiple messages where the same IP is present, is this something that can be done with pm_evcorr? i hace tried and not yet able to get this functionality, if possible pls provide a quick example. thanks.

hi , 2 days ago i started getting this error : ERROR ### ASSERTION FAILED at line 33 in xm_gelf.c/xm_gelf_writer_udp(): "deflateInit(&strm, Z_DEFAULT_COMPRESSION) == Z_OK" ### INFO reconnecting in 1 seconds   anu idea ? tks .

Dear community, I'm currently working on parsing MS Exchange logs and sending them via GELF to my graylog instance. I'd like to convert the sender- and recipient-address field to lowercase. Sounds pretty easy, in fact, I need help :( my current config looks like this (below). Any help is appreciated. I've tried to work with "Exec       $sender-address = lc($sender-address);" within the input as well as Output backet - neither did work. define BASEDIR C:\Program Files\Microsoft\Exchange Server\V14\TransportRoles\Logs\MessageTracking <Extension csv> Module xm_csv Fields $date-time, $client-ip, $client-hostname, $server-ip, $server-hostname, $source-context, $connector-id, $exchange_source, $event-id, $internal-message-id, $message-id, $recipient-address, $recipient-status, $total-bytes, $recipient-count, $related-recipient-address, $reference, $message-subject, $sender-address, $return-path, $message-info, $directionality, $tenant-id, $original-client-ip, $original-server-ip, $custom-data FieldTypes string, string, string, string, string, string, string, string, string, integer, string, string, string, integer, integer, string, string, string, string, string, string, string, string, string, string, string Delimiter , </Extension> <Input in_exchange> Module im_file File '%BASEDIR%\MSGTRK????????*-*.LOG' SavePos TRUE Exec if $raw_event =~ /HealthMailbox/ drop(); Exec if $raw_event =~ /^#/ drop(); Exec csv->parse_csv(); </Input> <Output out_exchange> Module om_udp Host graylog.local Port 12203 OutputType GELF Exec $SourceName = 'exchange_msgtrk_log'; </Output> <Route exchange> Path in_exchange => out_exchange </Route>

I am trying to use NXlog to ship Event Logs and IIS logs to Graylog. No matter what I do in NXlog the IIS log timestamp in Graylog is the same as the EventReceivedTime. However, the Event Log timestamps are correct. Here is a link to my config on pastebin. All the configs I found on Google set the EventTime the same way. See anything wrong? $EventTime = parsedate($date + " " + $time); \ I tried doing a WireShark capture to see what the difference was. NXlog compresses the data before sending it. I haven't taken the time to learn how to uncompressit. I also tried setting the timestamp manually. That didn't work either... $timestamp = integer(parsedate($date + " " + $time))/1000; \

I am trying to figure out how many events are coming in per hour on a given a input module named win. I have searched around and haven't found any definitive solution. Most of what I have seen implements the create_stat function. But from there, I am lost. Here is my current config for the input, output, and route. How would I implement this feature into what I currently have? My end goal is to calculate EPS and write it out to log_info every hour with a message saying something like: EPS calulcated: 3,019 <Input win>     module          im_tcp         host            0.0.0.0         port            524         Exec            parse_syslog();         Exec            log_info("Severity Windows Collector: " + $SyslogSeverity + ", Hostname: " + $Hostname); </Input> <Output winout>         Module          om_file         CreateDir    true         File            '%WINLOG%'         <Schedule>                 Every           60 sec                 Exec            if (file_size('%WINLOG%') >= 2G) \                                 { \                                        file_cycle('%WINLOG%', 500); \                                         winout->reopen(); \                                 }         </Schedule> </Output> <Route 5>     Path            win => winout, Grid </Route>

NxLog Community Edition 2.8.1248 sometimes requires manual service restart,strangely,even reboot of OS doesnot help for: ERROR fatal connection error, reconnection will not be attempted (statuscode: 731004); apr_sockaddr_info failed for ____; The requested name is valid, but no data of the requested type was found. Can you please check,confirm, provide some fix &/or release updated version ...

On a Window 2012 server I'm collecting system events and then sending them to a remote server using OutputType GELF. This works fine on my serverrs behind a firewall however I have a AWS server that I would like to log and send logs over a TLS connection. Here's what my working Output looks like: <Output out>     Module      om_udp     Host        XXX.XXX.XXX.XXX     Port        12201     OutputType  GELF </Output> <Route 1>     Path      insql, in => out </Route> I have tested configs like below without success, the SSL connection is made but no logs are sent and the machine just repeats connections over and over to my graylog server. <Output out>     Module      om_ssl     Host        XXX.XXX.XXX.XXX     Port        12201     CAFile     %ROOT%\cert\ca.pem     CertFile    %ROOT%\cert\client-cert.pem     CertKeyFile    %ROOT%\cert\client-key.pem     OutputType  GELF </Output> <Route 1>     Path      insql, in => out </Route> Any ideas on how to proxy a "OutputType GELF" within a TLS/SSL connection? Thanks, Chipmunk  

Hi, I've started playing with NXlog and have found strange behaviour. This is my configuration: [code] <Input b-logs>     Module      im_file     File "d:\\Temp\\Logs\\test.txt"          SavePos TRUE     InputType LineBased     Exec if ( $raw_event =~ /^#/ )    \     {    \         $raw_event="ok";    \     }    \     else    \     {    \         $raw_event="bad";    \     } </Input>   <Output b-logs-out>     Module      om_tcp     Host        192.168.0.8     Port        8888 </Output> <Route 1>     Path        b-logs => b-logs-out </Route> [/code] I wrote simple python server that listens on TCP port 8888 on machine 192.168.0.8. What I want to achieve is to display "OK" when line in my test log file, begins with "#" and "BAD" when it is not. It works at the beginning. Output looks like following example: OK OK BAD OK OK Then I'm trying to change character "#" to any other and it stop working. Then I revert changes, put "#" into nxlog.conf again (restart Windows service) and instead of BAD or OK messages, NXLog sends entire line from log file. I'm getting following results: #line1 #Line2 TestLine3 #Comment at line4 I can't find reason of this behaviour.  Could you please give me any hint what may be the reason? My exampe of configuration is not useful, but I just want to understand how nxlog works to be able to use it to more sophisticated tasks in future.

I cannot compile nxlog on Solaris. On configure i got: checking for apr_socket_create in -lapr-1... no configure: error: libapr-1 not found If I have APR in /usr/apr etc, I am trying: ./configure CFLAGS="-I/usr/apr/1.3/include -I/usr/apr-util/1.3/include" LDFLAGS="-L/usr/apr/1.3/lib/sparcv9 -L/usr/apr-util/1.3/lib/sparcv9" --with-apr=/usr/apr/1.3/ --with-included-apr --prefix=/export/home/user/compiled And there is the same.