Hello, I'm having trouble centralize logs because my storage system performs poorly. I have several nxlog-ce agents sending logs with om_tcp module and server I have a nxlog-ce recording these logs with the module om_file in a shared directory on my NAS. What recommendation for improving the performance of my solution as a whole? Use another distributed file system? Using an unstructured database? Exsitem more options? Obs .: I need to be able to retrieve the log in its original format, I need to be fault-tolerant and I need to have high write performance.

I'm using NXLog to read a log file and send it to Logstash.  This works fine, but some of the log lines are duplicates.  They're in separate lines, but the content is exactly the same.  I can't change the way the logs are written to the log file, so the only way is to fix it either with NXLog before it gets send, or in Logstash when it arrives, which I prefer not to do. I see NXLog does have a function for this, but it's not working for me.  I've tried this in my config file. <Processor norepeat> Module pm_norepeat </Processor> <Route 1> Path in => norepeat => out </Route> This is abviously not working for me, am I maybe missing something here?

So i have configured nxlog with https to talk with logstash.  I got an error on nxlog side " ERROR unexpected data from server (64 bytes)"  I looked it up and it looks like nxlog got https://github.com/lamby/pkg-nxlog-ce/blob/master/src/modules/output/http/om_http.c#L6462  So i did check up the http input on logstash side , but i have not understand it  https://github.com/logstash-plugins/logstash-input-http#L118 Looks like logstash should send respond code, but i dont see it. Remember i dont have ruby skills.  Can someone explain to me what might be wrong here ? Or how to fix this issue ?

I've seen some posts from about a year ago that NXLog is unable to parse attributes using xm_xml, I just wanted to check if this is still true? I am running NXLog as a service on Windows machines and want to be able to parse the following message, is it possible? <log4j:event logger="com.sentry.test.LogContextListener" timestamp="1437661699866" level="TRACE" thread="localhost-startStop-1"> <log4j:message><![CDATA[This is a trace message about how we should use C#]]></log4j:message> </log4j:event>

Hi guys, could You please share experience on filtering windows event logs. I have the folowing configuration in input tag: Query <QueryList>\ <Query Id="0">\ <Select Path="Application">*</Select>\ <Select Path="System">*</Select>\ <Select Path="Security">*</Select>\ </Query>\ </QueryList> # For windows 2003 and earlier use the following: #   Module      im_mseventlog #Exec if ($raw_event =~ /INFO\s+62464/) drop(); # Query za event logove (uzima samo definirane) #Query <QueryList>\ #<Query Id='1'>\ #<Select Path='Application'>*[System[(EventID='32068')]]</Select>\ #<Select Path='System'>*[System[(EventID='7001')]]</Select>\ #</Query>\ #</QueryList> ## Level 1 (ID=30  Critical)     severity level events # Level 2 (ID=40  Error)        severity level events # Level 3 (ID=50  Warning)      severity level events # Level 4 (ID=80  Information)  severity level events # Level 5 (ID=100 Verbose)      severity level events # Drop 4, i 5 level severity logs Exec if ($EventType == 'VERBOSE') OR ($EventType == 'Verbose') drop(); Exec if ($EventType == 'INFORMATION') OR ($EventType == 'Information') drop(); Exec if $raw_event =~ /INFO\s+4648/ drop(); .. I planed to filter specific events by entering name of 'eventtype' as in the example above. I would like to filter all Verbose,Information, and Warning levels. This unfortunately doesnt filter security INFORMATION level. Could You please please give proper example of filtering logs by severity  and by specific event_ID? Thank you very much. Appreciate the help.

I am working in Product development company in india. I have downloaded nxlog community edition and setup to transfer our logs into logstash server. Our environment details as below. OS - Windows 7 Professional. nxLog installed as service.   Everything working as we expected. Now what i want to ask is, can we take this setup into our production environment?  Is Community edition fully free of cost or is there any license has to be purchase for this (nxLog Community edition)? Please give us license details about nxLog community edtion in Production environment.    Thanks Sugumar J  

I have a stack of  nxlog -> om_ssl -> tcp input ->logstash.  And after i do kill -9 i lose a bunch of messages. i looked up buffered data in logstash and switch it to 1. tested it and if i read file i can lose about max 5 messages. So i tried to look up in nxlog, data_timeout is there 30 sec.  so as i looked up with data lose over time  it looks it might be it. Is there a chance to change this value after installing rpm ?  or i have to compile it with changed value in the code.  

Hi, is it possiable to access IIS logs located on a network share?  for example, right now I'm using:     Module    im_file     File    "D:\\Logs\\W3SVC1\\u_ex*"     SavePos  TRUE   and it works. But if i change it to:     Module    im_file     File    "\\netapp-ams-cifs\\IISlogs\\Trader1\\W3SVC1\\u_ex*"     SavePos  TRUE   i see an error on the nxlog saying: ERROR failed to open directory: \netapp-ams-cifs\IISlogs\Trader1\W3SVC1: The system cannot find the path specified.  Can anyone help please? Thanks

Hey all, I'm trying to install nxlog rpm on redhat 6. Once I run the rpm I get error message :  " Failed dependencies: libdbi >= 0.8.1 is needed by nxlog-ce-2.9.1347-1.x86_64" When I'm trying install libdbi 0.8.1 I get error on   dependencies collision with a more advanced version of libdbi. The only way to continue the installation is to delete the newer version - and it is very problematic. Someone experienced the problem?   Thanks a lot.

On the site only source zip for version 2.8.1248 is available Thanks,   Angelo.

Hi, I am evaluating nxlog on Windows but I've run into a snag. I have logs organized like this: D:\ServerLogs\2015-07-16\*.log Where the directory is rotated based on today's date. Now, if I use the recursive parameter set to true for the file watcher, including this D:\ServerLogs\*.log works. However, I don't want to include all files dues to size and/or relevance. I only want to include some of them. As far as I can gather, the recursive parameter only works when there's a wildcard on the filename. What I need is to be able to include the following: D:\ServerLogs\*\Log1.log so it follows the directory rotation on that file alone. I can't figure out how to accomplish this with nxlog. Please advise. Thank you.

Hi, I'm using nxlog community edition nxlog-ce-2.9.1347, I have few questions related om_http module. We have a centralized log server(Log Insight) and 10 application servers. Log Insight server exposes a REST API to post the logs data.I'm using NXLOG as a log forwarder to Log Insight Server from all my application servers. Please clarify the following questions. POST URL : http://loginsight:9000/api/v1/messages/ingest/4C4C4544-0037-5910-805A-C4C04F585831 Request Body: {"messages": [{  "fields": [  {"name": "Channel", "content": "Security"},  {"name": "EventID", "content": "4688"},  {"name": "EventRecordID", "content": "33311266"},  {"name": "Keywords", "content": "Audit Success"},  {"name": "Level", "content": "Information"},  {"name": "OpCode","content": "Info"},  {"name": "ProcessID", "content": "4"},  {"name": "ProviderName", "content": "Microsoft-Windows-Security-Auditing"},  {"name": "Task", "content": "Process Creation"},  {"name": "ThreadID", "content": "64"}  ],  "text": "A new process has been created.",  "timestamp": 1396622879241  }  ] } 1. How do I format my log data into the request as mentioned above in NXLOG. Request should be formated into json with fields and the data accordingly.    I'm able to parse IIS logs,eventlogs and logs from files into JSON but got stuck with calling REST API with request body. Please find my nxlog.conf below. 2. Is it a good idea to directly send the log data to log server via REST API? if not what are the disadvantages.  3. Does om_http module support retrying logic\buffering just in case REST API is down or doesn't respond. 4. What is the best architecture for sending the logs to centralized server, I see lot of people online follow NXLOG=>LOG STASH=>ELASTICSEARCH or Some Centralized server(Log Insight in my case). 5. Should I use pm_buffer, as my log files will be rotated after certain memory limit is reached in case if the REST API is down or om_http module handles this automatically.      Following configuration reads IIS, event logs and logs from files. ## This is a sample configuration file. See the nxlog reference manual about the ## configuration options. It should be installed locally and is also available ## online at http://nxlog.org/nxlog-docs/en/nxlog-reference-manual.html ## Please set the ROOT to the folder your nxlog was installed into, ## otherwise it will not start. #define ROOT C:\Program Files\nxlog define ROOT C:\Program Files (x86)\nxlog Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data LogFile %ROOT%\data\nxlog.log <Extension json> Module xm_json </Extension> <Extension w3c> # Map the fields from the IIS log file (you can open the IIS log file to see the header and know what fields to map) Module xm_csv Fields $date, $time, $s-ip, $cs-method, $cs-uri-stem, $cs-uri-query, $s-port, $cs-username, $c-ip, $cs(User-Agent), $sc-status, $sc-substatus, $sc-win32-status, $time-taken FieldTypes string, string, string, string, string, string, integer, string, string, string, integer, integer, integer, integer Delimiter ' ' </Extension> <Extension multiline> Module xm_multiline HeaderLine /^\d{4}\-\d{2}\-\d{2} \d{2}\:\d{2}\:\d{2}.\d{3}/ </Extension> <Extension charconv> Module xm_charconv AutodetectCharsets utf-8, utf-16, utf-32, iso8859-2 </Extension> <Input eventlog> Module im_msvistalog ReadFromLast True Module im_msvistalog Query <QueryList>\ <Query Id="0">\ <Select Path="Application">*</Select>\ <Select Path="System">*</Select>\ <Select Path="Security">*</Select>\ </Query>\ </QueryList> </Input> <Input iis> Module im_file File 'C:\inetpub\logs\LogFiles\W3SVC1\u_ex*.log' ReadFromLast TRUE Exec if $raw_event =~ /^#/ drop(); \ else \ { \ w3c->parse_csv(); \ $EventTime = parsedate($date + " " + $time); \ to_json (); \ } </Input> <Input webconsole> Module im_file File 'C:\Stash\WebConsole.log' InputType multiline SavePos TRUE # The call to convert_fields automatically converts the input to utf-8 Exec convert_fields("AUTO","utf-8"); \ if $raw_event =~ /^(\d{4}\-\d{2}\-\d{2} \d{2}\:\d{2}\:\d{2}.\d{3}) \[(\S+)\] \[(\S+)\] \[(\S+)\] \[(\S+)\] \[(.*)\] \[(.*)\] (.*)/s \ { \ $time = $1; \ $hostname = $2; \ $activityId = $3; \ $userIddeviceId = $4; \ $threadId = $5; \ $level = $6; \ $logger = $7; \ $message = $8; \ to_json(); \ } \ else \ { \ drop(); \ } </Input> <Output eventlog-out> Module om_tcp Host 127.0.0.1 Port 3515 Exec $EventReceivedTime = integer($EventReceivedTime) / 1000000; \ to_json(); </Output> <Output iis-out> Module om_tcp Host 127.0.0.1 Port 3516 </Output> <Output general-out> Module om_tcp Host 127.0.0.1 Port 3517 </Output> <Route 1> Path eventlog => eventlog-out </Route> <Route 2> Path iis => iis-out </Route> <Route 3> Path webconsole => general-out </Route> Sample logs(Webconsole.log): 2015-07-10 10:24:17.424    [20EX15736]    [00000000-0000-0000-0000-000000000000]    [0000000-0000000]    [00008]    [Info ]    [TestModule]    Testing log stash3     2015-07-10 10:24:17.425    [20EX15736]    [00000000-0000-0000-0000-000000000000]    [0000000-0000000]    [00008]    [Info ]    [TestModule]    Testing log stash4     2015-07-10 10:24:17.448    [20EX15736]    [00000000-0000-0000-0000-000000000000]    [0000000-0000000]    [00008]    [Error]    [TestModule]    *** EXCEPTION *** System.DivideByZeroException: Attempted to divide by zero.    at TCPPublisher.Program.Main(String[] args) in c:\Users\test\Documents\Visual Studio 2013\Projects\TCPDemo\TCPPublisher\Program.cs:line 26   Thanks in advance! Mohan G      

Hey, nxlog.log dosent create.     the folder /var/log/nxlog/  is empty.   My environmet is RedHat 6. What can be the reason ?    Thanks.

Hi, i have follwing logentry: ------------- New entry -------------------- line 1 line 2 line 3 line x {blank line} ------------- New entry -------------------- line 1 line 2 line 3 line x ​{blank line}   I try to parse this with the multiline extension with following configuration: <Extension multiline>     Module    xm_multiline     HeaderLine    /^--/     EndLine    /^$/ </Extension> <Input in>     Module    im_file     File    "input.txt"     SavePos    FALSE     ReadFromLast TRUE     InputType    multiline        Exec    if $raw_event !~ /^--/ drop();     Exec        $raw_event = replace($raw_event, "\r\n", ";"); </Input> <Output out>     Module    om_file     File    "output.txt" </Output> <Route 1>     Path    in => out </Route> When i write one line and save then the input-file nxlog outputs only the header. When i write the complete entry a once nxlog works as expected. What iam doing wrong?    

I have the following sql_exec command outline in my config: <Output out>     Module      om_odbc     ConnectionString    DSN=nxlog;     <Exec>       sql_exec("INSERT INTO eventlog (hostname) VALUES (?)", '$Hostname');     </Exec> </Output> This is exactly how the 2 examples in the documentation have it set up, I cannot seem to get it to work though I get the following error in my error log: procedure 'sql_exec()' does not exist or takes different arguments Can anyone shed any light on why this would be happening or exactly what arguements it is looking for?  Thanks      

hey, I have a problem reading the Apache access log file. After I tried to do tcpdump I noticed that the file not read  and not sent to the destination. I gave full permissions to the file but the situation has not changed. nxlog.conf structure : <Input in1>     Module im_file     File "/data/srv/httpd/logs/access_log_web1"       SavePos TRUE </Input> <Output fileout1>  Module  om_udp    Host  192.168.10.1      Port      630 </Output>   <Route 1> Path        in1   =>    fileout1 </Route>   * **There is no nxlog log file exist in /var/log/nxlog/   Thanks

TL;DR summary: NXLog won't load my certificate - why? I've set up an NXlog CE server to act as an encrypted entrypoint for a Graylog server. After creating a self-signed certificate and adding it to the configuration, I get an unexpected error in the log when NXlogs starts up. 2015-07-10 06:49:22 INFO reloading configuration and restarting modules 2015-07-10 06:49:22 ERROR SSL error, Failed to open certfile: "/certificates/client.pem";No such file or directory 2015-07-10 06:49:22 INFO configuration reloaded successfully. I assumed it might be a file rights issue, but looking at the files, it seems like the rights are actually a little too loose for comfort: root@e757dae37501:/# ll /certificates/ total 20 drwxr-xr-x 2 nxlog nxlog 4096 Jul 10 06:32 ./ drwxr-xr-x 49 root root 4096 Jul 10 06:32 ../ -rw-r--r-- 1 nxlog nxlog 1387 Jul 9 15:51 ca.pem -rw-r--r-- 1 nxlog nxlog 887 Jul 9 15:54 client.key -rw-r--r-- 1 nxlog nxlog 1001 Jul 9 15:55 client.pem My NXlog configuration looks like this: ... <Input inssl> Module im_ssl Host 0.0.0.0 Port 1515 InputType Binary CAFile "/certificates/ca.pem" CertFile "/certificates/client.pem" CertKeyFile "/certificates/client.key" KeyPass secret RequireCert FALSE AllowUntrusted TRUE </Input> ... I've tried looking at the source code for the SSL module, but it looks like the line that fails is not related to validation but simple file access. I have no idea how to proceed. Suggestions?

Trying to run nxlog-ce 2.8.1248 on AIX 7.1. Compilation is succesful, nxlog binnary is created, but can not run. There are messages like: "ERROR Failed to load module from /usr/local/libexec/nxlog/modules/extension/xm_syslog.so, Could not load module /usr/local/libexec/nxlog/modules/extension/xm_syslog.so.; System error: No such file or directory;DSO load file" There are static libraries (.a) in directories but dynamic (.so) not.

Hi,   We have deployed nxlog 2.8.1248 for Windows on many computers (so far about 200) using Windows 2008 R2, 2012 and 2012 R2. it appears we are facing multiple issues randomly: - Service is crashing expectably and log an error 1067 and after a few attempt, it can restart successfully - Service doesn't want stat and log error 1053 at each try (uninstall/reinstall of the client doesn't help) - The service is running but seems to be hanging but we have the following messages on the nxlog log: ERROR EvtUpdateBookmark failed: Unrecognized Win32 error code 6 ERROR EvtNext failed with error 14: Unrecognized Win32 error code 14   Any idea?   Thanks   PM

I am having this weird encoding problem. I am exporting utf8 characters into a local mysql table. when i export from nxlog into a file the characters seem ok. when i switch to dbi export its jiberish. i tried debugging the queries from mysql into a log to check if the characters arrive in the db intact and the characters seem fine there. collation in mysql is utf8. any ideas?