I have a stack of  nxlog -> om_ssl -> tcp input ->logstash.  And after i do kill -9 i lose a bunch of messages. i looked up buffered data in logstash and switch it to 1. tested it and if i read file i can lose about max 5 messages. So i tried to look up in nxlog, data_timeout is there 30 sec.  so as i looked up with data lose over time  it looks it might be it. Is there a chance to change this value after installing rpm ?  or i have to compile it with changed value in the code.  

Hi, is it possiable to access IIS logs located on a network share?  for example, right now I'm using:     Module    im_file     File    "D:\\Logs\\W3SVC1\\u_ex*"     SavePos  TRUE   and it works. But if i change it to:     Module    im_file     File    "\\netapp-ams-cifs\\IISlogs\\Trader1\\W3SVC1\\u_ex*"     SavePos  TRUE   i see an error on the nxlog saying: ERROR failed to open directory: \netapp-ams-cifs\IISlogs\Trader1\W3SVC1: The system cannot find the path specified.  Can anyone help please? Thanks

Hey all, I'm trying to install nxlog rpm on redhat 6. Once I run the rpm I get error message :  " Failed dependencies: libdbi >= 0.8.1 is needed by nxlog-ce-2.9.1347-1.x86_64" When I'm trying install libdbi 0.8.1 I get error on   dependencies collision with a more advanced version of libdbi. The only way to continue the installation is to delete the newer version - and it is very problematic. Someone experienced the problem?   Thanks a lot.

On the site only source zip for version 2.8.1248 is available Thanks,   Angelo.

Hi, I am evaluating nxlog on Windows but I've run into a snag. I have logs organized like this: D:\ServerLogs\2015-07-16\*.log Where the directory is rotated based on today's date. Now, if I use the recursive parameter set to true for the file watcher, including this D:\ServerLogs\*.log works. However, I don't want to include all files dues to size and/or relevance. I only want to include some of them. As far as I can gather, the recursive parameter only works when there's a wildcard on the filename. What I need is to be able to include the following: D:\ServerLogs\*\Log1.log so it follows the directory rotation on that file alone. I can't figure out how to accomplish this with nxlog. Please advise. Thank you.

Hi, I'm using nxlog community edition nxlog-ce-2.9.1347, I have few questions related om_http module. We have a centralized log server(Log Insight) and 10 application servers. Log Insight server exposes a REST API to post the logs data.I'm using NXLOG as a log forwarder to Log Insight Server from all my application servers. Please clarify the following questions. POST URL : http://loginsight:9000/api/v1/messages/ingest/4C4C4544-0037-5910-805A-C4C04F585831 Request Body: {"messages": [{  "fields": [  {"name": "Channel", "content": "Security"},  {"name": "EventID", "content": "4688"},  {"name": "EventRecordID", "content": "33311266"},  {"name": "Keywords", "content": "Audit Success"},  {"name": "Level", "content": "Information"},  {"name": "OpCode","content": "Info"},  {"name": "ProcessID", "content": "4"},  {"name": "ProviderName", "content": "Microsoft-Windows-Security-Auditing"},  {"name": "Task", "content": "Process Creation"},  {"name": "ThreadID", "content": "64"}  ],  "text": "A new process has been created.",  "timestamp": 1396622879241  }  ] } 1. How do I format my log data into the request as mentioned above in NXLOG. Request should be formated into json with fields and the data accordingly.    I'm able to parse IIS logs,eventlogs and logs from files into JSON but got stuck with calling REST API with request body. Please find my nxlog.conf below. 2. Is it a good idea to directly send the log data to log server via REST API? if not what are the disadvantages.  3. Does om_http module support retrying logic\buffering just in case REST API is down or doesn't respond. 4. What is the best architecture for sending the logs to centralized server, I see lot of people online follow NXLOG=>LOG STASH=>ELASTICSEARCH or Some Centralized server(Log Insight in my case). 5. Should I use pm_buffer, as my log files will be rotated after certain memory limit is reached in case if the REST API is down or om_http module handles this automatically.      Following configuration reads IIS, event logs and logs from files. ## This is a sample configuration file. See the nxlog reference manual about the ## configuration options. It should be installed locally and is also available ## online at http://nxlog.org/nxlog-docs/en/nxlog-reference-manual.html ## Please set the ROOT to the folder your nxlog was installed into, ## otherwise it will not start. #define ROOT C:\Program Files\nxlog define ROOT C:\Program Files (x86)\nxlog Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data LogFile %ROOT%\data\nxlog.log <Extension json> Module xm_json </Extension> <Extension w3c> # Map the fields from the IIS log file (you can open the IIS log file to see the header and know what fields to map) Module xm_csv Fields $date, $time, $s-ip, $cs-method, $cs-uri-stem, $cs-uri-query, $s-port, $cs-username, $c-ip, $cs(User-Agent), $sc-status, $sc-substatus, $sc-win32-status, $time-taken FieldTypes string, string, string, string, string, string, integer, string, string, string, integer, integer, integer, integer Delimiter ' ' </Extension> <Extension multiline> Module xm_multiline HeaderLine /^\d{4}\-\d{2}\-\d{2} \d{2}\:\d{2}\:\d{2}.\d{3}/ </Extension> <Extension charconv> Module xm_charconv AutodetectCharsets utf-8, utf-16, utf-32, iso8859-2 </Extension> <Input eventlog> Module im_msvistalog ReadFromLast True Module im_msvistalog Query <QueryList>\ <Query Id="0">\ <Select Path="Application">*</Select>\ <Select Path="System">*</Select>\ <Select Path="Security">*</Select>\ </Query>\ </QueryList> </Input> <Input iis> Module im_file File 'C:\inetpub\logs\LogFiles\W3SVC1\u_ex*.log' ReadFromLast TRUE Exec if $raw_event =~ /^#/ drop(); \ else \ { \ w3c->parse_csv(); \ $EventTime = parsedate($date + " " + $time); \ to_json (); \ } </Input> <Input webconsole> Module im_file File 'C:\Stash\WebConsole.log' InputType multiline SavePos TRUE # The call to convert_fields automatically converts the input to utf-8 Exec convert_fields("AUTO","utf-8"); \ if $raw_event =~ /^(\d{4}\-\d{2}\-\d{2} \d{2}\:\d{2}\:\d{2}.\d{3}) \[(\S+)\] \[(\S+)\] \[(\S+)\] \[(\S+)\] \[(.*)\] \[(.*)\] (.*)/s \ { \ $time = $1; \ $hostname = $2; \ $activityId = $3; \ $userIddeviceId = $4; \ $threadId = $5; \ $level = $6; \ $logger = $7; \ $message = $8; \ to_json(); \ } \ else \ { \ drop(); \ } </Input> <Output eventlog-out> Module om_tcp Host 127.0.0.1 Port 3515 Exec $EventReceivedTime = integer($EventReceivedTime) / 1000000; \ to_json(); </Output> <Output iis-out> Module om_tcp Host 127.0.0.1 Port 3516 </Output> <Output general-out> Module om_tcp Host 127.0.0.1 Port 3517 </Output> <Route 1> Path eventlog => eventlog-out </Route> <Route 2> Path iis => iis-out </Route> <Route 3> Path webconsole => general-out </Route> Sample logs(Webconsole.log): 2015-07-10 10:24:17.424    [20EX15736]    [00000000-0000-0000-0000-000000000000]    [0000000-0000000]    [00008]    [Info ]    [TestModule]    Testing log stash3     2015-07-10 10:24:17.425    [20EX15736]    [00000000-0000-0000-0000-000000000000]    [0000000-0000000]    [00008]    [Info ]    [TestModule]    Testing log stash4     2015-07-10 10:24:17.448    [20EX15736]    [00000000-0000-0000-0000-000000000000]    [0000000-0000000]    [00008]    [Error]    [TestModule]    *** EXCEPTION *** System.DivideByZeroException: Attempted to divide by zero.    at TCPPublisher.Program.Main(String[] args) in c:\Users\test\Documents\Visual Studio 2013\Projects\TCPDemo\TCPPublisher\Program.cs:line 26   Thanks in advance! Mohan G      

Hey, nxlog.log dosent create.     the folder /var/log/nxlog/  is empty.   My environmet is RedHat 6. What can be the reason ?    Thanks.

Hi, i have follwing logentry: ------------- New entry -------------------- line 1 line 2 line 3 line x {blank line} ------------- New entry -------------------- line 1 line 2 line 3 line x ​{blank line}   I try to parse this with the multiline extension with following configuration: <Extension multiline>     Module    xm_multiline     HeaderLine    /^--/     EndLine    /^$/ </Extension> <Input in>     Module    im_file     File    "input.txt"     SavePos    FALSE     ReadFromLast TRUE     InputType    multiline        Exec    if $raw_event !~ /^--/ drop();     Exec        $raw_event = replace($raw_event, "\r\n", ";"); </Input> <Output out>     Module    om_file     File    "output.txt" </Output> <Route 1>     Path    in => out </Route> When i write one line and save then the input-file nxlog outputs only the header. When i write the complete entry a once nxlog works as expected. What iam doing wrong?    

I have the following sql_exec command outline in my config: <Output out>     Module      om_odbc     ConnectionString    DSN=nxlog;     <Exec>       sql_exec("INSERT INTO eventlog (hostname) VALUES (?)", '$Hostname');     </Exec> </Output> This is exactly how the 2 examples in the documentation have it set up, I cannot seem to get it to work though I get the following error in my error log: procedure 'sql_exec()' does not exist or takes different arguments Can anyone shed any light on why this would be happening or exactly what arguements it is looking for?  Thanks      

hey, I have a problem reading the Apache access log file. After I tried to do tcpdump I noticed that the file not read  and not sent to the destination. I gave full permissions to the file but the situation has not changed. nxlog.conf structure : <Input in1>     Module im_file     File "/data/srv/httpd/logs/access_log_web1"       SavePos TRUE </Input> <Output fileout1>  Module  om_udp    Host  192.168.10.1      Port      630 </Output>   <Route 1> Path        in1   =>    fileout1 </Route>   * **There is no nxlog log file exist in /var/log/nxlog/   Thanks

TL;DR summary: NXLog won't load my certificate - why? I've set up an NXlog CE server to act as an encrypted entrypoint for a Graylog server. After creating a self-signed certificate and adding it to the configuration, I get an unexpected error in the log when NXlogs starts up. 2015-07-10 06:49:22 INFO reloading configuration and restarting modules 2015-07-10 06:49:22 ERROR SSL error, Failed to open certfile: "/certificates/client.pem";No such file or directory 2015-07-10 06:49:22 INFO configuration reloaded successfully. I assumed it might be a file rights issue, but looking at the files, it seems like the rights are actually a little too loose for comfort: root@e757dae37501:/# ll /certificates/ total 20 drwxr-xr-x 2 nxlog nxlog 4096 Jul 10 06:32 ./ drwxr-xr-x 49 root root 4096 Jul 10 06:32 ../ -rw-r--r-- 1 nxlog nxlog 1387 Jul 9 15:51 ca.pem -rw-r--r-- 1 nxlog nxlog 887 Jul 9 15:54 client.key -rw-r--r-- 1 nxlog nxlog 1001 Jul 9 15:55 client.pem My NXlog configuration looks like this: ... <Input inssl> Module im_ssl Host 0.0.0.0 Port 1515 InputType Binary CAFile "/certificates/ca.pem" CertFile "/certificates/client.pem" CertKeyFile "/certificates/client.key" KeyPass secret RequireCert FALSE AllowUntrusted TRUE </Input> ... I've tried looking at the source code for the SSL module, but it looks like the line that fails is not related to validation but simple file access. I have no idea how to proceed. Suggestions?

Trying to run nxlog-ce 2.8.1248 on AIX 7.1. Compilation is succesful, nxlog binnary is created, but can not run. There are messages like: "ERROR Failed to load module from /usr/local/libexec/nxlog/modules/extension/xm_syslog.so, Could not load module /usr/local/libexec/nxlog/modules/extension/xm_syslog.so.; System error: No such file or directory;DSO load file" There are static libraries (.a) in directories but dynamic (.so) not.

Hi,   We have deployed nxlog 2.8.1248 for Windows on many computers (so far about 200) using Windows 2008 R2, 2012 and 2012 R2. it appears we are facing multiple issues randomly: - Service is crashing expectably and log an error 1067 and after a few attempt, it can restart successfully - Service doesn't want stat and log error 1053 at each try (uninstall/reinstall of the client doesn't help) - The service is running but seems to be hanging but we have the following messages on the nxlog log: ERROR EvtUpdateBookmark failed: Unrecognized Win32 error code 6 ERROR EvtNext failed with error 14: Unrecognized Win32 error code 14   Any idea?   Thanks   PM

I am having this weird encoding problem. I am exporting utf8 characters into a local mysql table. when i export from nxlog into a file the characters seem ok. when i switch to dbi export its jiberish. i tried debugging the queries from mysql into a log to check if the characters arrive in the db intact and the characters seem fine there. collation in mysql is utf8. any ideas?

Hello, I am having an issue with the Program name not population with anything on our syslog server.  The service name is just blank.  Below is my config file.  Am I missing something?   ## Please set the ROOT to your nxlog installation directory   #define ROOT C:\Program Files\nxlog define ROOT C:\Program Files (x86)\nxlog   Moduledir %ROOT%\modules CacheDir  %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir  %ROOT%\data LogFile %ROOT%\data\nxlog.log   <Extension syslog>   Module xm_syslog   </Extension>   # Monitor application log files #<Input watchfile> #  Module im_file #  # File 'C:\\path\\to\\*.log' #  Exec $Message = $raw_event; # Exec if file_name() =~ /.*\\(.*)/ $SourceName = $1; # SavePos TRUE   #  Recursive TRUE #</Input>   # Monitor Windows event logs #<Input eventlog>   # Uncomment for Windows Vista/2008 or later  #  Module im_msvistalog      # Uncomment for Windows 2000 or later   # Module im_mseventlog #</Input>   #<Processor eventlog_transformer> #  Module pm_transformer #  Exec $Hostname = hostname(); #  OutputFormat syslog_rfc5424   #</Processor>   <Output syslogout_centreon>   Module om_udp   Host 10.10.103.112   Port 514 </Output> # Monitor CME FlatFile <Input watchfile_test>   Module im_file   File 'C:\\logs\\test.txt'   Exec $Message = $raw_event;   Exec $SyslogSeverityValue = 6;   Exec if $raw_event =~ /INFO/ drop();    Exec if file_name() =~ /.*\\(.*)/ $SourceName = $1;    Exec if $raw_event =~ /WARNING/ $SyslogSeverityValue = 4;    Exec if $raw_event =~ /ERROR/ $SyslogSeverityValue = 3;    Exec if $raw_event =~ /CRITICAL/ $SyslogSeverityValue = 2;   Exec if $raw_event =~ /ALERT/ $SyslogSeverityValue = 1;    SavePos TRUE     Recursive TRUE   PollInterval 10 </Input> <Processor filewatcher_transformer_test>   Module pm_transformer       #Uncomment to override the program name    Exec $SourceName = 'test';        Exec $Hostname = hostname();   OutputFormat syslog_rfc5424 </Processor> # Path to send Syslog message for test Flat File Generator <Route cme_flat_file_generator>   Path watchfile_test => filewatcher_transformer_test => syslogout_centreon </Route>              

I am using the following im_file configuration to try to collect Windows DHCP Server logs: ## Input module for Microsoft DHCP server audit logs <Input dhcp>     Module im_file     File "C:\\Windows\\System32\\Dhcp\\DhcpSrvLog-*.log"     SavePos TRUE     PollInterval 180     Exec to_syslog_bsd(); </Input> I also tried this without escaping the backslashes and even with "/" characters instead. I also tried using a specific filename but nothing seems to work, since I get the "input file does not exist" error. When I try the same config but with the location being at C:\Dhcp\DhcpSrvLog-*.log, everything works. nxlog service is being run as LocalSystem. Any hints on what I would need to do next to get the logs working from their native location?

Hi, We're evaluating nxlogn to forward event logs and IIS logs from same server as SYSLOG format to centralised log server. The event logs are reaching correctly, but not the IIS logs. Please see the configuration settings. Your help would be highly appreciated ============================================================ define ROOT C:\Program Files (x86)\nxlog Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data LogFile %ROOT%\data\nxlog.log <Extension _syslog>     Module      xm_syslog </Extension> <Input eventlog>     Module      im_msvistalog # For windows 2003 and earlier use the following: #   Module      im_mseventlog </Input> <Extension w3c>     Module    xm_csv     Fields    $date, $time, $site, $dstip, $HTTPMethod, $URIStem, $URIQuery, $port, $username, $srcip, $UserAgent, $HTTPStatus, $SubStatus, $win32Status     FieldTypes    string, string, string, string, string, string, string, string, string, string, string, string, string, string,     Delimiter    ' ' </Extension> <Input IIS_Log>   Module    im_file     File    "C:\WINDOWS\System32\LogFiles\W3SVC1\ex*"     ReadFromLast TRUE         #Drop info legend lines         Exec    if $raw_event =~ /^#/ drop();\         else\         {\             w3c->parse_csv();\             $EventTime = parsedate($date + " " + $time);\         } </Input> <Output out>     Module      om_udp     Host        X.X.X.X     Port        514     Exec        to_syslog_snare(); </Output> <Route 1>     Path        eventlog => out </Route> <Route 2>     Path        IIS_Log => out </Route

I'm currently using the TCP output of NXLog (v2.9.1347) to ship Windows Server 2008 R2 eventlogs to Logstash (v1.4.2) in JSON format; lately I found that NXLog crashes if Logstash has been unavailable for some time and then became available, although it ships a few logs before crashing. This event is logged in the eventlog: Faulting application name: nxlog.exe, version: 0.0.0.0, time stamp: 0x54fedd1a Faulting module name: ntdll.dll, version: 6.1.7601.18247, time stamp: 0x521ea8e7 Exception code: 0xc0000005 Fault offset: 0x0005e8d1 Faulting process id: 0x4e4 Faulting application start time: 0x01d0a2b5080df49c Faulting application path: C:\Program Files (x86)\nxlog\nxlog.exe Faulting module path: C:\Windows\SysWOW64\ntdll.dll Report Id: 7ebdb4d7-1036-11e5-909f-005056a30012 To reproduce the issue, just have NXLog ship logs to Logstash and then stop Logstash for about an hour then start it, NXLog crashes soon after. Any idea what might be causing this?

My problem is I cannot collect ADDS or DNS events with Nxlog and send them to an ELK server. In the Nxlog config for the DC and DNS server I have the following Query <QueryList>\ <Query Id="0">\ <Select Path="Security">*</Select>\ <Suppress Path="Security">*[System[(EventID=4624 or EventID=4776 or EventID=4634 or EventID=4672 or EventID=4688 or EventID=4769)]]</Suppress>\ <Select Path="System">*[System/Level=2]</Select>\ <Select Path="Microsoft-Windows-ActiveDirectory_DomainService">*</Select>\ <Select Path="Microsoft-Windows-DNS-Server-Service">*</Select>\ </Query>\ </QueryList> The config file works correctly without the Active Directory and DNS paths. The desired Security and System logs go to ELK correctly. I have also tried leaving only the ADDS or DNS paths in the config file with no luck. I don't think I have the correct paths for ADDS and DNS in the config and that is my problem. My Google-fu and Bing-fu hasn't found any results giving me the Event ID channel for ADDS and DNS events. I've only found the Event ID channels for Application, Security, System, and Setup. Any suggestions? I'm up for any! The DC\DNS server and the ELK server are running on Windows Server 2012. ELK install is running the latest stable releases of ELK. Thanks!

I am trying out the enterprise edition, and could not find documentation for reading event logs directly from .evtx file only, can anyone help in a sample config.