Hello everybody, I'm trying to collect Doubletake logs from a Centos 6.x server. I have that kind of logs ; 01/31/2016  21:15:26.000000 25786 140456885212928     48  2          0  Ops Skipped:        308390 01/31/2016  21:15:26.000000 25786 140456885212928     49  2          0  Total Mirror Ops:   321057 01/31/2016  21:15:26.000000 25786 140456885212928     50  2          0  Elapsed Time:          2318 seconds 01/31/2016  21:15:26.000000 25786 140456885212928     51  2          0  Paused Time:          0 seconds 01/31/2016  21:15:26.000000 25786 140456885212928     52  2          0  Total number of pauses:          0 01/31/2016  21:44:31.000000 25786 140456771974912     53  2         77  Connection lost with IP address ip://127.0.0.1 : 1501 01/31/2016  21:44:32.000000 25786 140457057756928     54  2     700000  Server Monitor was successfully stopped 01/31/2016  21:44:32.000000 25786 140457057756928     55  2      51503  Source module Stopped 01/31/2016  21:44:32.000000 25786 140457057756928     56  2      52503  Stopping all targets 01/31/2016  21:44:32.000000 25786 140457057756928     57  2      52503  Target module Stopped 00/00/0000  00:00:00.0000 Start of logfile  00/00/0000  00:00:00.0000 Application starting 01/31/2016  21:44:44.000000  9976 139690482448128      1  2          0  Buffer allocator limit is: 67108864 bytes 01/31/2016  21:44:44.000000  9976 139690482448128      2  2          0  QMemoryBufferMax size is: 268435456 bytes 01/31/2016  21:44:45.000000  9976 139690482448128      3  2          0  ActivationCode is valid: "6uvb-kyqa-tgar-wpeu-0t52-ubuv". 01/31/2016  21:44:45.000000  9976 139690482448128      4  2          0  Source Failover is allowed 01/31/2016  21:44:45.000000  9976 139690482448128      5  2          0  Target Failover is allowed 01/31/2016  21:44:45.000000  9976 139690482448128      6  2          0  Source Full Server Failover is allowed 01/31/2016  21:44:45.000000  9976 139690482448128      7  2          0  Target Full Server Failover is allowed 01/31/2016  21:44:45.000000  9976 139690482448128      8  2          0  Source Replication is allowed 01/31/2016  21:44:45.000000  9976 139690482448128      9  2          0  Target Replication is allowed 01/31/2016  21:44:45.000000  9976 139690482448128     10  2          0  Heartbeat Transmission started on port 1500 (interval=3seconds) 01/31/2016  21:44:48.000000  9976 139690482448128     11  2         69  Kernel Started on bl-db01.marsathletic.com  ip://10.10.100.75 : 1500  Version: 7.1.1.1255.0 01/31/2016  21:44:48.000000  9976 139690482448128     12  2     504002  Double-Take has successfully found /dev/dtrep0 01/31/2016  21:44:48.000000  9976 139690482448128     13  2      52501  Target module loaded successfully 01/31/2016  21:44:48.000000  9976 139690482448128     14  2          0  Disabling all replication from the driver 01/31/2016  21:44:48.000000  9976 139690482448128     15  2          0  Returning default addr: 10.10.100.75 : 1500 01/31/2016  21:44:48.000000  9976 139690482448128     16  2         71  Originator Attempting ip://10.10.150.10 : 1500 01/31/2016  21:44:48.000000  9976 139689629570816     17  2         73  Connected to  IP address ip://10.10.150.10 : 1500 01/31/2016  21:44:48.000000  9976 139690413819648     18  2         75  Connection resumed with IP address ip://10.10.150.10 : 1500 01/31/2016  21:44:48.000000  9976 139690482448128     19  2         80  Auto-reconnecting Lvra_0d341950-6cba-412c-834d-8afa3426cc83 to ip://10.10.150.10 : 1500::/var/lib/mysql/ -> /opt/dbtk/mnt/.job-0d341950-6cba-412c-834d-8afa3426cc83/var/lib/mysql/;/boot/ -> /opt/dbtk/mnt/.job-0d341950-6cba-412c-834d-8afa3426cc83/boot/;/ -> /opt/dbtk/mnt/.job-0d341950-6cba-412c-834d-8afa3426cc83/; 01/31/2016  21:44:48.000000  9976 139690482448128     20  2        800  Transmission manually resumed by client 01/31/2016  21:44:48.000000  9976 139690482448128     21  2          0  Returning default addr: 10.10.100.75 : 1500 01/31/2016  21:44:48.000000  9976 139690482448128     22  2         87  Starting replication of set Lvra_0d341950-6cba-412c-834d-8afa3426cc83 for connection 1 01/31/2016  21:44:48.000000  9976 139690482448128     23  2          0  Activating replication on / 01/31/2016  21:44:48.000000  9976 139690482448128     24  2          0  Activating replication on /boot 01/31/2016  21:44:48.000000  9976 139690482448128     25  2          0  Activating replication on /var/lib/mysql 01/31/2016  21:44:48.000000  9976 139690482448128     26  2          0  Disabling replication on /var/log/DT 01/31/2016  21:44:48.000000  9976 139690482448128     27  2          0  Disabling replication on /var/cache/DT 01/31/2016  21:44:48.000000  9976 139690482448128     28  2     500000  Starting a connection for a Linux Virtual Recovery job. 01/31/2016  21:44:48.000000  9976 139690482448128     29  2     500000  Lvra_0d341950-6cba-412c-834d-8afa3426cc83 is connected to ip://10.10.150.10 : 1500::/var/lib/mysql/ -> /opt/dbtk/mnt/.job-0d341950-6cba-412c-834d-8afa3426cc83/var/lib/mysql/;/boot/ -> /opt/dbtk/mnt/.job-0d341950-6cba-412c-834d-8afa3426cc83/boot/;/ -> /opt/dbtk/mnt/.job-0d341950-6cba-412c-834d-8afa3426cc83/; using compression level 1 (1) 01/31/2016  21:44:48.000000  9976 139690482448128     30  2     500000  Auto-Reconnect success. ConID = 1 01/31/2016  21:44:48.000000  9976 139690482448128     31  2      51501  Source module loaded successfully 01/31/2016  21:44:48.000000  9976 139690482448128     32  2          0  Detected RedHat configuration for failover persistence. 01/31/2016  21:44:53.000000  9976 139690311149312     33  2         72  Connection request from IP address 10.10.150.10 01/31/2016  21:44:53.000000  9976 139690311149312     34  2      99001  Telling peer IP: ip://10.10.150.10 : 1500 that conditions are OK to proceed. 01/31/2016  21:45:01.000000  9976 139690269189888     35  2        800  Local connection accepted, spinning up new local listen thread 01/31/2016  21:45:01.000000  9976 139690269189888     36  2         72  Responding to request from IP address 127.0.0.1 : 1502 using 127.0.0.1 : 1501 01/31/2016  21:45:01.000000  9976 139690269189888     37  2     600002  User :lms: has FULL access (2) 01/31/2016  21:45:04.000000  9976 139689361135360     38  2          0  Repset contains 41751738977 byte(s) to mirror 01/31/2016  21:45:04.000000  9976 139689361135360     39  2          0  Repset requires 324078 ops to mirror 01/31/2016  21:45:04.000000  9976 139689361135360     40  2         94  Delete Orphans Started <1> 01/31/2016  21:45:04.000000  9976 139689361135360     41  2         89  Mirror Started, Differences, Block Checksum <1> Nxlog is able to send the logs to Graylog via GELF UDP but I cant display the complete line on Graylog. All I can see is a cutted piece of line like ; 01/31/2016  21:15:26.000000 25786 140456885212928 I cant display more than this. I will be very happy if somebody can help to solve this. Thank you. Mehmet   Here is my nxlog.conf ######################################## # Global directives                    # ######################################## User root Group nxlog LogFile /var/log/nxlog/nxlog.log LogLevel INFO ######################################## # Modules                              # ######################################## <Extension gelf>     Module      xm_gelf </Extension> <Input Doubletake>     Module      im_file     File        "/tmp/2.log"     SavePos     TRUE </Input> <Output graylog_out>     Module      om_udp     Host        192.168.2.94     Port        12201     OutputType  GELF </Output> ######################################## # Routes                               # ######################################## <Route 1>     Path        Doubletake => graylog_out </Route>  

Scenario: I have NXLog EE installed on a host in Windows domain. I need to read DHCP logs from the DC(s), UNC path: \\<server name>\C$\Windows\System32\dhcp\DhcpSrvLog-*.log Since it is not possible to specify alternate credentials for accessing remote files (as it is for eventlog, i.e. im_msvistalog module), nxlog has to be started using an account with special privileges on the DC's file system - 4 options:  1. for nxlog service, use domain admin account (local admin role does not exist on DC)      - nxlog.conf - use UNC path: `\\<server name>\C$\Windows\System32\dhcp\DhcpSrvLog-*.log`  2. for nxlog service, use local admin account on the agent's host + share C:\Windows\System32\dhcp\ on the DC, enabling read only permissions for nxlog account only      - nxlog.conf - use share name: `\\<server name>\dhcp\DhcpSrvLog-*.log`  3. install nxlog agent on the DC, run nxlog as a service, use local admin account  4. smaller footprint ? -> install http://nxlog-ce.sourceforge.net/nxlog-docs/en/nxlog-reference-manual.html#nxlog_processor on the DC None of these options are win-wins for customer production environment, as they require opening the restricted environment of the DC. My question is: are there any nxlog configuration options, which would enable me to fetch the file remotely, similar to these for DC's Security event log?: <Input dc1> Module im_msvistalog RemoteServer <ip> Remoteuser <user> RemotePassword <pwn> RemoteDomain <domain> Query <QueryList><Query Id="0" Path="Security"><Select Path="Security">*</Select></Query></QueryList> </Input>

Sorry,my english is not good. I occur a problem with om_file module. my config as following: #define Log receive from Network <Input LogNetIn>     Module      im_tcp     host        0.0.0.0     port        1514     <Exec>         if $raw_event =~ /^"([^"]+)"/ {             $ModuleName=$1 ;         } else if $raw_event =~ /ModuleName=([^\d]+)/{             $ModuleName=$1 ;         } else {             file_write('%ROOT%\data\debug.txt',$raw_event +"\r\n");         }         if ($ModuleName=="HB") {             HB_IL_CSV->parse_csv();             HB_CSV->to_csv();             $StorePathFileName='%ROOT%\data\heartbeat.log';         }else if ($ModuleName=="IL") {             HB_IL_CSV->parse_csv();             IL_CSV->to_csv();             $StorePathFileName='%ROOT%\data\nxlog_agents.log';         }else if ($ModuleName=="WinEventIn") {             WinEventNetIn_CSV->parse_csv();             WinEventFileOut_CSV->to_csv();             $StorePathFileName='%ServersLogStorePath%\'+$Department+'\'+$Section+'\'+ $SourceHostName + '\WinEventLog\'+ $Channel +'\' + strftime($EventTime,"%Y%m%d")+'_'+ $SourceHostName + '_' + $Channel+ '.csv';         }else if ($ModuleName=="WebLog" or $ModuleName=="TransLog" or $ModuleName=="OthersLog") {             LogKvp->parse_kvp();             $raw_event=$Message;             $StorePathFileName='%ServersLogStorePath%\'+$Department+'\'+$Section+'\'+ $Hostname + '\' + $ModuleName +'\' + $ParentDir + '\' + $FileName;         }else {             file_write('%ROOT%\data\debugModuleName.txt',$ModuleName +"\r\n");             $StorePathFileName='%ROOT%\data\UnknownLog.log';         }                  create_stat("stat", "RATE", 1); add_stat("stat", 1);     </Exec>     <Schedule>         Every   1 sec         Exec    log_info("EPS: " + get_stat("stat"));     </Schedule> </Input> #define Log Network Receive Buffer <Processor LogNetInBuffer>     Module      pm_buffer     # 4096MB buffer     MaxSize    4194304     Type    Mem     # warn at 10MB     WarnLimit    10240     #<Schedule>     #    Every   10 sec     #    Exec    log_info("BufferSize: " + buffer_size() + ",BufferCount: " +  buffer_count());     #</Schedule> </Processor> #define Log Output File <Output LogStoreFile>      Module      om_file      CreateDir TRUE      File        $StorePathFileName </Output> <Route LogRoute>     Path   LogNetIn => LogNetInBuffer => LogStoreFile </Route> When I  use ultraedit text editor to open the file "20160126_VMNXLOGCLT_Security.csv". I found nxlog.log of ServerSide has as following error. 2016-01-27 17:23:29 ERROR failed to open \\apstor1\Eventlog\Servers\WinEventLog\Security\20160126_VMNXLOGCLT_Security.csv; The process cannot access the file because it is being used by another process.   2016-01-27 17:23:30 INFO EPS: 129 2016-01-27 17:23:31 INFO EPS: 124 2016-01-27 17:23:31 INFO EPS: 124 2016-01-27 17:23:33 INFO EPS: 129 2016-01-27 17:23:33 INFO EPS: 129 2016-01-27 17:23:34 ERROR failed to open \\apstor1\Eventlog\Servers\WinEventLog\Security\20160126_VMNXLOGCLT_Security.csv; The process cannot access the file because it is being used by another process.   2016-01-27 17:23:35 ERROR last message repeated 2 times 2016-01-27 17:23:35 INFO EPS: 126 The Server Side nxlog process will not output any file and it continuous consume memory, I must restart nxlog services that recover work. So, how do I solve this problem ? Regards  

Here is a sample event when using to_syslog_snare() in the nxlog.conf: <14>Jan 27 10:03:39 event_computer MSWinEventLog        1        Security        32630749        Wed Jan 27 10:03:39 2016        4624        Microsoft-Windows-Security-Auditing        N/A        N/A        Success Audit        event_computer        Logon                An account was successfully logged on.    Subject:   Security ID:  S-1-0-0   Account Name:  -   Account Domain:  -   Logon ID:  0x0    Logon Type:   3    Impersonation Level:  Impersonation    New Logon:   Security ID:  S-1-5-21-2705889813-1605608894-1661845433-43745   Account Name:  account_name   Account Domain:  account_domain   Logon ID:  0x23820B882   Logon GUID:  {00000000-0000-0000-0000-000000000000}    Process Information:   Process ID:  0x0   Process Name:  -    Network Information:   Workstation Name: workstation_name   Source Network Address: source_address   Source Port:  54241    Detailed Authentication Information:   Logon Process:  NtLmSsp    Authentication Package: NTLM   Transited Services: -   Package Name (NTLM only): NTLM V2   Key Length:  0    This event is generated when a logon session is created. It is generated on the computer that was accessed.    The subject fields indicate the account on the local system which requested the logon. This is most commonly a service such as the Server service, or a local process such as Winlogon.exe or Services.exe.    The logon type field indicates the kind of logon that occurred. The most common types are 2 (interactive) and 3 (network).    The New Logon fields indicate the account for whom the new logon was created, i.e. the account that was logged on.    The network fields indicate where a remote logon request originated. Workstation name is not always available and may be left blank in some cases.    The impersonation level field indicates the extent to which a process in the logon session can impersonate.    The authentication information fields provide detailed information about this specific logon request.   - Logon GUID is a unique identifier that can be used to correlate this event with a KDC event.   - Transited services indicate which intermediate services have participated in this logon request.   - Package name indicates which sub-protocol was used among the NTLM protocols.   - Key length indicates the length of the generated session key. This will be 0 if no session key was requested.        35284558 My issue is that I would NOT want to collect the "informational text" representing the event - in this case everything starting from the string "This event is generated---" all the way up until "--was requested." Before I go any deeper into this, let me state that in the logs of this format I call the "<14>Jan 27 10:03:39 event_computer MSWinEventLog        1        Security        32630749        Wed Jan 27 10:03:39 2016        4624        Microsoft-Windows-Security-Auditing        N/A        N/A        Success Audit        event_computer        Logon" portion of the whole log message the HEADER, and the rest is called MESSAGE. Putting it another way, I would like to forward the message using syslog in a format constructed according to the pseudocode below: parse fields from windows event /* e.g. SubjectUserName, LogonType, IpAddress, etc. */ /* print the header "as is" already in the to_syslog_snare() format, i.e. from "<14>---" until and including "---Logon" print HEADER /* e.g. event_time,event_computer,event_type,event_id,... */ for all fields parsed     print "'field_name=field_value'" /* e.g. SubjectUserName=value,LogonType=value,IpAddress=value,... /*      The reason I would like to do this is that the informational text, which gets appended to some Windows events (not all, it seems), takes a lot of space, and we do not really need this information text for anything. Another way to do this would be to statically list all the fields POSSIBLY found in an Windows event and construct the message that way, but this would often leave me with a lot of empty key-value pairs. THUS I would only like to print out those fields that were found in that specific log message while leaving out the informational message. I do acknowledge, though, that especially Application and System events might not contain most or any of the fields that are present in a Security log event. Take for example the following System log event: <14>Jan 27 11:09:21 event_computer MSWinEventLog        1        System        32633951        Wed Jan 27 11:09:21 2016        7036        Service Control Manager        N/A        N/A        Information        event_computer        N/A                The Remote Registry service entered the stopped state.        319889 In the example above, the "header" portion of the whole message only contains the string "The Remote Registry service entered the stopped state." I do hope, though, that the variable where this string is stored is actually the same that hosted the string "An account was successfully logged on.", which would mean that my approach in the pseudocode would still work (i.e. the array or list of fields that is iterated and printed would only contain one field. The HEADER portion of the field is exactly the same in all messages. The description of to_syslog_snare() in the nxlog documentation states: "Create a SNARE Syslog formatted log message in $raw_event. Uses the following fields to construct $raw_event: $EventTime, $Hostname, $SeverityValue, $FileName, $EventID, $SourceName, $AccountName, $AccountType, $EventType, $Category, $Message." Thus when reflecting back to what I said, it seems that what I call the HEADER includes all the fields from $EventTime to (and including) $Category - this I would like to keep as it is. But according to the documentation, the $Message variable actually then holds all the other information in the log, or what I call the MESSAGE portion. So I guess the question is that can the contents of the $Message variable be further filtered, as it obviously is constructed from e.g. EventData's Data fields listed below. I would like to only change the $Message contents so that it would never contain the informational text if there exists such a message in a given log message, and that preferably the Data fields inside $Message would be formatted using key-value pairs instead of the to_syslog_snare format seen in the first example (one or more whitespace as delimiter).

Hello Everybody,  We are currently using nxlog to send Windows logs to QRadar SIEM utsing TLS. It works fine, but I receive extra lines in QRadar. I receive empty logs (containing Cg== on base 64, which seems to be a carriage return or a line break).  The problem appears only when using module om_ssl, not whith om_tcp or om_udp. We tried to remove line break or carriage return using nxlog configuration, but  we still have the behaviour.  Has anybody seen the same problem ? How could I correct this behaviour ?  You'll find below our current configuration   define ROOT C:\Program Files (x86)\nxlog Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data LogFile %ROOT%\data\nxlog.log <Extension syslog> Module xm_syslog </Extension> <Input internal>     Module    im_msvistalog     ReadFromLast TRUE </Input> <Output ssl_out>     Module    om_ssl     CertFile    C:\CERTDIR\syslog-tls.cert     CertKeyFile    C:\CERTDIR\syslog-tls.key     AllowUntrusted TRUE     Host    10.0.0.1     Port    6514     Exec    to_syslog_snare(); </Output> <Route 1>     Path    internal => ssl_out </Route>   Thank you !    

We have been using Snare Lite Open source for years, and understand it no longer meets PCI 3.0 compliance as it does not log enough event information. Snare Enterprise logs significantly more information, and would be PCI 3.0 compliant, but is expensive. I wondered if anyone here would know if the NXlog CE edition, which is free, logs MORE event info than the Snare Lite Open source version that we could use it to replace Snare Lite Open source,  and be confident it would comply with minimum PCI 3.0 requirements. I see the comparison of Snare lite open source with enterprise here to see the differences, but haven't found a similar chart for NXlog CE to compare. https://www.intersectalliance.com/why-snare-enterprise/  

Hi, I am suresh, i have installed nxlog agent on one of my Windows 2008 R2 print server and it transfer Application, Security & System Eventlogs to LogRhythm log collector.  My question is, will it also transfer the print logs (stored in Event Viewer -> Applications and Service Logs -> Microsoft -> Windows -> Print Service -> Operational) to my LogRhythm log collector? or not?

Hello nxlog world, Shamed to say, I've spent entire yesterday trying to figure out how to read Windows DHCP log files and ship the events to ElasticSearch. Problem was with using direct path for folder C:\Windows\System32\dhcp\. Managed to get nxlog to read by sharing the folders (read-only permissions) to the user account used for nxlog service account logon. As the events were not showing in ES, I'm stuck with trying to write the events into another file, in order to confirm that the source files are being read correctly.   OS: Win Srv 2008 R2 Ent nxlog: v 2.9.1347 Here is the nxlog.conf: define ROOT C:\Program Files (x86)\nxlog Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data LogFile %ROOT%\data\nxlog.log # debugging only: LogLevel DEBUG NoCache TRUE <Input msdhcp> Module im_file File '\\DC5\dhcp\DhcpSrvLog-*.log' SavePos TRUE InputType LineBased Exec if $raw_event =~ /^#/ drop(); Exec $message = $raw_event; </Input> <Input dns> Module im_file File '\\DC5\dns\dns.log' SavePos TRUE InputType LineBased Exec if $raw_event =~ /^#/ drop(); Exec $message = $raw_event; </Input> <Output file_test> Module om_file File 'C:\Program Files (x86)\nxlog\data\test_file_output.txt' # Sync TRUE OutputType LineBased </Output> <Route test> Path msdhcp,dns => file_test </Route> As a result, only DNS events are written in the output file: 21.1.2016. 11:34:00 A6A8 PACKET 0000000003B27E90 UDP Snd 192.168.105.12 3f0d R Q [8085 A DR NOERROR] A (8)PLANKING(3)lab(5)rador(0) 21.1.2016. 11:34:00 A6A8 PACKET 0000000003EDA2C0 UDP Rcv 192.168.105.12 3c32 Q [0001 D NOERROR] A (8)PLANKING(3)lab(5)rador(0) 21.1.2016. 11:34:00 A6A8 PACKET 0000000003EDA2C0 UDP Snd 192.168.105.12 3c32 R Q [8085 A DR NOERROR] A (8)PLANKING(3)lab(5)rador(0) ..but only the new ones, i.e. as the source DNS log file is being appended. I have tried modifying the SavePos parameter to FALSE of both input modules, but to no avail - same result.   Questions: 1. What would be the correct configuration of global NoCache and module specific SavePos parameters, in order to read and output the complete source file, regardless of prior attempts? 2. What is the reason DHCP logs (using wildcard) are not being read (or at least written in the output), as opposed to the same configuration for DNS logs?     I will provide nxlog debug level log if needed. No visible errors there.   Any help greatly appreciated!

Hi,  This question follow an old one: https://nxlog.co/question/1252/nxlog-omssl-infinite-reconnection-saas-log-centralizer We are trying to connect NXLog with Logmatic.io with a SSL connection. The certificate provided by Logmatic is still valid and works with Rsyslog, Syslog-NG or Open-SSL.. (you can find the certificate here : https://doc.logmatic.io/docs/logging-from-windows) However, no matter how hard we tried to connect NXLog we had no succes, and we are now stuck without any ideas of what can we do/test to make it work. Our config hasn't changed: <Output out> Module om_ssl Host api.logmatic.io Port 10515 CAFile <path_to_your_.crt_file> ####Add the API key before the event Exec $raw_event="<your_api_key> "+$raw_event; </Output> We tried a lot more configuration with AllowUntrusted TRUE/FALSE, CertFile, etc... But the result is always the same: 2016-01-06 16:19:56 INFO nxlog-ce-2.9.1504 started 2016-01-06 16:19:56 INFO connecting to api.logmatic.io:10515 2016-01-06 16:19:57 INFO successfully connected to api.logmatic.io:10515 2016-01-06 16:19:57 INFO reconnecting in 1 seconds 2016-01-06 16:19:57 ERROR SSL certificate verification failed: unable to get local issuer certificate (err: 20) 2016-01-06 16:19:58 INFO connecting to api.logmatic.io:10515 2016-01-06 16:19:58 INFO successfully connected to api.logmatic.io:10515 2016-01-06 16:19:58 INFO reconnecting in 1 seconds 2016-01-06 16:19:58 ERROR SSL certificate verification failed: unable to get local issuer certificate (err: 20) We wiresharked the connection to logmatic (which doesn't work) and the connection to Loggly (which worked) both with the same configuration file. https://drive.google.com/folderview?id=0B7GCHhnklusNcjgxM3ozeTZseVU&usp=sharing The only difference were in the configuration settings (host, port, certificate..) It seems that the Handshake between Nxlog and logmatic.io server is the root of this bug. Do you know why ?  Since it's Nxlog which refuse to extablish the connection, do you know why the option "AllowUntrusted false" in Nxlog.conf doesn't work ?  Sorry to bother you again with that but we tried everything and are now short of ideas :/ Many Thanks in advance !  

Hi, I recently started using Nxlog with Windows event log. I'm able to get events off of the Security log and onto my server, however, I've noticed something peculiar. The value of the 'ProcessID' field is always 4, no matter what process is generating the event. My nxlog.conf has the following sections: <Extension json> Module xm_json </Extension> <Input security_log> Module im_msvistalog SavePos TRUE ReadFromLast TRUE Channel Security Query * </Input> <Output out> Module om_tcp Host 10.1.11.50 Port 9000 Exec to_json(); </Output> <Route 1> Path security_log => out </Route> And the resulting output looks like: { "EventTime":"2016-01-18 10:32:13", "Hostname":"dev-collector.dev.local", "Keywords":-9214364837600034816, "EventType":"AUDIT_SUCCESS", "SeverityValue":2, "Severity":"INFO", "EventID":4689, "SourceName":"Microsoft-Windows-Security-Auditing", "ProviderGuid":"{54849625-5478-4994-A5BA-3E3B0328C30D}", "Version":0, "Task":13313, "OpcodeValue":0, "RecordNumber":355, "ProcessID":4, "ThreadID":48, "Channel":"Security", "Message":"A process has exited.\r\n\r\nSubject:\r\n\tSecurity ID:\t\tS-1-5-21-3503561835-834928460-339998367-1000\r\n\tAccount Name:\t\tachung\r\n\tAccount Domain:\t\tDEV-COLLECTOR\r\n\tLogon ID:\t\t0x727518c\r\n\r\nProcess Information:\r\n\tProcess ID:\t0xb64\r\n\tProcess Name:\tC:\\Windows\\System32\\calc.exe\r\n\tExit Status:\t0x0", "Category":"Process Termination", "Opcode":"Info", "SubjectUserSid":"S-1-5-21-3503561835-834928460-339998367-1000", "SubjectUserName":"achung", "SubjectDomainName":"DEV-COLLECTOR", "SubjectLogonId":"0x727518c", "Status":"0x0", "ProcessName":"C:\\Windows\\System32\\calc.exe", "EventReceivedTime":"2016-01-18 10:32:14", "SourceModuleName":"security_log", "SourceModuleType":"im_msvistalog" } The value of the "ProcessID" field is 4, when it should be 2916 (0xb64), as shown in the "Message" field. Is this a configuration issue or possibly a bug with one of the modules? Thanks, Allen

Hi!   I'm new to nxlog and try to find a way to clear the Windows eventlogs during or after read/export with nxlog. I've browsed through the manual, but can't find out how, is there a way to do this with a nxlog module/command? Regards, Mikael  

Hello - I am testing nxlog to send windows based syslogs to papertrail. The following code seems to produce the logs, however, the log header entries are logging with (date, external ip address, "logger:")  on each line. To be consistent with the other logs being collected, would like these headers to be (date, HOSTNAME of originator, Process originator) how can I change the header that nxlog is pushing out to papertrail? The information that I would like to populate the header is within the message body but am not clear how to populate the fields to the headers output? Here is an example of the current output line: Jan 11 10:25:30 159.88.217.201 logger:  good dog <14>1 2016-01-11T10:25:28.998751-05:00 Win8MacVM1 Microsoft-Windows-Kernel-General 3116 - [NXLOG@14506 Keywords="-9223372036854775792" EventType="INFO" EventID="1" ProviderGuid="{A68CA8B7-004F-D7B6-A698-07E2DE0F1F5D}" Version="1" Task="0" OpcodeValue="0" RecordNumber="33715" ThreadID="3480" Channel="System" Domain="NT AUTHORITY" AccountName="SYSTEM" UserID="SYSTEM" AccountType="User" Opcode="Info" NewTime="2016-01-11T15:25:28.998234400Z" OldTime="2016-01-11T15:25:28.998234400Z" Reason="3" EventReceivedTime="2016-01-11 10:25:30" SourceModuleName="in" SourceModuleType="im_msvistalog"] The system time has changed to ‎2016‎-‎01‎-‎11T15:25:28.998234400Z from ‎2016‎-‎01‎-‎11T15:25:28.998234400Z.    Change Reason: System time adjusted to the new time zone. Here is how I would like it to be: Jan 11 10:25:30 Win8MacVM1 Microsoft-Windows-Kernel-General   good dog <14>1 2016-01-11T10:25:28.998751-05:00 Win8MacVM1 Microsoft-Windows-Kernel-General 3116 - [NXLOG@14506 Keywords="-9223372036854775792" EventType="INFO" EventID="1" ProviderGuid="{A68CA8B7-004F-D7B6-A698-07E2DE0F1F5D}" Version="1" Task="0" OpcodeValue="0" RecordNumber="33715" ThreadID="3480" Channel="System" Domain="NT AUTHORITY" AccountName="SYSTEM" UserID="SYSTEM" AccountType="User" Opcode="Info" NewTime="2016-01-11T15:25:28.998234400Z" OldTime="2016-01-11T15:25:28.998234400Z" Reason="3" EventReceivedTime="2016-01-11 10:25:30" SourceModuleName="in" SourceModuleType="im_msvistalog"] The system time has changed to ‎2016‎-‎01‎-‎11T15:25:28.998234400Z from ‎2016‎-‎01‎-‎11T15:25:28.998234400Z.    Change Reason: System time adjusted to the new time zone. Any clues from the community on how to accomplish this? *** Code snippet to follow *** ## UDP Papertrail Test define ROOT C:\Program Files (x86)\nxlog define ROOT_STRING C:\Program Files (x86)\\nxlog    Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data LogFile %ROOT%\data\nxlog.log # Include fileop while debugging, also enable in the output module below <Extension fileop>     Module      xm_fileop </Extension>   <Extension syslog>     Module      xm_syslog </Extension>   # Enable json extension <Extension json>     Module      xm_json </Extension>   <Input in>     Module      im_msvistalog     Exec    to_json();          ReadFromLast TRUE     SavePos     TRUE     Query       <QueryList>\                     <Query Id="0">\                         <Select Path="Application">*</Select>\                         <Select Path="System">*</Select>\                         <Select Path="Security">*</Select>\                     </Query>\                 </QueryList>     </Input>   <Output out>    Module      om_udp    Host xxxx.papertrailapp.com     Port NNNNN #Test to write unique entry in front of each message but did not change the actual header    Exec to_syslog_ietf();\ $raw_event = " good dog "+$raw_event; #Use the following line for debugging (uncomment the fileop extension above as well)    Exec file_write("C:\\Program Files (x86)\\nxlog\data\\nxlog_output.log",  $raw_event); </Output> <Route 1>     Path        in => out </Route> *** Code snippet end ***

Hi, Where I can get nxlog agent package for CentOS 5 ?. Only packages available for verion 6 and 7 on download page. Could you please share the download link for the package that support centos 5.x ? 

Sorry for the activity, I hope this to be the last question - I've been playing with nxlog-ce for a week already. How can I get a count of (unintentionally) dropped records? Something syslog-ng reports in "mark" messages... Do I have to create a buffer processor, drop the records "intentionally" in it and implement my own counter in script? Or does nxlog exhibit somehow its internal flow control stats? Milan

Hi, please advise me how to configure the following setup: several inputs (UDP, TCP, internal) joined into common path (i_udp, i_tcp, i_internal => p_pattern ...), one global processor (pattern, filter)  module, several outputs (file, UDP, TCP), every message is delivered to one of outputs according to processor's decision. I call it "route branching". The solutions I have come to so far: "broadcast": p_pattern => o_file, o_udp, o_tcp -- every output having an Exec filter to drop() messages that should be sent via another output, "reroute": i_null => o_file; i_null => o_udp; i_null => o_tcp and use Exec reroute() in p_pattern. The (1) is very inefficient. Otoh, (2) introduces problems with flow control, as the flow control is disabled when using reroute() (a new "feature" of 2.9.1504). So is there an efficient way with flow control enabled? ;-) Milan 

Hello, will you, please, publish recent source code, the one the latest CE packages were built from? I made quite a few tests with 2.9.1504 on Debian and now would like to patch some minor problems of pm_pattern (and contributing back), but last published sources are 2.8.1248. And I can not get a changelog between theese two versions either (the changelog packaged in the binary package, as mentioned in the previous post, is too brief): nxlog-ce (2.9.1504) unstable; urgency=low SVN snapshot release. -- Botond Botyanszki <boti@nxlog.org> Tue, 08 Dec 2015 14:02:37 +0100 nxlog-ce (0.1-1) unstable; urgency=low Initial Release. -- Botond Botyanszki <boti@nxlog.org> Sat, 06 Nov 2010 18:24:10 +0200 Local variables: mode: debian-changelog End:  

Where can I find the NXLOG code history for the various releases? I am currently on NXLOG 2.9.1347 and need to know what's new and fixed in 2.9.1504 but cannot locate where this information is. Thank You,

Hi  I'm recently tried to use NXLog community edition to maintain log files. Basically the idea is to rewrite log files one to one from many services on many hosts to one centralized server.  In my test scenario I've created 3 services on one host and started propagate logs. Logs were rewriten succesfully but what I observed: - It took 40 minutes to rewrite 1GB of logs (1GB connection with 60MB transfer between hosts) - NXLog CPU usage on host where the logs were stored was about 10% (Intel Core i7) - In Sysinternals ProcessMonitor I've observed that for one data read (65000 bytes) 16:02:26,4218297 nxlog.exe 12884 ReadFile D:\app\Logs\service1.service\Logs_151130_12.log SUCCESS Offset: 490 945 000, Length: 65 000 there is about 450 partial data writes by 30/90/120 bytes which makes write process high ineffective 16:02:26,4219115 nxlog.exe 12884 LockFile \\192.168.199.46\Shared\logs\service1\Logs_151130_12.log SUCCESS Exclusive: True, Offset: 0, Length: 4 294 967 295, Fail Immediately: False 16:02:26,4219594 nxlog.exe 12884 QueryStandardInformationFile \\192.168.199.46\Shared\logs\service1\Logs_151130_12.log SUCCESS AllocationSize: 490 930 673, EndOfFile: 490 930 673, NumberOfLinks: 1, DeletePending: False, Directory: False 16:02:26,4219996 nxlog.exe 12884 WriteFile \\192.168.199.46\Shared\logs\service1\Logs_151130_12.log SUCCESS Offset: 490 930 673, Length: 84, Priority: Normal 16:02:26,4228112 nxlog.exe 12884 UnlockFileSingle \\192.168.199.46\Shared\logs\service1\Logs_151130_12.log SUCCESS Offset: 0, Length: 4 294 967 295 Is that lack of performance or do I have missed something. Usage of NXLog with such results is unacceptable for me. My config file: #define ROOT C:\Program Files\nxlog define ROOT C:\Program Files (x86)\nxlog Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data LogFile %ROOT%\data\nxlog.log define LOG_SERVER \\\\192.168.199.46 define LOGS_PATH d:\\App\\Logs\\\\*.log <Extension _syslog> Module xm_fileop Module xm_exec </Extension> <Input logs> Module im_file File "%LOGS_PATH%" InputType LineBased PollInterval 15 SavePos TRUE CloseWhenIdle TRUE Recursive TRUE Exec $fileName = file_basename(file_name()); Exec if file_name() =~ /^.*Logs\\(.*?)\.service/ \ { \ $serviceName = $1; \ } </Input> <Output out> Module om_file CreateDir TRUE OutputType LineBased File "%LOG_SERVER%\\Shared\\Logs\\" + $serviceName + "\\" + $fileName </Output> <Route 1> Path logs => out </Route>

Hi all, I tried to use the gelf extension with the tcp module, but it doesn't work. I saw tht the commercial version includes a special tcp gelf enxtension. Supports the community edition the usage of GELF over TCP? Kind regards Michael

Hello, nxlog was sending to nagios log server just fine for a few days and then all of a sudden we started seeing the below errors in the nxlog.log file that are preventing logs from being sent. ERROR om_tcp send failed; An existing connection was forcibly closed by the remote host. ERROR couldn't connect to tcp socket on xx.xx.xxx.xx:3515; No connection could be made because the target machine actively refused it. Is this something that you would support or would we need to go through Nagios support instead? We have already tried restarting the nxlog service but the issue remains.   Thank you