Dear all, I have a lot of text logs and I want to parse them with RegEx in NXlog comm.edition. Somehow I figured out how to parse them but I dont know how to put them in variables in the config file. In my example ; I parsed first 4 fields with regex like "date time ID product_code" and I have the regex statement. 01/29/2016  09:13:01.000000  1344 140334835169024     49  2          0  Target data state for connection 1 from ip://10.10.100.72 : 1500 has changed because a mirror has been stopped. What is the delimiter which is separating fields in regex ? My regex code is ; [0-2][0-9]\/[[1-2][0-9]\/[1-2]0[1-2][1-9]\s\s\w+:\w+:\w+\.\w+\s\s\w+\s\w+ I want to do something like this in my conf file thank you  $Time = $1;\                 $CStatus = $2;\                 $Process = $3;\                 $Process_result = $4;\  

I installed NXLOG onto our windows server. I setup INPUT to send the c:\squid\var\logs\access.log to our graylog server. I restarted the NXLOG service. On the graylog service, i still keep getting windows event log instead of the squid proxy logs. Has anyone encountered this before?  

Hello ,      I'm trying get specific data from some logs of hadoop with REGEX and I recieved this error: ERROR invalid keyword: Output at C:\Program Files (x86)\nxlog\conf\nxlog.conf:45       Here is my config file: define ROOT C:\Program Files (x86)\nxlog #  Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data LogFile %ROOT%\data\nxlog.log # <Extension gelf>     Module         xm_gelf </Extension> <Extension fileop>     Module         xm_fileop </Extension> <Extension json>     Module      xm_json </Extension> <Extension multi>     Module      xm_multiline     HeaderLine  /^(\d+-\d+-\d+\s\d+:\d+:\d+,\d+)/     EndLine        /(.*)/ </Extension> # <Input hadoop>   Module         im_file   File             "E:\\Hadoop\\test\\*.*"   SavePos         TRUE   Recursive     TRUE   InputType        multi      Exec      if $raw_event =~/^(\d+-\d+-\d+\s\d+:\d+:\d+,\d+)\s(?:INFO|ERROR|WARN)\s(org.apache.hadoop.\w+.\w+):\s(.*)/g\             {\                 $Time = $1;\                 $CStatus = $2;\                 $Process = $3;\                 $Process_result = $4;\                 to_json();\             }\             else\             {\                 drop();\             }\ </Input> <Output graylog>     Module      om_udp     Host        10.101.78.224     Port        12201     OutputType    GELF       #Use the following line for debugging (uncomment the fileop extension above as well)     #Exec file_write("C:\\Program Files (x86)\\nxlog\\data\\nxlog_output.log", $raw_event); </Output> <Route eventlog>     Path        hadoop => graylog </Route> Anyone know what is bad in this config file?. THank you.

There is bug in im_msvistalog.c (around line 560): if (ConvertSidToStringSid(imconf->renderbuf[EvtSystemUserID].SidVal, &sidstr)) <p><span style="font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; line-height: 15.2727px; white-space: pre;">{</span></p> <p><span class="pl-c1" style="box-sizing: border-box; color: rgb(0, 134, 179); font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; line-height: 15.2727px; white-space: pre;">nx_logdata_set_string</span><span style="font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; line-height: 15.2727px; white-space: pre;">(logdata, </span><span class="pl-s" style="box-sizing: border-box; color: rgb(24, 54, 145); font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; line-height: 15.2727px; white-space: pre;"><span class="pl-pds" style="box-sizing: border-box;">"</span>UserID<span class="pl-pds" style="box-sizing: border-box;">"</span></span><span style="font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; line-height: 15.2727px; white-space: pre;">, user); </span><span style="color: rgb(255, 0, 0); font-family: 'Open Sans', Arial, sans-serif; font-size: 14px; font-weight: bold; line-height: 20px;">&lt;&lt;&lt; There Should be sidstr instead of user</span></p> <p><span class="pl-c1" style="box-sizing: border-box; color: rgb(0, 134, 179); font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; line-height: 15.2727px; white-space: pre;">LocalFree</span><span style="font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; line-height: 15.2727px; white-space: pre;">(sidstr);</span></p> <p><span style="font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; line-height: 15.2727px; white-space: pre;">}</span></p> <p>&nbsp;</p> </td> </tr> <tr> <td class="blob-code blob-code-inner js-file-line" style="box-sizing: border-box; padding: 0px 10px; position: relative; vertical-align: top; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 12px; white-space: pre; overflow: visible; word-wrap: normal;">&nbsp;</td> </tr> </tbody>

We are using Windows event collector which is pulling in from over 400 hundred servers. We have configured both disk and memory buffers and looks like nxlog peaks at 2GB memory and then starts to crash and no longer sends logs. I am seeing the following messages in the nxlog log. When using mem only buffer 2016-01-29 17:46:52 ERROR EvtNext failed with error 14: Not enough storage is available to complete this operation.   2016-01-29 17:46:52 ERROR EvtUpdateBookmark failed: The handle is invalid.   2016-01-29 17:46:52 ERROR EvtNext failed with error 14: Not enough storage is available to complete this operation.   2016-01-29 17:46:52 ERROR EvtUpdateBookmark failed: The handle is invalid.   2016-01-29 17:46:52 ERROR EvtCreateRenderContext failed; Not enough storage is available to complete this operation. I've adjusted the buffer to use both disk and mem and now getting this...  2016-01-29 18:03:51 ERROR couldn't connect to tcp socket on IP:3515; An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was full.   Did we reach some sort of limitation or perhaps just too much logs incoming for it to handle? Using version: nxlog-ce-2.9.1504  

Hello, is there any guideline for install the community edition in AIX?

Hello everybody, I'm trying to collect Doubletake logs from a Centos 6.x server. I have that kind of logs ; 01/31/2016  21:15:26.000000 25786 140456885212928     48  2          0  Ops Skipped:        308390 01/31/2016  21:15:26.000000 25786 140456885212928     49  2          0  Total Mirror Ops:   321057 01/31/2016  21:15:26.000000 25786 140456885212928     50  2          0  Elapsed Time:          2318 seconds 01/31/2016  21:15:26.000000 25786 140456885212928     51  2          0  Paused Time:          0 seconds 01/31/2016  21:15:26.000000 25786 140456885212928     52  2          0  Total number of pauses:          0 01/31/2016  21:44:31.000000 25786 140456771974912     53  2         77  Connection lost with IP address ip://127.0.0.1 : 1501 01/31/2016  21:44:32.000000 25786 140457057756928     54  2     700000  Server Monitor was successfully stopped 01/31/2016  21:44:32.000000 25786 140457057756928     55  2      51503  Source module Stopped 01/31/2016  21:44:32.000000 25786 140457057756928     56  2      52503  Stopping all targets 01/31/2016  21:44:32.000000 25786 140457057756928     57  2      52503  Target module Stopped 00/00/0000  00:00:00.0000 Start of logfile  00/00/0000  00:00:00.0000 Application starting 01/31/2016  21:44:44.000000  9976 139690482448128      1  2          0  Buffer allocator limit is: 67108864 bytes 01/31/2016  21:44:44.000000  9976 139690482448128      2  2          0  QMemoryBufferMax size is: 268435456 bytes 01/31/2016  21:44:45.000000  9976 139690482448128      3  2          0  ActivationCode is valid: "6uvb-kyqa-tgar-wpeu-0t52-ubuv". 01/31/2016  21:44:45.000000  9976 139690482448128      4  2          0  Source Failover is allowed 01/31/2016  21:44:45.000000  9976 139690482448128      5  2          0  Target Failover is allowed 01/31/2016  21:44:45.000000  9976 139690482448128      6  2          0  Source Full Server Failover is allowed 01/31/2016  21:44:45.000000  9976 139690482448128      7  2          0  Target Full Server Failover is allowed 01/31/2016  21:44:45.000000  9976 139690482448128      8  2          0  Source Replication is allowed 01/31/2016  21:44:45.000000  9976 139690482448128      9  2          0  Target Replication is allowed 01/31/2016  21:44:45.000000  9976 139690482448128     10  2          0  Heartbeat Transmission started on port 1500 (interval=3seconds) 01/31/2016  21:44:48.000000  9976 139690482448128     11  2         69  Kernel Started on bl-db01.marsathletic.com  ip://10.10.100.75 : 1500  Version: 7.1.1.1255.0 01/31/2016  21:44:48.000000  9976 139690482448128     12  2     504002  Double-Take has successfully found /dev/dtrep0 01/31/2016  21:44:48.000000  9976 139690482448128     13  2      52501  Target module loaded successfully 01/31/2016  21:44:48.000000  9976 139690482448128     14  2          0  Disabling all replication from the driver 01/31/2016  21:44:48.000000  9976 139690482448128     15  2          0  Returning default addr: 10.10.100.75 : 1500 01/31/2016  21:44:48.000000  9976 139690482448128     16  2         71  Originator Attempting ip://10.10.150.10 : 1500 01/31/2016  21:44:48.000000  9976 139689629570816     17  2         73  Connected to  IP address ip://10.10.150.10 : 1500 01/31/2016  21:44:48.000000  9976 139690413819648     18  2         75  Connection resumed with IP address ip://10.10.150.10 : 1500 01/31/2016  21:44:48.000000  9976 139690482448128     19  2         80  Auto-reconnecting Lvra_0d341950-6cba-412c-834d-8afa3426cc83 to ip://10.10.150.10 : 1500::/var/lib/mysql/ -> /opt/dbtk/mnt/.job-0d341950-6cba-412c-834d-8afa3426cc83/var/lib/mysql/;/boot/ -> /opt/dbtk/mnt/.job-0d341950-6cba-412c-834d-8afa3426cc83/boot/;/ -> /opt/dbtk/mnt/.job-0d341950-6cba-412c-834d-8afa3426cc83/; 01/31/2016  21:44:48.000000  9976 139690482448128     20  2        800  Transmission manually resumed by client 01/31/2016  21:44:48.000000  9976 139690482448128     21  2          0  Returning default addr: 10.10.100.75 : 1500 01/31/2016  21:44:48.000000  9976 139690482448128     22  2         87  Starting replication of set Lvra_0d341950-6cba-412c-834d-8afa3426cc83 for connection 1 01/31/2016  21:44:48.000000  9976 139690482448128     23  2          0  Activating replication on / 01/31/2016  21:44:48.000000  9976 139690482448128     24  2          0  Activating replication on /boot 01/31/2016  21:44:48.000000  9976 139690482448128     25  2          0  Activating replication on /var/lib/mysql 01/31/2016  21:44:48.000000  9976 139690482448128     26  2          0  Disabling replication on /var/log/DT 01/31/2016  21:44:48.000000  9976 139690482448128     27  2          0  Disabling replication on /var/cache/DT 01/31/2016  21:44:48.000000  9976 139690482448128     28  2     500000  Starting a connection for a Linux Virtual Recovery job. 01/31/2016  21:44:48.000000  9976 139690482448128     29  2     500000  Lvra_0d341950-6cba-412c-834d-8afa3426cc83 is connected to ip://10.10.150.10 : 1500::/var/lib/mysql/ -> /opt/dbtk/mnt/.job-0d341950-6cba-412c-834d-8afa3426cc83/var/lib/mysql/;/boot/ -> /opt/dbtk/mnt/.job-0d341950-6cba-412c-834d-8afa3426cc83/boot/;/ -> /opt/dbtk/mnt/.job-0d341950-6cba-412c-834d-8afa3426cc83/; using compression level 1 (1) 01/31/2016  21:44:48.000000  9976 139690482448128     30  2     500000  Auto-Reconnect success. ConID = 1 01/31/2016  21:44:48.000000  9976 139690482448128     31  2      51501  Source module loaded successfully 01/31/2016  21:44:48.000000  9976 139690482448128     32  2          0  Detected RedHat configuration for failover persistence. 01/31/2016  21:44:53.000000  9976 139690311149312     33  2         72  Connection request from IP address 10.10.150.10 01/31/2016  21:44:53.000000  9976 139690311149312     34  2      99001  Telling peer IP: ip://10.10.150.10 : 1500 that conditions are OK to proceed. 01/31/2016  21:45:01.000000  9976 139690269189888     35  2        800  Local connection accepted, spinning up new local listen thread 01/31/2016  21:45:01.000000  9976 139690269189888     36  2         72  Responding to request from IP address 127.0.0.1 : 1502 using 127.0.0.1 : 1501 01/31/2016  21:45:01.000000  9976 139690269189888     37  2     600002  User :lms: has FULL access (2) 01/31/2016  21:45:04.000000  9976 139689361135360     38  2          0  Repset contains 41751738977 byte(s) to mirror 01/31/2016  21:45:04.000000  9976 139689361135360     39  2          0  Repset requires 324078 ops to mirror 01/31/2016  21:45:04.000000  9976 139689361135360     40  2         94  Delete Orphans Started <1> 01/31/2016  21:45:04.000000  9976 139689361135360     41  2         89  Mirror Started, Differences, Block Checksum <1> Nxlog is able to send the logs to Graylog via GELF UDP but I cant display the complete line on Graylog. All I can see is a cutted piece of line like ; 01/31/2016  21:15:26.000000 25786 140456885212928 I cant display more than this. I will be very happy if somebody can help to solve this. Thank you. Mehmet   Here is my nxlog.conf ######################################## # Global directives                    # ######################################## User root Group nxlog LogFile /var/log/nxlog/nxlog.log LogLevel INFO ######################################## # Modules                              # ######################################## <Extension gelf>     Module      xm_gelf </Extension> <Input Doubletake>     Module      im_file     File        "/tmp/2.log"     SavePos     TRUE </Input> <Output graylog_out>     Module      om_udp     Host        192.168.2.94     Port        12201     OutputType  GELF </Output> ######################################## # Routes                               # ######################################## <Route 1>     Path        Doubletake => graylog_out </Route>  

Scenario: I have NXLog EE installed on a host in Windows domain. I need to read DHCP logs from the DC(s), UNC path: \\<server name>\C$\Windows\System32\dhcp\DhcpSrvLog-*.log Since it is not possible to specify alternate credentials for accessing remote files (as it is for eventlog, i.e. im_msvistalog module), nxlog has to be started using an account with special privileges on the DC's file system - 4 options:  1. for nxlog service, use domain admin account (local admin role does not exist on DC)      - nxlog.conf - use UNC path: `\\<server name>\C$\Windows\System32\dhcp\DhcpSrvLog-*.log`  2. for nxlog service, use local admin account on the agent's host + share C:\Windows\System32\dhcp\ on the DC, enabling read only permissions for nxlog account only      - nxlog.conf - use share name: `\\<server name>\dhcp\DhcpSrvLog-*.log`  3. install nxlog agent on the DC, run nxlog as a service, use local admin account  4. smaller footprint ? -> install http://nxlog-ce.sourceforge.net/nxlog-docs/en/nxlog-reference-manual.html#nxlog_processor on the DC None of these options are win-wins for customer production environment, as they require opening the restricted environment of the DC. My question is: are there any nxlog configuration options, which would enable me to fetch the file remotely, similar to these for DC's Security event log?: <Input dc1> Module im_msvistalog RemoteServer <ip> Remoteuser <user> RemotePassword <pwn> RemoteDomain <domain> Query <QueryList><Query Id="0" Path="Security"><Select Path="Security">*</Select></Query></QueryList> </Input>

Sorry,my english is not good. I occur a problem with om_file module. my config as following: #define Log receive from Network <Input LogNetIn>     Module      im_tcp     host        0.0.0.0     port        1514     <Exec>         if $raw_event =~ /^"([^"]+)"/ {             $ModuleName=$1 ;         } else if $raw_event =~ /ModuleName=([^\d]+)/{             $ModuleName=$1 ;         } else {             file_write('%ROOT%\data\debug.txt',$raw_event +"\r\n");         }         if ($ModuleName=="HB") {             HB_IL_CSV->parse_csv();             HB_CSV->to_csv();             $StorePathFileName='%ROOT%\data\heartbeat.log';         }else if ($ModuleName=="IL") {             HB_IL_CSV->parse_csv();             IL_CSV->to_csv();             $StorePathFileName='%ROOT%\data\nxlog_agents.log';         }else if ($ModuleName=="WinEventIn") {             WinEventNetIn_CSV->parse_csv();             WinEventFileOut_CSV->to_csv();             $StorePathFileName='%ServersLogStorePath%\'+$Department+'\'+$Section+'\'+ $SourceHostName + '\WinEventLog\'+ $Channel +'\' + strftime($EventTime,"%Y%m%d")+'_'+ $SourceHostName + '_' + $Channel+ '.csv';         }else if ($ModuleName=="WebLog" or $ModuleName=="TransLog" or $ModuleName=="OthersLog") {             LogKvp->parse_kvp();             $raw_event=$Message;             $StorePathFileName='%ServersLogStorePath%\'+$Department+'\'+$Section+'\'+ $Hostname + '\' + $ModuleName +'\' + $ParentDir + '\' + $FileName;         }else {             file_write('%ROOT%\data\debugModuleName.txt',$ModuleName +"\r\n");             $StorePathFileName='%ROOT%\data\UnknownLog.log';         }                  create_stat("stat", "RATE", 1); add_stat("stat", 1);     </Exec>     <Schedule>         Every   1 sec         Exec    log_info("EPS: " + get_stat("stat"));     </Schedule> </Input> #define Log Network Receive Buffer <Processor LogNetInBuffer>     Module      pm_buffer     # 4096MB buffer     MaxSize    4194304     Type    Mem     # warn at 10MB     WarnLimit    10240     #<Schedule>     #    Every   10 sec     #    Exec    log_info("BufferSize: " + buffer_size() + ",BufferCount: " +  buffer_count());     #</Schedule> </Processor> #define Log Output File <Output LogStoreFile>      Module      om_file      CreateDir TRUE      File        $StorePathFileName </Output> <Route LogRoute>     Path   LogNetIn => LogNetInBuffer => LogStoreFile </Route> When I  use ultraedit text editor to open the file "20160126_VMNXLOGCLT_Security.csv". I found nxlog.log of ServerSide has as following error. 2016-01-27 17:23:29 ERROR failed to open \\apstor1\Eventlog\Servers\WinEventLog\Security\20160126_VMNXLOGCLT_Security.csv; The process cannot access the file because it is being used by another process.   2016-01-27 17:23:30 INFO EPS: 129 2016-01-27 17:23:31 INFO EPS: 124 2016-01-27 17:23:31 INFO EPS: 124 2016-01-27 17:23:33 INFO EPS: 129 2016-01-27 17:23:33 INFO EPS: 129 2016-01-27 17:23:34 ERROR failed to open \\apstor1\Eventlog\Servers\WinEventLog\Security\20160126_VMNXLOGCLT_Security.csv; The process cannot access the file because it is being used by another process.   2016-01-27 17:23:35 ERROR last message repeated 2 times 2016-01-27 17:23:35 INFO EPS: 126 The Server Side nxlog process will not output any file and it continuous consume memory, I must restart nxlog services that recover work. So, how do I solve this problem ? Regards  

Here is a sample event when using to_syslog_snare() in the nxlog.conf: <14>Jan 27 10:03:39 event_computer MSWinEventLog        1        Security        32630749        Wed Jan 27 10:03:39 2016        4624        Microsoft-Windows-Security-Auditing        N/A        N/A        Success Audit        event_computer        Logon                An account was successfully logged on.    Subject:   Security ID:  S-1-0-0   Account Name:  -   Account Domain:  -   Logon ID:  0x0    Logon Type:   3    Impersonation Level:  Impersonation    New Logon:   Security ID:  S-1-5-21-2705889813-1605608894-1661845433-43745   Account Name:  account_name   Account Domain:  account_domain   Logon ID:  0x23820B882   Logon GUID:  {00000000-0000-0000-0000-000000000000}    Process Information:   Process ID:  0x0   Process Name:  -    Network Information:   Workstation Name: workstation_name   Source Network Address: source_address   Source Port:  54241    Detailed Authentication Information:   Logon Process:  NtLmSsp    Authentication Package: NTLM   Transited Services: -   Package Name (NTLM only): NTLM V2   Key Length:  0    This event is generated when a logon session is created. It is generated on the computer that was accessed.    The subject fields indicate the account on the local system which requested the logon. This is most commonly a service such as the Server service, or a local process such as Winlogon.exe or Services.exe.    The logon type field indicates the kind of logon that occurred. The most common types are 2 (interactive) and 3 (network).    The New Logon fields indicate the account for whom the new logon was created, i.e. the account that was logged on.    The network fields indicate where a remote logon request originated. Workstation name is not always available and may be left blank in some cases.    The impersonation level field indicates the extent to which a process in the logon session can impersonate.    The authentication information fields provide detailed information about this specific logon request.   - Logon GUID is a unique identifier that can be used to correlate this event with a KDC event.   - Transited services indicate which intermediate services have participated in this logon request.   - Package name indicates which sub-protocol was used among the NTLM protocols.   - Key length indicates the length of the generated session key. This will be 0 if no session key was requested.        35284558 My issue is that I would NOT want to collect the "informational text" representing the event - in this case everything starting from the string "This event is generated---" all the way up until "--was requested." Before I go any deeper into this, let me state that in the logs of this format I call the "<14>Jan 27 10:03:39 event_computer MSWinEventLog        1        Security        32630749        Wed Jan 27 10:03:39 2016        4624        Microsoft-Windows-Security-Auditing        N/A        N/A        Success Audit        event_computer        Logon" portion of the whole log message the HEADER, and the rest is called MESSAGE. Putting it another way, I would like to forward the message using syslog in a format constructed according to the pseudocode below: parse fields from windows event /* e.g. SubjectUserName, LogonType, IpAddress, etc. */ /* print the header "as is" already in the to_syslog_snare() format, i.e. from "<14>---" until and including "---Logon" print HEADER /* e.g. event_time,event_computer,event_type,event_id,... */ for all fields parsed     print "'field_name=field_value'" /* e.g. SubjectUserName=value,LogonType=value,IpAddress=value,... /*      The reason I would like to do this is that the informational text, which gets appended to some Windows events (not all, it seems), takes a lot of space, and we do not really need this information text for anything. Another way to do this would be to statically list all the fields POSSIBLY found in an Windows event and construct the message that way, but this would often leave me with a lot of empty key-value pairs. THUS I would only like to print out those fields that were found in that specific log message while leaving out the informational message. I do acknowledge, though, that especially Application and System events might not contain most or any of the fields that are present in a Security log event. Take for example the following System log event: <14>Jan 27 11:09:21 event_computer MSWinEventLog        1        System        32633951        Wed Jan 27 11:09:21 2016        7036        Service Control Manager        N/A        N/A        Information        event_computer        N/A                The Remote Registry service entered the stopped state.        319889 In the example above, the "header" portion of the whole message only contains the string "The Remote Registry service entered the stopped state." I do hope, though, that the variable where this string is stored is actually the same that hosted the string "An account was successfully logged on.", which would mean that my approach in the pseudocode would still work (i.e. the array or list of fields that is iterated and printed would only contain one field. The HEADER portion of the field is exactly the same in all messages. The description of to_syslog_snare() in the nxlog documentation states: "Create a SNARE Syslog formatted log message in $raw_event. Uses the following fields to construct $raw_event: $EventTime, $Hostname, $SeverityValue, $FileName, $EventID, $SourceName, $AccountName, $AccountType, $EventType, $Category, $Message." Thus when reflecting back to what I said, it seems that what I call the HEADER includes all the fields from $EventTime to (and including) $Category - this I would like to keep as it is. But according to the documentation, the $Message variable actually then holds all the other information in the log, or what I call the MESSAGE portion. So I guess the question is that can the contents of the $Message variable be further filtered, as it obviously is constructed from e.g. EventData's Data fields listed below. I would like to only change the $Message contents so that it would never contain the informational text if there exists such a message in a given log message, and that preferably the Data fields inside $Message would be formatted using key-value pairs instead of the to_syslog_snare format seen in the first example (one or more whitespace as delimiter).

Hello Everybody,  We are currently using nxlog to send Windows logs to QRadar SIEM utsing TLS. It works fine, but I receive extra lines in QRadar. I receive empty logs (containing Cg== on base 64, which seems to be a carriage return or a line break).  The problem appears only when using module om_ssl, not whith om_tcp or om_udp. We tried to remove line break or carriage return using nxlog configuration, but  we still have the behaviour.  Has anybody seen the same problem ? How could I correct this behaviour ?  You'll find below our current configuration   define ROOT C:\Program Files (x86)\nxlog Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data LogFile %ROOT%\data\nxlog.log <Extension syslog> Module xm_syslog </Extension> <Input internal>     Module    im_msvistalog     ReadFromLast TRUE </Input> <Output ssl_out>     Module    om_ssl     CertFile    C:\CERTDIR\syslog-tls.cert     CertKeyFile    C:\CERTDIR\syslog-tls.key     AllowUntrusted TRUE     Host    10.0.0.1     Port    6514     Exec    to_syslog_snare(); </Output> <Route 1>     Path    internal => ssl_out </Route>   Thank you !    

We have been using Snare Lite Open source for years, and understand it no longer meets PCI 3.0 compliance as it does not log enough event information. Snare Enterprise logs significantly more information, and would be PCI 3.0 compliant, but is expensive. I wondered if anyone here would know if the NXlog CE edition, which is free, logs MORE event info than the Snare Lite Open source version that we could use it to replace Snare Lite Open source,  and be confident it would comply with minimum PCI 3.0 requirements. I see the comparison of Snare lite open source with enterprise here to see the differences, but haven't found a similar chart for NXlog CE to compare. https://www.intersectalliance.com/why-snare-enterprise/  

Hi, I am suresh, i have installed nxlog agent on one of my Windows 2008 R2 print server and it transfer Application, Security & System Eventlogs to LogRhythm log collector.  My question is, will it also transfer the print logs (stored in Event Viewer -> Applications and Service Logs -> Microsoft -> Windows -> Print Service -> Operational) to my LogRhythm log collector? or not?

Hello nxlog world, Shamed to say, I've spent entire yesterday trying to figure out how to read Windows DHCP log files and ship the events to ElasticSearch. Problem was with using direct path for folder C:\Windows\System32\dhcp\. Managed to get nxlog to read by sharing the folders (read-only permissions) to the user account used for nxlog service account logon. As the events were not showing in ES, I'm stuck with trying to write the events into another file, in order to confirm that the source files are being read correctly.   OS: Win Srv 2008 R2 Ent nxlog: v 2.9.1347 Here is the nxlog.conf: define ROOT C:\Program Files (x86)\nxlog Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data LogFile %ROOT%\data\nxlog.log # debugging only: LogLevel DEBUG NoCache TRUE <Input msdhcp> Module im_file File '\\DC5\dhcp\DhcpSrvLog-*.log' SavePos TRUE InputType LineBased Exec if $raw_event =~ /^#/ drop(); Exec $message = $raw_event; </Input> <Input dns> Module im_file File '\\DC5\dns\dns.log' SavePos TRUE InputType LineBased Exec if $raw_event =~ /^#/ drop(); Exec $message = $raw_event; </Input> <Output file_test> Module om_file File 'C:\Program Files (x86)\nxlog\data\test_file_output.txt' # Sync TRUE OutputType LineBased </Output> <Route test> Path msdhcp,dns => file_test </Route> As a result, only DNS events are written in the output file: 21.1.2016. 11:34:00 A6A8 PACKET 0000000003B27E90 UDP Snd 192.168.105.12 3f0d R Q [8085 A DR NOERROR] A (8)PLANKING(3)lab(5)rador(0) 21.1.2016. 11:34:00 A6A8 PACKET 0000000003EDA2C0 UDP Rcv 192.168.105.12 3c32 Q [0001 D NOERROR] A (8)PLANKING(3)lab(5)rador(0) 21.1.2016. 11:34:00 A6A8 PACKET 0000000003EDA2C0 UDP Snd 192.168.105.12 3c32 R Q [8085 A DR NOERROR] A (8)PLANKING(3)lab(5)rador(0) ..but only the new ones, i.e. as the source DNS log file is being appended. I have tried modifying the SavePos parameter to FALSE of both input modules, but to no avail - same result.   Questions: 1. What would be the correct configuration of global NoCache and module specific SavePos parameters, in order to read and output the complete source file, regardless of prior attempts? 2. What is the reason DHCP logs (using wildcard) are not being read (or at least written in the output), as opposed to the same configuration for DNS logs?     I will provide nxlog debug level log if needed. No visible errors there.   Any help greatly appreciated!

Hi,  This question follow an old one: https://nxlog.co/question/1252/nxlog-omssl-infinite-reconnection-saas-log-centralizer We are trying to connect NXLog with Logmatic.io with a SSL connection. The certificate provided by Logmatic is still valid and works with Rsyslog, Syslog-NG or Open-SSL.. (you can find the certificate here : https://doc.logmatic.io/docs/logging-from-windows) However, no matter how hard we tried to connect NXLog we had no succes, and we are now stuck without any ideas of what can we do/test to make it work. Our config hasn't changed: <Output out> Module om_ssl Host api.logmatic.io Port 10515 CAFile <path_to_your_.crt_file> ####Add the API key before the event Exec $raw_event="<your_api_key> "+$raw_event; </Output> We tried a lot more configuration with AllowUntrusted TRUE/FALSE, CertFile, etc... But the result is always the same: 2016-01-06 16:19:56 INFO nxlog-ce-2.9.1504 started 2016-01-06 16:19:56 INFO connecting to api.logmatic.io:10515 2016-01-06 16:19:57 INFO successfully connected to api.logmatic.io:10515 2016-01-06 16:19:57 INFO reconnecting in 1 seconds 2016-01-06 16:19:57 ERROR SSL certificate verification failed: unable to get local issuer certificate (err: 20) 2016-01-06 16:19:58 INFO connecting to api.logmatic.io:10515 2016-01-06 16:19:58 INFO successfully connected to api.logmatic.io:10515 2016-01-06 16:19:58 INFO reconnecting in 1 seconds 2016-01-06 16:19:58 ERROR SSL certificate verification failed: unable to get local issuer certificate (err: 20) We wiresharked the connection to logmatic (which doesn't work) and the connection to Loggly (which worked) both with the same configuration file. https://drive.google.com/folderview?id=0B7GCHhnklusNcjgxM3ozeTZseVU&usp=sharing The only difference were in the configuration settings (host, port, certificate..) It seems that the Handshake between Nxlog and logmatic.io server is the root of this bug. Do you know why ?  Since it's Nxlog which refuse to extablish the connection, do you know why the option "AllowUntrusted false" in Nxlog.conf doesn't work ?  Sorry to bother you again with that but we tried everything and are now short of ideas :/ Many Thanks in advance !  

Hi, I recently started using Nxlog with Windows event log. I'm able to get events off of the Security log and onto my server, however, I've noticed something peculiar. The value of the 'ProcessID' field is always 4, no matter what process is generating the event. My nxlog.conf has the following sections: <Extension json> Module xm_json </Extension> <Input security_log> Module im_msvistalog SavePos TRUE ReadFromLast TRUE Channel Security Query * </Input> <Output out> Module om_tcp Host 10.1.11.50 Port 9000 Exec to_json(); </Output> <Route 1> Path security_log => out </Route> And the resulting output looks like: { "EventTime":"2016-01-18 10:32:13", "Hostname":"dev-collector.dev.local", "Keywords":-9214364837600034816, "EventType":"AUDIT_SUCCESS", "SeverityValue":2, "Severity":"INFO", "EventID":4689, "SourceName":"Microsoft-Windows-Security-Auditing", "ProviderGuid":"{54849625-5478-4994-A5BA-3E3B0328C30D}", "Version":0, "Task":13313, "OpcodeValue":0, "RecordNumber":355, "ProcessID":4, "ThreadID":48, "Channel":"Security", "Message":"A process has exited.\r\n\r\nSubject:\r\n\tSecurity ID:\t\tS-1-5-21-3503561835-834928460-339998367-1000\r\n\tAccount Name:\t\tachung\r\n\tAccount Domain:\t\tDEV-COLLECTOR\r\n\tLogon ID:\t\t0x727518c\r\n\r\nProcess Information:\r\n\tProcess ID:\t0xb64\r\n\tProcess Name:\tC:\\Windows\\System32\\calc.exe\r\n\tExit Status:\t0x0", "Category":"Process Termination", "Opcode":"Info", "SubjectUserSid":"S-1-5-21-3503561835-834928460-339998367-1000", "SubjectUserName":"achung", "SubjectDomainName":"DEV-COLLECTOR", "SubjectLogonId":"0x727518c", "Status":"0x0", "ProcessName":"C:\\Windows\\System32\\calc.exe", "EventReceivedTime":"2016-01-18 10:32:14", "SourceModuleName":"security_log", "SourceModuleType":"im_msvistalog" } The value of the "ProcessID" field is 4, when it should be 2916 (0xb64), as shown in the "Message" field. Is this a configuration issue or possibly a bug with one of the modules? Thanks, Allen

Hi!   I'm new to nxlog and try to find a way to clear the Windows eventlogs during or after read/export with nxlog. I've browsed through the manual, but can't find out how, is there a way to do this with a nxlog module/command? Regards, Mikael  

Hello - I am testing nxlog to send windows based syslogs to papertrail. The following code seems to produce the logs, however, the log header entries are logging with (date, external ip address, "logger:")  on each line. To be consistent with the other logs being collected, would like these headers to be (date, HOSTNAME of originator, Process originator) how can I change the header that nxlog is pushing out to papertrail? The information that I would like to populate the header is within the message body but am not clear how to populate the fields to the headers output? Here is an example of the current output line: Jan 11 10:25:30 159.88.217.201 logger:  good dog <14>1 2016-01-11T10:25:28.998751-05:00 Win8MacVM1 Microsoft-Windows-Kernel-General 3116 - [NXLOG@14506 Keywords="-9223372036854775792" EventType="INFO" EventID="1" ProviderGuid="{A68CA8B7-004F-D7B6-A698-07E2DE0F1F5D}" Version="1" Task="0" OpcodeValue="0" RecordNumber="33715" ThreadID="3480" Channel="System" Domain="NT AUTHORITY" AccountName="SYSTEM" UserID="SYSTEM" AccountType="User" Opcode="Info" NewTime="2016-01-11T15:25:28.998234400Z" OldTime="2016-01-11T15:25:28.998234400Z" Reason="3" EventReceivedTime="2016-01-11 10:25:30" SourceModuleName="in" SourceModuleType="im_msvistalog"] The system time has changed to ‎2016‎-‎01‎-‎11T15:25:28.998234400Z from ‎2016‎-‎01‎-‎11T15:25:28.998234400Z.    Change Reason: System time adjusted to the new time zone. Here is how I would like it to be: Jan 11 10:25:30 Win8MacVM1 Microsoft-Windows-Kernel-General   good dog <14>1 2016-01-11T10:25:28.998751-05:00 Win8MacVM1 Microsoft-Windows-Kernel-General 3116 - [NXLOG@14506 Keywords="-9223372036854775792" EventType="INFO" EventID="1" ProviderGuid="{A68CA8B7-004F-D7B6-A698-07E2DE0F1F5D}" Version="1" Task="0" OpcodeValue="0" RecordNumber="33715" ThreadID="3480" Channel="System" Domain="NT AUTHORITY" AccountName="SYSTEM" UserID="SYSTEM" AccountType="User" Opcode="Info" NewTime="2016-01-11T15:25:28.998234400Z" OldTime="2016-01-11T15:25:28.998234400Z" Reason="3" EventReceivedTime="2016-01-11 10:25:30" SourceModuleName="in" SourceModuleType="im_msvistalog"] The system time has changed to ‎2016‎-‎01‎-‎11T15:25:28.998234400Z from ‎2016‎-‎01‎-‎11T15:25:28.998234400Z.    Change Reason: System time adjusted to the new time zone. Any clues from the community on how to accomplish this? *** Code snippet to follow *** ## UDP Papertrail Test define ROOT C:\Program Files (x86)\nxlog define ROOT_STRING C:\Program Files (x86)\\nxlog    Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data LogFile %ROOT%\data\nxlog.log # Include fileop while debugging, also enable in the output module below <Extension fileop>     Module      xm_fileop </Extension>   <Extension syslog>     Module      xm_syslog </Extension>   # Enable json extension <Extension json>     Module      xm_json </Extension>   <Input in>     Module      im_msvistalog     Exec    to_json();          ReadFromLast TRUE     SavePos     TRUE     Query       <QueryList>\                     <Query Id="0">\                         <Select Path="Application">*</Select>\                         <Select Path="System">*</Select>\                         <Select Path="Security">*</Select>\                     </Query>\                 </QueryList>     </Input>   <Output out>    Module      om_udp    Host xxxx.papertrailapp.com     Port NNNNN #Test to write unique entry in front of each message but did not change the actual header    Exec to_syslog_ietf();\ $raw_event = " good dog "+$raw_event; #Use the following line for debugging (uncomment the fileop extension above as well)    Exec file_write("C:\\Program Files (x86)\\nxlog\data\\nxlog_output.log",  $raw_event); </Output> <Route 1>     Path        in => out </Route> *** Code snippet end ***

Hi, Where I can get nxlog agent package for CentOS 5 ?. Only packages available for verion 6 and 7 on download page. Could you please share the download link for the package that support centos 5.x ? 

Sorry for the activity, I hope this to be the last question - I've been playing with nxlog-ce for a week already. How can I get a count of (unintentionally) dropped records? Something syslog-ng reports in "mark" messages... Do I have to create a buffer processor, drop the records "intentionally" in it and implement my own counter in script? Or does nxlog exhibit somehow its internal flow control stats? Milan