Hello nxlog world. My application has a custom log file on a Windows 2012 box that I need to send to Graylog which has a GELF UDP input running. The log file is multiline and there is a specific header and footer that I'm seperating the log file entries with. As of right now despite using the multiline module, when my logs arrive in Graylog it is parsing each line of the log file individually instead of by header and footer. Here is my nxlog.conf: define ROOT C:\Program Files (x86)\nxlog Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data LogFile %ROOT%\data\nxlog.log <Extension gelf>     Module xm_gelf </Extension> <Extension multiline>     Module    xm_multiline         HeaderLine    /\d{4}-\d{2}-\d{2}\s\d*:\d\d:\d*,\d{3}\s[a-zA-Z]*\s\s\S\s\[Begin Lead\]/         EndLine        /\d{4}-\d{2}-\d{2}\s\d*:\d\d:\d*,\d{3}\s[a-zA-Z]*\s\s\S\s\[End Lead\]/ </Extension> <Extension json>     Module    xm_json </Extension> <Input im_msvistalog>     Module      im_msvistalog     Query   <QueryList>\             <Query Id="0">\                     <Select Path="System">*[System[(Level=1 or Level=2 or Level=3)]]</Select>\                     <Select Path="Application">*[System[(Level=1 or Level=2 or Level=3)]]</Select>\             </Query>\         </QueryList>     Exec if $raw_event =~ /(\d\d\d\d\-\d\d-\d\d \d\d:\d\d:\d\d)/ $EventTime = parsedate($1); </Input> <Input im_file>     Module   im_file     File "C:\\LOGS\\application_service.log"     SavePos FALSE      ReadFromLast FALSE     InputType multiline </Input> <Output om_udp>     Module      om_udp     Host        10.25.32.21     Port        12201     OutputType  GELF </Output> <Route 1>     Path    im_msvistalog  => om_udp </Route> <Route 2>     Path im_file => om_udp </Route> The application_service.log file looks like this: 2017-06-20 17:25:50,168 INFO  – [Begin Lead] 2017-06-20 17:25:50,168 INFO  – Getting lead by id '5551212' 2017-06-20 17:25:51,215 INFO  – Lead field = First Name. Value = Michael 2017-06-20 17:25:54,778 INFO  – Lead field = Last Name. Value = Westin 2017-06-20 17:25:54,793 INFO  – Lead field = CustomerNumber. Value = 99999 2017-06-20 17:26:05,887 INFO  – [End Lead] 2017-06-20 18:25:50,168 INFO  – [Begin Lead] 2017-06-20 18:25:50,168 INFO  – Getting lead by id '4279551' 2017-06-20 18:25:51,215 INFO  – Lead field = First Name. Value = Emmit 2017-06-20 18:25:54,778 INFO  – Lead field = Last Name. Value = Stussy 2017-06-20 18:25:54,793 INFO  – Lead field = CustomerNumber. Value = 94231 2017-06-20 18:26:05,887 INFO  – [End Lead] 2017-06-20 19:25:50,168 INFO  – [Begin Lead] 2017-06-20 19:25:50,168 INFO  – Getting lead by id '0081227' 2017-06-20 19:25:51,215 INFO  – Lead field = First Name. Value = Saul 2017-06-20 19:25:54,778 INFO  – Lead field = Last Name. Value = Goodman 2017-06-20 19:25:54,793 INFO  – Lead field = CustomerNumber. Value = 33487 2017-06-20 19:26:05,887 INFO  – [End Lead] Ideally when it his Graylog I'd like the content between [Begin Lead] and [End Lead] to be stored in one field.  But with my nxlog config above it is putting each line of the log file in its own message ID in Graylog.  Do I have my config wrong?  Is there a better method to parse the log file and send to Graylog?  Any help is appreciated.

Hi, Using two tables in mySQL: table1 and table2 with same structure.  Field          Type          Length        Key -------------------------------------------------------- id               INT            11               Primary text1          CHAR        20 number1    INT            11 Initial situation: Content of table1, table2 is empty  Table1                            table2 --------------------------       --------------------- 1    string1        100        <no records> 2    string2        200 3    string3        300     My nxlog.conf: define ROOT C:\Program Files (x86)\nxlog define CERT %ROOT%\cert Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data LogFile %ROOT%\data\nxlog.log #PersistLogqueue    TRUE #SyncLogqueue        TRUE CacheFlushInterval    always CacheSync        TRUE #LogLevel  DEBUG <Input    testIn>     Module  im_odbc #    Pollinterval  60     ConnectionString  DSN=mySQL;uid=***;pwd=***;database=nxlog     SQL   SELECT  id as vid ,\                   text1 as vtext1 ,\                   number1 as vnumber1 \           FROM table1 where id > ?     SavePos TRUE </Input>               <Output     testOut>         Module        om_odbc         ConnectionString DSN=mySQL;uid=***;pwd=***;database=nxlog         <Exec>           if (sql_exec("INSERT INTO table2(text1,number1) VALUES (?,?)",$vtext1,$vnumber1)== TRUE) {};         </Exec> </Output> <Route 1>     Path testIn => testOut </Route> After I start nxlog service both tables look like this. Table1                                     table2 -------------------                        ------------------- 1    string1        100                1    string1        100 2    string2        200                2    string2        200 3    string3        300                3    string3        300 So far so good. But when I manualy add a record to table1 both tables look like this: Table1                                    table2 -------------------                        ------------------- 1    string1        100                1    string1        100 2    string2        200                2    string2        200 3    string3        300                3    string3        300 4    string4        400                4    string3        400 As you can see in table2: the string has the value of record3. Number is OK. When I keep adding records to table1 it keeps on taking the string of record 3 but number is correct. After restart of nxlog service and manualy adding a record to table1 I get this: Table1                                    table2 -------------------                        ------------------- 1    string1        100                1    string1        100 2    string2        200                2    string2        200 3    string3        300                3    string3        300 4    string4        400                4    string3        400 5    string5        500                5    <empty>     500 table2 string has no value and number is correct. Any ideas someone?  Wout  

NXLog IM_MSVistaLog module collects the Rendered Event log rather than the raw XML Windows Event Log.   Is there a configuration option in the NXLog agent or IMVistaLog module to enable collecting the original Windows XML Event Log rather than the Rendered Event Log? Best Regards, Chris   Edit: Think I worked this out. Appears to collect the XML data but also the rendered log field. This would lead questions to be: 1) Can you disable or filter out the Message field? It's not needed. 2) Can you collect the Windows Event Log fields in the order they're written, e.g., the Provider field From testing the Provider field is renamed as Sourcename and collected out of order from the original Windows Event Log. Reason for the above is have multiple upstream systems that require the original log format, and hence testing viability to use NXLog to retrieve Windows Event Logs.

Hello, we’re looking at using multiple instances of nxlog.exe on the same Windows machine. What is the supported method for this please? If I create a new process we receive the ERROR Service is already running log entry. Thanks

hi! I am sending data from csv format and the nxlog start to gather after start/resume of my pc. Sometimes I'm having a trouble when the csv file is not ready yet and my nxlog already start. it will throw some parsing error. Is there a way that when the csv file is already ready, nxlog will also refresh. Thank you.  

I've got a route where the input is from im_tcp and I'd like to output the incoming data to a set of files, but select a specific file based on the content of the data. I can easily do this in om_perl with something that examines the data, figures out the correct file, then stuffs it into that file, but that would appear to require a file open and a file close per event, which seems non-optimal at scale.  So - is the code identified by the PerlCode directive basically run for each event, or is it stateful so I can maintain an array of open files and just stuff the event in the correct one? Alternatively, can I use xm_perl to find the correct filename and set that as an environment valiable that could be consumed by om_file? I'm open to other efficient methods for accomplishing this, if there are ideas out there.  Thanks!

Hello everyone, How can I tell nxlog to send logs to our server every 5 mins? Is there any trigger on NXLog to send the logs to the server?   Thank you.

Hi! I just want to ask if when I update my nxlog to the current version, do I need to revise my code? Is there a list of versions with release notes available? Upon saerching in the website, I couldn't find the list of versions. Only the latesd version is available. Thank you in advance.   

Hi,   We are using the Community NXlog to forward our domain controller security events into Graylog and all was going well until the Microsoft May Security rollup and we now see 100% CPU when Graylog is running. Is anyone aware of a bug / something else that is causing this? Thanks Peter.

It does not appear to be possible for NxLog Community to add fields to a JSON document which contain complex fields because parse_json() converts those complex fields to strings, thereby breaking the document as it is sent upstream. Alernatively, I'm doing something wrong ;). Let's say the NxLog is reading a file with a JSON document on each line and I want to add a custom field. I understand I would do the following: Exec parse_json(); Exec $new_field = 'too sexy'; Exec to_json(); If the original JSON line is  {"field1":1, "field2":2, "fieldComplex":{"a":3,"b",4}} I would expect my resuting document, after NxLog injection, to be: {"field1":1, "field2":2, "fieldComplex":{"a":3,"b",4}, "new_field":"too sexy"} But that's not what happens. The complex field is converted to text and the resulting document is: {"field1":1, "field2":2, "fieldComplex":"{'a':3,'b',4}", "new_field":"too sexy"} Can this behaviour be avoided? Thanks, Geoff

  hello I had an error  while execting nxlog on windows server  2017-05-11 15:34:51 ERROR invalid keyword: { at C:\Program Files (x86)\nxlog\conf\nxlog.conf:38   NXLOG.CONF define ROOT C:\Program Files (x86)\nxlog Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data LogFile %ROOT%\data\nxlog.log <Extension _syslog>     Module      xm_syslog </Extension> <Input in> Module  im_file File 'C:\DNSlogs\DNSlogs.log' SavePos TRUE ReadFromLast TRUE PollInterval 5 CloseWhenIdle TRUE # The following line is the pre-parsing of DNS log. It turns the normal DNS log line into a more readable and easy for parsing with OSSIM format  Exec if $raw_event =~/(\d+\/\d+\/\d+\s+\d+\:\d+\:\d+)\s+.*(Snd|Rcv)\s+(\d+\.\d+\.\d+\.\d+)\s+\S+\s+\w+\s+\w+\s+\S+\s+\w+\s+\S+\s+(\w+)\s+\(\d+\)(\S+)\(\d+\)/\  { \                   $EventTime = parsedate($1); \                   $SourceName = $3; \                   $ProcessID = $2; \                   $Message = $5; \                   if $Message =~ s/\(\d+\)/./g log_info("removed all whitespace in SourceName"); \                 } </Input> <Output out>     Module      om_udp     Host        80.176.253.71     Port        514     Exec        to_syslog_bsd(); </Output> <Route 1>     Path        in => out </Route> thx for help 

Hello, I am using nxlog to parse log files. Everything is working perfectly, however, when the log file is rotated, the nxlog application has errors in the parse of the logs. I did a research, and found the following document: https://nxlog.co/support-tickets/xmcsv-fails-parse-line-when-cr-split-lf-and-its-integer Apparently this bug had been solved in the past. Is it possible that it has returned in some file merge? Or, could it be a new bug?   2017-05-10 17:00:03 WARNING input file was deleted: D:\Program Files\Microsoft\Exchange\V15\TransportRoles\Logs\MessageTracking\MSGTRKMD2017042106-1.LOG 2017-05-10 17:00:03 WARNING input file was deleted: D:\Program Files\Microsoft\Exchange\V15\TransportRoles\Logs\MessageTracking\MSGTRKMD2017042101-1.LOG 2017-05-10 17:00:03 WARNING input file was deleted: D:\Program Files\Microsoft\Exchange\V15\TransportRoles\Logs\MessageTracking\MSGTRKMD2017042103-1.LOG 2017-05-10 17:00:03 WARNING input file was deleted: D:\Program Files\Microsoft\Exchange\V15\TransportRoles\Logs\MessageTracking\MSGTRKMD2017042105-1.LOG 2017-05-10 17:00:03 ERROR procedure 'parse_csv' failed at line 64, character 43 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; Not enough fields in CSV input, expected 27, got 1 in input '#Software: Microsoft Exchange Server' 2017-05-10 17:00:39 WARNING input file was deleted: D:\Program Files\Microsoft\Exchange\V15\TransportRoles\Logs\MessageTracking\MSGTRKMS2017041020-1.LOG 2017-05-10 17:00:39 ERROR procedure 'parse_csv' failed at line 64, character 43 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; Not enough fields in CSV input, expected 27, got 1 in input '#Software: Microsoft Exchange Server' https://nxlog.co/support-tickets/xmcsv-fails-parse-line-when-cr-split-lf-and-its-integer <Input exmsgtrk>    Module im_file    File "%EXMSGTRK%\MSGTRK????????*-*.LOG"    SavePos TRUE    InputType LineBased    Exec    if $raw_event =~ /HealthMailbox/ drop();    Exec    if $raw_event =~ /Mbx_/ drop();    Exec    if $raw_event =~ /^#/ drop();    Exec    ExtMessageTracking->parse_csv();    Exec    $EventTime = parsedate($EmailTime);    Exec    $Message = "Subject: " + $EmailSubject; </Input> Version: nxlog-ce-2.9.1716.msi

I have a very basic setup. I was easily able to get the general syslog functionality working. I have been unable to get the file transport working. I've spent several days trying alternative configurations and Googling for help; all to no avail. I also tested with om_file - trying to just grab the file and output it locally - the outcome was just a blank file. Any help will be greatly appreciated. Here is my config: define ROOT C:\Program Files (x86)\nxlog Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data LogFile %ROOT%\data\nxlog.log LogLevel INFO <Extension gelf>     Module  xm_gelf </Extension> <Extension _syslog> module xm_syslog </extension> <Input 1>     Module    im_file     file    "C:\\MSSQL\\ERRORLOG" </Input> <Output 2>     Module om_tcp     Host 192.168.1.50     Port 5550     OutputType  GELF_TCP </Output> <Route 3>   Path 1 => 2 </Route>    

Is it possible to apply a filter that will allow for a windows event (e.g. 4624) to be dropped if it contains a machine name (appended with $) but send all others?  The line that contains the payload is: JCEF_srcUser:  xxxxx$  xxxx would need to be a wildcard to be inclusive of all machine accounts.  My reasoning is that for our environment we can't completely drop 4624 altogether but the inclusion of machine accounts creates the bulk of the events (over 90%) which is just noise for my puposes.  Thanks in advance!

Hi all, I am evaluating NXLOG and could not find if the om_tcp module supports TCP server (listening mode) in stead of TCP client (connect mode). I have a gui which can connect to a tcp server and display live syslog events so a om_tcp_server is required where the GUI can connect to. Regards, Erwin

I would like to propose a possibility to have multiple config files in a folder (like how telegraf works) "Configuration file locations The location of the configuration file can be set via the --config command line flag. Telegraf will also pick up all files matching the pattern *.conf if the -config-directory command line flag is used. On most systems, the default locations are /etc/telegraf/telegraf.conf for the main configuration file and /etc/telegraf/telegraf.d for the directory of configuration files." I belive this would enhance NXlog by making it a lot easier to implement where there is a need for different log sources. It will allow the owner/maintainer of a function or application to create their own nxlog config for their needs. IE The operating system team manages the config file for the system logs The database team manages the config file for the logging required for example SQL server The web server team manages the logging for IIS/Apache The benefit over having it in one file and have to edit that file for each change in logging needs are: No risk for disruption of other logging on the same system. adding logging for a new app is as easy as dropping a file into the config dir. Very easy for a deployment tool. Clear responsibility - Our app - our config file. Makes paas/saas scenarios a lot easier

Hi, First time post so please go easy.... I have a simplet test Windows 2008 server with nxlog installed with the following config file. define ROOT C:\Program Files (x86)\nxlog Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data LogFile %ROOT%\data\nxlog.log <Extension _syslog>     Module      xm_syslog </Extension> <Input file_in>     Module im_file     File 'D:\\01\\syslogtest.txt'     Exec $Message = $raw_event;     SavePos True     ReadFromLast True </Input> <Processor file_transformer>     Module pm_transformer     Exec $SourceName = 'offline_testing';     OutputFormat syslog_bsd </Processor> <Output file_out>     Module      om_udp     Host        syslog.domain.com     Port        10000 </Output> <Route file_route>     Path        file_in => file_transformer => file_out </Route>   I'm sending my logs to a syslog server running syslog-ng. When the server is available, everything is good.  What I write to the local Windows text file appears on the syslog-ng server. I enable Windows firewall to simulate the syslog-ng server down. I write to the local Windows text file and nothing appears in syslog-ng.  As expected. I remove the firewall rule and here is where my problem lies.  I don't see what was written to the local text file while the syslog-ng was unavailable on the syslog-ng server. If I start writing to the text file after disabling the firewall, I see the new stuff coming in but not what was written while the syslog server was "offline". Can anybody help figure out why this is not working as expected? Regards, Victor

hello: the nxlog  DEBUG  log  are: section 1 2017-04-24 16:15:40 DEBUG check file: /u01/app/oracle/diag/rdbms/testdb/testdb/trace/alert_testdb.log 2017-04-24 16:15:40 DEBUG mtime of file '/u01/app/oracle/diag/rdbms/testdb/testdb/trace/alert_testdb.log' changed 2017-04-24 16:15:40 DEBUG file '/u01/app/oracle/diag/rdbms/testdb/testdb/trace/alert_testdb.log' was truncated 2017-04-24 16:15:40 INFO input file /u01/app/oracle/diag/rdbms/testdb/testdb/trace/alert_testdb.log was truncated, restarting from beginning 2017-04-24 16:15:40 DEBUG file /u01/app/oracle/diag/rdbms/testdb/testdb/trace/alert_testdb.log already opened ... section 2 2017-04-24 16:15:42 DEBUG check file: /u01/app/oracle/diag/rdbms/testdb/testdb/trace/alert_testdb.log 2017-04-24 16:15:42 DEBUG mtime of file '/u01/app/oracle/diag/rdbms/testdb/testdb/trace/alert_testdb.log' changed 2017-04-24 16:15:42 DEBUG file size of '/u01/app/oracle/diag/rdbms/testdb/testdb/trace/alert_testdb.log' increased since last read 2017-04-24 16:15:42 DEBUG file '/u01/app/oracle/diag/rdbms/testdb/testdb/trace/alert_testdb.log' has unread data (540975520 > 540709280) 2017-04-24 16:15:42 DEBUG file /u01/app/oracle/diag/rdbms/testdb/testdb/trace/alert_testdb.log already opened ... section 3 2017-04-24 16:15:44 DEBUG check file: /u01/app/oracle/diag/rdbms/testdb/testdb/trace/alert_testdb.log 2017-04-24 16:15:44 DEBUG mtime of file '/u01/app/oracle/diag/rdbms/testdb/testdb/trace/alert_testdb.log' changed 2017-04-24 16:15:44 DEBUG file '/u01/app/oracle/diag/rdbms/testdb/testdb/trace/alert_testdb.log' was truncated 2017-04-24 16:15:44 INFO input file /u01/app/oracle/diag/rdbms/testdb/testdb/trace/alert_testdb.log was truncated, restarting from beginning 2017-04-24 16:15:44 DEBUG file /u01/app/oracle/diag/rdbms/testdb/testdb/trace/alert_testdb.log already opened the section2-3  is repeated，and continues to loop. please help to figure out the reasons，thank you!

I do use NXLog to parse the Exchange logs and send them to my Graylog.  Time to time, Graylog still receive the line #Software: Microsoft Exchange...    Note that the line is sent once per hours or two hours.  It seems that the check if $raw_event =~ /^#/ drop(); fail to drop the #line randomly. Bellow part of my config for the input: <Input in_MSGTRK>     Module          im_file     File            "\\\\xxxxxxx\\d$\\Exchsvr\\TransportRoles\\Logs\\\MessageTracking\\\\*.log"     SavePos         TRUE     ReadFromLast     TRUE     Recursive        TRUE     CloseWhenIdle   TRUE     InputType       LineBased         PollInterval    10     Exec if $raw_event =~ /HealthMailbox/ drop();     Exec if $raw_event =~ /^#/ drop();     Exec ParseEXCHANGESMSGTRK->parse_csv();     </Input>   Any clue, help or solution is welcome   

I'm having issues leveraging Ansible's win_package to perform installs.  It seems the CE installer ommits the required entries in HKLM:\Software\Microsoft\Windows\CurrentVersion\Uninstall. For more info: https://docs.ansible.com/ansible/latest/win_package_module.html https://docs.microsoft.com/windows/desktop/Msi/uninstall-registry-key https://groups.google.com/d/msg/ansible-project/sgT8C55-eqE/S-UkP3HGCgAJ https://groups.google.com/d/msg/ansible-project/i615RqrZrAs/G8SeGvSQAwAJ