Hi All, I downloaded the NXlog Community Version source and compiled and installed in Ubuntu. I wanted to create a Windows pacakge from the source. So i run the script ./wininst.sh from the packaging/windows/ folder. running the wininst.sh script is showing errors, Can someone let me how to rectify this issue,  test -z "//input" || /bin/mkdir -p "C:\\\\msys\\\\1.0\\\\nxlog-tmp//input"  /bin/bash ../../../../libtool   --mode=install /usr/bin/install -c   im_file.la 'C:\\msys\\1.0\\nxlog-tmp//input' libtool: install: /usr/bin/install -c .libs/im_file.so C:\\msys\\1.0\\nxlog-tmp//input/im_file.so /usr/bin/install: cannot create regular file 'C:\msys\1.0\nxlog-tmp//input/im_file.so': No such file or directory Makefile:291: recipe for target 'install-im_fileLTLIBRARIES' failed make[5]: *** [install-im_fileLTLIBRARIES] Error 1 make[5]: Leaving directory '/nxlog-ce-2.8.1248/src/modules/input/file' Makefile:456: recipe for target 'install-am' failed make[4]: *** [install-am] Error 2 make[4]: Leaving directory '/nxlog-ce-2.8.1248/src/modules/input/file' Makefile:288: recipe for target 'install-recursive' failed make[3]: *** [install-recursive] Error 1 make[3]: Leaving directory '/nxlog-ce-2.8.1248/src/modules/input' Makefile:288: recipe for target 'install-recursive' failed make[2]: *** [install-recursive] Error 1 make[2]: Leaving directory '/nxlog-ce-2.8.1248/src/modules' Makefile:288: recipe for target 'install-recursive' failed make[1]: *** [install-recursive] Error 1 make[1]: Leaving directory '/nxlog-ce-2.8.1248/src' Makefile:411: recipe for target 'install-recursive' failed make: *** [install-recursive] Error 1

Hi all Inherited this system, so sorry if my terminology isn't right or I'm not super clear on what i'm asking for. I am trying to get logs to ship to my SOC CTA.  It works in shipping logs to Nagios but not to the CTA.  I've tried updating the conf file from a server that is working, but every time I restart the service, 5 minutes later the conf file rolls back to its initial config.  What am I doing wrong?  Is the settings for the conf file resetting because of a script issue? Here is my sample conf file that keeps overwriting the work I've done on it.  Thanks all in advance # Please set the ROOT to the folder your nxlog was installed into, ## otherwise it will not start. #define ROOT C:\Program Files\nxlog define ROOT C:\Program Files (x86)\nxlog define CERT %ROOT%\cert   Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data LogFile %ROOT%\data\nxlog.log   # Include fileop while debugging, also enable in the output module below #<Extension fileop> #    Module xm_fileop #</Extension>   <Extension json>     Module      xm_json </Extension>   <Extension syslog>     Module xm_syslog </Extension>   <Input internal>     Module im_internal </Input>   # Watch your own files <Input file1>     Module   im_file     File     '%ROOT%\data\nxlog.log'     SavePos  TRUE     Exec     $Message = $raw_event; </Input>   # Windows Event Log <Input eventlog> # Uncomment im_msvistalog for Windows Vista/2008 and later     Module im_msvistalog     Exec if $EventID IN (5156) drop();   # Uncomment im_mseventlog for Windows XP/2000/2003 #   Module im_mseventlog </Input> <Input MSDNSLOGIN>  Module im_file  File 'C:\Windows\Sysnative\dns\dns.log' # 32-bit applications can access the native system directory by substituting %windir%\Sysnative for %windir%\System32. WOW64 recognizes Sysnative as a special alias used to indicate that the file system should not redirect the access. This mechanism is flexible and easy to use, therefore, it is the recommended mechanism to bypass file system redirection. Note that 64-bit applications cannot use the Sysnative alias as it is a virtual directory not a real one.  InputType LineBased  SavePos TRUE  ReadFromLast FALSE  PollInterval 1 # The input is assumed to contain log messages separated by newlines. Thus if an LF (\n) or CRLF (\r\n) is found, the function considers that it has reached the end of the log message.  Exec if $raw_event == '' drop();  ##drops blank messages  Exec $SourceName = "MSDNSLog";$Message = $raw_event; </Input> <Input MSDHCP>  Module im_file  File 'C:\Windows\Sysnative\dhcp\DhcpSrvLog-*.log' # 32-bit applications can access the native system directory by substituting %windir%\Sysnative for %windir%\System32. WOW64 recognizes Sysnative as a special alias used to indicate that the file system should not redirect the access. This mechanism is flexible and easy to use, therefore, it is the recommended mechanism to bypass file system redirection. Note that 64-bit applications cannot use the Sysnative alias as it is a virtual directory not a real one.  InputType LineBased  SavePos TRUE  ReadFromLast FALSE  PollInterval 1 # The input is assumed to contain log messages separated by newlines. Thus if an LF (\n) or CRLF (\r\n) is found, the function considers that it has reached the end of the log message.  Exec if $raw_event == '' drop();  ##drops blank messages  Exec $SourceName = "DHCPLog";$Message = $raw_event; </Input>   <Output out1>     Module om_udp     Host 10.*.*.*     Port 3***          Exec  $tmpmessage = $Message; delete($Message); rename_field("tmpmessage","message");     Exec  $raw_event = to_json();          # Uncomment for debug output     # Exec file_write('%ROOT%\data\nxlog_output.log', $raw_event + "\n"); </Output> <Output out2>     Module om_udp     Host 10.*.*.*         Port 514 # Transmit in Snare format on local6.info         Exec $SyslogFacilityValue = 22;to_syslog_snare(); </Output> <Output out3>     Module      om_udp     Host        10.*.*.*     Port        514 # Create our RFC3164 compliant syslog line and transmit on local5.info     Exec        $SyslogFacilityValue = 21;$Severity = 'INFO';to_syslog_bsd(); </Output> <Route 1>     Path internal, file1, eventlog  => out1 </Route> <Route 2>     Path        eventlog => out2 </Route> <Route 3>     Path   MSDNSLOGIN, MSDHCP => out3 </Route>   #<Route 1> Note that OUTPUT 2 & 3 are the OUTPUTS not working #    Path internal, file1, eventlog, MSDNSLOGIN, MSDHCP  => out1, out2 #</Route>   Error Log 2017-06-26 13:39:19 ERROR failed to open directory: C:\Windows\Sysnative\dhcp: The system cannot find the path specified.   2017-06-26 13:39:19 WARNING Module MSDHCP has no input files to read 2017-06-26 13:39:21 WARNING input file does not exist: C:\Windows\Sysnative\dns\dns.log 2017-06-26 13:39:27 WARNING last message repeated 2 times 2017-06-26 13:39:35 WARNING input file does not exist: C:\Windows\Sysnative\dns\dns.log 2017-06-26 13:39:51 WARNING input file does not exist: C:\Windows\Sysnative\dns\dns.log 2017-06-26 13:40:23 WARNING input file does not exist: C:\Windows\Sysnative\dns\dns.log 2017-06-26 13:41:28 WARNING input file does not exist: C:\Windows\Sysnative\dns\dns.log 2017-06-26 13:43:37 WARNING input file does not exist: C:\Windows\Sysnative\dns\dns.log 2017-06-26 13:47:54 WARNING input file does not exist: C:\Windows\Sysnative\dns\dns.log 2017-06-26 13:56:27 WARNING input file does not exist: C:\Windows\Sysnative\dns\dns.log 2017-06-26 14:13:32 WARNING input file does not exist: C:\Windows\Sysnative\dns\dns.log 2017-06-26 14:47:41 WARNING input file does not exist: C:\Windows\Sysnative\dns\dns.log

Hello, I'm using NXLOG to send data from CSV files to Elasticsearch. It's working correctly; Other numbers with decimals are being sent as Strings. What FieldType should I use to work with numbers with decimal places (such as: decimal, double, float)? Thank you. <Extension ExtData>      Module xm_csv      Fields $Time,$Id,$Budget,$Tax      FieldTypes string,integer,DECIMAL,DOUBLE      Delimiter ',' </Extension>

I am experimenting running nxlog-processor against a configuration file which is working OK for foreground use, but it does not terminate after pickup, or if no new files are present. I have configured it to monitor a folder in Windows with a wildcarded  path    \....\\*.JSON   Also with multiline input. Are there any limitations, special tips for single-run of nxlog-processor?       p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 13.0px Helvetica} p.p2 {margin: 0.0px 0.0px 0.0px 0.0px; font: 10.0px Helvetica}

Hello nxlog world. My application has a custom log file on a Windows 2012 box that I need to send to Graylog which has a GELF UDP input running. The log file is multiline and there is a specific header and footer that I'm seperating the log file entries with. As of right now despite using the multiline module, when my logs arrive in Graylog it is parsing each line of the log file individually instead of by header and footer. Here is my nxlog.conf: define ROOT C:\Program Files (x86)\nxlog Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data LogFile %ROOT%\data\nxlog.log <Extension gelf>     Module xm_gelf </Extension> <Extension multiline>     Module    xm_multiline         HeaderLine    /\d{4}-\d{2}-\d{2}\s\d*:\d\d:\d*,\d{3}\s[a-zA-Z]*\s\s\S\s\[Begin Lead\]/         EndLine        /\d{4}-\d{2}-\d{2}\s\d*:\d\d:\d*,\d{3}\s[a-zA-Z]*\s\s\S\s\[End Lead\]/ </Extension> <Extension json>     Module    xm_json </Extension> <Input im_msvistalog>     Module      im_msvistalog     Query   <QueryList>\             <Query Id="0">\                     <Select Path="System">*[System[(Level=1 or Level=2 or Level=3)]]</Select>\                     <Select Path="Application">*[System[(Level=1 or Level=2 or Level=3)]]</Select>\             </Query>\         </QueryList>     Exec if $raw_event =~ /(\d\d\d\d\-\d\d-\d\d \d\d:\d\d:\d\d)/ $EventTime = parsedate($1); </Input> <Input im_file>     Module   im_file     File "C:\\LOGS\\application_service.log"     SavePos FALSE      ReadFromLast FALSE     InputType multiline </Input> <Output om_udp>     Module      om_udp     Host        10.25.32.21     Port        12201     OutputType  GELF </Output> <Route 1>     Path    im_msvistalog  => om_udp </Route> <Route 2>     Path im_file => om_udp </Route> The application_service.log file looks like this: 2017-06-20 17:25:50,168 INFO  – [Begin Lead] 2017-06-20 17:25:50,168 INFO  – Getting lead by id '5551212' 2017-06-20 17:25:51,215 INFO  – Lead field = First Name. Value = Michael 2017-06-20 17:25:54,778 INFO  – Lead field = Last Name. Value = Westin 2017-06-20 17:25:54,793 INFO  – Lead field = CustomerNumber. Value = 99999 2017-06-20 17:26:05,887 INFO  – [End Lead] 2017-06-20 18:25:50,168 INFO  – [Begin Lead] 2017-06-20 18:25:50,168 INFO  – Getting lead by id '4279551' 2017-06-20 18:25:51,215 INFO  – Lead field = First Name. Value = Emmit 2017-06-20 18:25:54,778 INFO  – Lead field = Last Name. Value = Stussy 2017-06-20 18:25:54,793 INFO  – Lead field = CustomerNumber. Value = 94231 2017-06-20 18:26:05,887 INFO  – [End Lead] 2017-06-20 19:25:50,168 INFO  – [Begin Lead] 2017-06-20 19:25:50,168 INFO  – Getting lead by id '0081227' 2017-06-20 19:25:51,215 INFO  – Lead field = First Name. Value = Saul 2017-06-20 19:25:54,778 INFO  – Lead field = Last Name. Value = Goodman 2017-06-20 19:25:54,793 INFO  – Lead field = CustomerNumber. Value = 33487 2017-06-20 19:26:05,887 INFO  – [End Lead] Ideally when it his Graylog I'd like the content between [Begin Lead] and [End Lead] to be stored in one field.  But with my nxlog config above it is putting each line of the log file in its own message ID in Graylog.  Do I have my config wrong?  Is there a better method to parse the log file and send to Graylog?  Any help is appreciated.

Hi, Using two tables in mySQL: table1 and table2 with same structure.  Field          Type          Length        Key -------------------------------------------------------- id               INT            11               Primary text1          CHAR        20 number1    INT            11 Initial situation: Content of table1, table2 is empty  Table1                            table2 --------------------------       --------------------- 1    string1        100        <no records> 2    string2        200 3    string3        300     My nxlog.conf: define ROOT C:\Program Files (x86)\nxlog define CERT %ROOT%\cert Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data LogFile %ROOT%\data\nxlog.log #PersistLogqueue    TRUE #SyncLogqueue        TRUE CacheFlushInterval    always CacheSync        TRUE #LogLevel  DEBUG <Input    testIn>     Module  im_odbc #    Pollinterval  60     ConnectionString  DSN=mySQL;uid=***;pwd=***;database=nxlog     SQL   SELECT  id as vid ,\                   text1 as vtext1 ,\                   number1 as vnumber1 \           FROM table1 where id > ?     SavePos TRUE </Input>               <Output     testOut>         Module        om_odbc         ConnectionString DSN=mySQL;uid=***;pwd=***;database=nxlog         <Exec>           if (sql_exec("INSERT INTO table2(text1,number1) VALUES (?,?)",$vtext1,$vnumber1)== TRUE) {};         </Exec> </Output> <Route 1>     Path testIn => testOut </Route> After I start nxlog service both tables look like this. Table1                                     table2 -------------------                        ------------------- 1    string1        100                1    string1        100 2    string2        200                2    string2        200 3    string3        300                3    string3        300 So far so good. But when I manualy add a record to table1 both tables look like this: Table1                                    table2 -------------------                        ------------------- 1    string1        100                1    string1        100 2    string2        200                2    string2        200 3    string3        300                3    string3        300 4    string4        400                4    string3        400 As you can see in table2: the string has the value of record3. Number is OK. When I keep adding records to table1 it keeps on taking the string of record 3 but number is correct. After restart of nxlog service and manualy adding a record to table1 I get this: Table1                                    table2 -------------------                        ------------------- 1    string1        100                1    string1        100 2    string2        200                2    string2        200 3    string3        300                3    string3        300 4    string4        400                4    string3        400 5    string5        500                5    <empty>     500 table2 string has no value and number is correct. Any ideas someone?  Wout  

NXLog IM_MSVistaLog module collects the Rendered Event log rather than the raw XML Windows Event Log.   Is there a configuration option in the NXLog agent or IMVistaLog module to enable collecting the original Windows XML Event Log rather than the Rendered Event Log? Best Regards, Chris   Edit: Think I worked this out. Appears to collect the XML data but also the rendered log field. This would lead questions to be: 1) Can you disable or filter out the Message field? It's not needed. 2) Can you collect the Windows Event Log fields in the order they're written, e.g., the Provider field From testing the Provider field is renamed as Sourcename and collected out of order from the original Windows Event Log. Reason for the above is have multiple upstream systems that require the original log format, and hence testing viability to use NXLog to retrieve Windows Event Logs.

Hello, we’re looking at using multiple instances of nxlog.exe on the same Windows machine. What is the supported method for this please? If I create a new process we receive the ERROR Service is already running log entry. Thanks

hi! I am sending data from csv format and the nxlog start to gather after start/resume of my pc. Sometimes I'm having a trouble when the csv file is not ready yet and my nxlog already start. it will throw some parsing error. Is there a way that when the csv file is already ready, nxlog will also refresh. Thank you.  

I've got a route where the input is from im_tcp and I'd like to output the incoming data to a set of files, but select a specific file based on the content of the data. I can easily do this in om_perl with something that examines the data, figures out the correct file, then stuffs it into that file, but that would appear to require a file open and a file close per event, which seems non-optimal at scale.  So - is the code identified by the PerlCode directive basically run for each event, or is it stateful so I can maintain an array of open files and just stuff the event in the correct one? Alternatively, can I use xm_perl to find the correct filename and set that as an environment valiable that could be consumed by om_file? I'm open to other efficient methods for accomplishing this, if there are ideas out there.  Thanks!

Hello everyone, How can I tell nxlog to send logs to our server every 5 mins? Is there any trigger on NXLog to send the logs to the server?   Thank you.

Hi! I just want to ask if when I update my nxlog to the current version, do I need to revise my code? Is there a list of versions with release notes available? Upon saerching in the website, I couldn't find the list of versions. Only the latesd version is available. Thank you in advance.   

Hi,   We are using the Community NXlog to forward our domain controller security events into Graylog and all was going well until the Microsoft May Security rollup and we now see 100% CPU when Graylog is running. Is anyone aware of a bug / something else that is causing this? Thanks Peter.

It does not appear to be possible for NxLog Community to add fields to a JSON document which contain complex fields because parse_json() converts those complex fields to strings, thereby breaking the document as it is sent upstream. Alernatively, I'm doing something wrong ;). Let's say the NxLog is reading a file with a JSON document on each line and I want to add a custom field. I understand I would do the following: Exec parse_json(); Exec $new_field = 'too sexy'; Exec to_json(); If the original JSON line is  {"field1":1, "field2":2, "fieldComplex":{"a":3,"b",4}} I would expect my resuting document, after NxLog injection, to be: {"field1":1, "field2":2, "fieldComplex":{"a":3,"b",4}, "new_field":"too sexy"} But that's not what happens. The complex field is converted to text and the resulting document is: {"field1":1, "field2":2, "fieldComplex":"{'a':3,'b',4}", "new_field":"too sexy"} Can this behaviour be avoided? Thanks, Geoff

  hello I had an error  while execting nxlog on windows server  2017-05-11 15:34:51 ERROR invalid keyword: { at C:\Program Files (x86)\nxlog\conf\nxlog.conf:38   NXLOG.CONF define ROOT C:\Program Files (x86)\nxlog Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data LogFile %ROOT%\data\nxlog.log <Extension _syslog>     Module      xm_syslog </Extension> <Input in> Module  im_file File 'C:\DNSlogs\DNSlogs.log' SavePos TRUE ReadFromLast TRUE PollInterval 5 CloseWhenIdle TRUE # The following line is the pre-parsing of DNS log. It turns the normal DNS log line into a more readable and easy for parsing with OSSIM format  Exec if $raw_event =~/(\d+\/\d+\/\d+\s+\d+\:\d+\:\d+)\s+.*(Snd|Rcv)\s+(\d+\.\d+\.\d+\.\d+)\s+\S+\s+\w+\s+\w+\s+\S+\s+\w+\s+\S+\s+(\w+)\s+\(\d+\)(\S+)\(\d+\)/\  { \                   $EventTime = parsedate($1); \                   $SourceName = $3; \                   $ProcessID = $2; \                   $Message = $5; \                   if $Message =~ s/\(\d+\)/./g log_info("removed all whitespace in SourceName"); \                 } </Input> <Output out>     Module      om_udp     Host        80.176.253.71     Port        514     Exec        to_syslog_bsd(); </Output> <Route 1>     Path        in => out </Route> thx for help 

Hello, I am using nxlog to parse log files. Everything is working perfectly, however, when the log file is rotated, the nxlog application has errors in the parse of the logs. I did a research, and found the following document: https://nxlog.co/support-tickets/xmcsv-fails-parse-line-when-cr-split-lf-and-its-integer Apparently this bug had been solved in the past. Is it possible that it has returned in some file merge? Or, could it be a new bug?   2017-05-10 17:00:03 WARNING input file was deleted: D:\Program Files\Microsoft\Exchange\V15\TransportRoles\Logs\MessageTracking\MSGTRKMD2017042106-1.LOG 2017-05-10 17:00:03 WARNING input file was deleted: D:\Program Files\Microsoft\Exchange\V15\TransportRoles\Logs\MessageTracking\MSGTRKMD2017042101-1.LOG 2017-05-10 17:00:03 WARNING input file was deleted: D:\Program Files\Microsoft\Exchange\V15\TransportRoles\Logs\MessageTracking\MSGTRKMD2017042103-1.LOG 2017-05-10 17:00:03 WARNING input file was deleted: D:\Program Files\Microsoft\Exchange\V15\TransportRoles\Logs\MessageTracking\MSGTRKMD2017042105-1.LOG 2017-05-10 17:00:03 ERROR procedure 'parse_csv' failed at line 64, character 43 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; Not enough fields in CSV input, expected 27, got 1 in input '#Software: Microsoft Exchange Server' 2017-05-10 17:00:39 WARNING input file was deleted: D:\Program Files\Microsoft\Exchange\V15\TransportRoles\Logs\MessageTracking\MSGTRKMS2017041020-1.LOG 2017-05-10 17:00:39 ERROR procedure 'parse_csv' failed at line 64, character 43 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; Not enough fields in CSV input, expected 27, got 1 in input '#Software: Microsoft Exchange Server' https://nxlog.co/support-tickets/xmcsv-fails-parse-line-when-cr-split-lf-and-its-integer <Input exmsgtrk>    Module im_file    File "%EXMSGTRK%\MSGTRK????????*-*.LOG"    SavePos TRUE    InputType LineBased    Exec    if $raw_event =~ /HealthMailbox/ drop();    Exec    if $raw_event =~ /Mbx_/ drop();    Exec    if $raw_event =~ /^#/ drop();    Exec    ExtMessageTracking->parse_csv();    Exec    $EventTime = parsedate($EmailTime);    Exec    $Message = "Subject: " + $EmailSubject; </Input> Version: nxlog-ce-2.9.1716.msi

I have a very basic setup. I was easily able to get the general syslog functionality working. I have been unable to get the file transport working. I've spent several days trying alternative configurations and Googling for help; all to no avail. I also tested with om_file - trying to just grab the file and output it locally - the outcome was just a blank file. Any help will be greatly appreciated. Here is my config: define ROOT C:\Program Files (x86)\nxlog Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data LogFile %ROOT%\data\nxlog.log LogLevel INFO <Extension gelf>     Module  xm_gelf </Extension> <Extension _syslog> module xm_syslog </extension> <Input 1>     Module    im_file     file    "C:\\MSSQL\\ERRORLOG" </Input> <Output 2>     Module om_tcp     Host 192.168.1.50     Port 5550     OutputType  GELF_TCP </Output> <Route 3>   Path 1 => 2 </Route>    

Is it possible to apply a filter that will allow for a windows event (e.g. 4624) to be dropped if it contains a machine name (appended with $) but send all others?  The line that contains the payload is: JCEF_srcUser:  xxxxx$  xxxx would need to be a wildcard to be inclusive of all machine accounts.  My reasoning is that for our environment we can't completely drop 4624 altogether but the inclusion of machine accounts creates the bulk of the events (over 90%) which is just noise for my puposes.  Thanks in advance!

Hi all, I am evaluating NXLOG and could not find if the om_tcp module supports TCP server (listening mode) in stead of TCP client (connect mode). I have a gui which can connect to a tcp server and display live syslog events so a om_tcp_server is required where the GUI can connect to. Regards, Erwin

I would like to propose a possibility to have multiple config files in a folder (like how telegraf works) "Configuration file locations The location of the configuration file can be set via the --config command line flag. Telegraf will also pick up all files matching the pattern *.conf if the -config-directory command line flag is used. On most systems, the default locations are /etc/telegraf/telegraf.conf for the main configuration file and /etc/telegraf/telegraf.d for the directory of configuration files." I belive this would enhance NXlog by making it a lot easier to implement where there is a need for different log sources. It will allow the owner/maintainer of a function or application to create their own nxlog config for their needs. IE The operating system team manages the config file for the system logs The database team manages the config file for the logging required for example SQL server The web server team manages the logging for IIS/Apache The benefit over having it in one file and have to edit that file for each change in logging needs are: No risk for disruption of other logging on the same system. adding logging for a new app is as easy as dropping a file into the config dir. Very easy for a deployment tool. Clear responsibility - Our app - our config file. Makes paas/saas scenarios a lot easier