The following config is working for me to send application event logs to server via tcp, but it is sending duplicate messages for every Event Viewer event/entry. config below as follows. User nxlog Group nxlog LogFile /var/log/nxlog/nxlog.log LogLevel INFO <Extension exec> Module xm_exec </Extension> <Extension syslog> Module xm_syslog </Extension> <Input in> Module im_tcp Port 1514 Host 0.0.0.0 Exec parse_syslog_bsd(); </Input> <Output out> Module om_file File "/application/logs/" + $Hostname + "" + "app.log" <Exec> if out->file_size() > 1G { set_var('newfile', file_name() + strftime(now(), '%Y%m%d%H%M%S')); rotate_to(get_var('newfile')); exec_async('/bin/bzip2', get_var('newfile')); } </Exec> </Output> <Route 1> Path in => out </Route>

Hi I'm using nxlog version 2.10.2102. I have created the following Input: <Input Test> Module im_file File "D:\TEST\Data\posWS\UploadLogs\61\\." SavePos TRUE <Exec> if $raw_event =~ /[^,]+/g; { $datetime = $1; $log_level = $2; $version = $3; $company_id = $4; $device_code = $5; $device_manufacturer = $6; $device_board = $7; $sdk_version = $8; $process_id = $9; $thread_id = $10; $class_name = $11; $method_name = $12; $file_name = $13; $line = $14; $description = $15; $stack_trace = $16; $EventTime = parsedate($datetime); $SourceName = "POS_Device"; } delete($datetime); </Exec> </Input> When I start the nxlog service, I receive the following error: ERROR Couldn't parse Exec block at C:\Program Files (x86)\nxlog\conf\nxlog.conf:73; couldn't parse statement at line 74, character 29 in C:\Program Files (x86)\nxlog\conf\nxlog.conf; syntax error, unexpected ;, expecting ( I have tested the regex on https://regex101.com/ and it is working fine (flavour = pcre). Sample data would be: 2018-07-19 10:26:07.830,ASSERT,3.0.2420InternalTest,145,352099091773370,samsung,universal7870,24,8521,1,CrashHandler,uncaughtException,CrashHandler.java,32,"UncaughtException","java.lang.NullPointerException: Attempt to invoke virtual method 'boolean bll.core.Product.isDiscountProduct()' on a null object reference;at bll.core.Sale$SaleItem.isDiscountItem(Sale.java:5514);at bll.core.Sale.containsSpecial(Sale.java:7685);at activity.SalesControlDisplaySale.setItemsPayments(SalesControlDisplaySale.java:454);at activity.SalesControlDisplaySale.displaySale(SalesControlDisplaySale.java:316);at activity.SalesControlDisplaySale.access$400(SalesControlDisplaySale.java:96);at activity.SalesControlDisplaySale$1.onItemSelected(SalesControlDisplaySale.java:213);at android.widget.AdapterView.fireOnSelected(AdapterView.java:1319);at android.widget.AdapterView.dispatchOnItemSelected(AdapterView.java:1308);at android.widget.AdapterView.-wrap1(AdapterView.java);at android.widget.AdapterView$SelectionNotifier.run(AdapterView.java:1275);at android.os.Handler.handleCallback(Handler.java:751);at android.os.Handler.dispatchMessage(Handler.java:95);at android.os.Looper.loop(Looper.java:154);at android.app.ActivityThread.main(ActivityThread.java:6776);at java.lang.reflect.Method.invoke(Native Method);at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:1496);at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1386);" If I do not use the "/g" (Global) at the end, it only returns the first match. Any help would be greatly appropriated. Many thanks Stephen

Hello, I have a strange situation: I have a folder to index, which has some files, each file has a single line, a JSON encoded string. If I turn the DEBUG level, I can see that the files match the pattern for files, im_file_add_file function is called, but no event reaches my INPUT. <Input api> Module im_file SavePos FALSE ReadFromLast FALSE PollInterval 0.25 #ActiveFiles 1000 #CloseWhenIdle TRUE File "/var/www/project/current/var/api_logs/default/2018-07-25/qm_sync/*" <Exec> log_info(file_name()); $file = file_name(); #if $file =~ /^\/var\/www\/project\/current\/var\/api_logs\/default\/2018-07-25\/qm_sync\/[^\/]*\/([^\/]*)\/.*/ #{ $user = $1; file_write("/var/log/nxlog/nxlog.log", $raw_event); if $raw_event =~ /^(.*"requestBody":\{[^\[]*"[^"]*":)\[([^\]]*)\](.*)$/ { $raw_event = $1 + '"[' + replace($2, '"', '\"') + ']"' + $3; } file_write("/var/log/nxlog/nxlog.log", $raw_event); parse_json(); $timestamp = $requestTimestamp; $eventTime = strftime($EventReceivedTime, '%Y-%m-%d %H:%M:%S'); $host = hostname_fqdn(); rename_field('timestamp', '@timestamp'); delete($EventReceivedTime); delete($SourceModuleName); delete($SourceModuleType); to_json(); #} else { # file_write("/var/log/nxlog/nxlog_parse_failed.log", "File: " + $file + "; Message: " + $raw_event + "\n"); # drop(); #} $indexType = 'nxlog-api-external'; </Exec> </Input> If I enter a file with VIM, and just write and quit, without changing anything, the file gets indexed. Any idea why this is happening? I am using the same instance of NxLog CE to index other files and it works fine with them

Hello, I'm trying to use the log rotation functionality of nxlog but when nxlog tries to rotate is get the follow error message: Error failed to rename file from C:\Users\me\Desktop\pfirewall.log to C:\Users\me\Desktop\pfirewall.log.1: The process cannot access the file because it is being used by another process. Is there anyway to get around this? Here is my nxlog.conf define ROOT C:\Program Files (x86)\nxlog Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data LogFile %ROOT%\data\nxlog.log define FWLOG C:\Users\me\Desktop\pfirewall.log <Extension _syslog> Module xm_syslog </Extension> <Extension fileop> Module xm_fileop </Extension> <Input FWLOG> Module im_file File '%FWLOG%' InputType LineBased SavePos TRUE </Input> <Output OUT> Module om_file File '%FWLOG%' &lt;Schedule&gt; Every 10 sec Exec if (file_size('%FWLOG%') &gt;= 1K) \ { \ file_cycle('%FWLOG%',10); \ OUT-&gt;reopen(); \ } &lt;/Schedule&gt; </Output> <Route 1> Path FWLOG => OUT </Route>

Hello, Is there a way to import .evt files with nxlog? Using im_file doesn't throw errors, but data doesnt seem to ship. I also tried using: module im_mseventlog file "file_path" but that didn't work either. Thanks for your help.

Hello, Has anyone successfully configured the new event tracing for windows (ETW) input module in nxlog 4.0 to collect Windows DNSServer events? I configured it in nxlog, but the output file doesn't show most of the DNS queries being made. When I look at the nxlog output and compare it with a trace session in Event Viewer, Event Viewer shows all of the events but nxlog is missing almost all of them. There are a few entries in the nxlog file, but not many. I can't seem to reproduce the scenario that causes them to be included in the nxlog output file. Info on setup: Server 2016 datacenter, v1607 nxlog 4.0.3735-x64 Related nxlog config: <Input winetw> Module im_etw Provider Microsoft-Windows-DNSServer </Input> <Output file> Module om_file File 'C:\Windows\Logs\nxlog\test.txt' </Output> <Route messages_to_file> Path winetw => file </Route>

Hello, I'm trying to avoid having duplicate logs send to my OSSIM server. I tried using the pm_norepeat module but to no avail. Here the line I added in nxlog.conf file : <Processor sans_doublons> Module pm_norepeat </Processor> <Route route_windows_logs> Path in_windows_events => sans_doublons => out_alienvault_csv </Route> I also tried adding "CheckFields raw_event" in the processor, but I still get duplicate logs. Does anyone know what could be the problem ? Thanks

I'm using nxlog-ce-2.10.5 on AIX 6.5. Configure and make with default options. After building successfully, I tried to start nxlog with im_file and om_file, like the following: <Input messages> Module im_file File "/home/logtest/*.log" </Input> <Output file> Module om_file File "/home/logtest/1.out" </Output> <Route messages_to_tcp> Path messages => tcp </Route> But nxlog shows some error like this: 2018-07-16 08:45:16 ERROR Failed to load module from /usr/local/libexec/nxlog/modules/input/im_file.so, rtld: 0712-001 Symbol nx_assertion_failed was referenced; from module /usr/local/libexec/nxlog/modules/input/im_file.so(), but a runtime definition; of the symbol was not found.;rtld: 0712-001 Symbol nx_module_add_logdata_input was referenced; from module /usr/local/libexec/nxlog/modules/input/im_file.so(), but a runtime definition; of the symbol was not found.;rtld: 0712-001 Symbol nx_log was referenced; from module /usr/local/libexec/nxlog/modules/input/im_file.so(), but a runtim;DSO load failed 2018-07-16 08:45:16 ERROR Failed to load module from /usr/local/libexec/nxlog/modules/output/om_file.so, rtld: 0712-001 Symbol nx_assertion_failed was referenced; from module /usr/local/libexec/nxlog/modules/output/om_file.so(), but a runtime definition; of the symbol was not found.;rtld: 0712-001 Symbol nx_log was referenced; from module /usr/local/libexec/nxlog/modules/output/om_file.so(), but a runtime definition; of the symbol was not found.;rtld: 0712-001 Symbol nx_pool_create_child was referenced; from module /usr/local/libexec/nxlog/modules/output/om_file.so(), but a runtime de;DSO load failed 2018-07-16 08:45:16 WARNING no functional input modules! 2018-07-16 08:45:16 ERROR module 'messages' is not declared at nxlog.conf:12 2018-07-16 08:45:16 ERROR module 'tcp' is not declared at nxlog.conf:12 2018-07-16 08:45:16 ERROR route messages_to_tcp is not functional without input modules, ignored at nxlog.conf:12 2018-07-16 08:45:16 WARNING no routes defined! 2018-07-16 08:45:16 INFO nxlog-ce-2.10.5 started What did I miss?

I'm having this newbie problem with sending logs from Windows Server 2008 R2. I tried to search a solution from here and from the user guide but can't find a solution. I'm getting this error message to nxlog.log: WARNING Module filein1 has no input files to read My config is this: <Extension multiline1> Module xm_multiline HeaderLine /^\d{4}\-\d{2}\-\d{2}\s\d{2}\:\d{2}\:\d{2}\,\d{3}\s\d+/ #escape lines with whitespaces Exec if $raw_event =~ /^\s*$/ drop(); </Extension> <Input filein1> Module im_file File "C:\Program Files (x86)\CA\Some Manager\Connector Server\jcs\logs\LDAP_DYN\jcs_conn" InputType multiline1 SavePos TRUE Exec $SourceName = 'SomeManager'; </Input> <Output out> Module om_tcp Host 192.168.1.1 Port 1468 Exec to_syslog_ietf(); </Output> <Route 1> Path filein1 => out </Route> What am i missing here? NXLog version is 4.0.3735.

I am trying to use NXlog CE to forward Exchange 2016 message tracking logs via syslog. I am trying to use the example shown here https://nxlog.co/documentation/nxlog-user-guide#exchange_transport_logs When I create the conf file and attempt to run it the service starts but there are several errors logged in the nxlog.log file. My config file is as follows Panic Soft #NoFreeOnExit TRUE define ROOT C:\Program Files (x86)\nxlog define CERTDIR %ROOT%\cert define CONFDIR %ROOT%\conf define LOGDIR %ROOT%\data define LOGFILE %LOGDIR%\nxlog.log LogFile %LOGFILE% Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data <Extension syslog> Module xm_syslog </Extension> define BASEDIR C:\Program Files\Microsoft\Exchange Server\V15 <Extension csv_parser> Module xm_csv Fields date-time, client-ip, client-hostname, server-ip, server-hostname, \ source-context, connector-id, source, event-id, \ internal-message-id, message-id, network-message-id, \ recipient-address, recipient-status, total-bytes, recipient-count, \ related-recipient-address, reference, message-subject, \ sender-address, return-path, message-info, directionality, \ tenant-id, original-client-ip, original-server-ip, custom-data, \ transport-traffic-type, log-id, schema-version </Extension> <Input messagetracking> Module im_file File '%BASEDIR%\TransportRoles\Logs\MessageTracking\MSGTRK*.LOG' <Exec> if $raw_event =~ /^(\xEF\xBB\xBF)?(date-time,|#)/ drop(); else { csv_parser->parse_csv(); $EventTime = parsedate(${date-time}); } </Exec> </Input> <Output out1> Module om_udp Host 10.1.1.1 Port 514 Exec to_syslog_snare(); </Output> <Route 1> Path messagetracking => out1 </Route> The errors logged in the nxlog.log file are as follows. 2018-07-12 18:06:10 ERROR Couldn't parse Exec block at C:\Program Files (x86)\nxlog\conf\nxlog.conf:39; couldn't parse statement at line 44, character 36 in C:\Program Files (x86)\nxlog\conf\nxlog.conf; invalid character: '$' (0x24) 2018-07-12 18:06:10 ERROR module 'messagetracking' has configuration errors, not adding to route '1' at C:\Program Files (x86)\nxlog\conf\nxlog.conf:59 2018-07-12 18:06:10 ERROR route 1 is not functional without input modules, ignored at C:\Program Files (x86)\nxlog\conf\nxlog.conf:59 2018-07-12 18:06:10 WARNING no routes defined! 2018-07-12 18:06:10 WARNING not starting unused module messagetracking 2018-07-12 18:06:10 WARNING not starting unused module out1 2018-07-12 18:06:10 INFO nxlog-ce-2.10.2102 started I have yet to come across a working example of how to forward exchange logs and was hoping someone may be able to assist. I cant see what i am missing here.

Hello, Where can I find Nxlog-ce source code for Windows?

Hi, i have a very basic question. i have to use nxlog as a logshipper on the (solaris) client side to logstash (elk). Within this configuration there are some bigger logfiles with multilines to parse (java traces / xml ...). Should i handle the multiline on the nxlog side or on the logstash receiver side? thanks in advance Andreas

Hi, Does NXLog supports sending log files to Amazon S3 cloud storage? Thanks, Amishai

Hi, I have configured nxlog with Graylog2, i am seeing below warning in nxlog.log, wat could be the issue 2018-07-03 11:34:03 WARNING string limit (1048576) exceeded while trying to append 2018-07-03 11:41:37 INFO inode changed for '/var/log/Store/Store.log' (96469298->96469281): reopening possibly rotated file 2018-07-03 12:03:22 INFO inode changed for '/var/log/Store/Store.log' (96469281->96469298): reopening possibly rotated file

Can someone please let me know if Community Edition support TSL v 1.2 as currently i'm using 1.0 and that does not seem to be working after one of the vendors blocked it due to vulnerability in v 1.0. Can you please let me know what options i have now. Thanks, Zee

Hi I'm using nxlog v2.9.1716. I've created the following input: <Input in> Module im_file File "C:\Program Files\LogFiles\*.log" SavePos TRUE Recursive TRUE Exec if $raw_event =~ /^#/ drop(); Exec if $raw_event =~ ^([^;]+);([^;]+);([^;]+)(?&gt;;([^;]+);([^;]+);([^;]+);([^;]+);([^;]+);([^;]+);([^;]+);([^;]+);([^;]+);(.+)$)?/gx; \ { \ $date = $1; \ $time = $2; \ $site-instance = $3; \ $event = $4; \ $client-ip = $5; \ $via-header = $6; \ $http-x-forwarded-for = $7; \ $host-header = $8; \ $additional-info-1 = $9; \ $additional-info-2 = $10; \ $additional-info-3 = $11; \ $additional-info-4 = $12; \ $additional-info = $13; \ $EventTime = parsedate($date + &quot; &quot; + $time); \ $SourceName = &quot;WAF&quot;; \ } </Input> The regex being used has been successfully tested with https://regex101.com/ Sample data below: 2018-06-28 ; 10:23:52 ; W3SVC2 ; OnPreprocHeaders ; 10.10.10.10 ; ; 8.8.8.8 ; my.domain.com ; GET ; /account/login ; ALERT: '/account/' not allowed in URL ; HTTP/1.0 ; 0 ; ; Actional Intermediary When I start the nxlog service, I get the following error: 2018-06-28 16:44:51 ERROR Couldn't parse Exec block at C:\Program Files (x86)\nxlog\conf\nxlog.conf:89; couldn't parse statement at line 89, character 24 in C:\Program Files (x86)\nxlog\conf\nxlog.conf; syntax error 2018-06-28 16:44:51 ERROR module 'in' has configuration errors, not adding to route '2' at C:\Program Files (x86)\nxlog\conf\nxlog.conf:116 2018-06-28 16:44:51 ERROR route 2 is not functional without input modules, ignored at C:\Program Files (x86)\nxlog\conf\nxlog.conf:116 2018-06-28 16:44:51 WARNING not starting unused module in 2018-06-28 16:44:51 INFO nxlog-ce-2.9.1716 started 2018-06-28 16:44:51 INFO reconnecting in 1 seconds I also tried the following: <Input in> Module im_file File "C:\Program Files\AQTRONIX Webknight\LogFiles\*.log" SavePos TRUE Recursive TRUE <Exec> if $Message =~ /^#/ drop(); $Message =~ ^(?<date>[^;]+);(?<time>[^;]+);(?<site_instance>[^;]+)(?>;(?<event>[^;]+);(?<client_ip>[^;]+);(?<via_header>[^;]+);(?<http_x_forwarded_for>[^;]+);(?<host_header>[^;]+);(?<additional_info_1>[^;]+);(?<additional_info_2>[^;]+);(?<additional_info_3>[^;]+);(?<additional_info_4>[^;]+);(?<additional_info>.+)$)? /gx; </Exec> </Input> But I receive the following error on starting nxlog: 2018-06-28 17:15:54 ERROR Couldn't parse Exec block at C:\Program Files (x86)\nxlog\conf\nxlog.conf:70; couldn't parse statement at line 72, character 15 in C:\Program Files (x86)\nxlog\conf\nxlog.conf; syntax error 2018-06-28 17:15:54 ERROR module 'in' has configuration errors, not adding to route '2' at C:\Program Files (x86)\nxlog\conf\nxlog.conf:100 2018-06-28 17:15:54 ERROR route 2 is not functional without input modules, ignored at C:\Program Files (x86)\nxlog\conf\nxlog.conf:100 2018-06-28 17:15:54 WARNING not starting unused module in 2018-06-28 17:15:54 INFO nxlog-ce-2.9.1716 started I tried various syntax changes, but just cannot see the issue. This is the first time I've tried using a regex with nxlog. Any help or guidance much appreciated.

I have been trying to get NXLog to send Syslog entries from the Teamviewer "Connections_incoming.txt" log file. It is a tab delimited file. I found this site which supplied the basic code for the task: https://gist.github.com/idefux/949e84c8ec8d4db1775c which i couldn't get working as expected. To cut a long story short, i have discovered that the $raw_event is often blank, so the Syslog entries do not contain the necessary information. I have tried to google this issue, but i have not been able to find the information to figure this one out. There has been times when $raw_event did contain the information required from the original log file, but it is not reliable. Can someone please give me some suggestions on how to get information into the $raw_event function? On the positive side, i do get a reliable Syslog entry whenever someone accesses the computer through Teamviewer. Using the code below, i get the message "Teamviewer Login Event" Below is my current configuration. The if-else statement is designed to be as simple as possible to try fault-find the issue define ROOT C:\Program Files (x86)\nxlog Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data LogFile %ROOT%\data\nxlog.log <Extension _syslog> Module xm_syslog </Extension> <Input in_teamviewer> Module im_file File 'c:\Program Files (x86)\TeamViewer\Connections_incoming.txt' SavePos TRUE ReadFromLast TRUE PollInterval 10 <Exec> $Hostname = hostname(); $SeverityValue = 5; $MessageSourceAddress = hostname(); $SyslogFacilityValue = 4; $SourceName = 'TeamViewer'; $EventTime = parsedate($5 + '-' + $4 + '-' + $3 + ' ' + $6); $user = $2; if ($raw_event == '') $Message = $raw_event + "Teamviewer Login Event"; else $Message = $raw_event + '['+ file_name() + ']'; to_syslog_bsd(); </Exec> </Input> <Output out_syslog> Module om_udp Host localhost Port 514 </Output> <Route 1> Path in_teamviewer => out_syslog </Route>

Hi, I'm using nxlog ce latest version with centos 7 with latest patches to read files from a directory that is a cifs mount to a windows server. Logs are in w3c format. However from times to times, I'm getting the information in nxlog log file "INFO input file '' was truncated, restarting from the beginning", but the file isn't truncated. Using strace with options -tfp (since debug option of nxlog doesn't give me much information) i get this: pid 7675] 12:52:58 close(16 <unfinished ...> [pid 7676] 12:52:58 <... stat resumed> {st_mode=S_IFREG|0400, st_size=144080774, ...}) = 0 [pid 7676] 12:52:58 stat("", <unfinished ...> [pid 7675] 12:52:58 <... close resumed> ) = 0 [pid 7676] 12:52:58 <... stat resumed> {st_mode=S_IFREG|0400, st_size=54218426, ...}) = 0 [pid 7675] 12:52:58 stat("", <unfinished ...> [pid 7676] 12:52:58 stat("", <unfinished ...> [pid 7675] 12:52:58 <... stat resumed> {st_mode=S_IFREG|0400, st_size=277116066, ...}) = 0 [pid 7675] 12:52:58 fstat(26, {st_mode=S_IFREG|0400, st_size=277116066, ...}) = 0 [pid 7676] 12:52:58 <... stat resumed> {st_mode=S_IFREG|0400, st_size=141529135, ...}) = 0 [pid 7676] 12:52:58 stat("", {st_mode=S_IFREG|0400, st_size=148346198, ...}) = 0 My nxlog configuration is this: User nxlog Group nxlog LogFile /var/log/nxlog/nxlog.log LogLevel INFO define DIR Module xm_syslog Module im_file File '%DIR%/_*_.w3c' SavePos TRUE ReadFromLast TRUE Recursive TRUE RenameCheck TRUE CloseWhenIdle TRUE PollInterval 5 DirCheckInterval 10 ActiveFiles 4 <Exec> to_syslog_bsd(); </Exec> Module om_tcp Host Port 514 Path in1 => out_syslog-ng Can you understand why nxlog says that file is truncated when it's not?

Unable to consume AWS s3 log file using NXlog

Hi, I want to send syslog from Windows Server 2012 R2 (using NxLog) from my SIEM (FORTISIEM)