I've got some data that comes in with a somewhat unusual format.  It's a set of fixed fields, followed by a variable length set of keys, followed by a set of values.  It looks something like this (but with more fields): col1, col2, col3, description(key1; key2; ...;keyn), val1, val2, ..., valn I'm trying to transform this into something more like: a=col1, b=col2, c=col3, key1=val1, key2=val2, ..., keyn=valn I've actually got this working by using Exec and a bit of perl that I wrote that tears apart $raw_event and writes the modified logline to a domain socket, where a second instance Route is listening and sends the log over the network to its destination.  My problem is that this is not terribly performant, since it starts a perl process per log line.  I've had trouble figuring out another way to do this, mostly because the number of keys/values is variable. Any suggestions on ways this might be done that are likely to have better performance?

Hello there! I am using nxlog for tailing specific files (im_file) and output them via GELF_TCP to Graylog which goes good except 1 thing > short_messages get truncated to 64 chars which, if I understand right, is related to this directive (or I might be wrong): ShortMessageLength >This optional directive can be used to specify the length of the short_message field. This defaults to 64 if the directive is not explicitly specified. If the field short_message or ShortMessage is present, it will not be truncated. The question might be fairly simple (to someone who knows): how to use this directive? could someone please write an example how to use it so short_messages do not get truncated? Many thanks in advance! P.S. Same problem here > https://groups.google.com/forum/#!topic/graylog2/wUQIaFdUlZs

I have a client who wants to use Loggly with NXLOG using UNC Paths to shared servers instead of absolute paths but has hit a road block. Instead of using something like this: C:\files\logs\mylog.txt He wants to do something like this: \\computername\windowsshare\c$\mylog.txt When he has the nxlog service set to Local System and points to an absolute path to a local file it works fine. But when he changes the service to be running as a Domain User and then sets his UNC path in the config file he gets an access denied error. He has demonstrated that that Domain User has access to the file by logging in as that user and then opening the path to the file. FYI, he has found that putting \\ in the config file causes problems but using \\\ seems to work. Thanks. -Bill

Greetings, I'm trying to filter event viewer logs by the source name using the following configuration: <Input EventLog>     Module      im_msvistalog     Query       <QueryList>\                     <Query Id="0">\                         <Select Path="Application">*</Select>\                         <Select Path='System'>*[System[(SourceName="Service Control Manager")]]</Select>\                     </Query>\                 </QueryList> </Input>   However it's not working. When I try and filter by Event ID that works no problem. Any assistance would be much appreciated.  

Hi, I have several routes, outputs, and inputs and would like to have a schedule block that effects all logs. Is this possible? My current implementation is to put the schedule block in each Output. It works, but I was seeing if there was a way to clean it up and get it into one scheduler. Thank you!

I'm having an alert every time the om_http sends a message to a web service. Has anyone ever experienced this? LOG: 2015-08-19 16:45:18 INFO connecting to localhost:80 2015-08-19 16:45:18 WARNING http server disconnected while reading the response 2015-08-19 16:45:18 INFO reconnecting in 0 seconds CONF: <Output outATM>     Module              om_http     URL                 http://localhost:80/modules/AtmProcessorMT/index.php </Output>

I am trying to rotate a log file in the Output module. Here is my configuration below: Easier on the eyes (pastebin) ######################################## # Global directives # ######################################## User nxlog Group nxlog LogFile /var/log/nxlog/nxlog.log LogLevel INFO #SuppressRepeatingLogs FALSE define WINLOG /logserv/collections/windows-collector/windows-collector-log.log ######################################## # Modules # ######################################## <Extension fileop> Module xm_fileop </Extension> <Extension _syslog> Module xm_syslog </Extension> <Input windows-collector-log> module im_tcp host 0.0.0.0 port 524 Exec parse_syslog(); Exec log_info("Severity Windows Collector: " + $SyslogSeverity + ", Hostname: " + $Hostname); </Input> <Output windows-collector-log-out> Module om_file CreateDir true File '%WINLOG%' <Schedule> Every 30 sec Exec if (file_size('%WINLOG%') >= 100M) file_cycle('%WINLOG%',500); </Schedule> </Output> <Route 5> Path windows-collector-log => windows-collector-log-out </Route>   I am getting an error saying that the file does not exist when the rotation is executed. 2015-08-19 13:22:23 ERROR failed to determine file size of '/logserv/collections/windows-collector/windows-collector-log.log': No such file or directory

Hi,  I am writing here to inquire about nxLog for centralized logging implementation in production environment.  We have tested nxLog community edition in development environment and We're very much interested in implementing nxLog in production(US/UK client) environment too. I'd appreciate, if someone provides me a detailed product terms and purchase/subscription cost incurred for purchasing nxLog community & enterprise edition. I need to take a decision in the coming few days so it’s really very important that I receive this information as soon as possible. Awaiting reply.  Best Regards, Sreeram

Hi, is it possible to configure at the om_tcp moule  a "closewhenidle". A persistent connection to my destination ist not good when no data sent. 

Hey all,  I was wondering if rindex worked on nxlog, I’m not having luck getting it to work, and all my searches come up empty. I’m looking to parse a username after a "|"  create_var('char'); \ set_var('char', '|'); \ create_var('index_num'); \ set_var('index_num', rindex($UserName, get_var('char'))); \

Hello, Because of my lack of encryption knowledge, I search and found these instructions that I followed in order to create an SSL connection between an nxlog client (Windows server 2008 R2) and a graylog server. So I transfered the "nxlog-ca.crt" to the client and indicated to the graylog server "nxlog-ca.key" as the TLS private key file and "nxlog-ca.crt" as the TLS cert file. Here is the nxlog.conf : define ROOT C:\Program Files (x86)\nxlog define CERTDIR %ROOT%\cert Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data LogFile %ROOT%\data\nxlog.log <Extension _syslog>     Module      xm_gelf </Extension> <Input in>     Module      im_msvistalog         Query    <QueryList>\             <Query Id="0">\                 <Select Path="Application">*[System[(Level=1  or Level=2 or Level=3)]]</Select>\                 <Select Path="Security">*[System[(Level=1  or Level=2 or Level=3)]]</Select>\                 <Select Path="System">*[System[(Level=1  or Level=2 or Level=3)]]</Select>\                 <Select Path="HardwareEvents">*[System[(Level=1  or Level=2 or Level=3)]]</Select>\             </Query>\         </QueryList> </Input> <Output sslout>    Module          om_ssl    Host            host_ip_address    Port            12201    CAFile          %CERTDIR%\nxlog-ca.crt    OutputType      GELF_TCP    AllowUntrusted  FALSE </Output> <Route 1>     Path        in => sslout </Route>   But when I launch "nxlog.exe -f" here is the error : nxlog.exe -f 2015-08-04 12:23:05 INFO nxlog-ce-2.9.1347 started 2015-08-04 12:23:05 INFO connecting to host_ip_address:12201 2015-08-04 12:23:05 INFO successfully connect to host_ip_address:12201 2015-08-04 12:23:05 INFO remote socket was closed during SSL handshake 2015-08-04 12:23:05 INFO reconnecting in 1 seconds   And That's it. What am I missing ? I read in the documentation that all the files about the certificates are in "pem" format but when I create it from certtool I have "crt" and "key" format files.   Thank you.  

Hi, I was wondering if anyone could help me please. I want to use PowerShell to do a silent install of NxLog, I have tried different ways of using MSIEXEC in powershell and command line but I just can't seem to get it working. Any advice would be great.   Thanks.

Sorry I am new to this. Where in the config file do you set it to pull from MSWinEventLog?   Thanks in advance!

We've had some clients where we install & deploy the MSI via group policy -- using the latest version 2.9.1347. On Windows 10 -- it looks like the install succeeds (all the nx log files exist in Program Files (x86), etc.) -- but the service never gets installed. We don't see the nx log service anywhere in services.msc -- and don't see any logs saying that its install failed. We've now seen this on multiple Windows 10 machines. Has anyone else -- and is there a workaround?

Hello I write input module for nxlog. I have wrote function to read data but I don't know how to tranfer data further. Which function should I call? nx_logdata_set_string?

Hi, my ODBC import won't work, checked everything but always this error occurs: ERROR im_odbc couldn't connect to the database, IM014:1:0:[Microsoft][ODBC Driver Manager] The specified DSN contains an architecture mismatch between the Driver and Application Config: <Input ODBC> Module im_odbc ConnectionString DSN=S_ODBC;database=SophosSecurity; SavePos TRUE PollInterval 5 IdIsTimestamp FALSE </Input> best regards

I'm using NXLog to read log files and send to to Logstash. Normally this works fine, but I'm now trying to send logs from a file, where the new events gets added at the top of the file, not the bottom. Now it's not sending anything. This is from my NXLog config. <Input file> Module im_file File "C:\\TEMP\\export.txt" InputType LineBased Exec $Message = $raw_event; SavePos TRUE ReadFromLast TRUE Exec if $raw_event =~ /^#/ drop(); Is it possible to read from bottom to top?

Are there any numbers about how nxLog performs when it is processing a high rate of messages being placed into a log file? Right now we have a couple of incidents which resulted in a few thousand messages being logged per second. I assume this is more than nxLog can handle but am wondering about any performance testing that has been run

Hi, I need to use im_dbi import module of nxlog. This module isn't part of actual setup, so where can I get it? regards Christian

Hi all, after installd nxlog-ce-2.9.1347-1_rhe6.x86_64.rpm on Red Hat I tried to start the service "/etc/init.d/nxlog start" and I'm get the following error: " Starting nxlog deamon... /usr/bin/nxlog: symbol lookup error: /usr/bin/nxlog: undefiend symbol: apr_pool_create_unmanaged_ex   " Does anyone have an idea?