i defined an input for im_file that is being sent as syslog i have multiple files in a folder what i want to do is read each file and on EOF copy to another folder. didn`t find the option to identify EOF

Hi, I'm trying to use the xm_multiline module with nxlog to forward content of a logfile to logstash The log contains different xml elements which are properly indented (opening and closing elements are located at the start of the line) . E.g. <data version="x" xmlns:bla="http://www.example.com/bla"> <val:InfoSet> ... ... ... </val:InfoSet> </data> <message ...> <ns>bla</ns> ... ... </message> Because the elements have different names, I can only use < and </ to find the start and end line. I was hoping a filter like this should be enough to select the correct lines: HeaderLine /^</ EndLine /^<// But somehow nxlog gets confused with the / in the regex pattern. I also tried escaping which dindn't help. More testing showed that it needs at least one letter. I tried to specify all letters via regex but that didn't work: HeaderLine /^<[a-z]/ Only way that seems to work is to specify all letters in the square braket (with the exception of the lettern, which breaks).  HeaderLine /^<[abcdefghijklmopqrstuvwxyz]/ (left out n) Here all my test results. These lines worked: HeaderLine /^<m/ EndLine /^</m/ HeaderLine /^<m/ EndLine /^<\/m/ HeaderLine /^<[abcdefghijklm]/ EndLine /^<\/[abcdefghijklm]/ HeaderLine /^<[abcdefghijklmo]/ EndLine /^<\/[abcdefghijklmo]/ HeaderLine /^<[abcdefghijklmopqrstuvwxyz]/ (left out n) EndLine /^<\/[abcdefghijklmopqrstuvwxyz]/ HeaderLine /^<[abcdefghijklmopqrstuvwxyz]/ (left out n + not escaped EndLine /^</[abcdefghijklmopqrstuvwxyz]/ These lines didn't work: HeaderLine /^</ EndLine /^</m/ HeaderLine /^<[a-z]/ EndLine /^</m/ HeaderLine /^<\w/ EndLine /^</m/ HeaderLine /^<[abcdefghijklmn]/ EndLine /^<\/[abcdefghijklmn]/ HeaderLine /^<[bcdefghijklmn]/ EndLine /^<\/[bcdefghijklmn]/ HeaderLine /^<[abcdefghijklmopqrstuvwxyzn]/ EndLine /^<\/[abcdefghijklmopqrstuvwxyzn]/ HeaderLine /^<[abcdefghijklmnopqrstuvwxyz]/ EndLine /^</[abcdefghijklmnopqrstuvwxyz]/ Right now I still have a problem because many of my bessages start with <n. I think this is a bug in the module. Can you confirm so I can open a ticket? Thanks   Fyi, this is a duplicate of http://stackoverflow.com/questions/27429234/which-headerline-and-endline-for-multiline-xml-with-different-elements  

Hello Team, I'm looking to use nxlog to retrieve a table entries from MS SQL , To be clear I do not require any of the database server logging only the info contained within a few tables within a database on the server. I have followed the setup and battled first under windows and then linux versions of nxlog. I am now trying to use libdbi-freetds without success after having out of memory upon using the mysql hooks. Here is my im_dbi without the user and pass for security. <Input DB01-elog>     Module      im_dbi     SavePos     TRUE     SQL         SELECT AllXml FROM db_table     Driver      freetds     Option      host 192.168.148.227     Option      port 49000 (non standard port changed from 1433)     Option      username domain\username     Option      password password     Option      dbname database </Input>   The SQL server im attempting to connect to is MS SQL Server 2014, I also attempted this using mysql however recieved out of memory from the libdbi during execution of the query. The OS machine has 8GB of mem available currently upped from 4GB when the out of memory initially occured. I'm secretly hoping someone has run into this before and has been able to successfully retrieve table data and log it from MS SQL Server 2014. If not im happy to work with the team to resolution on this. Many Thanks, GC

Hello! I need to make some changes in sources of modules. Ok - I found necessary code, changed it. But how to compile windows version? No documentation found :(

Hi Please help to exclude/ignore logs for a particular application   Thanks Virender

We are using nxlog to collect eventlog information. Some entries can be large, in fact some message are split over several entries as a workaround for the maximum eventlog entry size. However, these large entries seem to hang nxlog so that it stops processing new entries. Typical error messages are: --------------------------------------- 2014-10-27 17:10:32 ERROR EvtNext failed with error 1734: The array bounds are invalid.   2014-10-27 17:10:33 ERROR EvtUpdateBookmark failed: The handle is invalid. ---------------------------------------- Why is this? Is there any workaround?  

We are using NXlog to send Windows EventLogs and IIS logs to Logstash (ELK).  It works very well indeed with only one problem - at 2am (we suspect IIS log rotation, or maybe weird Netscaler packet) every day the NXlog service crashes on ALL servers that are sending IIS logs.  We have some other servers only sending EventLogs - no crashing occurs.  I have set the Windows NXlog service to restart on a crash, so service is only interrupted for a minute, but here's the weird part - NXlog re-sends a (random?) huge chunk of messages.  I say 'random', because check out this list of time received vs. event time: (these are all from the same server) 2014-10-28T06:00:54.074Z & 2014-10-25 04:18:08 2014-10-28T06:00:54.074Z & 2014-10-25 03:58:08 2014-10-28T06:00:54.072Z & 2014-10-25 03:20:02 2014-10-28T06:00:54.058Z & 2014-10-25 03:10:33 2014-10-28T06:00:54.058Z & 2014-10-25 03:18:10 The weird time received jumping forward/backward in time continues (perhaps it's an ELK stack processing oddity)...keeping in mind we're talking hundreds of entries over a 1-2 minute period.  On our UAT servers I observed that this large chunk of logs contains logs from the last week or more - but it's not consistent enough to say 'it's re-sending everything from the last week'. Anyways - what causes the crash is bad input, like weird bad characters...I am sure the following copy/paste won't do it justice.   If anyone has ideas, I'm open!!  Happy to either fix the weird log dumps, or fix the bad input data.  We're continuing to investigate, but I thought it important (and not Google-friendly) enough to post here.  A number of Google hits for questions like mine with no answers (or 'no bad input filtering, is a bug, sorry'). (disregard the timestamp) 2014-10-22 02:51:39 Not enough fields in CSV input, expected 22, got 1 in input 'PK ' 2014-10-22 02:51:39 Not enough fields in CSV input, expected 22, got 1 in input 'U@¦Ä-*¸H¶ ÜH—ÙùPÜÙ¡žŒÛŠ”¹‘ðè²Þè' 2014-10-22 02:51:39 Not enough fields in CSV input, expected 22, got 2 in input '¹'O‚¨ˆÌ(ˆÌÈF&È”™™ØÌDEéA3)B3éA3±¡é#eEff63AO‚È,ŠÈ,‚È,zÈ,ŠÈ,ld¢Ùé1³g3³&ŸI™½ 2:2KBõÔPD2£F‚³ËäL¡¢PÜÙÁ¢' 2014-10-22 02:51:39 cannot parse integer, invalid modifier: '+' 2014-10-22 02:51:39 Not enough fields in CSV input, expected 22, got 1 in input '%6»Ô^È=ùÙý™y¨(wv°(#2Ñ¡³%ÍdlhúH±ýŽ• .ê13' 2014-10-22 02:51:39 Not enough fields in CSV input, expected 22, got 2 in input '2³Éòì' 2014-10-22 02:51:39 cannot parse integer, invalid modifier: '³' 2014-10-22 02:51:39 Not enough fields in CSV input, expected 22, got 2 in input 'zÈ ‘©¨€Ê È́ÍLž' 2014-10-22 02:51:39 Not enough fields in CSV input, expected 22, got 1 in input 'PY°‘ zb'ETè[1]ÈÏNÏÿºÿA£Ó@….€þ‹Ô~WO÷?G÷7]' 2014-10-22 02:51:39 Not enough fields in CSV input, expected 22, got 1 in input 'ß;º' 2014-10-22 02:51:39 Not enough fields in CSV input, expected 22, got 1 in input 'ŠÌlº…;;X”q‹' 2014-10-22 02:51:39 Not enough fields in CSV input, expected 22, got 1 in input '>›™5LEptzÌ\(2sÁf&*Jš•' 2014-10-22 02:51:39 Not enough fields in CSV input, expected 22, got 1 in input 'M°xU™•ÍL”‰' 2014-10-22 02:51:39 Not enough fields in CSV input, expected 22, got 5 in input 'M´xUš•MMwóÀ†fM>“‑3-è1ÓL™fzÌ4Óc¦™ 3Íô˜i¦ÇÌÈf&š]dCô$ÈÌ$ÈÌ$ÈÌ$È̤ÈÌ$ÈÌÄf&È$h,ë13 2SQ[1]YaC3øH‚̤K °v…L?:º    

Trying to configure the NXLog forwarder to load balance it's output stream. I tried creating 2 output stanzas out1 and out2 with the appropriate IPs and then routed as Path   eventlog => out1, out2. This seemed to clone the output stream as logs were going to both receivers. What would be the proper way to tell NXLog to use one IP if available, else another IP?

Currently process and transform the windows event/iis logs on the client, however as I have more servers I am wondering about routing everything to a central point using the binary format and then processing them into the relevant tables in to a mysql db. I am struggling with at which stage this filtering and tansforming is done and what the route should look like. Do I use the patern filter in a process stage and then use and if statement in the route based on the patern id? Clients im_msvistalog => om_tcp (binary) w3c extension(im_file) => om_tcp (binary) Server im_tcp => ?????????? => ?????? (om_dbi but based on source message type evntlog table |syslog table |iis log table|apache log table|security log table) Can you point me in the right direction?

I am trying to use the perl extension in Windows, but nxlog complains that it cannot find the xm_perl.dll. The complaint is correct, the file is not there. My question is why? Does the windows version not support using perl?

Hi,  I really would like some assistance in the forum -  Here is the scenario: Installed nxlog.c.e in Windows 2008 R2 Used the query list -to get security logs Added that in the module - started the server - everything is fine. I tested this in my Windows mahcine - to my Redhat server. Works. The issue happens when the Windows machine is registered to the domain. Once the service is started in windows, the syslog server creates host folders which include the Windows client name  as well as a bunch of other stuff. Such as Authlite, Process, 0 00 1  etc etc.. list goes on. Has anyone encountered this issue?  Kindly advise.   Additional information:  This issue only occurs with windows client - registered in the domain.      When the nxlog forwards information to my rsyslog server - (which uses the template dyn$ to create host folders)  I get folders with 0 00 Authlite ... etc etc.  Is there anyway to stop this of fix it.  Again it only happens with Windows client registered in the domain.  

I am unable to find the dependencies for wheezy. apt-get -f install simply removes nxlog. My version of debian is : Linux chis-logcollector 2.6.32-5-amd64 #1 SMP Sun Sep 23 10:07:46 UTC 2012 x86_64 GNU/Linux Install pkg: nxlog-ce_2.8.1248_amd64.debian-wheezy.deb Is there a plan to include dependencies as this is a rather flaky way of installation? Thanks Ash Kumar

Hello, It seems that there is some kind of conflict or bug in nxlog-ce-2.8.1248 (I'm not sure about earlier versions) where, upon boot, nxlog gets stuck in a loop initiating connections to a remote machine (Logstash in my case using om_tcp). I only became aware of this after installing nxlog on server that had the group policy "Audit Filtering Platform Connection" enabled on successful connections; this actually generated so many events that it filled up my elasticsearch cluster (I've attached a sample of the event the filled the cluster). The issue doesn't happen when nxlog service is configured for "Automatic (Delayed Start)". I hope this helps. Thanks