Hi Team,

Hi,

I'm using nxlog to send logs from Windows eventlog to elasticsearch, and using Kibana view.

I'm getting all the message as it is in the 'Message' column, I want to re-order it so the hostname parameter will be the windows server (and not the elasticsearch server), add 'Type' to the messages, etc.

this is the configuration file of nxlog:

* 55.2.110.4=elasticsearch server

<Extension json>
 Module xm_json
</Extension>

# Nxlog internal logs
<Input internal>
   Module im_internal
   Exec $EventReceivedTime = integer($EventReceivedTime) / 1000000;
</Input>
 
# Windows Event Log
<Input eventlog>
# Uncomment im_msvistalog for Windows Vista/2008 and later
   Module im_msvistalog
 
# Uncomment im_mseventlog for Windows XP/2000/2003
#   Module im_mseventlog
 
   Exec $EventReceivedTime = integer($EventReceivedTime) / 1000000;
</Input>
 
<Output out>
   Module om_http
   URL  https://55.2.110.4:443
   HTTPSAllowUntrusted    TRUE
</Output>
 
<Route 1>
   Path internal, eventlog => out
</Route>

this is an example message from kibana:

 

Field Action Value @timestamp   2015-09-08T07:35:47.064Z @version   1 _id   AU-r4dtqVULqkki94YkZ _index   logstash-2015.09.08 _type   logs host   55.2.110.4 http_port   5005 message  

2015-09-08 07:35:43 dc-prod-zone-a.organization.com AUDIT_SUCCESS 4634 An account was logged off. Subject: Security ID: S-1-5-21-1595779987-1987268195-2987234418-1104 Account Name: DC-PROD-ZONE-C$ Account Domain: ORGANIZATION Logon ID: 0x679381 Logon Type: 3 This event is generated when a logon session is destroyed. It may be positively correlated with a logon event using the Logon ID value. Logon IDs are only unique between reboots on the same computer.

 

Thanks a lot

Hi,

I am running NXlog CE version 2.9.1347 on Windows 2012 R2.  The service keeps crashing with the following;

Log Name:      Application
Source:        Application Error
Date:          07/09/2015 09:26:42
Event ID:      1000
Task Category: (100)
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      myServer
Description:
Faulting application name: nxlog.exe, version: 0.0.0.0, time stamp: 0x54fedd1a
Faulting module name: ntdll.dll, version: 6.3.9600.17668, time stamp: 0x54c846bb
Exception code: 0xc0000005
Fault offset: 0x000195da
Faulting process id: 0x3504
Faulting application start time: 0x01d0e94f3d2521e0
Faulting application path: C:\Program Files (x86)\nxlog\nxlog.exe
Faulting module path: C:\Windows\SYSTEM32\ntdll.dll
Report Id: 8c5cf6ce-5542-11e5-80c1-005056bc12a5
Faulting package full name: 
Faulting package-relative application ID: 
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Application Error" />
    <EventID Qualifiers="0">1000</EventID>
    <Level>2</Level>
    <Task>100</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2015-09-07T09:26:42.000000000Z" />
    <EventRecordID>123651</EventRecordID>
    <Channel>Application</Channel>
    <Computer>myServer</Computer>
    <Security />
  </System>
  <EventData>
    <Data>nxlog.exe</Data>
    <Data>0.0.0.0</Data>
    <Data>54fedd1a</Data>
    <Data>ntdll.dll</Data>
    <Data>6.3.9600.17668</Data>
    <Data>54c846bb</Data>
    <Data>c0000005</Data>
    <Data>000195da</Data>
    <Data>3504</Data>
    <Data>01d0e94f3d2521e0</Data>
    <Data>C:\Program Files (x86)\nxlog\nxlog.exe</Data>
    <Data>C:\Windows\SYSTEM32\ntdll.dll</Data>
    <Data>8c5cf6ce-5542-11e5-80c1-005056bc12a5</Data>
    <Data>
    </Data>
    <Data>
    </Data>
  </EventData>
</Event>

I have seen other posts about a hotfix that fixes another app crashing issue.  But does it also fix this one?

Thanks,

Darren.

Hello 

I have created a regex expression to extract values from my logs, I have difficulties in writing these values in the nxlog output

The issue is that the output file is identical to the input log, not sure what I'm doing wrong here, any help is welcome

My nxlog.conf is as follow

<Input in>
        Module  im_file
        File    "C:\\tmp\example-log.txt"
        SavePos  TRUE
        Recursive TRUE
        InputType       multiline
        exec if $raw_event =~ /(?:\{"([0-9]+?), ([0-9]+?), "(.+?)", "(.+?)", "(?:.+?)", "(.+?)\n(?:[a-zA-Z0-9:,"\n\r\s\f\t\-\{\}\.\(\)]+?)New\sLogon:\s*(?:.+?)\n\s*Account\sName:\s*(.+?)\n(?:[a-zA-Z0-9:,"\n\r\s\f\t\-\{\}\.\(\)]+?)Source\sNetwork\sAddress:\s*([0-9\.]{7,15})\n(?:[a-zA-Z0-9:,"\n\r\s\f\t\-\{\}\.\(\)]+?)"\})/ \
    { \
    $timestamp = $1; \
    $event = $2; \
    $status = $3; \
    $type = $4; \
    $short = $5; \
    $user = $6; \
    $source = $7; \
    }
</Input>

#<Output out>
#    Module      om_tcp
#    Host        192.168.1.15
#    Port        12201
#    OutputType  GELF_TCP
#</Output>

<Processor one>
    Module      pm_null
    Exec        $raw_event = $timestamp + $event + $status + $type + $short + $user + $source;
</Processor>

<Output out>
    Module    om_file
    File    'C:\\tmp\output'
</Output>

<Route 1>
    Path        in => one => out
</Route>

Thanks heaps

Mark

Hy, I need some help in configuring my nxlog in windows XP.

System:
S.O. -> windows XP
NXLOG Version -> 2.8.1248

LOGS:
We use a log file for each event and these log files are stored in an hierarchical directory as: f:\year\month\day\accxxxx.txt (e.g. f:\2015\09\03\event000001.txt)
​

nxlog.conf excerpts:

<Extension multiline>
    Module xm_multiline
    HeaderLine '<?xml version="1.0" encoding="UTF-8" standalone="yes"?>'
</Extension>

<Input in>
    Module       im_file
    File         'E:\\hl7\\' + strftime(now(),"%Y\\\\%m\\\\%d") + '\\AccEvent*.txt'

    SavePos      TRUE
    ReadFromLast TRUE
    PollInterval 15
    Recursive    TRUE

    InputType    multiline

</Input>

I tested these cases:
1. (service nxlog active), add new file in  f:\2015\09\03\-> nxlog find the new file and analyze [OK]
2. Stop service nxlog, add new file in f:\2015\09\03\, start service -> nxlog don't find the new file. [X]
3. (service nxlog active), rename a file  in f:\2015\09\03\ already analyzed -> nxlog find the file renamed and analyze (in the log warning input file deleted) [OK]
4. (service nxlog active), update an existing file (same name, change only datetime last modified) -> nxlog don't analyze [OK]

I need to resolve step 2.
If I set ReadFromLast =FALSE when I restart service all fiels are all analyzed and that is not the desiderata.

Thanks in advance for your help.

Dario

Hello,

I think this is an easy answer, but I am having some issues.  I am trying to read in a log file and send out a syslog message.  If the line in the file contains the word error, I want a syslog message to be sent.  If the file does not contain that word, just drop the message.  What would be the proper way to do this?  Below is what I currently have.

<Input watchfile_%service%>
  Module im_file
  File '%servicepath%'
  Exec $Message = $raw_event;
  Exec $SyslogSeverityValue = 6;
  Exec if file_name() =~ /.*\\(.*)/ $SourceName = $1; 
  Exec if $raw_event =~ /ERROR/ $SyslogSeverityValue = 3; 
  SavePos TRUE  
  Recursive TRUE
  PollInterval 10
</Input>

Thank You

Hi,

I am trying to read multiline xml files (results of ELMAH logs in ASP.NET), but for some of them I am unable to do so, as they contain lots of information. This results in errors like this:

2015-09-02 08:50:50 ERROR data size (119671) is over the limit (65000), will be truncated

I tried googling for a resolution to this problem, but I couldn't find any information anywhere about a data size limit in nxlog. Could some please advise what I could do to fix this?

Hello,

Recently, i ran into a problem with file locking. One application here is generating some log file, one per client, when nxlog try to read file, i got error :

ERROR failed to open LOGFILEPATH; Le processus ne peut pas accéder au fichier car ce fichier est utilisé par un autre processus.  

Initaly my configuration was :

<Input Orbis>
    Module im_file
    File 'LogDirectory\\*'
    SavePos TRUE
    ReadFromLast FALSE
    CloseWhenIdle TRUE
    InputType orbisLog
    Exec if $raw_event !~ /^L:.*/  drop();
    Exec $ClientNAme = replace(file_basename(file_name()),'.LOG','');

</Input>

I have changed the PollIntervall to 60 and 3600, no effect... Except that at PollInterval 3600, nxlog grab 40% of CPU.

Is there a way to insctruct nxlog to retry to read the file ?

Hi ,

I am trying to compile nxlog on solairs 11 sparc platform.

I am encoutering the following errors while compiling.

configure: error: libapr-1 not found

gcc: error: unrecognized command line option '-mt'

I used the following parameters :: ./configure APRCONFIG=/usr/apr/1.3/bin/apr-1-config

Please let me know if any one of you have successfully compiled it on Solaris 11.

Thanking you with regards.

Hello,

I'm quite new to nxlog, so forgive me if my question is  trivial but I'm having hard time to get the values I extract from my  logs using exec and a regex. I have a very large stash of old windows logs in text file (in multiline format), what I want to do is use nxlog to load them in graylog, but I want to format the log in a different way.

It works perfectly when I do not use the the exec and the regexp, but it fails with the message:

2015-08-31 12:12:42 ERROR invalid keyword: $timestamp at C:\Program Files (x86)\nxlog\conf\nxlog.conf:36

The regex works when I test it using http://www.regexr.com/

The error seems in the way I'm trying to assign/write the variables matched by the regex

My nxlog.conf is like the below:

## This is a sample configuration file. See the nxlog reference manual about the
## configuration options. It should be installed locally and is also available
## online at http://nxlog.org/nxlog-docs/en/nxlog-reference-manual.html

## Please set the ROOT to the folder your nxlog was installed into,
## otherwise it will not start.

#define ROOT C:\Program Files\nxlog
define ROOT C:\Program Files (x86)\nxlog

Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log

<Extension gelf>
    Module       xm_gelf
</Extension>

<Extension multiline>
    Module      xm_multiline
    HeaderLine  /^{/
    EndLine     /^}/
</Extension>

<Input in>
        Module  im_file
        File    "C:\\tmp\\\\example-log.txt"
        SavePos  TRUE
        Recursive TRUE
        InputType       multiline
        exec if $raw_event =~ /(?:\{"([0-9]+?), ([0-9]+?), "(.+?)", "(.+?)", "(?:.+?)","(.+?)\n(?:[a-zA-Z0-9:,"\n\r\s\f\t\-\{\}\.\(\)]+?)New\sLogon:\s*(?:.+?)\n\s*Account\sName:\s*(.+?)\n(?:[a-zA-Z0-9:,"\n\r\s\f\t\-\{\}\.\(\)]+?)Source\sNetwork\sAddress:\s*([0-9\.]{7,15})\n(?:[a-zA-Z0-9:,"\n\r\s\f\t\-\{\}\.\(\)]+?)"\})/g; 
     
    $timestamp = $1; 
    $event = $2; 
    $status = $3; 
    $type = $4; 
    $short = $5; 
    $user = $6; 
    $source = $7; 
    
</Input>

#<Output out>
#    Module      om_tcp
#    Host        192.168.1.15
#    Port        12201
#    OutputType  GELF_TCP
#</Output>

<Output out>
    Module    om_file
    File    'C:\\tmp\\output'
</Output>

<Route 1>
    Path        in => out
</Route>

 Could someone kindly help me out on this? I tried to read the nxlog manual but I need to confess that I could not understand what I'm doing wrong 

Basically I just would the output to be composed by the raw message and the few fields I match with the regex

Thanks very much!

Mark

I've got some data that comes in with a somewhat unusual format.  It's a set of fixed fields, followed by a variable length set of keys, followed by a set of values.  It looks something like this (but with more fields):

col1, col2, col3, description(key1; key2; ...;keyn), val1, val2, ..., valn

I'm trying to transform this into something more like:

a=col1, b=col2, c=col3, key1=val1, key2=val2, ..., keyn=valn

I've actually got this working by using Exec and a bit of perl that I wrote that tears apart $raw_event and writes the modified logline to a domain socket, where a second instance Route is listening and sends the log over the network to its destination.  My problem is that this is not terribly performant, since it starts a perl process per log line.  I've had trouble figuring out another way to do this, mostly because the number of keys/values is variable.

Any suggestions on ways this might be done that are likely to have better performance?

Hello there!

I am using nxlog for tailing specific files (im_file) and output them via GELF_TCP to Graylog which goes good except 1 thing > short_messages get truncated to 64 chars which, if I understand right, is related to this directive (or I might be wrong):

ShortMessageLength >This optional directive can be used to specify the length of the short_message field. This defaults to 64 if the directive is not explicitly specified. If the field short_message or ShortMessage is present, it will not be truncated.

The question might be fairly simple (to someone who knows): how to use this directive? could someone please write an example how to use it so short_messages do not get truncated?

Many thanks in advance!

P.S. Same problem here > https://groups.google.com/forum/#!topic/graylog2/wUQIaFdUlZs

I have a client who wants to use Loggly with NXLOG using UNC Paths to shared servers instead of absolute paths but has hit a road block.

Instead of using something like this: C:\files\logs\mylog.txt

He wants to do something like this: \\computername\windowsshare\c$\mylog.txt

When he has the nxlog service set to Local System and points to an absolute path to a local file it works fine.

But when he changes the service to be running as a Domain User and then sets his UNC path in the config file he gets an access denied error.

He has demonstrated that that Domain User has access to the file by logging in as that user and then opening the path to the file.

FYI, he has found that putting \\ in the config file causes problems but using \\\ seems to work.

Thanks.

-Bill

Greetings,

I'm trying to filter event viewer logs by the source name using the following configuration:

<Input EventLog>
    Module      im_msvistalog
    Query       <QueryList>\
                    <Query Id="0">\
                        <Select Path="Application">*</Select>\
                        <Select Path='System'>*[System[(SourceName="Service Control Manager")]]</Select>\
                    </Query>\
                </QueryList>
</Input>

However it's not working. When I try and filter by Event ID that works no problem. Any assistance would be much appreciated.

Hi,

I have several routes, outputs, and inputs and would like to have a schedule block that effects all logs. Is this possible? My current implementation is to put the schedule block in each Output. It works, but I was seeing if there was a way to clean it up and get it into one scheduler.

Thank you!

I'm having an alert every time the om_http sends a message to a web service. Has anyone ever experienced this?

LOG:
2015-08-19 16:45:18 INFO connecting to localhost:80
2015-08-19 16:45:18 WARNING http server disconnected while reading the response
2015-08-19 16:45:18 INFO reconnecting in 0 seconds

CONF:
<Output outATM>
    Module              om_http
    URL                 http://localhost:80/modules/AtmProcessorMT/index.php
</Output>

I am trying to rotate a log file in the Output module. Here is my configuration below:

Easier on the eyes (pastebin)

########################################

# Global directives                    #

########################################

User nxlog

Group nxlog



LogFile /var/log/nxlog/nxlog.log

LogLevel INFO

#SuppressRepeatingLogs FALSE



define WINLOG /logserv/collections/windows-collector/windows-collector-log.log



########################################

# Modules                              #

########################################

<Extension fileop>

        Module          xm_fileop

</Extension>



<Extension _syslog>

        Module          xm_syslog

</Extension>







<Input windows-collector-log>

        module          im_tcp

        host            0.0.0.0

        port            524

        Exec            parse_syslog();

        Exec            log_info("Severity Windows Collector: " + $SyslogSeverity + ", Hostname: " + $Hostname);

</Input>



<Output windows-collector-log-out>

        Module          om_file

        CreateDir    true

        File            '%WINLOG%'



     <Schedule>

                Every           30 sec

                Exec            if (file_size('%WINLOG%') >= 100M) file_cycle('%WINLOG%',500);

     </Schedule>

</Output>



<Route 5>

    Path            windows-collector-log => windows-collector-log-out

</Route>

I am getting an error saying that the file does not exist when the rotation is executed.

2015-08-19 13:22:23 ERROR failed to determine file size of '/logserv/collections/windows-collector/windows-collector-log.log': No such file or directory

Hi, 

I am writing here to inquire about nxLog for centralized logging implementation in production environment. 

We have tested nxLog community edition in development environment and We're very much interested in implementing nxLog in production(US/UK client) environment too. I'd appreciate, if someone provides me a detailed product terms and purchase/subscription cost incurred for purchasing nxLog community & enterprise edition.

I need to take a decision in the coming few days so it’s really very important that I receive this information as soon as possible. Awaiting reply. 

Best Regards,

Sreeram

Hi,

is it possible to configure at the om_tcp moule  a "closewhenidle". A persistent connection to my destination ist not good when no data sent. 

Hey all, 

I was wondering if rindex worked on nxlog, I’m not having luck getting it to work, and all my searches come up empty. I’m looking to parse a username after a "|" 

create_var('char'); \
set_var('char', '|'); \
create_var('index_num'); \
set_var('index_num', rindex($UserName, get_var('char'))); \