Having a reoccurring problem where I get the following error and my output file quits updating for a few minutes then picks back up again. As a side note, I was having a problem with the eventlog because of an entry that was being written that had an object serialized as xml in the message. That is the exclusion in my application log. Since then, the event log has been working flawlessly.

I found a similar post except that in my instance I have already use a more specific filter.

http://nxlog.org/question/531/iis-log-bad-characters-causes-nxlog-crash-causing-random-event-dump

Log 2015-03-06 13:10:32 INFO nxlog-ce-2.8.1248 started 2015-03-06 13:14:07 ERROR if-else failed at line 70, character 336 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; procedure 'parse_csv' failed at line 70, character 226 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; Not enough fields in CSV input, expected 17, got 0 in input ''

2015-03-06 13:28:30 ERROR if-else failed at line 70, character 336 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; procedure 'parse_csv' failed at line 70, character 226 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; Not enough fields in CSV input, expected 17, got 0 in input ''

2015-03-06 13:50:07 ERROR if-else failed at line 70, character 336 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; procedure 'parse_csv' failed at line 70, character 226 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; Not enough fields in CSV input, expected 17, got 0 in input ''

2015-03-06 14:07:39 ERROR if-else failed at line 70, character 336 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; procedure 'parse_csv' failed at line 70, character 226 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; Not enough fields in CSV input, expected 17, got 0 in input ''

Line 70 refers to the closing bracket in
<Input default_iis_in>
 Module im_file
 File "D:\inetpub\logs\LogFiles\W3SVC1\u_ex*.log"
    ReadFromLast TRUE
    Exec if $raw_event=~/^#/ drop();                         \
   else                                                \
   {                                                   \
    w3c->parse_csv();                               \
    $EventTime = parsedate($date + " " + $time);    \
   }
</Input>

Full config

## This is a sample configuration file. See the nxlog reference manual about the
## configuration options. It should be installed locally and is also available
## online at http://nxlog.org/nxlog-docs/en/nxlog-reference-manual.html

## Please set the ROOT to the folder your nxlog was installed into,
## otherwise it will not start.

#define ROOT C:\Program Files\nxlog
define ROOT C:\Program Files (x86)\nxlog

Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log

<Extension w3c>
    Module      xm_csv
    Fields $date, $time, $s-ip, $cs-method, $cs-uri-stem, $cs-uri-query, $s-port, $cs-username, $c-ip, $cs(User-Agent), $cs(Referer), $sc-status, $sc-substatus, $sc-win32-status, $sc-bytes, $cs-bytes, $time-taken
    FieldTypes  string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string
    Delimiter ' '
    QuoteChar   '"'
    EscapeControl FALSE
    UndefValue  -
</Extension>

<Extension json>
    Module xm_json
</Extension>

<Extension fileop>
    Module      xm_fileop

    <Schedule>
        Every   10 min
        Exec    file_truncate("evt.json");
    </Schedule>

    <Schedule>
        Every   10 min
        Exec    file_truncate("iis-default.json");
    </Schedule>

</Extension>

<Input evt_in>
    Module      im_msvistalog
    Query       <QueryList>\
                    <Query Id="0">\
                        <Select Path="Application">*</Select>   \
                    </Query>\
                </QueryList>
    ReadFromLast TRUE
    Exec        if $SourceName=~/TfsDeployer/ drop();           \
                else                                            \
                {                                               \
                    to_json();                                  \
                }
</Input>

<Input default_iis_in>
 Module im_file
 File "D:\inetpub\logs\LogFiles\W3SVC1\u_ex*.log"
    ReadFromLast TRUE
    Exec if $raw_event=~/^#/ drop();                         \
   else                                                \
   {                                                   \
    w3c->parse_csv();                               \
    $EventTime = parsedate($date + " " + $time);    \
   }
</Input>

<Output evt_out>
    Module om_file
    File "evt.json"
</Output>

<Output default_iis_out>
    Module om_file
    Exec $raw_event = to_json();
    File "iis-default.json"
</Output>

<Route 1>
    Path        evt_in => evt_out
</Route>

<Route 2>
    Path default_iis_in => default_iis_out
</Route>

Any help would be appreciated. Thank you.

Hello,

I'm finding sometimes the message "non-active modification on ....<input_filename>". 

Could anyone give us explanations about it ?

My nxlog is v2.8.1248 on Windows Server 2008 R2.

Thanks

Patrick

Hi,

I'm trying to foward log events with NXlog to logstash over a SSL connection. Therefore, I generated my own CA certificate and a certificate to be used by logstash for the SSL connection. NXlog is installed on a Ubuntu 14.04 (virtual) machine with the CA certificate added as '/etc/ssl/certs/logstash-ca.crt'.

I tested the connection with

openssl s_client -CAfile /etc/ssl/certs/logstash-ca.cert -connect localhost:5000

and everything works fine (all entered text lines are received by logstash).

However, when I start NXlog with the appropriate om_ssl output configuration it fails with

ERROR SSL certificate verification failed: unable to get local issuer certificate (err: 20)

My NXlog configuration looks like this:

User nxlog
Group nxlog

LogFile /var/log/nxlog/nxlog.log
LogLevel DEBUG

# Modules
<Extension _syslog>
    Module xm_syslog
</Extension>
<Extension json>
    Module xm_json
</Extension>

# Nxlog internal logs
<Input internal>
    Module im_internal
    Exec $EventReceivedTime = integer($EventReceivedTime) / 1000000; to_json();
</Input>

<Output sslout>
    Module          om_ssl
    Host            localhost
    Port            5000
    CertFile        "/etc/ssl/certs/logstash-ca.crt"
    OutputType      LineBased
</Output>

<Route logstash>
    Path internal => sslout
</Route>

When I start NXlog with

sudo nxlog -f -c nxlog.conf

the debug output looks like this:

2015-03-05 20:37:05 DEBUG module sslout got 1 poll events
2015-03-05 20:37:05 DEBUG Module sslout can read
2015-03-05 20:37:05 DEBUG [...]
2015-03-05 20:37:05 DEBUG PROCESS_EVENT: READ (sslout)
2015-03-05 20:37:05 DEBUG om_ssl read
2015-03-05 20:37:05 DEBUG doing handshake
2015-03-05 20:37:05 DEBUG verify callback (ok: 0)
2015-03-05 20:37:05 DEBUG preverification returned non-OK: unable to get local issuer certificate
2015-03-05 20:37:05 DEBUG STOP: sslout
2015-03-05 20:37:05 DEBUG remove socket [21]
2015-03-05 20:37:05 DEBUG executing statements
2015-03-05 20:37:05 DEBUG [...]
2015-03-05 20:37:05 INFO reconnecting in 1 seconds
2015-03-05 20:37:05 ERROR [om_ssl.c:532/io_err_handler()] -;[ssl.c:168/nx_ssl_check_io_error()] SSL certificate verification failed: unable to get local issuer certificate (err: 20)

I have no idea why the verification work with OpenSSL but fails with NXlog (I'm using nxlog-ce-2.8.1248 and OpenSSL version is 1.0.1f).

Any help is highly appreciated.

define ROOT C:\Program Files (x86)\nxlog

Moduledir %ROOT%\modules

CacheDir %ROOT%\data

Pidfile %ROOT%\data\nxlog.pid

SpoolDir %ROOT%\data

LogFile %ROOT%\data\nxlog.log

#### Extensions - load and configure global extendsions

<Extension eW3c>

    Module xm_csv

    Fields $date, $time, $s-ip, $cs-method, $cs-uri-stem, $cs-uri-query, $s-port, $cs-username, $c-ip, $csUser-Agent, $sc-status, $sc-substatus, $sc-win32-status, $time-taken

    FieldTypes string, string, string, string, string, string, integer, string, string, string, integer, integer, integer, integer

    Delimiter ' '

    QuoteChar '"'

    EscapeControl FALSE

    UndefValue -

</Extension>

<Extension eFileOp>

    Module xm_fileop

</Extension>

####

#### Inputs

<Input iIIS-fulfillment.Api.1>

    Module    im_file

    File     "D:\\APShared\\Log\\W3SVC1\\\\*"

    SavePos    TRUE

    

#    Exec if $raw_event =~ /^#/ drop();

    Exec file_write("C:\\temp\\nxlog_RawIIS.log", $raw_event+"some text man"); \

        eW3c->parse_csv(); \

        $SourceName = "IIS-fulfillment.Api"; \

        file_write("C:\\temp\\nxlog_IIS-csv.log", $raw_event);

</Input>

####

#### Processor

####

#### Outputs

<Output oIISFile>

    module    om_file

    File    "c:\\temp\\nx_iis.out"

</Output>

####

#### Routes

<Route rIISFile>

    path     iIIS-fulfillment.Api.1 => oIISFile

</Route>

NXLOG.log

2015-03-04 12:13:45 INFO nxlog-ce-2.8.1248 started

2015-03-04 12:14:11 ERROR procedure 'parse_csv' failed at line 40, character 94 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; Not enough fields in CSV input, expected 14, got 0 in input ''

"C:\\temp\\nxlog_RawIIS.log"

contains: some text man

C:\\temp\\nxlog_IIS-csv.log - File not created.

Help.  Where's my $raw_event data?

I have xml style messages that can have thousands of <CUID>xxx</CUID> statements in them so I need to cut out from the middle of the message: from the first string of <CUID> to the last </CUID>. Is this possible?

Hello,
I have followed instructions to install nxlog in a Windows Server Core 2012 machine (no UI only command prompt).

However, when I try to run the service I get: service could not be started

Do you know what I could try to fix this?

This Windows is the instance created automatically by Amazon Beanstalk and without a UI I find this difficult to debug.. so any suggestion is appreciated

Thanks

Dear NXlog community,

I am using nxlog on a windows 2003 environment and i am having some problems with windows failed authentication events.  All entry's with EventID 675 contain the AccountName "SYSTEM" in stead of the username that the failed authentication is for. I couldnt get it to work with pattern matching in nxlog but as i have never used this before i am probably doing something wrong. I would really like to get some statistics of this and get the user name in the AccountName field.

For example kibana is reporting:

AccountName        SYSTEM
AccountType        User
Category        Account Logon
CategoryNumber        9
Domain            NT AUTHORITY
EventID            675
EventType        AUDIT_FAILURE
FileName        Security
Hostname        SomeHostName
Severity        ERROR
SeverityValue        4
SourceModuleName    eventlog
SourceModuleType    im_mseventlog
SourceName        Security
host            SomeHostName.SomeDomain
message            Pre-authentication failed:
             User Name: [username]
             User ID: %{some user id}
             Service Name: krbtgt/office Pre-Authentication
             Type: 0x0 Failure Code: 0x19
             Client Address: [ip address]

Any help is appreciated!

Hello,

I am trying to connect through SSL to ship logs but nxlog only attempts to connect through SSLv3, not TLS. Any way to force TLS 1.1 or 1.2 handshakes?

Thank you

I'm noticing something weird with my NXLog endpoints. If I have my NXLog agents pull windows event logs and transfer them over the network it loses the event time somewhere along the way. My current setup looks like this:

Remote Agent: Windows Event Log -> im__mseventlog -> om_SSL (Type Binary) (to Log Collector)

Log Collector: im_SSL (Type Binary) -> om_UDP (Type GELF) (to Graylog server)

When Graylog is offline it will buffer correctly, but after Graylog comes back online, NXlog will send the logs with the current time.

Hi guys, because the event log doesn't subscribe to debug logs and because I have an app that mangles it's entries I was wondering if you can run a powershell command encapsulated in an if.

eg:

if $SourceName == "IHateThisApp" \

{ \

$Message = *encapsulated powershell command* ;  }

Of if someone know a way to get the debug info out and parsed into nxlog.

Hi

I read the help but still don't understand how to send this log that would like to call impaxauditlog from a windowsbox text file (that is overwritten) to my syslog server. Below are some examples from the log and my current nxlog config setting.

Basically the entries take two lines one starts with !SYS and the other with <IHEYr4>. So based on the example the "!SYS 2015-02-23 11:51:49,478"  is the time the event was logged and "<TimeStamp>2015-02-23T10:08:59+02:00</TimeStamp>" is the real event datetime that I would like use as syslog message datetime (you can ask AGFA for why the difference:) if this is not possible then of course I can settle with the 2015-02-23 11:51:49,478.

In essence the message itself should be everything that is between <IHEYr4></IHEYr4> tags BUT there is one big but - I need to parse the message a little and cut out everything that is between fist <CUID> and last </CUID> as there can be thousands of these and this is just wasting my diskspace. So the syslog message should look somethng like this:

2015-02-23T10:08:59+02:00 windowsbox impaxauditlog <IHEYr4><DicomQuery><CUID></CUID><SyntaxUID>LittleIndianImplicit</SyntaxUID></DicomQuery><Host>xx.xx.xx.xx</Host><TimeStamp>2015-02-23T10:08:59+02:00</TimeStamp></IHEYr4>

Example:

!SYS 2015-02-23 11:51:49,478 - veebr 23 10:08:59 172.16.10.33 <?xml version="1.0" encoding="UTF-8" ?>
<IHEYr4><DicomQuery><CUID>1.3.12.2.1107.5.1.4.65588.30000015021704415668700013901</CUID><CUID>1.3.12.2.1107.5.1.4.65588.30000015021706453421800094763</CUID><SyntaxUID>LittleIndianImplicit</SyntaxUID></DicomQuery><Host>xx.xx.xx.xx</Host><TimeStamp>2015-02-23T10:08:59+02:00</TimeStamp></IHEYr4>

Main part of my current config:

<Input in>

    Module      im_msvistalog
</Input>

<Extension syslog>
Module xm_syslog
</Extension>

<Output sslout>
Module om_ssl
Host xx.xx.xx.xx
Port 10514
Exec to_syslog_ietf();
CAFile %CERTDIR%/ca.cert.pem
CertFile %CERTDIR%/cert.pem
CertKeyFile %CERTDIR%/key.pem
AllowUntrusted TRUE
OutputType Syslog_TLS
</Output>

<Route 1>
    Path        in => sslout
</Route>

Hello,

Having an issue with using om_ssl from a Windows NXlog client to a Linux ELK server.

The issue lies with the encryption part, the connection is established on the associated destination SSL port I set but no SSL traffic can be see/captured.

For information here is the logstash config and the NXlog config, I believe the issue lies with the certs.

input {
  tcp {
    port => 5000
    type => syslog
  }
  udp {
    port => 5000
    type => syslog
  }
  tcp {
    port => 5001
    type => syslog
    ssl_cacert => "/etc/pki/tls/certs/rootCA.pem"
    ssl_cert => "/etc/pki/tls/certs/logstash.crt"
    ssl_key => "/etc/pki/tls/private/logstash.key"
    ssl_enable => true

nxlog.conf:
<Output syslogout>
#This is for TCP non-SSL traffic
  Module om_tcp
  Host 192.168.0.20
  Port 5000
</Output>

<Output omsslout>
#This is for SSL traffic only, omit all "#"
  Module          om_ssl
  Host            192.168.0.20
  Port            5001
  CAFile          %CERTDIR%\rootCA.pem
  OutputType LineBased
  AllowUntrusted FALSE
</Output>

<Route 1>
  Path eventlog => eventlog_transformer => omsslout
</Route>

Only included relative SSL parts as everything else works over TCP and UDP.

Many thanks

Hello, is there any chance that nxlog supports Output om_ssl with OutputType GELF? As descriped here https://www.mail-archive.com/nxlog-ce-users@lists.sourceforge.net/msg00397.html currently NXLog only supports GELF via om_udp .

It would be great, because Graylog now (19. February 2015) supports TLS for all standard TCP and AMQP inputs.

I am getting an error message to nxlog.log. 2015-02-17 08:16:23 INFO nxlog-ce-2.8.1248 started 2015-02-17 08:16:35 ERROR Couldn't read next event, corrupted eventlog?; The data is invalid. And after this error no more events or log messages are generated. However I can read event log with event viewer and I can see new events. These are messages in my graylog2: 2015-02-17 08:16:35.000 wintoosa Couldn't read next event, corrupted eventlog? The data is inval 2015-02-17 08:16:23.000 wintoosa nxlog-ce-2.8.1248 started

Seems that nxlog is running but it can't handle events after this error. How can I fix this ?

My very basic nxlog.conf:

define ROOT C:\Program Files (x86)\nxlog Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data LogFile %ROOT%\data\nxlog.log #LogLevel DEBUG

<Extension syslog> Module xm_syslog </Extension>

<Extension gelf> Module xm_gelf </Extension>

<Input internal> Module im_internal </Input>

<Input eventlog> Module im_msvistalog

For windows 2003 and earlier use the following:

Module im_mseventlog

</Input>

<Output out> Module om_udp Host 10.0.0.103 Port 12900 outputType GELF </Output>

<Route 1> Path internal, eventlog => out </Route>

Botond,

I would like to write logs out to a message queue such as kafka or redis or rabbitmq etc. Is there a road map for that? Alternately would a workaround such as writing out in JSON format to a om_tcp destination work?

Can I also track the state of a tcp connection and send output to an alternate destination? I could write an external shell that would track the nxlog.log for "connection lost/made" and decision off that but I suspect that would be very expensive.

Thanks

Ash 

Hi everyone,

We have an internal application on a windows box which logs in a way which is slightly mangled. All other windows logs come through nxlog but on these messages the entire message field gets dropped.

Is there a way to parse using ifs? eg:

if $service == "Homegrown app" {

}

So that I can either change the output format or parse them untouched into a file etc...

Does anyone know if there are any characters that don't get escaped properly that might mangle the json output?

Sorry for not giving a huge amount of information, I have to keep this as agnostic as I can.

Is there a way to use the Community Edition to log to MS SQL Server? I have tried writing a Perl script to dump the text log files into the database, but have had no luck. I understand that you can use im_odbc and om_odbc in the Enterprise version, but not the Community Edition (CE). Has anyone had any luck doing this in the CE?

Hi,

I am trying to parse a log4net file into json.

Here's my sample log4net:

----------------

2015-01-27 01:06:18,859 [7] ERROR Web.Cms.Content.Base.Taxonomy.TaxonomyDetectionProvider [(null)] - Get taxonomy Type Failed for Tools
2015-01-27 06:34:31,051 [26] ERROR www.Status404 [(null)] - ErrorId: 20150127_102b01c6-3208-48c5-8c8b-ae4f92cf2b20
    UserAgent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.99 Safari/537.36
    HostAddress: 192.168.10.2
    RequestUrl: /ErrorPages/404.aspx
    MachineName: QA01
    Raw Url:/undefined/
    Referrer: http://qa1.www.something.com/toolset.aspx

2015-01-27 06:34:33,270 [26] DEBUG Web.Caching.Core.CacheManagerBase [(null)] - Custom CacheProvider:Web.Caching.Core.AppFabricCacheManager,Web.Caching.Core Disabled

Now I am using xm_multiline to capture each log entries.

----------------

<Extension multiline>
    Module        xm_multiline
    HeaderLine    /^\d{4}\-\d{2}\-\d{2} \d{2}\:\d{2}\:\d{2},\d{3}/
    EndLine        /\r?\n\r?\n^\d{4}\-\d{2}\-\d{2} \d{2}\:\d{2}\:\d{2},\d{3}/
</Extension>

I use a regex to capture the timestamp as the header then I use a regex to capture twice newline then the next timestamp as endline. However it still treat the second and last entry as ONE log entry.

Here's the output:

----------------

{  
   "EventReceivedTime":"2015-01-27 01:06:35",
   "SourceModuleName":"log4net",
   "SourceModuleType":"im_file",
   "time":"2015-01-27 01:06:18,859",
   "thread":"7",
   "level":"ERROR",
   "logger":"Web.Cms.Content.Base.Taxonomy.TaxonomyDetectionProvider",
   "ndc":"(null)",
   "message":"Get taxonomy Type Failed for Tools"
}{  
   "EventReceivedTime":"2015-01-27 06:34:35",
   "SourceModuleName":"log4net",
   "SourceModuleType":"im_file",
   "time":"2015-01-27 06:34:31,051",
   "thread":"26",
   "level":"ERROR",
   "logger":"www.Status404",
   "ndc":"(null)",
   "message":"  ErrorId: 20150127_102b01c6-3208-48c5-8c8b-ae4f92cf2b20\r\n  UserAgent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.99 
Safari/537.36\r\n  HostAddress: 192.168.10.2\r\n  RequestUrl: /ErrorPages/404.aspx\r\n  MachineName: QA01\r\n  
Raw Url:/undefined/\r\n  Referrer: http://qa1.www.something.com/toolset.aspx\r\n\r\n2015-01-27 06:34:33,270 [26] DEBUG Web.Caching.Core.CacheManagerBase [(null)] - Custom CacheProvider:Web.Caching.Core.AppFabricCacheManager,Web.Caching.Core Disabled"
}

I used this to produce that output:

----------------

Exec        if $raw_event =~ /^(\d{4}\-\d{2}\-\d{2} \d{2}\:\d{2}\:\d{2},\d{3}) \[(\S+)\] (\S+) (\S+) \[(\S+)\] \- (.*)/s \
                { \
                    $time = $1; \
                    $thread = $2; \
                    $level = $3; \
                    $logger = $4; \
                    $ndc = $5; \
                    $message = $6; \
                    to_json(); \
                } \
                else \
                { \
                    drop(); \
                }

I've also tried to tweak it by using this to avoid the combining the last two entries as one. However I am not able to get the last entry anymore.

----------------

Exec        if $raw_event =~ /^(\d{4}\-\d{2}\-\d{2} \d{2}\:\d{2}\:\d{2},\d{3}) \[(\S+)\] (\S+) (\S+) \[(\S+)\] \- ([\s\S]*?)(\r?\n\r?\n|$)/ \
                { \
                    $time = $1; \
                    $thread = $2; \
                    $level = $3; \
                    $logger = $4; \
                    $ndc = $5; \
                    $message = $6; \
                    to_json(); \
                } \
                else \
                { \
                    drop(); \
                }​

Will there be a Nxlog for Centos 7.

If so are there an expected date.

BTW has anybody succeded in compiling the tar file. I simply cannot figure out how to build the RPM files.

Regards

Jesper