Would this be the correct placement to add the filter event ID string?  Should anything esle be commented out?

#Windows Event Logging of Security,System and Application Logs  

Module      im_mseventlog  

Exec to_syslog_snare();

Exec if $EventID NOT IN (528, 529, 567, 592, 601, 602, 608, 612, 636, 7034, 7035, 7036, 7040, 4097, 64004, 2, 3005) drop();

</Input>

 

 

 

When NXLog ships a Windows event, it appears to be changing the Windows original severity level, and replacing it with SeverityValue and Severity with different values.   What is the mapping of these values?  If Windows has severity values, with "Level" being 4 for Information, 3 for Warning, 2 for Error and 1 for Critical, what is the nxlog created SeverityValue?

 

Also, I couldn't find an explaination of why this value is changing.  

Hi,

We are testing NXlog to ship the security logs to our security team.  We are using XP embedded and it working fine.  Security is asking us to only send specific event ID's.  I have looked at the documentation and it appears that we may not be able to do this with XP.  I was going to do the custom view but XP does not support that. We are supported on the embedded version of XP.

My question.  Has anyone tried to do this with XP or is it even possible?

Thanks,
David Martin

 

Hi all! Im new and would really appreciate any assistance/feedback on implementation... I'm trying to implement Audit Reduction in my environment which includes (4) Windows 7 boxes , (2) RHEL 7 boxes (SYSLOG) and (1) RHEL 6 box (SYSLOG)... We have all the logs writing to one of the Windows 7 workstations. The logs need to be manually pulled off and put in another location. This environment is replicated 7 times (7 systems with the same configuration with 7 appliances each) They ultimately get to a Dropbox that forwards them off to an ElasticSearch/Kibana instantiation....So 49~ devices/appliances reporting. I would like to know if the Community Edition would work or if I need to purchase Enterprise? Does NXlog need to be installed on each appliance? Thanks!

Hi Everyone,
New forum user, so sorry if I do not respect some uses rules…and I’m French by the way, so sorry about my bad English level too…;-)
I use Nxlog in the following context :
A machine A send periodically a file on a machine B (same file name)
Machine B is used like a first level of log centralization. Nxlog is installed on it
Then, Nxlog send log to Machine C who host Logstash
I’m looking for a process who remove input files after end of reading and processing, but I’m not sure about how process “in-out” operate, is-it after each line or after a n entire file reading ?
How can I test a “end of reading” condition ?
Right now, I’m going to test something like that :
<Extension fileop>
 Module xm_fileop
</Extension>

<Input in>
    Module im_file
    SavePos TRUE
    ReadFromLast TRUE
    PollInterval 900 # 15mn
    File "/var/log/inpufile*.log"
 Exec $InputFileName1 = file_name()
</Input>

<Output logstash>
    Module om_tcp
 Port 6002
Host 10.x.y.z
Exec file_remove($InputFileName1);
</Output>

<Route 1>
    Path in => logstash
</Route>

Is it correct ?

Thanks in advance for your help

Hi there,

 

Has any of you encountered this virus warning? The scan was done using TrendMicro HouseCall.

Is my nxlog infected? Thanks for your help

nxlog is crashing with the following error:

 

Faulting application name: nxlog.exe, version: 0.0.0.0, time stamp: 0x5666d55e

Faulting module name: ntdll.dll, version: 6.3.9600.18202, time stamp: 0x569e72c5

Exception code: 0xc0000005

Fault offset: 0x000192cb

Faulting process id: 0x1b60

Faulting application start time: 0x01d18540c8297bd3

Faulting application path: C:\Program Files (x86)\nxlog\nxlog.exe

Faulting module path: C:\Windows\SYSTEM32\ntdll.dll

Report Id: 06d89363-f134-11e5-80dd-005056a619fb

Faulting package full name:

Faulting package-relative application ID:

 

config file:

 

define ROOT C:\Program Files (x86)\nxlog

Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log

<Extension json>
    Module      xm_json
</Extension>

<Input eventlog>
    # Use 'im_mseventlog' for Windows XP and 2003
    Module      im_msvistalog
SavePos FALSE
ReadFromLast FALSE
 Query       <QueryList>\
                   <Query Id="0">\
                        <Select Path="Pool2PdfCreator.Produce">*</Select>\
                    </Query>\
                </QueryList>
</Input>

<Output out>
    Module      om_tcp
    Host        10.36.52.62
    Port        12201
    Exec        $EventTime = integer($EventTime) / 1000000; to_json();
    Exec log_info("RecordNumber: " + $RecordNumber);
</Output>

<Route r>
    Path        eventlog => out
</Route>

 

(during troubleshooting, I have narrowed down the query to one eventsource and also added

Exec log_info("RecordNumber: " + $RecordNumber);

to be able to pinpoint the exact entry that causes the issue. I was able to locate the entry that causes the crash. The strange thing is, it sometimes goes through, most of the times causes crash. I am not comfortable with sharing the entry here but I can send it via e-mail. This definitely looks like a bug.

 

nxlog version: nxlog-ce-2.9.1504

Using nxlog in front of logstash on a server.  On the same user, I want to use nxlog to replace rsyslog.  Seems pretty simple.  The only issue is the file format is slightly different than the what rsyslog outputs.  I see:

<78>May  6 13:50:01 CRON[19454]: (root) CMD (   /opt/observium/discovery.php -h new >> /dev/null 2>&1)

vs:

Jul 16 18:00:01 monitor01 CRON[6871]: (root) CMD (   /opt/observium/poller-wrapper.py 16 >> /dev/null 2>&1)

Main difference is the <NN> at the beginning of the line and the missing hostname (monitor01).  Here is my .conf:

<Input in_uds>
    Module    im_uds
    UDS       /dev/log
</Input>

<Input in_kernel>
    Module    im_kernel
</Input>

<Output out>
    Module    om_file
    File      "/var/log/syslog"
</Output>

<Route local_route>
    Path      in_uds, in_kernel => out
</Route>

Is there a simple change I can make to get the desired format?  Thanks.  -- Bud

 

 

I'm using NXlog to ship windows event logs to an ELK stack.  I need to preserve the datetime when the event happened <TimeCreated SystemTime=> that is stored in the event log.

 

However, the NXLog that is shipped doesn't preserve <TimeCreated SystemTime>, which I assume is because its invalid json.  How can I preserve this in my nxlog.conf?  Otherwise, I'm stuck with EventTime, which appears to be the datetime of when nxlog processes the event, not when the event happened.

 

How do I handle this?

Hello everyone,

This is my first time posting in this community forum, so any help would be greatly appreciated.

I've been working with NxLog and ElasticSearch for a few months now and I've had mostly no issues with it until very recently, where a new ElasticSearch index was created in order to accomodate the new structure of our logs. With that, we also had to update our existing nxlog.conf file.

We have about 3 different ElasticSearch endpoints with the same setup that we currently work with, and at one point during the week we had run out of storage space. After increasing the storage size for all 3 endpoints, two environments appeared to continue sending new information up to ElasticSearch with no further problems. However, the third environment's NxLog services appear to be stuck in both existing AWS instances and newly created instances, repeating the following NxLog log entry over and over:

2016-04-29 15:33:12 INFO connecting to search-stage-logging-udf7h4lq2bsm245ciawp2stcvu.us-east-1.es.amazonaws.com:80
2016-04-29 15:33:12 INFO reconnecting in 1 seconds
2016-04-29 15:33:12 ERROR ### PANIC at line 2456 in module.c/nx_module_pollset_add_socket(): "failed to add descriptor to pollset: Not enough space ; [cannot dump backtrace on this platform]" ###

This was the log entry that we initially discovered that alerted us we had run out of space in ElasticSearch. However, the ElasticSearch dashboard does not show a lack of space anymore, so it's a bit confusing why NxLog would continue to output log entries in the other two environments

Basically, I have two questions:
1. Is this a type of scenario where the NxLog service's working state has been stuck unable to see there is space available? Or does the fault lie with ElasticSearch not showing storage space correctly?
2. If the Nxlog service has been stuck in this state, is there a configuration or some other automated procedure for NxLog to get the service to restart itself on multiple failures?

 

If anyone has gone through a similar experience, any tips would be greatly appreciated. Thank you for your time.

The article on structured logging (https://nxlog.co/why-use-structured-logging) shows how you should use structured logging so that changes in log format is minimized.  The example of the sysmon event, process creates, shows what I think is a bug in NXLog.

 

The ProcessID in the "Message" is the ProcessID (25848) of the new process that sysmon sees created.  However, in the structured NXLog key/values, the ProcessID is that of sysmon itself (1680)

 

The only way to get the ProcessID of the process sysmon observed created, is to use regular expressions.  Can you verify that this is a bug in NXLog?

Hello Guys,

I have a question about NXLog IIS7 W3C logs. I set NXLog up and it works basicaly but the NXLog logs are full with error messages like this:

2016-04-26 09:46:36 ERROR if-else failed at line 64, character 257 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; procedure 'parse_csv' failed at line 64, character 113 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; couldn't parse integer: language=UK&region=802&idfwbonavigation=180173.2

It look, the IIS logs contain an "=" sign in the $cs-uri-query field, and NXLog wants integer after the "=", however the field set to be string in the config file.

Have you ever met anything like this? And if yes, what could be the solution?

 

NXLog extension and input config:

<Extension exiis>
    Module        xm_csv
    Fields        $date $time $s-ip $cs-method $cs-uri-stem $cs-uri-query $s-port $cs-username $c-ip $cs(User-Agent) $cs(Cookie) $cs(Referer) $sc-status $sc-substatus $sc-win32-status $sc-bytes $cs-bytes $time-taken
    FieldTypes    string, string, string, string, string, string, integer, string, string, string, string, string, integer, integer, integer, integer, integer, integer
    Delimiter    ' '
    QuoteChar    '"'
    EscapeControl    FALSE
    UndefValue    -
</Extension>

<Input IIS>
    Module        im_file
    File        "D:\\Logs\\IIS\\W3SVC300\\u_ex*"
    SavePos        TRUE
    Recursive    FALSE
    Exec if $raw_event =~ /^#/ drop();                                                                  \
        else                                                                                                         \
        {                                                                                                              \
            exiis->parse_csv();                                                                                \
            $EventTime = parsedate($date + " " + $time);                                          \
            $EventTime = strftime($EventTime, "%Y-%m-%dT%H:%M:%SZ");            \
        }
</Input>

 

Hello,

I'm kind of new to nxlog,

but is it possible to collect only Windows security logs ?

Hello,

I do admit I am totally lost about NXLog.conf for Windows 2K12 R2 machines.

The purpose is to filter some EventIDs from Security Event Log, for that I tried the below nslog.conf :

<Extension _syslog>

    Module      xm_syslog

</Extension>

<Input>
   Module      im_msvistalog
# For windows 2003 and earlier use the following:
#   Module      im_mseventlog

    Exec if ($EventID == 4634 or $EventID == 4624 or $EventID == 4672 or $EventID == 4801 or $EventID == 64 or $EventID == 7036 or $EventID == 7040) drop();\
    else\
    {\
        parse_syslog_ietf();\
        $Message = $FileName + ": " + $Message;\
        $SyslogFacility = syslog_facility_string(22);\
        $SyslogFacilityValue = syslog_facility_value("local6");\
        if ( $EventType == "INFO" ) $SyslogSeverityValue = 6;\
        if ( $EventType == "WARNING" ) $SyslogSeverityValue = 4;\
        if ( $EventType == "ERROR" ) $SyslogSeverityValue = 3;\
    }
</Input>

<Output out>
    Module      om_udp
    Host        10.1.1.39
    Port        1514
    Exec        to_syslog_snare();
</Output>

<Route 1>
    Path internal, eventlog => out
</Route>

 

 

Unfortunately despite the host and port are well set it doesn't work, and I also have these messages from nxlog.log :

xxxxxx WARNING no routes defined!
xxxxxx WARNING not starting unused module out

 

I would really appreciate any help

Am I missing something?

I see the source code, but no configuration options to compile the module under linux.

 

I am trying to parse AWS S3 Logs. They are in JSON format (One line no CR/LF) and Gzip'd. I need to ingest these into syslog TCP or UDP (Testing with file out) I can't get a reliable working nxlog.conf that will process the JSON files. # NXLOG define ROOT /usr/local/libexec/nxlog/ Pidfile /var/run/nxlog.pid LogFile ./nxlog.log define WORK /Users/cbitterfield/awslogs-project SpoolDir %WORK%/data CacheDir %WORK%/data LogLevel DEBUG Module xm_syslog Module xm_json Module im_file File "%WORK%/data19/*.json" Exec parse_json(); # Dump $raw_event Exec to_syslog_bsd(); SavePos FALSE ReadFromLast False Module om_file File "./output" Path in => out Yields the following errors and no output. 2016-04-10 22:13:00 DEBUG '^KEt^GC^D' does not match wildcard '859121128579_CloudTrail_ap-northeast-*.json' 2016-04-10 22:13:00 DEBUG checking '^KEt^GC^D' against wildcard '859121128579_CloudTrail_ap-northeast-*.json': 2016-04-10 22:13:00 DEBUG '^KEt^GC^D' does not match wildcard '859121128579_CloudTrail_ap-northeast-*.json' 2016-04-10 22:13:00 DEBUG checking '^KEt^GC^D' against wildcard '859121128579_CloudTrail_ap-northeast-*.json': 2016-04-10 22:13:00 DEBUG '^KEt^GC^D' does not match wildcard '859121128579_CloudTrail_ap-northeast-*.json' 2016-04-10 22:13:00 DEBUG checking '^KEt^GC^D' against wildcard '859121128579_CloudTrail_ap-northeast-*.json':

I am trying to read the table contents from DB2 table and send it as syslog. Is this possible with im_dbi module?

I was trying to test the database input module of NXlog in windows environment. After installed, I have verified the modules folder. I can see all other modules, but im_dbi and om_dbi are missing in the 2.8.x and 2.9.x versions, any idea please?

Dear all,

I have question below:

1 -  I want to get log from my Server 2008 r2 and i have installed nxlog in there. Beside, i have configed a file logstash config with some information as:

input {
  syslog {
    type => "WindowsEventLog"
    codec => json
    port => 3515 --> i open this port on Win Server
  }
}

filter {
  if [type] == "WindowsEventLog" {
    json {
      source => "message"
    }
    if [SourceModuleName] == "eventlog" {
      mutate {
        replace => [ "message", "%{Message}" ]
      }
      mutate {
        remove_field => [ "Message" ]
      }
    }
  }
}

output {
  elasticsearch {
    protocol {host => localhost}
    stdout {codec => rubydebug}
  }
}

And have a file config as nxlog

#define ROOT C:\\Program Files\\nxlog
#define ROOT_STRING C:\\Program Files\\nxlog
define ROOT C:\\Program Files (x86)\\nxlog
define ROOT_STRING C:\\Program Files (x86)\\nxlog
define CERTDIR %ROOT%\\cert
 
Moduledir %ROOT%\\modules
CacheDir %ROOT%\\data
Pidfile %ROOT%\\data\\nxlog.pid
SpoolDir %ROOT%\\data
LogFile %ROOT%\\data\\nxlog.log
 
# Include fileop while debugging, also enable in the output module below

#<Extension fileop>
#    Module      xm_fileop
#</Extension>
 
<Extension json>
    Module      xm_json
</Extension>
 
<Extension syslog>
    Module xm_syslog
</Extension>
 
<Input internal>
    Module im_internal
    Exec  $Message = to_json();
</Input>
    
<Input eventlog>
    Module      im_msvistalog
# Uncomment if you want only specific logs
#    Query       <QueryList>\
#                    <Query Id="0">\
#                        <Select Path="Application">*</Select>\
#                        <Select Path="System">*</Select>\
#                        <Select Path="Security">*</Select>\
#                    </Query>\
#                </QueryList>
</Input>

<Input file>
    Module    im_file
    File    "C:\\MyApp\\Logs\\mylog.json"
</Input>

<Input myapp>
    Module    im_file
    File    "C:\\MyApp\\Logs\\mylog.json"
    Exec    parse_json();
    Exec        $EventTime = parsedate($timestamp);
</Input>

<Input eventlog>
# Uncomment im_msvistalog for Windows 2008 and later
    Module im_msvistalog
 
#Uncomment im_mseventlog for Windows XP/Windows 7 and later
#Module im_mseventlog
 
    Exec  $Message = to_json();
</Input>
 
<Output elasticsearch>
    Module      om_http
    URL         http://elasticsearch:9200
    ContentType application/json
    Exec        set_http_request_path(strftime($EventTime, "/nxlog-%Y%m%d/" + $SourceModuleName)); rename_field("timestamp","@timestamp"); to_json();
</Output>

<Output out>
    Module om_tcp
    Host 10.151.130.114 --> this is address of Kibana and Logstash ( i config in a same server )
    Port 3515
 
    Exec to_syslog_ietf(); $raw_event = replace($raw_event, 'NXLOG@14506', '6701e99f-8724-4388-b2ac-cce6fd0eb03f@41058 tag="windows"] [', 1);
 
#Use the following line for debugging (uncomment the fileop extension above as well)
#Exec file_write("C:\\Program Files (x86)\\nxlog\\data\\nxlog_output.log",  $raw_event);
</Output>
 
<Route 1>
    Path internal, eventlog => out
</Route>

However, when i open kibana with it's address 10.151.130.114, don't have other data in there as well as result is No Results Found

I don't know where is my mistake. Pls, support me

2- Follow this web https://nxlog.co/docs/elasticsearch-kibana/using-nxlog-with-elasticsearch-and-kibana.html#idp54463840

in here, i don't know about om_elasticsearch module as well as om_http module. Where are they? and how can i config them?

Pls support me

Thanks and regds