Hi everyone,

Do you know if it is possible to schedule a module execution ?

Or does the Schedule function only takes Exec commands ?

Here is the solution I've come up with to schedule log sending:

<Input im_file_test>
        Module  im_file
        File    '/mnt/test/scheduler/*log'
        InputType       LineBased
        SavePos TRUE
        <Schedule>
                  When 0 12 * * *
                  Exec fileop->file_copy("/mnt/test/test.log", "/mnt/scheduler/test.log");
        </Schedule>
</Input>

<Output om_tcp_siem>
        Module  om_tcp
        Host    192.168.0.10
        Port    514
        OutputType      LineBased
</Output>

<Route RouteTestB>
​    Path        im_file_test => om_tcp_siem
</Route>

<Input im_file_test>
        <Schedule>
          When 0 12 * * *
          Module  im_file
          File    '/mnt/test/test.log'
          InputType       LineBased
          SavePos TRUE
       </Schedule>
</Input>

Hello.

Question is simple.

Have nxlog some module for working with mysql database?

Does anyone have nxlog usefully sending Microsoft DNS logs?

I have logging turned on, and I have tried with and without the details option checked.
Using it without the details is probably enough for us right now, as it shows the source and the requested URL.

However, when sent to syslog, only a blank line is sent.

The file output looks like below. The issue may be the space between each line?

8/12/2016 12:58:43 PM 0AE0 PACKET  000000F7524D7120 UDP Rcv x.x.x.x     5a68   Q [0001   D   NOERROR] A      (5)ctldl(13)windowsupdate(3)com(0)

8/12/2016 12:58:47 PM 0AE0 PACKET  000000F75221C070 UDP Rcv x.x.x.x     5a68   Q [0001   D   NOERROR] A      (5)ctldl(13)windowsupdate(3)com(0)

Any ideas?

Hello Guys,

I have a problem, that NXLog community edition sometimes sends 2 messages to GrayLog instead of 1, from a log file.
These are the messages, that came through together for example:

08 2016/12/02 13:13:28.581 Response: << MerchantId^XXXXXX~TransactionType^XX~OrderNumber^XXXXXXXX~StrId^XXXXXXXXXX~PTTID^XXXXXXXXXXX~MOP^XX~CurrencyId^XXX~Amount^XX.XX~AuthCode^XXXXXX~RequestType^X~MessageCode^XXXX~Message^XXXXXXXXX XXXXXXXX~CVNMessageCode^X~CVNMessage^XXXXXXXX/XXXXXX XXXXXXX >>
08 2016/12/02 13:13:28.581 ReportResult: (IDMMSITransaction = XXXXXXXX, Result = , MessageCode = XXXX)

This is the nxlog.conf part for this log:
 

<Extension exlogs>
    Module        xm_multiline
    HeaderLine    /^.. \d{4}\/\d{2}\/\d{2} \d{2}\:\d{2}\:\d{2}\.\d{3}/
</Extension>

<Input Logs>
    Module        im_file
    File        "D:\\path\\to\\logs\\log_*"
    SavePos        TRUE
    InputType    exlogs
</Input>

<Route logs-graylog>
    Path        Logs => graylog
</Route>

The problem doesn't happen with all the messages, but quite often and prevents me to create proper extractors.

Do you have any idea what could cause this?

Thanks,

Tamas Juhasz

tjuhasz@stanjames.com

~~Please confirm about HPUX server how to install NXLOG server or client.

Hello,

I'm catching up logs, but when they arrived to my Graylog, they take the actual date and not the old date. 

I saw my logs and they don't have format date. They're csv, and I have this field :

FW_date_time:10Nov2016;21:40:10

and I need to assign it on the timestamp, but I don't kno how.

Could you help me please to convert this on a good format?

I tried this (I have a field called: FW_date_time on my source csv file):

Exec          FW->parse_csv();

Exec            $EventReceivedTime =strptime($FW_date_time, '%Y %b %d %T');

Exec              $EventTime =strptime($FW_date_time, '%Y %b %d %T');

But it didn't work.

Thank you very much for your help.

Cordially,

Ana

I'm having some issues sending event logs from a windows 2012 R2 client using nxlog ce agent to a linux syslog-ng server.  The tcp connection appears to be getting dropped by syslog-ng due to an invalid frame header.

syslog-ng says:

2016-11-22T09:10:35+00:00 server syslog-ng[8317]: Invalid frame header; header=''
2016-11-22T09:10:35+00:00 server syslog-ng[8317]: Syslog connection closed; fd='8', client='AF_INET(ip.of.nxlog.client:55473)', local='AF_INET(0.0.0.0:6514)'
2016-11-22T09:10:36+00:00 server syslog-ng[8317]: Syslog connection accepted; fd='8', client='AF_INET(ip.of.nxlog.client:55474)', local='AF_INET(0.0.0.0:6514)'

nxlog says:

2016-11-22 15:24:27 ERROR om_tcp send failed; An established connection was aborted by the software in your host machine.  
2016-11-22 15:24:28 INFO connecting to ip.of.syslog-ng.server:6514
2016-11-22 15:24:28 INFO reconnecting in 1 seconds

I've tried several variations in the nxlog.conf file to work around this issue. 

Here is my nxlog.conf:

define ROOT C:\Program Files (x86)\nxlog

Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log

<Extension _syslog>
    Module      xm_syslog
</Extension>

<Input in>
    Module      im_msvistalog

# For windows 2003 and earlier use the following:
#   Module      im_mseventlog
    ReadFromLast TRUE
    <QueryXML>
       <QueryList>
         <Query Id='1'>
            <Select Path='Application'>*</Select>
            <Select Path='Security'>*</Select>
            <Select Path='System'>*</Select>
         </Query>
       </QueryList>
   </QueryXML>
</Input>

<Processor eventlog_transformer>
  Module pm_transformer
  Exec $Hostname = hostname();
 OutputFormat syslog_rfc5424 
</Processor>

<Output out>
   Module om_tcp
   Host        ip.of.syslog-ng.server
   Port 6514
  
    Exec $raw_event = replace($raw_event, "\r\n", " ");
    Exec $raw_event = replace($raw_event, "\t", " ");
    #Exec to_syslog_ietf();
    #Exec        to_syslog_bsd();
</Output>

<Route 1>
    Path        in => eventlog_transformer => out
</Route>

Here is the syslog-ng configuration:

@version:3.5
@include "scl.conf"

# syslog-ng configuration file.
#
# This should behave pretty much like the original syslog on RedHat. But
# it could be configured a lot smarter.
#
# See syslog-ng(8) and syslog-ng.conf(5) for more information.
#
# Note: it also sources additional configuration files (*.conf)
#       located in /etc/syslog-ng/conf.d/

#Options
#
## Warnings
#
#
options {
    create_dirs(yes);
    dir_perm(0755);
    perm(0644);

    log_msg_size(65536);
    log_fifo_size(10000);

    threaded(yes);
    ts_format(iso);

    keep_hostname(no);
    use_dns(no);
    dns_cache(no);
    use_fqdn(no);

    flush_lines(100);
    stats_freq(60);
    mark_freq(36400);

};

#Sources
source s_sys {
    system();
    internal();
};

source s_network {
    syslog( port(6514) flags(syslog-protocol)  transport("tcp") keep-alive(yes) so-keepalive(yes) so_rcvbuf(1073741824) log-fetch-limit(100) log-iw-size(100) max-connections(5000));
    syslog( port(514) transport("udp") so_rcvbuf(1073741824));

};

#Destinations
#destination d_all       { file("/var/log/logs/all_logs/$HOST/$HOST-$YEAR-$MONTH-$DAY-all_logs.log" create_dirs(yes)); };
destination d_all       { file("/var/log/logs/all_logs/$HOST/$HOST.log" create_dirs(yes)); };

#Filters

#Logs
#   { source(s_sys); filter(f_kernel); destination(d_cons); };
log { source(s_sys); destination(d_all); };
log { source(s_network); destination(d_all); flags(flow-control); };

# Source additional configuration files (.conf extension only)
@include "/etc/syslog-ng/conf.d/*.conf"

I can get things to work using udp, but can't get it to work over tcp.  Has anyone else had any success getting nxlog to send events from windows to a syslog-ng server over tcp?

Hello,

I have NXLOG installed, and use UDP to receive and sent the servers logs. However, when I run the netstat command I found that there are packets on error and other received through unknown ports. See the lines shown :

UDP:

 123234944 packets received

223432 packets to unknown port received.

523455 packet receive errors

2111 packets sent

Why does it happen? I didn't configure any other port on my NXLOG and the firewall rules only accept 514 port to communicate with my NXLOG server.

How could I verify what is this? Is it normal?

Thank you very much for your help and answer

Cordially,

Ana

After a bit of trial and and lots of reading, I managed to get Graylog2 working like a charm


I'm using NXLOG to send the logs to Graylog via GELF UDP


Right now I'm just testing and trying diffrent things.

Right now I';m testing with just 4 servers (have close to 100) has consumed a fair bit of space

So to evaluate usage, I figured I'd just send the security logs (these are all Windows Servers)

Original working configuration
__________________________________________________________
define ROOT C:\Program Files (x86)\nxlog
Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log
<Extension gelf>
Module xm_gelf
</Extension>
<Input in>
# Use ’im_mseventlog’ for Windows XP, 2000 and 2003
Module im_msvistalog
# Uncomment the following to collect specific event logs only
Query <QueryList>\
<Query Id="0">\
<Select Path="System">*</Select>\
<Select Path="Application">*</Select>\
<Select Path="Security">*</Select>\
</Query>\
</QueryList>
</Input>
<Output out>
Module om_udp
Host 10.60.10.62
Port 12201
OutputType GELF
</Output>
<Route r>
Path in => out
</Route>
_______________________________________________________

Now I put a REM statement at the beginning of the file

# Just capturing security logs

The service won't start.
If I rem out Application and System path, it won't start.

Any suggestions?

I am using nxlog-ce clients in windows machines, with om_ssl.  NXlog seems to keep the connection open, but does not send anything, except for the bursts for every Poll Interval.  The load balancer seems confused, and tries to send every second burst to a different server.  Is there any option to tell nxlog-ce that it should make a new connection for each burst, so that the load balancer can easily balance the load for different clients to different log servers?

Hello,

I have 2 infrastructures installed one on Amazon and another on premises. On Amazon every thing is OK, but when I send logs on premises I have a 5 to 7 minutes delayed and I loss almost 30% of the data,

I activated the buffer option and I pass through it before sending logs to my on premise server but it doesn't work

Somebody know how to correct that?

Thank you for your help and answer

Ana

We are pushing logs from file with the im_file module to logstash and then to elasticsearch. However, some of these logs only have second accuracy, and therefore not returned in order by elasticsearch when sorted by just the time. To get around this problem, we would like to add the position of the log event to a field that we store (for example beginning line number or byte offset within the source file). Is there a way I can add a separate field for this in the events pushed by nxlog, either in the input or output configuration?

I have the following regexp:

and a line from the log that I am trying to parse (there are some spaces at the beginning of the line):

 13:33:00.1205 [-] Persisting VDOM path /

trying to process a file using pm_pattern module with LogLevel DEBUG this is what I get:

2016-11-16 13:38:00 DEBUG regexp '^\s*(\d{2}:\d{2}:\d{2}\.\d{4})\s+(\[S+\])s+([\s\S]*)$' did not match against '  13:38:00.1857 [-] Persisting VDOM path /'

<pattern>
    <id>1</id>
    <name>log_line</name>
    <description>Regular log line</description>
    <matchfield>
        <name>raw_event</name>
        <type>regexp</type>
        <value>^\s*(\d{2}:\d{2}:\d{2}\.\d{4})\s+(\[S+\])s+([\s\S]*)$</value>                        
        <capturedfield>
            <name>EventTime</name>
            <type>string</type>
        </capturedfield>
        <capturedfield>
            <name>SourceName</name>
            <type>string</type>
        </capturedfield>
        <capturedfield>
            <name>Message</name>
            <type>string</type>
        </capturedfield>
    </matchfield>
</pattern>

What is wrong with my expression? 

Hello. 

The case is:

I am creating Stream with rules using API.

Question is:

Is there a way to set the input for a stream using GrayLog API?

Thanks.

Hi,

I want to store my logs in .evtx file in windows. I tried following configuration.

<Output out2>    
    Module      om_file
  File     '%ROOT%\tmp\test.evtx'

</Output>

This created evtx file but it was also opening with notepad, wordpad,etc. For security purpose, I want to make it open with MS EventViewer API only.

Is this possible using nxlog om_file module? Is there any plugin for nxlog to store data in .evtx files?

Hi,

I am a new at this and would like to know the command line argument that will let me specify what config file to use. I need to have this done during installation and avoid changing the file post installation.

Any help would be greatly appreciated.

Thanks.

Hi,

  I am totally new to nxlog and trying to set up a config file to watch a file on my local hard drive and send it to Loggly whenever it changes.  My conf file is below.  I copied this file from the loggly documentation and tried modifiying it as per their instructions so there may be some unnecessary lines in it. The file I want to watch and send is in the C:\\Program Files\\Autodesk\\Vault Professional 2016\\Explorer\\ folder.  The file name is JobProcessor.log.  When I uncomment the "#Exec file_write("C:\\Program Files\\Autodesk\\Vault Professional 2016\\Explorer\\JobProcessor.log",  $raw_event);" line, it does send it out so I know the connection is working.  But I have not been able to 'trigger' the file to process whenever there is a change to it.  Any advice is appreciated.  Thanks.

#define ROOT_STRING C:\\Program Files\\nxlog
define ROOT C:\\Program Files (x86)\\nxlog
define ROOT_STRING C:\\Program Files (x86)\\nxlog
define CERTDIR %ROOT%\\cert
 
Moduledir %ROOT%\\modules
CacheDir %ROOT%\\data
Pidfile %ROOT%\\data\\nxlog.pid
SpoolDir %ROOT%\\data
LogFile %ROOT%\\data\\nxlog.log
 
# Include fileop while debugging, also enable in the output module below
<Extension fileop>
    Module      xm_fileop
</Extension>
 
<Extension json>
    Module      xm_json
</Extension>
 
<Extension syslog>
    Module xm_syslog
</Extension>
 
<Input internal>
    Module im_internal
    Exec  $Message = to_json(); 
</Input>

##Module to watch a file
<Input file_watch_1>
    Module im_file
    File "C:\\Program Files\\Autodesk\\Vault Professional 2016\\Explorer\\"
    Exec   $SourceName = 'JobProcessor.log'; 
    SavePos TRUE

    ##include the message and add meta data
    Exec $Message = $raw_event;
    Exec to_syslog_ietf();
</Input>
 
<Output out>
    Module om_tcp
    Host logs-01.loggly.com
    Port 514
 
    Exec to_syslog_ietf();
    Exec $raw_event;

#Use the following line for debugging (uncomment the fileop extension above as well)
    #Exec file_write("C:\\Program Files\\Autodesk\\Vault Professional 2016\\Explorer\\JobProcessor.log",  $raw_event);
</Output>
 
<Route 1>
    #Path internal, eventlog => out
    Path    in => out
</Route>

Hi!

I have a problem using module im_odbc. I use sql query to receive some data from sql server. For example, query looks like "SELECT [Name] From [MyTable] WHERE [Id] > ?". In some moment, the "?" has stored incorrect value (I don't know how, but it's bigger then MAX(Id) value in my table) and sinse that moment, query has not return any data. 

Please, help me reset value of "?" (nxlog service restart not help). And, maybe, you can give me some recommendations, how to avoid this problem in future?

UPD: I reset value of "?" by renaming input name in config file. The first query after input renaming was executed with "0" as value of "?" and return all rows from table. But next time query executed with "?"-value "161027120023", wich bigger than max value of "Id" column in my table (max value was 161027115813 at that time). What am i doing wrong?

<Input odbc1>
    Module                im_odbc
    ConnectionString    DSN=im_odbc_test;database=MyDB;
    SQL                    SELECT [Name] From [MyTable] WHERE [Id] > ?;
    PollInterval        120
    SavePos                TRUE
    IdIsTimestamp        FALSE
</Input>

Hello,

This is jsut a quick question on the following lines of a config for Nxlog. Basically is the code suppressing all security events with the exception of those event IDs with in the brackets or are the event IDs within the brackets being suppress?

# Windows Event Log
<Input eventlog>
# Uncomment im_msvistalog for Windows Vista/2008 and later
   Module im_msvistalog
Query    <QueryList>\
                   <Query Id="0">\
                     <Select Path="Security">*</Select>\
                     <Suppress Path="Security">*[System[(EventID=4624 or EventID=4776 or EventID=4634 or EventID=4672 or EventID=4688 or EventID=4662 or EventID=5159) or EventID=5136 or EventID=4932 or EventID=4933]]</Suppress>\
                   </Query>\
        </QueryList>
   Exec $EventReceivedTime = integer($EventReceivedTime) / 1000000; to_json();
</Input>