Hi,

I've to use the same exec block multiple times in different paths. I've already tried it with a processor module, but unfortunately a processor module could only be used once. A possible solution is to implement them in the input modules, but I've more than one input module (syslog tcp/udp alone are already two) which ends in multiple code in different modules. But I want them once and want to use them multiple times.

Have someone an idea how to implement this?

Thanks in advance.

br

Hi,

My Server's TimeZone is EST(UTC-5) but the application running in the server has EET(UTC+2). The log file generated is names as <Application Date> ie todays log file is named as 20170717.log. Today's log will be truncated at 00:00 (UTC+2) and new log file 20170718.log will be created and old log file will remain in the same directory.

How can I configure NxLog to read the current log only

hi all!
im first time meet nxlog. it looks very simple and powerfull, but i need help!
i have cisco which can send syslog to my nxlog service. logs looks like
2017-06-21T07:19:49.214Z,"vpn.domain.com","Jun 21 2017 10:19:55: %ASA-4-722051: Group <GroupPolicy_domain> User <123456@domain.com> IP <213.87.163.123> IPv4 Address <10.217.130.23> IPv6 address <fc00::59> assigned to session"

please, help me write processor, which parse this string and get date (Jun 21 2017 10:19:55)? login (123456@domain.com) and external ip adress (213.87.163.123) and put them to text log file.

thanks!

Hello,

I need to parse logs mounted on file system NFS on centos.

Is NXLog reliable with NFS ?

Thanks,

Regards

I hava some data  like this 

--------------------------

a         9650     mktd     07:30     [OK    ] a   
b         9650     mktd     07:30     [OK    ] b   
c         9650     mktd      07:30     [OK    ] c    
d         9650     mktd     07:30     [OK    ] d

if data have change to this

a        9650     mktd     07:30    [OK    ] a
b        9650     mktd     07:30    [OK    ] b
c        0                                     [FAILED] c
d        0                                     [FAILED] d

how could I reload the file to nxlog

I have been to tried to use PollInterval ,but it seem to use for new file and new log entries

I also try to use file_touch,but both of them doesn't useful for me.

my scripts below

#define ROOT C:\Program Files\nxlog
define ROOT C:\Program Files (x86)\nxlog

Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log

<Extension json>
    Module      xm_json
</Extension>

<Extension fileop>
    Module      xm_fileop
</Extension>

<Extension exec>
    Module      xm_exec
</Extension>

<Extension kvp>
    Module       xm_kvp
    KVPDelimiter ,
    KVDelimiter  =
    EscapeChar   \\
</Extension>

<Input test>
 
     Module       im_file
     File         'D:\winapp\log\chk2.txt'
     SavePos      FALSE
     ReadFromLast FALSE
</Input>    

<Processor pro>
    Module       pm_null
    
    Exec         delete($SourceModuleName);                            \
                 delete($SourceModuleType);                            \
                 delete($EventReceivedTime);                            
                 
                
    <Exec>         
                if ($raw_event =~ /^(\w+)\s+(\d+)(.*)/) 
                 {                                                     
                     $app_name      = $1;                              
                     $pid           = integer($2);                     
                     $msg           = $3; 
                     to_json();                                        
                 }                                                     
                 else                                                  
                 {                                                     
                   drop();                                             
                 }                                                     
                 if ($pid == 0)                                        
                 {                                                     
                    if($msg =~ / \s+\[(\w+)\] (.*)/)                   
                    {                                                  
                       $status      = $1;                              
                       $event_time    = integer(now())/1000;           
                       delete($msg);                                   
                       to_json();                                      
                    }
                 }
                 if($pid != 0)                                         
                 {                                                     
                   if($msg =~ / (\w+)\s+(\d\d:\d\d)\s+\[(\w+)\s+\] (.*)/)  
                   {                                                   
                       $account     = $1;
                       $start_time  = $2;
                       $status      = $3;
                       $event_time    = integer(now())/1000;
                       delete($msg);                                   
                       to_json();                                      
                   }                                                   
                 }
                                                     
    </Exec>

</Processor>

<Output out>
    Module om_file                                                     
    File 'D:\winapp\log\pchk_test.txt'       
</Output>

<Route 1>
    Path        test => pro => out
</Route>

Thank you for help

I have to retain the new lines in a syslog.  I'm using NXLog to send logs from my laptop to a test syslog server.  I'm currently using the following:

<Extension _syslog>

    Module      xm_syslog

</Extension>


<Input in>

    Module      im_msvistalog

Exec to_syslog_ietf();

</Input>


<Processor rewrite>

    Module      pm_null

Exec        $Message = $EventID + "|" + $EventType + "|" + $Hostname + "|" + $SourceName + "|" + $AccountName + "|" + $AccountType + "|" + $Domain + "|" + $UserID + "|" + $raw_event;

</Processor>


<Output out>

    Module      om_udp

    Host        192.168.100.33

    Port        514

    #Exec        to_syslog_bsd();

</Output>


<Route 1>

    Path        in => rewrite => out

</Route>

I can get the logs to send with the \r\n intact is to remove the Exec to_syslog_snare(), then I loose all the other details about the log such as event id etc.  So I thought ok I'll construct my own by using Exec $Message = all the data fields I want....this doesn't work...so then I started playing with to_syslog_ieft and to_syslog_bsd() and they both strip out new lines.

What am I doing wrong?

Hi,

I need some help with nxlog.conf (Client side + Server side).

I need to send JSON file over syslog udp connection from a client to server and rebild JSON in server side.

I know that with nxlog it's posible to perform this but i don't know the correct configuration.

can you help me with this issue ?

Hi All,

I downloaded the NXlog Community Version source and compiled and installed in Ubuntu.

I wanted to create a Windows pacakge from the source. So i run the script ./wininst.sh from the packaging/windows/ folder.

running the wininst.sh script is showing errors, Can someone let me how to rectify this issue, 

test -z "//input" || /bin/mkdir -p "C:\\\\msys\\\\1.0\\\\nxlog-tmp//input"
 /bin/bash ../../../../libtool   --mode=install /usr/bin/install -c   im_file.la 'C:\\msys\\1.0\\nxlog-tmp//input'
libtool: install: /usr/bin/install -c .libs/im_file.so C:\\msys\\1.0\\nxlog-tmp//input/im_file.so

/usr/bin/install: cannot create regular file 'C:\msys\1.0\nxlog-tmp//input/im_file.so': No such file or directory

Makefile:291: recipe for target 'install-im_fileLTLIBRARIES' failed
make[5]: *** [install-im_fileLTLIBRARIES] Error 1
make[5]: Leaving directory '/nxlog-ce-2.8.1248/src/modules/input/file'
Makefile:456: recipe for target 'install-am' failed
make[4]: *** [install-am] Error 2
make[4]: Leaving directory '/nxlog-ce-2.8.1248/src/modules/input/file'
Makefile:288: recipe for target 'install-recursive' failed
make[3]: *** [install-recursive] Error 1
make[3]: Leaving directory '/nxlog-ce-2.8.1248/src/modules/input'
Makefile:288: recipe for target 'install-recursive' failed
make[2]: *** [install-recursive] Error 1
make[2]: Leaving directory '/nxlog-ce-2.8.1248/src/modules'
Makefile:288: recipe for target 'install-recursive' failed
make[1]: *** [install-recursive] Error 1
make[1]: Leaving directory '/nxlog-ce-2.8.1248/src'
Makefile:411: recipe for target 'install-recursive' failed
make: *** [install-recursive] Error 1

Hi all

Inherited this system, so sorry if my terminology isn't right or I'm not super clear on what i'm asking for. I am trying to get logs to ship to my SOC CTA.  It works in shipping logs to Nagios but not to the CTA.  I've tried updating the conf file from a server that is working, but every time I restart the service, 5 minutes later the conf file rolls back to its initial config.  What am I doing wrong?  Is the settings for the conf file resetting because of a script issue?

Here is my sample conf file that keeps overwriting the work I've done on it.  Thanks all in advance

# Please set the ROOT to the folder your nxlog was installed into,

## otherwise it will not start.
#define ROOT C:\Program Files\nxlog
define ROOT C:\Program Files (x86)\nxlog
define CERT %ROOT%\cert
 
Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log
 
# Include fileop while debugging, also enable in the output module below
#<Extension fileop>
#    Module xm_fileop
#</Extension>
 
<Extension json>
    Module      xm_json
</Extension>
 
<Extension syslog>
    Module xm_syslog
</Extension>
 
<Input internal>
    Module im_internal
</Input>
 
# Watch your own files
<Input file1>
    Module   im_file
    File     '%ROOT%\data\nxlog.log'
    SavePos  TRUE
    Exec     $Message = $raw_event;
</Input>
 
# Windows Event Log
<Input eventlog>
# Uncomment im_msvistalog for Windows Vista/2008 and later
    Module im_msvistalog
    Exec if $EventID IN (5156) drop();  
# Uncomment im_mseventlog for Windows XP/2000/2003
#   Module im_mseventlog
</Input>

<Input MSDNSLOGIN>
 Module im_file
 File 'C:\Windows\Sysnative\dns\dns.log'
# 32-bit applications can access the native system directory by substituting %windir%\Sysnative for %windir%\System32. WOW64 recognizes Sysnative as a special alias used to indicate that the file system should not redirect the access. This mechanism is flexible and easy to use, therefore, it is the recommended mechanism to bypass file system redirection. Note that 64-bit applications cannot use the Sysnative alias as it is a virtual directory not a real one.
 InputType LineBased
 SavePos TRUE
 ReadFromLast FALSE
 PollInterval 1
# The input is assumed to contain log messages separated by newlines. Thus if an LF (\n) or CRLF (\r\n) is found, the function considers that it has reached the end of the log message.
 Exec if $raw_event == '' drop();  ##drops blank messages
 Exec $SourceName = "MSDNSLog";$Message = $raw_event;
</Input>

<Input MSDHCP>
 Module im_file
 File 'C:\Windows\Sysnative\dhcp\DhcpSrvLog-*.log'
# 32-bit applications can access the native system directory by substituting %windir%\Sysnative for %windir%\System32. WOW64 recognizes Sysnative as a special alias used to indicate that the file system should not redirect the access. This mechanism is flexible and easy to use, therefore, it is the recommended mechanism to bypass file system redirection. Note that 64-bit applications cannot use the Sysnative alias as it is a virtual directory not a real one.
 InputType LineBased
 SavePos TRUE
 ReadFromLast FALSE
 PollInterval 1
# The input is assumed to contain log messages separated by newlines. Thus if an LF (\n) or CRLF (\r\n) is found, the function considers that it has reached the end of the log message.
 Exec if $raw_event == '' drop();  ##drops blank messages
 Exec $SourceName = "DHCPLog";$Message = $raw_event;
</Input>
 
<Output out1>
    Module om_udp
    Host 10.*.*.*
    Port 3***
    
    Exec  $tmpmessage = $Message; delete($Message); rename_field("tmpmessage","message");
    Exec  $raw_event = to_json();
    
    # Uncomment for debug output
    # Exec file_write('%ROOT%\data\nxlog_output.log', $raw_event + "\n");
</Output>

<Output out2>
    Module om_udp
    Host 10.*.*.*    
    Port 514
# Transmit in Snare format on local6.info
        Exec $SyslogFacilityValue = 22;to_syslog_snare();
</Output>

<Output out3>
    Module      om_udp
    Host        10.*.*.*
    Port        514
# Create our RFC3164 compliant syslog line and transmit on local5.info
    Exec        $SyslogFacilityValue = 21;$Severity = 'INFO';to_syslog_bsd();

</Output>

<Route 1>
    Path internal, file1, eventlog  => out1
</Route>

<Route 2>
    Path        eventlog => out2
</Route>

<Route 3>
    Path   MSDNSLOGIN, MSDHCP => out3
</Route>

 
#<Route 1> Note that OUTPUT 2 & 3 are the OUTPUTS not working
#    Path internal, file1, eventlog, MSDNSLOGIN, MSDHCP  => out1, out2
#</Route>

Error Log

2017-06-26 13:39:19 ERROR failed to open directory: C:\Windows\Sysnative\dhcp: The system cannot find the path specified.  
2017-06-26 13:39:19 WARNING Module MSDHCP has no input files to read
2017-06-26 13:39:21 WARNING input file does not exist: C:\Windows\Sysnative\dns\dns.log
2017-06-26 13:39:27 WARNING last message repeated 2 times
2017-06-26 13:39:35 WARNING input file does not exist: C:\Windows\Sysnative\dns\dns.log
2017-06-26 13:39:51 WARNING input file does not exist: C:\Windows\Sysnative\dns\dns.log
2017-06-26 13:40:23 WARNING input file does not exist: C:\Windows\Sysnative\dns\dns.log
2017-06-26 13:41:28 WARNING input file does not exist: C:\Windows\Sysnative\dns\dns.log
2017-06-26 13:43:37 WARNING input file does not exist: C:\Windows\Sysnative\dns\dns.log
2017-06-26 13:47:54 WARNING input file does not exist: C:\Windows\Sysnative\dns\dns.log
2017-06-26 13:56:27 WARNING input file does not exist: C:\Windows\Sysnative\dns\dns.log
2017-06-26 14:13:32 WARNING input file does not exist: C:\Windows\Sysnative\dns\dns.log
2017-06-26 14:47:41 WARNING input file does not exist: C:\Windows\Sysnative\dns\dns.log

Hello,

I'm using NXLOG to send data from CSV files to Elasticsearch.

It's working correctly; Other numbers with decimals are being sent as Strings.

What FieldType should I use to work with numbers with decimal places (such as: decimal, double, float)?

Thank you.

<Extension ExtData>
     Module xm_csv
     Fields $Time,$Id,$Budget,$Tax
     FieldTypes string,integer,DECIMAL,DOUBLE
     Delimiter ','
</Extension>

I am experimenting running nxlog-processor against a configuration file which is working OK for foreground use, but it does not terminate after pickup, or if no new files are present.

I have configured it to monitor a folder in Windows with a wildcarded  path    \....\\*.JSON   Also with multiline input.

Are there any limitations, special tips for single-run of nxlog-processor?

Hello nxlog world. My application has a custom log file on a Windows 2012 box that I need to send to Graylog which has a GELF UDP input running. The log file is multiline and there is a specific header and footer that I'm seperating the log file entries with. As of right now despite using the multiline module, when my logs arrive in Graylog it is parsing each line of the log file individually instead of by header and footer. Here is my nxlog.conf:

define ROOT C:\Program Files (x86)\nxlog

Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log

<Extension gelf>
    Module xm_gelf
</Extension>

<Extension multiline>
    Module    xm_multiline
        HeaderLine    /\d{4}-\d{2}-\d{2}\s\d*:\d\d:\d*,\d{3}\s[a-zA-Z]*\s\s\S\s\[Begin Lead\]/
        EndLine        /\d{4}-\d{2}-\d{2}\s\d*:\d\d:\d*,\d{3}\s[a-zA-Z]*\s\s\S\s\[End Lead\]/
</Extension>

<Extension json>
    Module    xm_json
</Extension>

<Input im_msvistalog>
    Module      im_msvistalog
    Query   <QueryList>\
            <Query Id="0">\
                    <Select Path="System">*[System[(Level=1 or Level=2 or Level=3)]]</Select>\
                    <Select Path="Application">*[System[(Level=1 or Level=2 or Level=3)]]</Select>\
            </Query>\
        </QueryList>
    Exec if $raw_event =~ /(\d\d\d\d\-\d\d-\d\d \d\d:\d\d:\d\d)/ $EventTime = parsedate($1);
</Input>

<Input im_file>
    Module   im_file
    File "C:\\LOGS\\application_service.log"
    SavePos FALSE
     ReadFromLast FALSE
    InputType multiline
</Input>

<Output om_udp>
    Module      om_udp
    Host        10.25.32.21
    Port        12201
    OutputType  GELF
</Output>

<Route 1>
    Path    im_msvistalog  => om_udp
</Route>

<Route 2>
    Path im_file => om_udp
</Route>

The application_service.log file looks like this:

2017-06-20 17:25:50,168 INFO  – [Begin Lead]
2017-06-20 17:25:50,168 INFO  – Getting lead by id '5551212'
2017-06-20 17:25:51,215 INFO  – Lead field = First Name. Value = Michael
2017-06-20 17:25:54,778 INFO  – Lead field = Last Name. Value = Westin
2017-06-20 17:25:54,793 INFO  – Lead field = CustomerNumber. Value = 99999
2017-06-20 17:26:05,887 INFO  – [End Lead]
2017-06-20 18:25:50,168 INFO  – [Begin Lead]
2017-06-20 18:25:50,168 INFO  – Getting lead by id '4279551'
2017-06-20 18:25:51,215 INFO  – Lead field = First Name. Value = Emmit
2017-06-20 18:25:54,778 INFO  – Lead field = Last Name. Value = Stussy
2017-06-20 18:25:54,793 INFO  – Lead field = CustomerNumber. Value = 94231
2017-06-20 18:26:05,887 INFO  – [End Lead]
2017-06-20 19:25:50,168 INFO  – [Begin Lead]
2017-06-20 19:25:50,168 INFO  – Getting lead by id '0081227'
2017-06-20 19:25:51,215 INFO  – Lead field = First Name. Value = Saul
2017-06-20 19:25:54,778 INFO  – Lead field = Last Name. Value = Goodman
2017-06-20 19:25:54,793 INFO  – Lead field = CustomerNumber. Value = 33487
2017-06-20 19:26:05,887 INFO  – [End Lead]

Ideally when it his Graylog I'd like the content between [Begin Lead] and [End Lead] to be stored in one field.  But with my nxlog config above it is putting each line of the log file in its own message ID in Graylog.  Do I have my config wrong?  Is there a better method to parse the log file and send to Graylog?  Any help is appreciated.

Hi,

Using two tables in mySQL: table1 and table2 with same structure. 

Field          Type          Length        Key
--------------------------------------------------------
id               INT            11               Primary
text1          CHAR        20
number1    INT            11

Initial situation: Content of table1, table2 is empty 

Table1                            table2
--------------------------       ---------------------
1    string1        100        <no records>
2    string2        200
3    string3        300    

My nxlog.conf:

define ROOT C:\Program Files (x86)\nxlog
define CERT %ROOT%\cert

Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log

#PersistLogqueue    TRUE
#SyncLogqueue        TRUE
CacheFlushInterval    always
CacheSync        TRUE

#LogLevel  DEBUG

<Input    testIn>
    Module  im_odbc
#    Pollinterval  60
    ConnectionString  DSN=mySQL;uid=***;pwd=***;database=nxlog
    SQL   SELECT  id as vid ,\
                  text1 as vtext1 ,\
                  number1 as vnumber1 \
          FROM table1 where id > ?
    SavePos TRUE
</Input>              

<Output     testOut>
        Module        om_odbc
        ConnectionString DSN=mySQL;uid=***;pwd=***;database=nxlog
        <Exec>
          if (sql_exec("INSERT INTO table2(text1,number1) VALUES (?,?)",$vtext1,$vnumber1)== TRUE) {};
        </Exec>
</Output>

<Route 1>
    Path testIn => testOut
</Route>

After I start nxlog service both tables look like this.

Table1                                     table2
-------------------                        -------------------
1    string1        100                1    string1        100
2    string2        200                2    string2        200
3    string3        300                3    string3        300

So far so good.

But when I manualy add a record to table1 both tables look like this:

Table1                                    table2
-------------------                        -------------------
1    string1        100                1    string1        100
2    string2        200                2    string2        200
3    string3        300                3    string3        300
4    string4        400                4    string3        400

As you can see in table2: the string has the value of record3. Number is OK.
When I keep adding records to table1 it keeps on taking the string of record 3 but number is correct.

After restart of nxlog service and manualy adding a record to table1 I get this:

Table1                                    table2
-------------------                        -------------------
1    string1        100                1    string1        100
2    string2        200                2    string2        200
3    string3        300                3    string3        300
4    string4        400                4    string3        400
5    string5        500                5    <empty>     500

table2 string has no value and number is correct.

Any ideas someone? 

Wout

NXLog IM_MSVistaLog module collects the Rendered Event log rather than the raw XML Windows Event Log.  

Is there a configuration option in the NXLog agent or IMVistaLog module to enable collecting the original Windows XML Event Log rather than the Rendered Event Log?

Best Regards,

Chris

Edit: Think I worked this out. Appears to collect the XML data but also the rendered log field. This would lead questions to be:

1) Can you disable or filter out the Message field? It's not needed.

2) Can you collect the Windows Event Log fields in the order they're written, e.g., the Provider field From testing the Provider field is renamed as Sourcename and collected out of order from the original Windows Event Log.

Reason for the above is have multiple upstream systems that require the original log format, and hence testing viability to use NXLog to retrieve Windows Event Logs.

Hello, we’re looking at using multiple instances of nxlog.exe on the same Windows machine. What is the supported method for this please?

If I create a new process we receive the ERROR Service is already running log entry.

Thanks

hi!

I am sending data from csv format and the nxlog start to gather after start/resume of my pc.

Sometimes I'm having a trouble when the csv file is not ready yet and my nxlog already start.
it will throw some parsing error.
Is there a way that when the csv file is already ready, nxlog will also refresh.

Thank you.

 

I've got a route where the input is from im_tcp and I'd like to output the incoming data to a set of files, but select a specific file based on the content of the data.

I can easily do this in om_perl with something that examines the data, figures out the correct file, then stuffs it into that file, but that would appear to require a file open and a file close per event, which seems non-optimal at scale.  So - is the code identified by the PerlCode directive basically run for each event, or is it stateful so I can maintain an array of open files and just stuff the event in the correct one?

Alternatively, can I use xm_perl to find the correct filename and set that as an environment valiable that could be consumed by om_file?

I'm open to other efficient methods for accomplishing this, if there are ideas out there.  Thanks!

Hello everyone,

How can I tell nxlog to send logs to our server every 5 mins?

Is there any trigger on NXLog to send the logs to the server?

Thank you.

Hi!

I just want to ask if when I update my nxlog to the current version, do I need to revise my code?

Is there a list of versions with release notes available? Upon saerching in the website, I couldn't find
the list of versions. Only the latesd version is available.

Thank you in advance. 

Hi,

We are using the Community NXlog to forward our domain controller security events into Graylog and all was going well until the Microsoft May Security rollup and we now see 100% CPU when Graylog is running.

Is anyone aware of a bug / something else that is causing this?

Thanks

Peter.