Hi, I recently started using Nxlog with Windows event log.

I'm able to get events off of the Security log and onto my server, however, I've noticed something peculiar.

The value of the 'ProcessID' field is always 4, no matter what process is generating the event.

My nxlog.conf has the following sections:

<Extension json>
    Module      xm_json
</Extension>

<Input security_log>
    Module      im_msvistalog
    SavePos     TRUE
    ReadFromLast    TRUE
    Channel     Security
    Query       *
</Input>

<Output out>
    Module      om_tcp
    Host        10.1.11.50
    Port        9000
    Exec        to_json();
</Output>

<Route 1>
    Path        security_log => out
</Route>

And the resulting output looks like:

{
"EventTime":"2016-01-18 10:32:13",
"Hostname":"dev-collector.dev.local",
"Keywords":-9214364837600034816,
"EventType":"AUDIT_SUCCESS",
"SeverityValue":2,
"Severity":"INFO",
"EventID":4689,
"SourceName":"Microsoft-Windows-Security-Auditing",
"ProviderGuid":"{54849625-5478-4994-A5BA-3E3B0328C30D}",
"Version":0,
"Task":13313,
"OpcodeValue":0,
"RecordNumber":355,
"ProcessID":4,
"ThreadID":48,
"Channel":"Security",
"Message":"A process has exited.\r\n\r\nSubject:\r\n\tSecurity ID:\t\tS-1-5-21-3503561835-834928460-339998367-1000\r\n\tAccount Name:\t\tachung\r\n\tAccount Domain:\t\tDEV-COLLECTOR\r\n\tLogon ID:\t\t0x727518c\r\n\r\nProcess Information:\r\n\tProcess ID:\t0xb64\r\n\tProcess Name:\tC:\\Windows\\System32\\calc.exe\r\n\tExit Status:\t0x0",
"Category":"Process Termination",
"Opcode":"Info",
"SubjectUserSid":"S-1-5-21-3503561835-834928460-339998367-1000",
"SubjectUserName":"achung",
"SubjectDomainName":"DEV-COLLECTOR",
"SubjectLogonId":"0x727518c",
"Status":"0x0",
"ProcessName":"C:\\Windows\\System32\\calc.exe",
"EventReceivedTime":"2016-01-18 10:32:14",
"SourceModuleName":"security_log",
"SourceModuleType":"im_msvistalog"
}

The value of the "ProcessID" field is 4, when it should be 2916 (0xb64), as shown in the "Message" field. Is this a configuration issue or possibly a bug with one of the modules?

Thanks,

Allen

Hi!

I'm new to nxlog and try to find a way to clear the Windows eventlogs during or after read/export with nxlog.

I've browsed through the manual, but can't find out how, is there a way to do this with a nxlog module/command?

Regards,

Mikael

Hello - I am testing nxlog to send windows based syslogs to papertrail. The following code seems to produce the logs, however, the log header entries are logging with (date, external ip address, "logger:")  on each line. To be consistent with the other logs being collected, would like these headers to be (date, HOSTNAME of originator, Process originator) how can I change the header that nxlog is pushing out to papertrail? The information that I would like to populate the header is within the message body but am not clear how to populate the fields to the headers output?

Here is an example of the current output line:

Jan 11 10:25:30 159.88.217.201 logger:  good dog <14>1 2016-01-11T10:25:28.998751-05:00 Win8MacVM1 Microsoft-Windows-Kernel-General 3116 - [NXLOG@14506 Keywords="-9223372036854775792" EventType="INFO" EventID="1" ProviderGuid="{A68CA8B7-004F-D7B6-A698-07E2DE0F1F5D}" Version="1" Task="0" OpcodeValue="0" RecordNumber="33715" ThreadID="3480" Channel="System" Domain="NT AUTHORITY" AccountName="SYSTEM" UserID="SYSTEM" AccountType="User" Opcode="Info" NewTime="2016-01-11T15:25:28.998234400Z" OldTime="2016-01-11T15:25:28.998234400Z" Reason="3" EventReceivedTime="2016-01-11 10:25:30" SourceModuleName="in" SourceModuleType="im_msvistalog"] The system time has changed to ‎2016‎-‎01‎-‎11T15:25:28.998234400Z from ‎2016‎-‎01‎-‎11T15:25:28.998234400Z.    Change Reason: System time adjusted to the new time zone.

Here is how I would like it to be:

Jan 11 10:25:30 Win8MacVM1 Microsoft-Windows-Kernel-General   good dog <14>1 2016-01-11T10:25:28.998751-05:00 Win8MacVM1 Microsoft-Windows-Kernel-General 3116 - [NXLOG@14506 Keywords="-9223372036854775792" EventType="INFO" EventID="1" ProviderGuid="{A68CA8B7-004F-D7B6-A698-07E2DE0F1F5D}" Version="1" Task="0" OpcodeValue="0" RecordNumber="33715" ThreadID="3480" Channel="System" Domain="NT AUTHORITY" AccountName="SYSTEM" UserID="SYSTEM" AccountType="User" Opcode="Info" NewTime="2016-01-11T15:25:28.998234400Z" OldTime="2016-01-11T15:25:28.998234400Z" Reason="3" EventReceivedTime="2016-01-11 10:25:30" SourceModuleName="in" SourceModuleType="im_msvistalog"] The system time has changed to ‎2016‎-‎01‎-‎11T15:25:28.998234400Z from ‎2016‎-‎01‎-‎11T15:25:28.998234400Z.    Change Reason: System time adjusted to the new time zone.

Any clues from the community on how to accomplish this?

*** Code snippet to follow ***

## UDP Papertrail Test

define ROOT C:\Program Files (x86)\nxlog
define ROOT_STRING C:\Program Files (x86)\\nxlog
  
Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log

# Include fileop while debugging, also enable in the output module below
<Extension fileop>
    Module      xm_fileop
</Extension>
 
<Extension syslog>
    Module      xm_syslog
</Extension>

 
# Enable json extension
<Extension json>
    Module      xm_json
</Extension>
 
<Input in>
    Module      im_msvistalog
    Exec    to_json();

    
    ReadFromLast TRUE
    SavePos     TRUE
    Query       <QueryList>\
                    <Query Id="0">\
                        <Select Path="Application">*</Select>\
                        <Select Path="System">*</Select>\
                        <Select Path="Security">*</Select>\
                    </Query>\
                </QueryList>    

</Input>

<Output out>
   Module      om_udp
   Host xxxx.papertrailapp.com
    Port NNNNN
#Test to write unique entry in front of each message but did not change the actual header
   Exec to_syslog_ietf();\
$raw_event = " good dog "+$raw_event;
#Use the following line for debugging (uncomment the fileop extension above as well)
   Exec file_write("C:\\Program Files (x86)\\nxlog\data\\nxlog_output.log",  $raw_event);
</Output>

<Route 1>
    Path        in => out
</Route>

*** Code snippet end ***

Hi, Where I can get nxlog agent package for CentOS 5 ?. Only packages available for verion 6 and 7 on download page. Could you please share the download link for the package that support centos 5.x ? 

Sorry for the activity, I hope this to be the last question - I've been playing with nxlog-ce for a week already.

How can I get a count of (unintentionally) dropped records? Something syslog-ng reports in "mark" messages... Do I have to create a buffer processor, drop the records "intentionally" in it and implement my own counter in script? Or does nxlog exhibit somehow its internal flow control stats?

Milan

Hi,

please advise me how to configure the following setup:

• several inputs (UDP, TCP, internal) joined into common path (i_udp, i_tcp, i_internal => p_pattern ...),
• one global processor (pattern, filter)  module,
• several outputs (file, UDP, TCP),
• every message is delivered to one of outputs according to processor's decision.

I call it "route branching". The solutions I have come to so far:

1. "broadcast": p_pattern => o_file, o_udp, o_tcp -- every output having an Exec filter to drop() messages that should be sent via another output,
2. "reroute": i_null => o_file; i_null => o_udp; i_null => o_tcp and use Exec reroute() in p_pattern.

The (1) is very inefficient. Otoh, (2) introduces problems with flow control, as the flow control is disabled when using reroute() (a new "feature" of 2.9.1504). So is there an efficient way with flow control enabled? ;-)

Milan 

Hello,

will you, please, publish recent source code, the one the latest CE packages were built from? I made quite a few tests with 2.9.1504 on Debian and now would like to patch some minor problems of pm_pattern (and contributing back), but last published sources are 2.8.1248. And I can not get a changelog between theese two versions either (the changelog packaged in the binary package, as mentioned in the previous post, is too brief):

nxlog-ce (2.9.1504) unstable; urgency=low

  * SVN snapshot release.

 -- Botond Botyanszki <boti@nxlog.org>  Tue, 08 Dec 2015 14:02:37 +0100

nxlog-ce (0.1-1) unstable; urgency=low

  * Initial Release.

 -- Botond Botyanszki <boti@nxlog.org>  Sat, 06 Nov 2010 18:24:10 +0200

Local variables:
mode: debian-changelog
End:

Where can I find the NXLOG code history for the various releases? I am currently on NXLOG 2.9.1347 and need to know what's new and fixed in 2.9.1504 but cannot locate where this information is.

Thank You,

Hi 

I'm recently tried to use NXLog community edition to maintain log files. Basically the idea is to rewrite log files one to one from many services on many hosts to one centralized server. 

In my test scenario I've created 3 services on one host and started propagate logs. Logs were rewriten succesfully but what I observed:
- It took 40 minutes to rewrite 1GB of logs (1GB connection with 60MB transfer between hosts)
- NXLog CPU usage on host where the logs were stored was about 10% (Intel Core i7)
- In Sysinternals ProcessMonitor I've observed that for one data read (65000 bytes)

16:02:26,4218297    nxlog.exe    12884    ReadFile    D:\app\Logs\service1.service\Logs_151130_12.log    SUCCESS    Offset: 490 945 000, Length: 65 000

there is about 450 partial data writes by 30/90/120 bytes which makes write process high ineffective

16:02:26,4219115    nxlog.exe    12884    LockFile    \\192.168.199.46\Shared\logs\service1\Logs_151130_12.log    SUCCESS    Exclusive: True, Offset: 0, Length: 4 294 967 295, Fail Immediately: False
16:02:26,4219594    nxlog.exe    12884    QueryStandardInformationFile    \\192.168.199.46\Shared\logs\service1\Logs_151130_12.log    SUCCESS    AllocationSize: 490 930 673, EndOfFile: 490 930 673, NumberOfLinks: 1, DeletePending: False, Directory: False
16:02:26,4219996    nxlog.exe    12884    WriteFile    \\192.168.199.46\Shared\logs\service1\Logs_151130_12.log    SUCCESS    Offset: 490 930 673, Length: 84, Priority: Normal
16:02:26,4228112    nxlog.exe    12884    UnlockFileSingle    \\192.168.199.46\Shared\logs\service1\Logs_151130_12.log    SUCCESS    Offset: 0, Length: 4 294 967 295

Is that lack of performance or do I have missed something. Usage of NXLog with such results is unacceptable for me.

My config file:

#define ROOT C:\Program Files\nxlog
define ROOT C:\Program Files (x86)\nxlog

Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log

define LOG_SERVER \\\\192.168.199.46
define LOGS_PATH d:\\App\\Logs\\\\*.log

<Extension _syslog>    
    Module      xm_fileop
    Module        xm_exec
</Extension>

<Input logs>
    Module im_file
    File "%LOGS_PATH%"
    InputType LineBased
    PollInterval 15
    SavePos TRUE
    CloseWhenIdle TRUE
    Recursive TRUE
    Exec $fileName = file_basename(file_name());
    Exec if file_name() =~ /^.*Logs\\(.*?)\.service/        \
    {                                                       \
        $serviceName = $1;                                  \
    }
</Input>

<Output out>    
    Module om_file
    CreateDir TRUE
    OutputType LineBased    
    File "%LOG_SERVER%\\Shared\\Logs\\" + $serviceName + "\\" + $fileName
</Output>

<Route 1>
    Path        logs => out
</Route>

Hi all,

I tried to use the gelf extension with the tcp module, but it doesn't work.
I saw tht the commercial version includes a special tcp gelf enxtension. Supports the community edition the usage of GELF over TCP?

Kind regards

Michael

Hello,
nxlog was sending to nagios log server just fine for a few days and then all of a sudden we started seeing the below errors in the nxlog.log file that are preventing logs from being sent.

ERROR om_tcp send failed; An existing connection was forcibly closed by the remote host.

ERROR couldn't connect to tcp socket on xx.xx.xxx.xx:3515; No connection could be made because the target machine actively refused it.

Is this something that you would support or would we need to go through Nagios support instead? We have already tried restarting the nxlog service but the issue remains.

Thank you 

Hello

When I call sql_exec with a bind parameter

if (sql_exec("INSERT INTO Events (Id) VALUES(?)", $ID) == TRUE) {}

I get the following error:

INFO SQLBindParam failed; HY104:1:0:[Microsoft][ODBC SQL Server Driver] Invalid precision value

Same commnad using string concatenatiion works fine:

if (sql_exec("INSERT INTO Events (Id) VALUES(" + $ID +")") == TRUE) {}

Could you help me with it?

Hello,

I have the following config file. What i want it to do is not send anything that has the word error in it.

<Input watchfile_m_NAME1>
  Module im_file
  File 'LOGLOCATION'
  Exec $Message = $raw_event;
  Exec if $raw_event !~ /ERROR/ drop();
  #Exec $SyslogSeverityValue = 6;
  Exec if file_name() =~ /.*\\(.*)/ $SourceName = $1; 
  #Exec if $raw_event =~ /ERROR/ $SyslogSeverityValue = 3;

The issue I am having is that it still sends the lines that say the word ERROR, but it sends them as an info message.  I wan them to not send them at all.  What am I doing wrong?

Hi,

We are trying to connect NXLog with Logmatic.io with a SSL connection. The certificate provided by Logmatic is valid and works with Rsyslog, Syslog-NG or Open-SSL.

However, no matter how hard we try to connect NXLog we have no success and not enough debugging information to troubleshoot it ourselves.

Here is our config:

<Output out>

    Module      om_ssl

    Host        api.logmatic.io

    Port        10515

    CAFile    <path_to_your_.crt_file>



####Add the API key before the event

    Exec     $raw_event="<your_api_key> "+$raw_event;

</Output>

We tried a lot more configuration with AllowUntrusted TRUE/FALSE, CertFile, etc... But the result is always the same:

2015-12-18 18:25:39 INFO connecting to api.logmatic.io:10515

2015-12-18 18:25:39 INFO successfully connected to api.logmatic.io:10515

2015-12-18 18:25:39 INFO reconnecting in 1 seconds

2015-12-18 18:25:40 INFO connecting to api.logmatic.io:10515

2015-12-18 18:25:40 INFO successfully connected to api.logmatic.io:10515

2015-12-18 18:25:41 INFO reconnecting in 1 seconds

2015-12-18 18:25:42 INFO connecting to api.logmatic.io:10515

2015-12-18 18:25:42 INFO successfully connected to api.logmatic.io:10515

Do you have any idea about what it could be?

You can find the certificate here: http://doc.logmatic.io/docs/logging-from-windows#section-enabling-security

Sorry to bother you with that but we spent a lot of time doing our homework before asking this question.

Thanks!!

I have a fairly simple question regarding sending an XML file to my SysLog Receiver (Nitro box).  Currently, I'm attempting to send this file to my Nitro box, but I'm not actually seeing the Syslog being sent to Nitro.  I have tcpdump watching for the packets being sent and I'm not getting any information across.  I verified that packets were being sent and captured to the Nitro box with a generic Syslog generator.  Here is my current .conf file.

## This is a sample configuration file. See the nxlog reference manual about the
## configuration options. It should be installed locally and is also available
## online at http://nxlog.org/nxlog-docs/en/nxlog-reference-manual.html

## Please set the ROOT to the folder your nxlog was installed into,
## otherwise it will not start.

#define ROOT C:\Program Files\nxlog
define ROOT C:\Program Files (x86)\nxlog

Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log

<Extension syslog>
 Module xm_syslog
</Extension>

<Extension xm_xml>
 Module xm_xml
</Extension>

<Input in>
Module im_file
File "C:\\Users\\Administrator\\Desktop\\NXLogTest\\test.xml"
</Input>

<Output out>
     Module om_udp
     Host xxx.xxx.xxx.xxx
 Port 514
 Exec to_syslog_bsd();
</Output>

<Route 1>
    Path        in => out
</Route>

After saving this file, I restart the service, but nothing is being sent.  I also checked the logs and there was no error or warnings, the service stops, exits, and restarts nicely.  I'd appreciate some help so I have a clear path moving forward.

Thanks!

Hi, 

I'm using im_file module to read windows log file to elasticsearch, and I need to get source file name and some fix string like log type to elastic search as well. I used below configuration, but i couldn't find the FileName in elasticsearch, can someone help? thanks a lot!

<Input TestFileInput>
    Module          im_file
    File              'E:\test\app*.log'
    Exec            $FileName = file_name();
</Input>

Hi,

I can't find the changelog for the community version: nxlog-ce-2.9.1504.

Where is it published?

thanks and br,

Patrick

Hi, I have a system that produces logs files. I send them via tcp to specific port. 

Directory path with logs looks like: 

C:\Logs\<ID>\*.log

Where ID is the unique 3-digit identifier. How to get this ID and send it with every log event?

My current config: 

define ROOT C:\Program Files (x86)\nxlog

Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log

<Input logs>  
    Module    im_file
    File    'C:\\Logs\\*\\*.log'
    ReadFromLast True
    SavePos True
    Recursive True
    Exec    if $raw_event =~ /^#/ drop();
</Input> 

<Processor buffer_logs>
    Module      pm_buffer
    MaxSize    102400
    Type    Mem
    WarnLimit    51200
</Processor>

<Output out_logs>  
    Module  om_tcp
    Host    localhost
    Port    10010
    OutputType  LineBased
</Output> 

<Route r_logs>  
    Path    logs => buffer_logs => out_logs
</Route>

Hi,

I was just wondering whether you intend to publish the source code for the current (2.9.1347) version?

Regards,

Al

Hi!

I have more than 5000 Handles in nxlog.exe

I analysed with Sysinternals Process Explorer. Most of Handles are from Type Thread.

When i look at threads i only see 10 threads in nxlog.exe but 5000 Thread Handles.

Can you verify that issue?

Version: nxlog-ce-2.9.1347