Hi Team, I have a single log source that is pumping around 40K EPS, which our NX server is unable to handle, my question is how do I increase the log ingestion capacity. Current setup on an AWS VM: Ubuntu 20.04 LTS 8 CPU, 32GB Ram, 32gb SSD As per my understanding we needed to increase the number of routes tied to the input, as well as the average event size and batch sizes, hence edited the nxlog.con file with following 1 input, 8 routes, 2048 byte average event size, 25000 event batch size.

Even with these settings, we are not processing more then 6k EPS.

Can anyone advice, what else we can do, please? Note: filtering of events at the source is not an option.

Hi,

The log folder is /opt/nxlog-manager/log

2 files will be there. nxlog-manager.log and nxlog-manager.err

How do I control the housekeeping of these 2 files ?

Please kindly advise. Thanks !

Hello

I'm trying to use xm_netflow in NXLog EE.

My configuration:

<Extension netflow> Module xm_netflow </Extension>

<Extension json> Module xm_json </Extension>

<Input in_10533_netflow_udp> Module im_udp Host 0.0.0.0 Port 10533 InputType netflow </Input>

<Route route_10533_netflow> Path in_10533_netflow_udp => out_file </Route>

<Output out_file> Module om_file File "/opt/nxlog/var/log/out.log" Exec to_json(); </Output>

But I get an error when I try run nxlog: Jun 24 12:27:50 xxx.evil.corp nxlog[3734]: 2022-06-24 12:27:50 ERROR [CORE|main] Invalid InputType 'netflow' at netflow.conf

What i do wrong?

RPMs:

rpm -qa | grep nxlog

nxlog-kafka-5.3.6735-1.el7_9.x86_64 nxlog-python-5.3.6735-1.el7_9.x86_64 nxlog-5.3.6735-1.el7_9.x86_64

We tested nxlog ce a few year back but did not end up using it in our company. We recently ran a vulnerability scan of the servers we had this installed on and they reported that nxlog was still installed. We can confirm that nxlog isn't visible in add/remove programs and the nxlog service isn't running. There is still a nxlog folder in c:\program files (x86). It only contains a data folder with two files in it. I'm assuming its registry entries are whats getting detected. Is there any documentation on what registry entries I should remove to completely remove this software?

After the upgrade from 5.5 to 5.6, the web UI is unable to load, jetty service unavailable.

Not sure why this is happening.

I have installed NXlog community free edition on my WEC. How do I forward logs from NXlog to WEC?

HELLO hello sir or madam i have a question about NXlog. when I got NXLOG from my central logs i run into problem like this error that i sent on attachment i look forward to your answering. i in advance say thank you from your favor. ERROR data size (67324) is over the limit (65000), will be truncated y

When installing nxlog- ce, I get problems with dependency for the package.

sudo apt install ./nxlog-ce_3.0.2272_ubuntu_focal_amd64.deb Reading package lists... Done Building dependency tree... Done Reading state information... Done Note, selecting 'nxlog-ce' instead of './nxlog-ce_3.0.2272_ubuntu_focal_amd64.deb' Some packages could not be installed. This may mean that you have requested an impossible situation or if you are using the unstable distribution that some required packages have not yet been created or been moved out of Incoming. The following information may help to resolve the situation:

The following packages have unmet dependencies. nxlog-ce : Depends: libperl5.30 (>= 5.30.0) but it is not installable Depends: libpython3.8 (>= 3.8.2) but it is not installable E: Unable to correct problems, you have held broken packages.

I can do a lot of trickery but was wondering if you will create a new upgrade soon? Pinning Python3.8 is a bit tricky and so is libperl5.30

Regards, Mats Tage Axelsson

Hi,

I installed recently the last version of NXLOG-CE (3.0.2284) on my windows server 2016 Standard. I noticed that the configcache is not functionnal in my case because of an erreor when nxlog service stop => "System Error 109 has occurred. The pipe has ended" My configcache.dat file is not updated and when I restart the service, a lot of old logs are sended to my logstash

Here's my nxlog conf file

Panic Soft

define ROOT C:\nxlog define CERTDIR %ROOT%\cert

define CONFDIR %ROOT%\conf define LOGDIR %ROOT%\data

define LOGFILE %LOGDIR%\nxlog.log LogFile %LOGFILE% LogLevel INFO

Moduledir %ROOT%\Modules Pidfile %ROOT%\nxlog.pid SpoolDir %ROOT%\SpoolDir CacheDir %ROOT%\CacheDir

<Extension _syslog> Module xm_gelf </Extension>

<Extension json> Module xm_json </Extension>

<Extension _charconv> Module xm_charconv AutodetectCharsets iso8859-2, utf-8, utf-16, utf-32 </Extension>

<Extension _exec> Module xm_exec </Extension>

<Extension _fileop> Module xm_fileop

# Check the size of our log file hourly, rotate if larger than 5MB
<Schedule>
    Every   1 hour
    Exec    if (file_exists('%LOGFILE%') and \
               (file_size('%LOGFILE%') >= 5M)) \
                file_cycle('%LOGFILE%', 8);
</Schedule>

# Rotate our log file every week on Sunday at midnight
<Schedule>
    When    @weekly
    Exec    if file_exists('%LOGFILE%') file_cycle('%LOGFILE%', 8);
</Schedule>

</Extension>

Collecting event log

<Input eventlog> Module im_msvistalog ReadFromLast TRUE SavePos TRUE Exec to_json(); </Input>

<Output ssl> Module om_ssl Host XX.XX.XX.XX CertFile %CERTDIR%\MyCertFile.crt Port XXXX AllowUntrusted TRUE </Output>

<Route 1> Path eventlog => ssl </Route>

Thank's a lot for your help

Hi, I have to forward tsmlogs to the server, I have below configuration . My audit team needs hostname printed when they receive the logs on their end. I have below configuration

<Output tsmout> Module om_tcp Host 10.24.8.23 Port 30133 Exec $FQDN = hostname_fqdn();

Exec parse_syslog_bsd();

</Output>

<Input tsmlogs> Module im_file File "/scripts/tsm/log/tsm.log" #File "/scripts/tsm/log/" + $FQDN + "-" + "tsm.log"

Exec $Hostname = hostname_fqdn();

</Input>

<Route tsmroute> Path tsmlogs => tsmout </Route>

How do I add hostname when they get forwarded.

Hi all,

NXLOG newbie. I'm trying to inject logs from a specific hostname into a specific folder. For example:

if hostname = x then store log in folder = y

I'm running a very bare-bones set up. I'd like to use the FQDN as the variable for hostname rather than an IP. Current config:

Panic Soft #NoFreeOnExit TRUE

define ROOT C:\Program Files\nxlog define CERTDIR %ROOT%\cert define CONFDIR %ROOT%\conf\nxlog.d define LOGDIR %ROOT%\data

define LOGFILE %LOGDIR%\nxlog.log LogFile %LOGFILE%

Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data

<Extension _syslog> Module xm_syslog </Extension>

<Extension _charconv> Module xm_charconv AutodetectCharsets iso8859-2, utf-8, utf-16, utf-32 </Extension>

<Extension _exec> Module xm_exec </Extension>

<Extension _fileop> Module xm_fileop

# Check the size of our log file hourly, rotate if larger than 5MB
<Schedule>
    Every   1 hour
    Exec    if (file_exists('%LOGFILE%') and \
               (file_size('%LOGFILE%') >= 5M)) \
                file_cycle('%LOGFILE%', 8);
</Schedule>

# Rotate our log file every week on Sunday at midnight
<Schedule>
    When    @weekly
    Exec    if file_exists('%LOGFILE%') file_cycle('%LOGFILE%', 8);
</Schedule>

</Extension>

<Extension syslog> Module xm_syslog </Extension>

<Extension fileop> Module xm_fileop </Extension>

<Input udp> Module im_udp Host 0.0.0.0 Port 514 Exec parse_syslog(); </Input>

<Output file> Module om_file File "C:/Syslogs/nxlog/" + $HostName + "/" + $HostName + ".log" CreateDir TRUE <Exec> if file_size("C:/Syslogs/nxlog/" + $HostName + "/" + $HostName + ".log") > 10M { file_cycle("C:/Syslogs/nxlog/" + $HostName + "/" + $HostName + ".log",2); reopen(); } </Exec> </Output>

<Route 1> Path udp => file </Route>

Snare compatible example configuration

Collecting event log

<Input in>

Module im_msvistalog

</Input>

Converting events to Snare format and sending them out over TCP syslog

<Output out>

Module om_tcp

Host 192.168.1.1

Port 514

Exec to_syslog_snare();

</Output>

Connect input 'in' to output 'out'

<Route 1>

Path in => out

</Route>

Hello,

I'm trying to send Windows events to two different syslog servers.

I have two output modules and one route. I see syslog packets getting sent to both collectors but the packets are not coming in the correct format

The configuration looks like

LogLevel INFO Logfile %LOGDIR%/nxlog.log

<Extension agent_managment> Module xm_soapadmin Connect 10.x.x.n Port 4041 SocketType SSL CAFile %CERTDIR%/agent-ca.pem AllowUntrusted FALSE RequireCert TRUE <ACL conf> Directory %CONFDIR% AllowRead TRUE AllowWrite TRUE </ACL> <ACL cert> Directory %CERTDIR% AllowRead TRUE AllowWrite TRUE </ACL> </Extension>

<Extension Extension_json> Module xm_json </Extension>

<Input eventlog> Module im_msvistalog SavePos TRUE ReadFromLast TRUE Exec if ($EventID == 5156) drop(); </Input>

<Input internal_mod> Module im_internal </Input>

<Output out> Module om_tcp Host 10.x.x.a Port 514 OutputType LineBased <Exec> $Hostname = string(host_ip()); to_syslog_ietf(); parse_syslog(); $Message = '@cee: ' + to_json(); to_syslog_bsd(); </Exec> </Output>

<Output outlcp> Module om_udp Host 10.x.x.b Port 514 OutputType Dgram Exec $Hostname = string(host_ip()); Exec to_syslog_snare();

</Output> <Route Syslog_lcp> Priority 1 Path eventlog, internal_mod => outlcp, out </Route>

I have an XML that I am trying to strip some data out so it can be pre-processed by software on another machine. XML file is being generated on a windows logging to an XML file. I am at bit of a lost I have tried too many things to list here. What I am trying to do is remove our domain name and our domain email address before it is sent to the machine to be pre-processed. I only want the username. Any records that have host\domain.com don’t need to be sent and I figured out how to drop that data. (number 4). If it helps I am running nxlog-ce-3.0.2284.
Here is an example of event: (Removed bunch of xml fields for clarity of this post) 1: <Event><Timestamp data_type="4">05/25/2022 12:45:43.806</Timestamp><Userid data_type="1">DOMAIN\username</Userid><IP-Address data_type="3">x.x.x.x</IP-Address><Endtimestamp data_type="5">05/25/2022 12:46:43.806</Endtimestamp> 2: <Event><Timestamp data_type="4">05/25/2022 12:45:43.806</Timestamp><Userid data_type="1">username</Userid><IP-Address data_type="3">x.x.x.x</IP-Address><Endtimestamp data_type="5">05/25/2022 12:46:43.806</Endtimestamp> 3: <Event><Timestamp data_type="4">05/25/2022 12:45:43.806</Timestamp><Userid data_type="1">username@domain.com</Userid><IP-Address data_type="3">x.x.x.x</IP-Address><Endtimestamp data_type="5">05/25/2022 12:46:43.806</Endtimestamp> 4: <Event><Timestamp data_type="4">05/25/2022 12:45:43.806</Timestamp><Userid data_type="1">host\domain.com</Userid><IP-Address data_type="3">x.x.x.x</IP-Address><Endtimestamp data_type="5">05/25/2022 12:46:43.806</Endtimestamp>

Nxlog.conf: #NoFreeOnExit TRUE

define ROOT C:\Program Files\nxlog define CERTDIR %ROOT%\cert define CONFDIR %ROOT%\conf define LOGDIR %ROOT%\data

define LOGFILE %LOGDIR%\nxlog.log LogFile %LOGFILE%

Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data

<Extension xmlparser> Module xm_xml </Extension>

<Extension json> Module xm_json </Extension>

<Input in> Module im_file File "C:\LogFiles\log*.log" InputType LineBased Exec $Message = $raw_event; SavePos TRUE
ReadFromLast TRUE <Exec>

Discard everything that doesn't seem to be an xml event

if $raw_event !~ /^<Event>/ drop(); if $raw_event =~ /^(.+)host(.+)/ drop();

parse_xml();

Convert to JSON

    to_json();

</Exec> </Input>

<Output out> Module om_udp Host yy.xx.xx.xx Port 514 </Output>

<Route 1> Path in => out </Route>

Hey All,

A bit of a newbee and trying to get NXLOG working with GrayLog. It is working and I'm seeing the information. The issue is that the information I'm seeing does not seem to match the PC's event logs.

Please see below Config File:

<Extension _exec> Module xm_exec </Extension>

<Extension _fileop> Module xm_fileop

# Check the size of our log file hourly, rotate if larger than 5MB
<Schedule>
    Every   1 hour
    Exec    if (file_exists('%LOGFILE%') and \
               (file_size('%LOGFILE%') >= 5M)) \
                file_cycle('%LOGFILE%', 8);
</Schedule>

# Rotate our log file every week on Sunday at midnight
<Schedule>
    When    @weekly
    Exec    if file_exists('%LOGFILE%') file_cycle('%LOGFILE%', 8);
</Schedule>

</Extension>

<Extension _gelf> Module xm_gelf </Extension>

<Input win> Module im_msvistalog </Input>

Converting events to Snare format and sending them out over TCP syslog

<Output graylog> Module om_udp Host X.X.X.X Port 3514 OutputType GELF </Output>

<Route graylog_route> Path win => graylog </Route>

Any ideas about what I'm doing wrong????

HTTP ERROR: 503 Problem accessing /nxlog-manager. Reason:

Service Unavailable

New install. Not sure why this is fighting me :)

I am betting I am simply missing a step. Anyone able to help?

Unpacking nxlog-manager (5.6.5633) ... dpkg-deb (subprocess): decompressing archive member: lzma error: compressed data is corrupt dpkg-deb: error: <decompress> subprocess returned error exit status 2 dpkg: error processing archive nxlog-manager-5.6.5633-jdk1.7.1.deb (--install): cannot copy extracted data for './opt/nxlog-manager/webapps/nxlog-manager/WEB-INF/lib/spring-js-resources-2.4.2.RELEASE.jar' to '/opt/nxlog-manager/webapps/nxlog-manager/WEB-INF/lib/spring-js-resources-2.4.2.RELEASE.jar.dpkg-new': unexpected end of file or stream Processing triggers for systemd (245.4-4ubuntu3.17) ... Errors were encountered while processing: nxlog-manager-5.6.5633-jdk1.7.1.deb

Dears,

I'm running nxlog-ce 3.0.2272 under Windows Server 2022 to search the Application event log for specific events using a custom query in order to forward them as GELF messages to some Graylog server. The connection is secured by SSL. With Server 2019, everything runs smoothly but with Server 2022, nxlog.exe keeps crashing after a few events have been collected and sent to Graylog. Worst fact is, that events are omitted and not transferred to Graylog.

The related event log entries (event id 1000):

Faulting application name: nxlog.exe, version: 0.0.0.0, time stamp: 0x00000000
Faulting module name: ntdll.dll, version: 10.0.20348.681, time stamp: 0x69d3cd31
Exception code: 0xc0000374
Fault offset: 0x0000000000103ad9
Faulting process id: 0x2b1c
Faulting application start time: 0x01d86901d76501a6
Faulting application path: C:\Program Files (x86)\nxlog\nxlog.exe
Faulting module path: C:\WINDOWS\SYSTEM32\ntdll.dll
Report Id: 362167c9-9922-4158-8d56-ee4bafd21e67
Faulting package full name: 
Faulting package-relative application ID: 





Faulting application name: nxlog.exe, version: 0.0.0.0, time stamp: 0x00000000
Faulting module name: RPCRT4.dll, version: 10.0.20348.707, time stamp: 0xd31f9dd8
Exception code: 0xc0000005
Fault offset: 0x00000000000272e3
Faulting process id: 0x1a24
Faulting application start time: 0x01d868da69310cd8
Faulting application path: C:\Program Files (x86)\nxlog\nxlog.exe
Faulting module path: C:\WINDOWS\System32\RPCRT4.dll
Report Id: 96a6d244-74ca-4f6f-8667-8bb5082a452a
Faulting package full name: 
Faulting package-relative application ID: 

Any idea?

Thanks ahead,
Elix

Has anyone installed NXLog Enterprise edition on Hyper-V? Any pros/cons if so?

Team,

Need your assistance on nxlog agent.

I have installed nxlog agent(Community Edition) in windows server and i could see the destination folder had created under c:/Program files (x86)/nxlog but when went to restart the nxlog service, i dont see the nxlog service inside services(service.msc) not sure where is the problem and need your assistance to fix it.

Version: nxlog-ce-2.11.2190

Thanks, Vignesh