Hello all. I wanted to know if anyone has had any luck or if it is possible to add a second <Output Out> configuration to the current nxlog.conf? Currently want to test a new log collector (Taegis) along side our current collector (Masergy) so we have streaming logs concurrently to each collector. 

e.g

Output out1>   Module om_tcp   Host 192.168.1.100   Port 514</Output> 

# Define the output to send logs to the second destination IP<Output out2>   Module om_tcp   Host 192.168.1.101   Port 514</Output> 

Thank you. 

Looking to test some ingest into a data lake to test searches adn dashboards.

hi,

I try  to DROP all messages, if they contains somewhere “/connection_status” or “/status”, but what ever I try, the filter won't fit on Nxlog, while it works in RegexTester

Example log:

Apr 25 11:15:11 nomad-cde cde_hpp-c7d794e5-6297-e29a-81d4-c284f52e8981: {"Hostname":"nomad-cde","ShortMessage":{"message":"Matched route \"status\".","context":{"route":"status","route_parameters":{"_route":"status","_controller":"App\\Controller\\Monitoring\\MonitoringController::getStatus"},"request_uri":"https://hpp.example.com/status","method":"GET"},"level":200,"level_name":"INFO","channel":"request","datetime":"2024-04-25T09:15:10.999734+00:00","extra":{}},"EventTime":"2024-04-25T11:15:11.000000+02:00","SeverityValue":6,"command":"/home/app/entrypoint.sh start php-fpm","container_id":"2baaa8302059bbcb881aac339807d429b1d91e1a117659e37cb4edfb7bb1eeca","container_name":"cde_hpp-c7d794e5-6297-e29a-81d4-c284f52e8981","image_id":"sha256:7ddb43de759bd8ef93975cf10b25289a4f8628587b6d31cbe3d16e8d39443572","image_name":"harbor.example.com/testing/hpp/test:433e16c7","tag":"production","EventReceivedTime":"2024-04-25T11:15:11.001689+02:00","SourceModuleName":"container","SourceName":"cde_hpp-c7d794e5-6297-e29a-81d4-c284f52e8981","SyslogFacility":6}

or

Apr 25 11:15:09 nomad-cde cde_iframes-3709bf3d-86a1-0264-fe9f-150ac2b14cdd: {"Hostname":"nomad-cde","ShortMessage":"172.16.0.40 - - [25/Apr/2024:09:15:09 +0000] \"GET /status HTTP/1.0\" 200 2 \"-\" \"-\" \"172.16.1.40\"","EventTime":"2024-04-25T11:15:09.479000+02:00","SeverityValue":6,"command":"/home/app/entrypoint.sh start php-fpm","container_id":"c8b7c9357b1bc195f6d88d09e4c329627bfe165debc09cfe4bbfd556fdab966c","container_name":"cde_iframes-3709bf3d-86a1-0264-fe9f-150ac2b14cdd","image_id":"sha256:be421273041ffa5d7b8be4963f91c0376d9829ba942b86341413c59105ae671c","image_name":"harbor.example.com/testing/iframes/test:3cb57629","tag":"production","EventReceivedTime":"2024-04-25T11:15:09.524068+02:00","SourceModuleName":"container","SourceName":"cde_iframes-3709bf3d-86a1-0264-fe9f-150ac2b14cdd","SyslogFacility":6}

or

Apr 25 11:15:09 nomad-cde cde_hpp-c7d794e5-6297-e29a-81d4-c284f52e8981: {"Hostname":"nomad-cde","ShortMessage":{"message":"Matched route \"connection_status\".","context":{"route":"connection_status","route_parameters":{"_route":"connection_status","_controller":"App\\Controller\\Monitoring\\MonitoringController::getStatusDB"},"request_uri":"https://web:4433/connection_status","method":"GET"},"level":200,"level_name":"INFO","channel":"request","datetime":"2024-04-25T09:15:09.603963+00:00","extra":{}},"EventTime":"2024-04-25T11:15:09.605000+02:00","SeverityValue":6,"command":"/home/app/entrypoint.sh start php-fpm","container_id":"2baaa8302059bbcb881aac339807d429b1d91e1a117659e37cb4edfb7bb1eeca","container_name":"cde_hpp-c7d794e5-6297-e29a-81d4-c284f52e8981","image_id":"sha256:7ddb43de759bd8ef93975cf10b25289a4f8628587b6d31cbe3d16e8d39443572","image_name":"harbor.example.com/testing/hpp/test:433e16c7","tag":"production","EventReceivedTime":"2024-04-25T11:15:09.634920+02:00","SourceModuleName":"container","SourceName":"cde_hpp-c7d794e5-6297-e29a-81d4-c284f52e8981","SyslogFacility":6}

I tried in the end the simplest one:

if $raw_event =~ /.*status.*/ drop();

 but it does not match. The config looks like this:

<Output syslog-container-server>
    Module om_udp
    Host ${user.logserver}
    Port 514
    <Exec>
        if $raw_event =~ /.*status.*/ drop();
        $Hostname = "nomad-cde";
        $message =~ s/-p[^\s]+/-pXXX/;
        delete($SourceModuleType);
        delete($MessageSourceAddress);
        delete($version);
        delete($created);
        $SourceName = $container_name;
        $SyslogFacility = $SeverityValue;
        to_json();
        to_syslog_bsd();
    </Exec>
</Output>

Can someone give me an hint, where I have to look ?

Good Morning All,

We would need to take advantage of the new features within NXLOG 6.0 EE.  Are there any instructions on to perform the upgrade from 5.0 to 6.0?  OR is this a revamp oft he whole environment and re-deployment of the agents?  I currently have 900 agents deployed and it would not make sense to re-deploy.

I'm currently using nxlog to collect windows event log and notice in the local log file there are time differences between event time and event received time. Event received time was about half an hour behind event time, any idea what would cause this to happen?

I noticed that many Azure heartbeat logs will send to SIEM, if i want to config the nxlog output file, how to filter it out and make it not send the logs to SIEM? Thanks.

Hi,

Understand that the Community Edition .msi installer are not digitally signed and there are previous discussion on this. 

Hope that I can some answers on where I can get the hashes for nxlog-ce-3.2.2329.msi to verify the file downloaded.

The following are the Hash values I got for my downloaded file

MD5: 31862b5f58bbd07c82fc5b3b507a3fd1

SHA1: 3b9ef0f6886d57601b9a072554cd78d7870f1866

Thank you very much.

Hello,

we using Nomad which sends logs in GELF format. We need to forward it to Rsyslog and also to Graylog. For Syslog I want to set $SourceName, which needs to be exracted from the JSON / GELF.

The config looks like this:

...

<Input container>
    Module        im_tcp
    ListenAddr    127.0.0.1:12202
    InputType     GELF_TCP
</Input>

...

<Output syslog-container-server>
    Module om_udp
    Host ${user.logserver}
    Port 514
    Exec to_json();
    Exec $message =~ s/-p[^\s]+/-pXXX/;
    Exec to_syslog_bsd();
</Output>

...

<Route container-to-syslog>
  Path container => syslog-container-buffer => syslog-container-server
</Route>

And the log on the rsyslog:

Apr 15 15:24:26 qh-a07-nomad-agent-03 {"version": "1.1","Hostname":"qh-a07-nomad-agent-03","ShortMessage":"[2024-04-15 13:24:26] app.DEBUG: Connected to redis...PONG [] []","EventTime":"2024-04-15T15:24:26.376000+02:00","SeverityValue":6,"command":"/home/app/entrypoint.sh start php-fpm","container_id":"f1...","container_name":"iframes-c77e666c-fd39-f6f6-4d57-b416a4a7e28a","created":"2024-04-12T08:58:36.870730597Z","image_id":"sha256:2a26fed9c075899cfe86d74f8f44c2729be0f392a96d10c938795fe84036506d","image_name":"repos/production/iframes/production:68c00192","tag":"production","MessageSourceAddress":"127.0.0.1","EventReceivedTime":"2024-04-15T15:24:26.376703+02:00","SourceModuleName":"container","SourceModuleType":"im_tcp"}

How can I extract `container_name`

and use for $SourceName = 'my_application';  

so that “my_application” is replaced with the content of "container_name ?

cu denny

This might seem as an odd thing, but I have a need where I want to combine syslog as well as json in the same message. Syslog should be combined (without the message field) with the complete $raw_event as json. I've successfully converted the entire thing to json with 

$json_message = to_json();

However when I attempt the same thing with to_syslog_ietf(); an error is thrown. How would I achieve this behaviour with CE?

Couldn't parse Exec block at C:\Path\nxlog.conf:58; couldn't parse statement at line 72, character 42 in C:\Path\nxlog.conf; function 'to_syslog_ietf()' does not exist or take different arguments.

Hello,

My current architecture is a windows nxlog agent sending logs to a remote syslog server. The agent is translating Windows event logs to json encapsulated syslog before sending them.

I've encountered an inconsistency with the hostname field of the sent log, most of the sent logs contain the hostname as expected, but some only contains the IP address which creates a mess on the sorting I made on the remote syslog server.

I haven't tried anything yet as I don't really know where to look for. My take is that it is a windows event log issue that can't be fixed but i'd like your opinions.

Thank you for your help.

• error message still remain: 

ERROR failed to subscribe to msvistalog events,access denied [error code : 5]; Access is denied

• Change the logon on account to administrator to start service 
• reinstall nxlog in Server 
• added local admin account in manage auditing and security log properties 

Hi,

I am getting the following error when using the AllowIP Directive in Enterprise Edition 6.2:

2024-04-02 15:17:42 ERROR [im_udp|SynologySyslog] invalid keyword: AllowIP at C:\Program Files\nxlog\conf\nxlog.conf:45

The config snippit containing this is:

<Input SynologySyslog>
	module		im_udp
	ListenAddr	0.0.0.0:514
	AllowIP 10.0.0.106
	<Exec>
		parse_syslog_ietf();
	</Exec>
</Input>

<Input SynologySyslog>module  im_udpListenAddr 0.0.0.0:514AllowIP 10.0.0.106<Exec> parse_syslog_ietf();</Exec></Input>

Any help would be greatly appreciated!!

I spun up a brand-new Linux instance in AWS. I downloaded the RHEL9 CE package and got it onto that instance. I installed it as:

yum -y localinstall nxlog-ce-3.2.2329_rhel9.x86_64.rpm

The problems:

1. Nothing gets installed to /opt/nxlog; NXLog gets installed instead to /etc/nxlog
2. There aren't any modules downloaded/installed

What am I missing?

Hello, Does anyone know how to install nxlog on a Windows 32-bit system? Windows Server 2003

Hi:

I am new to nxlog but I do haves sending windows events into graylog via nxlogs so I know some basics.I am know trying to parse csv exchange logs.

I am running the community version.

I realize I have no output or routing statements yet.

The log does not complain about the module xm_csv being found but does complain about module csv_parser not being found.

I used this as starting point: https://docs.nxlog.co/integrate/exchange.html using the community section for reference.If someone could offer any hints I would be most grateful.

--mikej

Version: nxlog-ce-3.2.2329

LOGFILE:

C:\Program Files\nxlog\data>type nxlog.log2024-03-25 15:15:51 ERROR Couldn't parse Exec block at C:\Program Files\nxlog\conf\nxlog.d\\protocol.conf:21; couldn't parse statement at line 25, character 27 in C:\Program Files\nxlog\conf\nxlog.d\\protocol.conf; module csv_parser not found2024-03-25 15:15:51 ERROR Couldn't parse Exec block at C:\Program Files\nxlog\conf\nxlog.d\\protocol.conf:34; couldn't parse statement at line 38, character 26 in C:\Program Files\nxlog\conf\nxlog.d\\protocol.conf; module csv_parser not found2024-03-25 15:15:51 WARNING not starting unused module smtp_receive2024-03-25 15:15:51 WARNING not starting unused module smtp_send2024-03-25 15:15:51 INFO nxlog-ce-3.2.2329 started

CONFIG FILE: protocol.conf - in nxlog.d

define BASEDIR C:\Program Files\Microsoft\Exchange Server\V15

#Software: Microsoft Exchange Server#Version: 15.0.0.0#Log-type: SMTP Receive Protocol Log#Date: 2024-03-25T19:00:26.686Z#Fields: date-time,connector-id,session-id,sequence-number,local-endpoint,remote-endpoint,event,data,context

## MJ number of fields matches count

<Extension csv>       Module    xm_csv       Fields    date-time, connector-id, session-id, sequence-number, \                 local-endpoint, remote-endpoint, event, data, context</Extension>

<Input smtp_receive>Module    im_file   File      '%BASEDIR%\TransportRoles\Logs\Hub\ProtocolLog\SmtpReceive\RECV*.LOG'<Exec> if $raw_event =~ /^(\xEF\xBB\xBF)?(date-time,|#)/ drop(); else {  csv_parser->parse_csv();  $EventTime = parsedate(${date-time}); }</Exec></Input>

<Input smtp_send>Module    im_fileFile      '%BASEDIR%\TransportRoles\Logs\Hub\ProtocolLog\SmtpSend\SEND*.LOG'<Exec>   if $raw_event =~ /^(\xEF\xBB\xBF)?(date-time,|#)/ drop();else{ csv_parser->parse_csv(); $EventTime = parsedate(${date-time});}</Exec></Input>

Hi team,

Our current .conf file has only one output module and sending logs to only one destination. Can we send the logs to 2 different destination parallelly(Specifically we need to send to Accenture MSS)

Regards, Anjani CM

Hello,

I am getting the following error message with the SSL configured using om_ssl . Has anyone encountered this issue in the past? The config works without SSL but I want to make SSL to work.Please note that some information has been modified to avoid sensitive information exposure.

2024-03-20 00:26:21 INFO connecting to destination_host:###
2024-03-20 00:26:21 INFO successfully connected to destination_host:###
2024-03-20 00:26:21 INFO reconnecting in 1 seconds
2024-03-20 00:26:21 ERROR SSL certificate verification failed: unable to get issuer certificate (err: 2)

This is my nxlog agent config code snippet:

...
<Output out_to_destination>
    Module          om_ssl
    Host    	    %OUTPUT_DESTINATION_HOST%
    Port    	    %OUTPUT_DESTINATION_PORT%
    Exec    	    $Message = to_json(); to_syslog_bsd();
    CAFile          %CERTDIR%\CA.pem
    CertFile        %CERTDIR%\client-cert.pem
    CertKeyFile     %CERTDIR%\client-key.pem
    AllowUntrusted  TRUE
</Output>
...

Is there anyway to bypass verification? Is this issue on the nxlog agent side?

Been searching the internet to see if anyone has asked this before.

Are there any plans for NXlog to support DTLS for secure low overhead forwarding?

Hi All, 

I am trying to test and evaluate the NXlog for collect the dns analytical log(ETL) and forward it to splunk directly. now I am using the community version of NXlog and get below error:

2024-03-14 10:35:31 ERROR Failed to load module from C:\Program Files\nxlog\modules\input\im_etw.dll, The specified module could not be found.  ; The specified module could not be found.  2024-03-14 10:35:31 ERROR invalid keyword: HTTPHeader at C:\Program Files\nxlog\conf\nxlog.conf:902024-03-14 10:35:31 ERROR module 'out_to_splunk' has configuration errors, not adding to route '1' at C:\Program Files\nxlog\conf\nxlog.conf:942024-03-14 10:35:31 ERROR route 1 is not functional without output modules, ignored at C:\Program Files\nxlog\conf\nxlog.conf:942024-03-14 10:35:31 WARNING no routes defined!

Could someone please help to point the error/misconfiguration from the below NXlog.conf? thanks.

nxlog.conf

Panic Soft
#NoFreeOnExit TRUE

define ROOT     C:\Program Files\nxlog
define CERTDIR  %ROOT%\cert
define CONFDIR  %ROOT%\conf\nxlog.d
define LOGDIR   %ROOT%\data

include %CONFDIR%\\*.conf
define LOGFILE  %LOGDIR%\nxlog.log
LogFile %LOGFILE%

Moduledir %ROOT%\modules
CacheDir  %ROOT%\data
Pidfile   %ROOT%\data\nxlog.pid
SpoolDir  %ROOT%\data

<Extension _syslog>
   Module      xm_syslog
</Extension>

<Extension _charconv>
   Module      xm_charconv
   AutodetectCharsets iso8859-2, utf-8, utf-16, utf-32
</Extension>

<Extension _exec>
   Module      xm_exec
</Extension>

<Extension _fileop>
   Module      xm_fileop

   # Check the size of our log file hourly, rotate if larger than 5MB
   <Schedule>
       Every   1 hour
       Exec    if (file_exists('%LOGFILE%') and \
                  (file_size('%LOGFILE%') >= 5M)) \
                   file_cycle('%LOGFILE%', 8);
   </Schedule>

   # Rotate our log file every week on Sunday at midnight
   <Schedule>
       When    @weekly
       Exec    if file_exists('%LOGFILE%') file_cycle('%LOGFILE%', 8);
   </Schedule>
</Extension>

# Snare compatible example configuration
# Collecting event log
# <Input in>
#     Module      im_msvistalog
# </Input>
# 
# Converting events to Snare format and sending them out over TCP syslog
# <Output out>
#     Module      om_tcp
#     Host        192.168.1.1
#     Port        514
#     Exec        to_syslog_snare();
# </Output>
# 
# Connect input 'in' to output 'out'
# <Route 1>
#     Path        in => out
# </Route>

<Input in_dns>
   Module      im_etw
   Provider    Microsoft-Windows-DNSServer
</Input>

<Input dns_analytical_log>
   Module      im_msvistalog
   Query       <QueryList>\
                   <Query Id="0">\
                       <Select Path="Microsoft-Windows-DNS-Server/Analytical">*</Select>\
                   </Query>\
               </QueryList>
</Input>

<Output out_to_splunk>
   Module      om_http
   URL         http://192.168.1.85:8088/services/collector
   ContentType application/json
   Exec        to_json();
   HTTPHeader  Authorization: 6aad1862-c232-4613-a248-bc58f0885ea8
</Output>

<Route 1>
   Path        dns_analytical_log => out_to_splunk
</Route>