With pm_null being marked for deprecation, does anyone know what the suggested alternative would be for creating modular configurations that use multiple processors?

e.ginput => p1 => p2 => output

Where p1 and p2 are pm_null modules that contain execs to perform some transformation on the event?

Hi something wrong with my API key? copy and paste to the config.  run the command below and get this error. Please help Thanks

error:

source ./master.cnf.sh && CALLED_FROM_MAKEFILE=1 ./scripts/00_check_api_key.sh[2024-11-17 17:39:21] [ERROR] Your NXLog Platform API key is invalid: MDE5MzI2MjktZGJmMy03ZmY2LThiZTMtM2Q3MDkxZjBmOTQzOm5ISlJCSVdpR1orR1RnZEUzaHUzenJHUVd2T2xBYlpHQTVGUUhLcVBuQmM9. Please double check your input.[2024-11-17 17:39:21] [ERROR] Status code returned by cloud platform.beta.nxlog.company: 401[2024-11-17 17:39:21] [INFO] You have 2 attempt(s) left to input a valid NXLog PLatform API keyEnter your NXLog PLatform API key: make: *** [Makefile:209: check-api-key] Error 1root@nxlog:/home/ubuntu# sudo nxp_manage.sh wizard 

Hello! 

I’m looking through the steps to “Enabling HTTPS for NXLog Manager” using a CA signed cert. The documentation provided is rather limited (https://docs.nxlog.co/manager/current/installation/https.html)

Any additional information you can share would be appreciated.Thanks!

I'm trying to configure an Exclude in im_file. I want to exclude logs with a date and number at the end of the filename. These files are already processed.Config:<Input PPS>  Module im_file  InputType multi_PPS  File '%LOGDIR%\\*.log'  # PPSPortaalManager_24-05-24_2.log  Exclude '*_??-??-??_*.log'  #. I've tried several notations, eg. double quotes instead of single, escaping the questionmarks. But none of them work. Searching for examples leads to 1 of 2 examples, which are very common examples, and none of them with wildcards.Using nxlog-ce-3.2.2329 .Has anyone a good, and working, example how to user Exclude in im_file?

I've been testing sending logs directly to Sentinel and am having a problem with NXLOG not liking the CA. The error is not one I'm finding a lot of online help with.

“no certificate or crl found” appears in the the log file, repeatedly.

Maybe an issue with the CA I'm pointing to? Is there a specific one for Azure Sentinel that I'm overlooking, and if so, where is that obtained?

Thanks in advance.

--B

 Hello there i try to forward logs radius to my elastic siem , i got this error when executing nxlog.exe -f  :2024-10-07 11:16:37 INFO nxlog-ce-3.2.2329 started2024-10-07 11:16:37 INFO reconnecting in 1 seconds2024-10-07 11:16:37 ERROR apr_sockaddr_info failed for 172.19.14.51  # IP du serveur distant:514; Unknow Host.

Here is the config (without ip for privacy) 

: <Extension _syslog>  

 Module      xm_syslog

</Extension>

<Extension xml>   Module      xm_xml

</Extension>

<Input radius_log>   

     Module      im_file 

     File        "D:\NPS\NPS Logs\IN2410.log" 

    SavePos     TRUE   

    ReadFromLast TRUE   

    PollInterval 1   

    Exec        parse_xml("/Event");

 </Input>

<Output remote_syslog>   

     Module      om_udp   Host         # IP SRV FORWARD   

     Port        514</Output>

<Route radius_to_remote>   

  Path        radius_log => remote_syslog

</Route>

Is it possible to get everything in the source folder not just one ?


 Because the name change everymonth , example : october  :IN2410.logNovember: IN2411.log December: IN2412.log   etc Please 

Thnaks a lot

Hi Team,

Need help!

We're unable to ingest the logs from the file ‘output.json,’ which is 150MB in size. But i am able to send the logs from file ‘output.json’ which is 10MB in size to the forwarder.

Below is the NXLog Community Edition Config file. Please help me is there anything to modify the Config file.

define ROOT C:\Program Files\nxlogdefine ADCONTEXT_OUTPUT_DESTINATION_ADDRESS <Forwarder IP>define ADCONTEXT_OUTPUT_DESTINATION_PORT <Port>Moduledir   %ROOT%\modulesCacheDir    %ROOT%\dataPidfile         %ROOT%\data\nxlog.pidSpoolDir     %ROOT%\dataLogFile       %ROOT%\data\nxlog.log<Extension json>    Module  xm_json</Extension><Input in_adcontext>   Module       im_file   File         "C:\AD\output.json"   DirCheckInterval 3600   PollInterval 3600</Input><Output out_chronicle_adcontext>   Module     om_tcp   Host       %ADCONTEXT_OUTPUT_DESTINATION_ADDRESS%   Port        %ADCONTEXT_OUTPUT_DESTINATION_PORT%</Output><Route ad_context_to_chronicle>    Path in_adcontext => out_chronicle_adcontext</Route>

Have installed nx onprem on ubuntu 22.04.5 LTS and cant login after the install finishes. Not sure how many pods are suppose to run but I see a postgress and vault 

1c9df1fc6f5d  nxlogacr.azurecr.io/vault:1.13.3                                3 hours ago  Up 2 hours ago (healthy)              nxlog-1_2_2-vault-1eef5bec91376  nxlogacr.azurecr.io/postgres:16.3-alpine  -c config_file=/e...  3 hours ago  Up 2 hours ago (healthy)              nxlog-1_2_2-postgres-1 

Below is what shows listening and I don't see any web services. 

dp    UNCONN  0       0                                      10.89.0.1:53          0.0.0.0:*      users:(("dnsmasq",pid=1291,fd=4))          udp    UNCONN  0       0                                  127.0.0.53%lo:53          0.0.0.0:*      users:(("systemd-resolve",pid=627,fd=13))  udp    UNCONN  0       0        [fe80::d433:a2ff:fe02:9e0d]%cni-podman1:53             [::]:*      users:(("dnsmasq",pid=1291,fd=10))         tcp    LISTEN  0       32                                     10.89.0.1:53          0.0.0.0:*      users:(("dnsmasq",pid=1291,fd=5))          tcp    LISTEN  0       4096                               127.0.0.53%lo:53          0.0.0.0:*      users:(("systemd-resolve",pid=627,fd=14))  tcp    LISTEN  0       128                                      0.0.0.0:22          0.0.0.0:*      users:(("sshd",pid=750,fd=3))              tcp    LISTEN  0       32       [fe80::d433:a2ff:fe02:9e0d]%cni-podman1:53             [::]:*      users:(("dnsmasq",pid=1291,fd=11))         tcp    LISTEN  0       128                                         [::]:22             [::]:*      users:(("sshd"

I see the following error in nxp.log

HA Mode                 standbyActive Node Address     <none>Raft Committed Index    31Raft Applied Index      31Error authenticating: error looking up token: Error making API request.

URL: GET http://0.0.0.0:8200/v1/auth/token/lookup-selfCode: 500. Errors:

I got the following during the install

[2024-09-25 00:28:55] [INFO] Vault container ID: 62f8bd5e1e00[2024-09-25 00:28:55] [INFO] Executing command (/init/bootstrap.sh) in container 62f8bd5e1e00...make: *** [Makefile:231: seed-vault] Error 2

Hello,

I'm attempting to install the NXLog Platform on-prem on a Ubuntu 24.04 LTS VM, but I am running into the following error on the nxlog-1_2_2-vault-1 container:

fetch https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.18/community/x86_64/APKINDEX.tar.gz
WARNING: updating and opening https://dl-cdn.alpinelinux.org/alpine/v3.18/main: temporary error (try again later)
WARNING: updating and opening https://dl-cdn.alpinelinux.org/alpine/v3.18/community: temporary error (try again later)
ERROR: unable to select packages:
  supervisor (no such package):
    required by: world[supervisor]
fetch https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.18/community/x86_64/APKINDEX.tar.gz
WARNING: updating and opening https://dl-cdn.alpinelinux.org/alpine/v3.18/main: temporary error (try again later)
WARNING: updating and opening https://dl-cdn.alpinelinux.org/alpine/v3.18/community: temporary error (try again later)
ERROR: unable to select packages:
  supervisor (no such package):
    required by: world[supervisor]
fetch https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/APKINDEX.tar.gz

When accessing this link from my host machine, I am able to download the .tar.gz, so it does not seem to be a network issue. Additionally, from the VM I am able to reach the internet perfectly fine to, e.g., run updates and I can cURL the URL from the VM as well.

Any assistance on this would be appreciated!Thanks

Hi

Here is my configuration. However, only Sysmon events are not working because they contain "/operational: " at the beginning of the message, which causes the events to be parsed incorrectly

So I want to know how to remove "/operational:  "

<Extension syslog>   Module          xm_syslog</Extension>

<Input in>   Module          im_msvistalog   ReadFromLast True    <QueryXML>            <QueryList>  <Query Id="0">   <Select Path="Application">*</Select>   <Select Path="System">*</Select>   <Select Path="Security">*</Select>   <Select Path="Windows PowerShell">*</Select>   <Select Path="Microsoft-Windows-Sysmon/Operational">*</Select>  </Query> </QueryList> </QueryXML>   Exec          $UnixTime = integer($EventTime)/1000; </Input>

<Output devo_relay> Module        om_tcp Host          192.168.29.133Port          13000 

I tried Exec if ($SourceName =~ /Microsoft-Windows-Sysmon\/Operational/) { $Message = replace($Message, "/operational: ", ""); }

and

 Exec    if ($Message =~ /\/operational: /) {                     $Message = replace($Message, "/operational: ", "");                 }

But all did not work

I'm successfully using this config format with im_file to read logs with todays date in the filename:

'\\server.domain\Logs\IN' + strftime(now(), "%y%m%d") + '.log'

One of our services writes its log for the previous day at 3am on the next day. The filename has yesterdays date. What's the easiest/neatest/most efficient way of reading this log please?

Hi folks,

We have an NXLog CE 3.2.2329 configuration using im_tcp with InputType LineBased, receiving blobs of JSON separated by newlines. The received data is passed to xm_perl and then relayed to an output.

Under some conditions where there may be thousands of events per second, there is a very rare chance for one event to be truncated at the end yielding an incomplete json blob.  We've verified that the input data coming from upstream is valid, fully formed JSON - the emitter encoding logs look OK. The preceeding and following lines are also received and parsed successfully. 

Has anybody else experienced similar behavior? At this point we suspect a bug in NXLog, considering that it occurs very infrequently (2-3 days between recurrences) and it only occurs during periods of relatively high load (normal load is ~150 events per second, load when issue occurs is ~800/s+)

Relevant configuration: 

<Extension gelf>

Module      xm_gelf

</Extension>

<Extension perl>

Module      xm_perl

PerlCode    /etc/nxlog/scripts/parse_mist.pl

</Extension>

<Input mist_tcp>

Module      im_tcp

Host        0.0.0.0

Port        32768

InputType   LineBased

<Exec>

perl_call("parse_mist");

if $nxlog_internal_drop_log drop();

</Exec>

</Input>

<Output graylog>

Module      om_udp

Host        x

Port        12201

OutputType  GELF

</Output>

<Route graylog>

Path       mist_tcp => graylog

</Route>

I've successfully installed the NXLog platform on Ubuntu 22.04.4 LTS, but I'm facing an issue with port 443 not being accessible. I’ve allowed port 443 through the firewall, but I still can't access the web console. I’ve checked the service status, and everything seems fine. Is there any configuration or additional steps I might have missed to get port 443 up and running?

Any help would be appreciated. Thanks in advance!@Ayodele@2Emeka Nwankwo 

I have setup with RHEL 7.9 (kernel  3.10.0-1160.108.1.el7.x86_64) / NXLog CE 3.2.2329 / Postgresql 15 vanilla / libdbi 8.4 with configuration:

User nxlog
Group nxlog

include /etc/nxlog/nxlog.d/*.conf
LogFile /var/log/nxlog/nxlog.log
LogLevel DEBUG

<Extension charconv>
    Module      xm_charconv
</Extension>

<Input sqlite3_dbi>
    Module      im_dbi
    SQL         SELECT * FROM vw_tp_message
    Driver      sqlite3
    Option      dbname term_refdata.db
    Option      sqlite3_dbdir /app/data/monitoring
    Option      sqlite3_timeout 1000
    PollInterval 5
    SavePos     TRUE
</Input>

<Output postgres_dbi>
    Module      om_dbi
    SQL         INSERT INTO msglog.msglog (facility, severity, hostname, timestamp, application, message) \
                VALUES ($SyslogFacility, $SyslogSeverity, $Hostname, '$EventTime', $SourceName, $Message)
    Driver      pgsql
    Option      host /var/run/postgresql
    Option      username srv_nxlog
    Option      dbname MSGLog
    Exec        $Message = convert($Message, "windows-1251", "utf-8");
</Output>

<Route sqlite_postgres>
     Path       sqlite3_dbi => postgres_dbi
</Route>

There are only 2 records in sqlite database:

$  sqlite3 -header /app/data/monitoring/term_refdata.db 'select * from vw_tp_message'
id|SyslogFacility|SyslogSeverity|Hostname|EventTime|SourceName|Message
1293441|USER|INFO|SERVER1-D1-CL|2024-09-12 00:18:22.540|tp_msg|Some text in Windows1251 encoding
1293442|USER|INFO|SERVER1-D1-CL|2024-09-12 00:41:04.677|tp_msg|Another text in Windows1251 encoding

And this is what I get in nxlog.log:

...
2024-09-12 16:54:59 DEBUG logdata missing or undef 'EventTime', setting to NULL
2024-09-12 16:54:59 DEBUG om_dbi SQL: INSERT INTO msglog.msglog (facility, severity, hostname, timestamp, application, message)                 VALUES ('USER', 'INFO', 'SERVER1-D1-CL', 'NULL', 'tp_msg', 'Some text in Windows1251 encoding')
2024-09-12 16:54:59 ERROR [om_dbi.c:256/om_dbi_write()] -;[om_dbi.c:85/om_dbi_error()] om_dbi failed to execute SQL statement "INSERT INTO msglog.msglog (facility, severity, hostname, timestamp, application, message)                 VALUES ('USER', 'INFO', 'SERVER1-D1-CL', 'NULL', 'tp_msg', 'Some text in Windows1251 encoding')". ERROR:  invalid input syntax for type timestamp: "NULL";LINE 1: ...          VALUES ('USER', 'INFO', 'SERVER1-D1-CL', 'NULL', 't...;                                                             ^;[errorcode: 0]
...
2024-09-12 16:54:59 DEBUG logdata missing or undef 'EventTime', setting to NULL
2024-09-12 16:54:59 DEBUG om_dbi SQL: INSERT INTO msglog.msglog (facility, severity, hostname, timestamp, application, message)                 VALUES ('USER', 'INFO', 'SERVER1-D1-CL', 'NULL', 'tp_msg', 'Р?нициализация СЏРґСЂР° после перезапуска')
2024-09-12 16:54:59 ERROR [om_dbi.c:256/om_dbi_write()] -;[om_dbi.c:85/om_dbi_error()] om_dbi failed to execute SQL statement "INSERT INTO msglog.msglog (facility, severity, hostname, timestamp, application, message)                 VALUES ('USER', 'INFO', 'SERVER1-D1-CL', 'NULL', 'tp_msg', 'Р?нициализация СЏРґСЂР° после перезапуска')". ERROR:  invalid input syntax for type timestamp: "NULL";LINE 1: ...          VALUES ('USER', 'INFO', 'ABACUS-D1-CL', 'NULL', 't...;                                                             ^;[errorcode: 0]
...
2024-09-12 16:54:59 ERROR [expr.c:189/nx_expr_statement_execute()] assignment failed at line 61, character 70 in /etc/nxlog/nxlog.conf. statement execution has been aborted;[expr.c:90/nx_expr_statement_assignment_execute()] -;[expr.c:509/nx_expr_evaluate()] function 'convert' failed at line 61, character 69 in /etc/nxlog/nxlog.conf. expression evaluation has been aborted;[expr.c:279/nx_expr_eval_func()] -;[xm_charconv_funcproc_cb.c:283/nx_expr_func__convert()] -;[str.c:106/_nx_string_new_size()] oversized string (1688710), limit is 1048576 bytes
2024-09-12 16:54:59 DEBUG logdata missing or undef 'EventTime', setting to NULL
2024-09-12 16:54:59 DEBUG om_dbi SQL: INSERT INTO msglog.msglog (facility, severity, hostname, timestamp, application, message)                 VALUES ('USER', 'INFO', 'SERVER1-D1-CL', 'NULL', 'tp_msg', 'Р В Р’В РВ<C2>
2024-09-12 16:54:59 ERROR [om_dbi.c:256/om_dbi_write()] -;[om_dbi.c:85/om_dbi_error()] om_dbi failed to execute SQL statement "INSERT INTO msglog.msglog (facility, severity, hostname, timestamp, application, message)                 VALUES ('USER', 'INFO', 'SERVER1-D1-CL', 'NULL', 'tp_msg', 'Р В Р’ВРР<E2><80>
...

As you can see, logdata variable $EventTime just disappear and $Message get corrupted after some retries (note: om_dbi retries crazy fast - about 50000 time every few seconds). If I change om_dbi to simple om_file then all works fine. Another note: this configuration worked fine some time since initial setup, but now error appear every time (maybe postgresql query time affects it?)

Please give me some advice. When I used nxlog to receive DHCP server logs, I found that the logs were missing and the nxlog logs showed error messages.

INFO inode changed for 'E:\DHCP\DhcpSrvLog-Fri.log' (56→56): reopening possibly rotated file

We currently have NXLog running on Windows 2019 with the file integrity module that monitors files in sub directories under a main directly.  I.E.  Say we have about 20 subdirectories for files under a directory called c:\code. If any file is changed/deleted/added under that directory, an alert message is sent out via the OM_UDP module to our siem. 

The problem I am looking to resolve, is that I will be copying over about 10 new directories with hundreds of files in each directory, and I don't want NXLog to generate hundreds of alerts because it found new files and directories.  Is there a way to “reset” or “refresh” the baseline after the new files/folders are copied over so it will know those files should be there?

Is there any ETA on Debian 12 support for NXLog Community Edition?

I havent been able to download the files for any of the community edition agents. https://nxlog.co/downloads/nxlog-ce#nxlog-community-edition , select the OS version and click Download.  5 sec later I get a pop up stating “Undefined”.  I have tried on different Browsers, platforms, and workstations and there is no change in the behavior.

Hi,

Could you please advise on how to replace the NXLog-Manager's self-signed certificate for the HTTPS console?

The steps in the following article do not apply to version 5.7.5:

https://docs.nxlog.co/manager/current/installation/https.html

Please help. Thanks.

Regards, Billy