I have a setup using NXlog instances as collectors in a large number of security zones. <Input in0> Module im_tcp Host XXX.XXX.XXX.XXX </Input> but for some reason this does not capture logs coming in on port 514 from Fortinet units; all other logs (from Windows and Linux servers) are received and processed just fine. tcpdump -nvvA host [Fortinet unit IP] shows log traffic coming in on the NIC from the given IP address. What am I missing?

Hello!I am working on integrating NXLog with Sumo Logic. I followed the instructions on how to integrate with Sumo Logic in the setup documentation, but the logs are not being forwarded to Sumo Logic correctly, even though I receive the INFO successfully connected message. I believe it may be related to the token. My input is from an SSL connection (which is successfully connecting, I tested by diverting it to a file which I'm currently using as an intermediary until I figure this out). I have tried both options below.    Module         om_ssl   Host              syslog.collection.us1.sumologic.com   Port               6514   CAFile           %CERTDIR%\digicert_ca.crt   Exec             to_syslog_ietf();    Exec             $raw_event =~ s/(\[.*])//g; \                        $raw_event = replace($raw_event, \                        '{', '[%SUMO_TOKEN%] {', 1); Module           om_ssl   Host                syslog.collection.us1.sumologic.com   Port                 6514   CAFile             %CERTDIR%\digicert_ca.crt   Exec                to_syslog_ietf();    Exec                $raw_event =~ s/(\[NXLOG@14506.*?\])//g; \                           $raw_event = replace($raw_event, \                           '{', '[%SUMO_TOKEN%] {', 1);

Hello everyone,New to NXlog, doing the trial now and on setup I have encountered an issue with setting up the mail server and stuck there… It is a ubuntu server (AWS EC2) with postfix installed that relays to google.Postfix works as when trying: echo "This is the body of the email" | mail -s "This is the subject line" user@ourdomain.com -r nxlog@ourdomain.com - I get the email, not an issue.When I go to the mail server config of the web UI I always get “Failed to send email”, also the same when I try to change the email or password. I see nothing in the mail log so it seems it is not even trying to send the email?The settings I try are:Host: 127.0.0.1Port: 25Sender: nxlog@ourdomain.comUsername/Password: (empty) - as we don't auth for the local interface, this is just a local relay Any advice on how to proceed since I'm stuck on this step… :( Thank you!

With pm_null being marked for deprecation, does anyone know what the suggested alternative would be for creating modular configurations that use multiple processors?e.ginput => p1 => p2 => outputWhere p1 and p2 are pm_null modules that contain execs to perform some transformation on the event?

Hi something wrong with my API key? copy and paste to the config.  run the command below and get this error. Please help Thankserror:source ./master.cnf.sh && CALLED_FROM_MAKEFILE=1 ./scripts/00_check_api_key.sh[2024-11-17 17:39:21] [ERROR] Your NXLog Platform API key is invalid: MDE5MzI2MjktZGJmMy03ZmY2LThiZTMtM2Q3MDkxZjBmOTQzOm5ISlJCSVdpR1orR1RnZEUzaHUzenJHUVd2T2xBYlpHQTVGUUhLcVBuQmM9. Please double check your input.[2024-11-17 17:39:21] [ERROR] Status code returned by cloud platform.beta.nxlog.company: 401[2024-11-17 17:39:21] [INFO] You have 2 attempt(s) left to input a valid NXLog PLatform API keyEnter your NXLog PLatform API key: make: *** [Makefile:209: check-api-key] Error 1root@nxlog:/home/ubuntu# sudo nxp_manage.sh wizard  

Hello! I’m looking through the steps to “Enabling HTTPS for NXLog Manager” using a CA signed cert. The documentation provided is rather limited (https://docs.nxlog.co/manager/current/installation/https.html)Any additional information you can share would be appreciated.Thanks!

I'm trying to configure an Exclude in im_file. I want to exclude logs with a date and number at the end of the filename. These files are already processed.Config:<Input PPS>  Module im_file  InputType multi_PPS  File '%LOGDIR%\\*.log'  # PPSPortaalManager_24-05-24_2.log  Exclude '*_??-??-??_*.log'  #. I've tried several notations, eg. double quotes instead of single, escaping the questionmarks. But none of them work. Searching for examples leads to 1 of 2 examples, which are very common examples, and none of them with wildcards.Using nxlog-ce-3.2.2329 .Has anyone a good, and working, example how to user Exclude in im_file?

I've been testing sending logs directly to Sentinel and am having a problem with NXLOG not liking the CA. The error is not one I'm finding a lot of online help with.“no certificate or crl found” appears in the the log file, repeatedly.Maybe an issue with the CA I'm pointing to? Is there a specific one for Azure Sentinel that I'm overlooking, and if so, where is that obtained?Thanks in advance.--B

 Hello there i try to forward logs radius to my elastic siem , i got this error when executing nxlog.exe -f  :2024-10-07 11:16:37 INFO nxlog-ce-3.2.2329 started2024-10-07 11:16:37 INFO reconnecting in 1 seconds2024-10-07 11:16:37 ERROR apr_sockaddr_info failed for 172.19.14.51  # IP du serveur distant:514; Unknow Host. Here is the config (without ip for privacy) : <Extension _syslog>   Module      xm_syslog</Extension><Extension xml>   Module      xm_xml</Extension><Input radius_log>        Module      im_file      File        "D:\NPS\NPS Logs\IN2410.log"     SavePos     TRUE       ReadFromLast TRUE       PollInterval 1       Exec        parse_xml("/Event"); </Input><Output remote_syslog>        Module      om_udp   Host         # IP SRV FORWARD        Port        514</Output><Route radius_to_remote>     Path        radius_log => remote_syslog</Route> Is it possible to get everything in the source folder not just one ? Because the name change everymonth , example : october  :IN2410.logNovember: IN2411.log December: IN2412.log   etc Please  Thnaks a lot

Hi Team,Need help!We're unable to ingest the logs from the file ‘output.json,’ which is 150MB in size. But i am able to send the logs from file ‘output.json’ which is 10MB in size to the forwarder.Below is the NXLog Community Edition Config file. Please help me is there anything to modify the Config file.define ROOT C:\Program Files\nxlogdefine ADCONTEXT_OUTPUT_DESTINATION_ADDRESS <Forwarder IP>define ADCONTEXT_OUTPUT_DESTINATION_PORT <Port>Moduledir   %ROOT%\modulesCacheDir    %ROOT%\dataPidfile         %ROOT%\data\nxlog.pidSpoolDir     %ROOT%\dataLogFile       %ROOT%\data\nxlog.log<Extension json>    Module  xm_json</Extension><Input in_adcontext>   Module    im_file   File         "C:\AD\output.json"   DirCheckInterval 3600   PollInterval 3600</Input><Output out_chronicle_adcontext>   Module    om_tcp   Host       %ADCONTEXT_OUTPUT_DESTINATION_ADDRESS%   Port        %ADCONTEXT_OUTPUT_DESTINATION_PORT%</Output><Route ad_context_to_chronicle>    Path in_adcontext => out_chronicle_adcontext</Route>

Have installed nx onprem on ubuntu 22.04.5 LTS and cant login after the install finishes. Not sure how many pods are suppose to run but I see a postgress and vault 1c9df1fc6f5d  nxlogacr.azurecr.io/vault:1.13.3                                3 hours ago  Up 2 hours ago (healthy)              nxlog-1_2_2-vault-1eef5bec91376  nxlogacr.azurecr.io/postgres:16.3-alpine  -c config_file=/e...  3 hours ago  Up 2 hours ago (healthy)              nxlog-1_2_2-postgres-1 Below is what shows listening and I don't see any web services. dp    UNCONN  0       0                                      10.89.0.1:53          0.0.0.0:*      users:(("dnsmasq",pid=1291,fd=4))          udp    UNCONN  0       0                                  127.0.0.53%lo:53          0.0.0.0:*      users:(("systemd-resolve",pid=627,fd=13))  udp    UNCONN  0       0        [fe80::d433:a2ff:fe02:9e0d]%cni-podman1:53             [::]:*      users:(("dnsmasq",pid=1291,fd=10))         tcp    LISTEN  0       32                                     10.89.0.1:53          0.0.0.0:*      users:(("dnsmasq",pid=1291,fd=5))          tcp    LISTEN  0       4096                               127.0.0.53%lo:53          0.0.0.0:*      users:(("systemd-resolve",pid=627,fd=14))  tcp    LISTEN  0       128                                      0.0.0.0:22          0.0.0.0:*      users:(("sshd",pid=750,fd=3))              tcp    LISTEN  0       32       [fe80::d433:a2ff:fe02:9e0d]%cni-podman1:53             [::]:*      users:(("dnsmasq",pid=1291,fd=11))         tcp    LISTEN  0       128                                         [::]:22             [::]:*      users:(("sshd"I see the following error in nxp.logHA Mode                 standbyActive Node Address     <none>Raft Committed Index    31Raft Applied Index      31Error authenticating: error looking up token: Error making API request.URL: GET http://0.0.0.0:8200/v1/auth/token/lookup-selfCode: 500. Errors:I got the following during the install[2024-09-25 00:28:55] [INFO] Vault container ID: 62f8bd5e1e00[2024-09-25 00:28:55] [INFO] Executing command (/init/bootstrap.sh) in container 62f8bd5e1e00...make: *** [Makefile:231: seed-vault] Error 2

Hello,I'm attempting to install the NXLog Platform on-prem on a Ubuntu 24.04 LTS VM, but I am running into the following error on the nxlog-1_2_2-vault-1 container:fetch https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/APKINDEX.tar.gz fetch https://dl-cdn.alpinelinux.org/alpine/v3.18/community/x86_64/APKINDEX.tar.gz WARNING: updating and opening https://dl-cdn.alpinelinux.org/alpine/v3.18/main: temporary error (try again later) WARNING: updating and opening https://dl-cdn.alpinelinux.org/alpine/v3.18/community: temporary error (try again later) ERROR: unable to select packages: supervisor (no such package): required by: world[supervisor] fetch https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/APKINDEX.tar.gz fetch https://dl-cdn.alpinelinux.org/alpine/v3.18/community/x86_64/APKINDEX.tar.gz WARNING: updating and opening https://dl-cdn.alpinelinux.org/alpine/v3.18/main: temporary error (try again later) WARNING: updating and opening https://dl-cdn.alpinelinux.org/alpine/v3.18/community: temporary error (try again later) ERROR: unable to select packages: supervisor (no such package): required by: world[supervisor] fetch https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/APKINDEX.tar.gz When accessing this link from my host machine, I am able to download the .tar.gz, so it does not seem to be a network issue. Additionally, from the VM I am able to reach the internet perfectly fine to, e.g., run updates and I can cURL the URL from the VM as well.Any assistance on this would be appreciated!Thanks

Hi, seems there is a bug in im_msvistalog module in nxlog-ce-3.0.2272 for Windows. After service has generated some output, stopping service sometimes give the message box "Broken pipe" and sometimes Nxlog write a malformed configcache.dat and following the start posts an ERROR failed to restore the saved position from bookmark xml (error:15008)" Tested on Windows 2012 R2 and Windows 2019 <Output out> Module om_file File '%LOGDIR%\Output' </Output> <Input _im_eventlog> Module im_msvistalog SavePos TRUE <QueryXML> <QueryList> <Query Id='1'> <Select Path='Security'>*</Select> </Query> </QueryList> </QueryXML> # Query <QueryList>\ # <Query Id='0'>\ # <Select Path="Security">*</Select>\ # </Query>\ # </QueryList> </Input> #<Output outNull> # Module om_null #</Output> <Route RouteA> Path _im_eventlog => out </Route> This issue make the module not worth to use because not saving the bookmark of the processed registry record, force it to generate same logs from start over each service restart. Best.

HiHere is my configuration. However, only Sysmon events are not working because they contain "/operational: " at the beginning of the message, which causes the events to be parsed incorrectlySo I want to know how to remove "/operational:  "<Extension syslog>   Module          xm_syslog</Extension><Input in>   Module          im_msvistalog   ReadFromLast True    <QueryXML>            <QueryList>  <Query Id="0">   <Select Path="Application">*</Select>   <Select Path="System">*</Select>   <Select Path="Security">*</Select>   <Select Path="Windows PowerShell">*</Select>   <Select Path="Microsoft-Windows-Sysmon/Operational">*</Select>  </Query> </QueryList> </QueryXML>   Exec          $UnixTime = integer($EventTime)/1000; </Input><Output devo_relay> Module        om_tcp Host          192.168.29.133Port          13000 I tried Exec if ($SourceName =~ /Microsoft-Windows-Sysmon\/Operational/) { $Message = replace($Message, "/operational: ", ""); }and Exec    if ($Message =~ /\/operational: /) {                     $Message = replace($Message, "/operational: ", "");                 }But all did not work

I'm successfully using this config format with im_file to read logs with todays date in the filename:'\\server.domain\Logs\IN' + strftime(now(), "%y%m%d") + '.log'One of our services writes its log for the previous day at 3am on the next day. The filename has yesterdays date. What's the easiest/neatest/most efficient way of reading this log please?

Hi folks,We have an NXLog CE 3.2.2329 configuration using im_tcp with InputType LineBased, receiving blobs of JSON separated by newlines. The received data is passed to xm_perl and then relayed to an output.Under some conditions where there may be thousands of events per second, there is a very rare chance for one event to be truncated at the end yielding an incomplete json blob.  We've verified that the input data coming from upstream is valid, fully formed JSON - the emitter encoding logs look OK. The preceeding and following lines are also received and parsed successfully. Has anybody else experienced similar behavior? At this point we suspect a bug in NXLog, considering that it occurs very infrequently (2-3 days between recurrences) and it only occurs during periods of relatively high load (normal load is ~150 events per second, load when issue occurs is ~800/s+)Relevant configuration: <Extension gelf>Module      xm_gelf</Extension><Extension perl>Module      xm_perlPerlCode    /etc/nxlog/scripts/parse_mist.pl</Extension><Input mist_tcp>Module      im_tcpHost        0.0.0.0Port        32768InputType   LineBased<Exec>perl_call("parse_mist");if $nxlog_internal_drop_log drop();</Exec></Input><Output graylog>Module      om_udpHost        xPort        12201OutputType  GELF</Output><Route graylog>Path       mist_tcp => graylog</Route>

I've successfully installed the NXLog platform on Ubuntu 22.04.4 LTS, but I'm facing an issue with port 443 not being accessible. I’ve allowed port 443 through the firewall, but I still can't access the web console. I’ve checked the service status, and everything seems fine. Is there any configuration or additional steps I might have missed to get port 443 up and running?Any help would be appreciated. Thanks in advance!@Ayodele@2Emeka Nwankwo 

I have setup with RHEL 7.9 (kernel  3.10.0-1160.108.1.el7.x86_64) / NXLog CE 3.2.2329 / Postgresql 15 vanilla / libdbi 8.4 with configuration:User nxlog Group nxlog include /etc/nxlog/nxlog.d/*.conf LogFile /var/log/nxlog/nxlog.log LogLevel DEBUG <Extension charconv> Module xm_charconv </Extension> <Input sqlite3_dbi> Module im_dbi SQL SELECT * FROM vw_tp_message Driver sqlite3 Option dbname term_refdata.db Option sqlite3_dbdir /app/data/monitoring Option sqlite3_timeout 1000 PollInterval 5 SavePos TRUE </Input> <Output postgres_dbi> Module om_dbi SQL INSERT INTO msglog.msglog (facility, severity, hostname, timestamp, application, message) VALUES ($SyslogFacility, $SyslogSeverity, $Hostname, '$EventTime', $SourceName, $Message) Driver pgsql Option host /var/run/postgresql Option username srv_nxlog Option dbname MSGLog Exec $Message = convert($Message, "windows-1251", "utf-8"); </Output> <Route sqlite_postgres> Path sqlite3_dbi => postgres_dbi </Route>There are only 2 records in sqlite database:$ sqlite3 -header /app/data/monitoring/term_refdata.db 'select * from vw_tp_message' id|SyslogFacility|SyslogSeverity|Hostname|EventTime|SourceName|Message 1293441|USER|INFO|SERVER1-D1-CL|2024-09-12 00:18:22.540|tp_msg|Some text in Windows1251 encoding 1293442|USER|INFO|SERVER1-D1-CL|2024-09-12 00:41:04.677|tp_msg|Another text in Windows1251 encodingAnd this is what I get in nxlog.log:... 2024-09-12 16:54:59 DEBUG logdata missing or undef 'EventTime', setting to NULL 2024-09-12 16:54:59 DEBUG om_dbi SQL: INSERT INTO msglog.msglog (facility, severity, hostname, timestamp, application, message) VALUES ('USER', 'INFO', 'SERVER1-D1-CL', 'NULL', 'tp_msg', 'Some text in Windows1251 encoding') 2024-09-12 16:54:59 ERROR [om_dbi.c:256/om_dbi_write()] -;[om_dbi.c:85/om_dbi_error()] om_dbi failed to execute SQL statement "INSERT INTO msglog.msglog (facility, severity, hostname, timestamp, application, message) VALUES ('USER', 'INFO', 'SERVER1-D1-CL', 'NULL', 'tp_msg', 'Some text in Windows1251 encoding')". ERROR: invalid input syntax for type timestamp: "NULL";LINE 1: ... VALUES ('USER', 'INFO', 'SERVER1-D1-CL', 'NULL', 't...; ^;[errorcode: 0] ... 2024-09-12 16:54:59 DEBUG logdata missing or undef 'EventTime', setting to NULL 2024-09-12 16:54:59 DEBUG om_dbi SQL: INSERT INTO msglog.msglog (facility, severity, hostname, timestamp, application, message) VALUES ('USER', 'INFO', 'SERVER1-D1-CL', 'NULL', 'tp_msg', 'Р?нициализация СЏРґСЂР° после перезапуска') 2024-09-12 16:54:59 ERROR [om_dbi.c:256/om_dbi_write()] -;[om_dbi.c:85/om_dbi_error()] om_dbi failed to execute SQL statement "INSERT INTO msglog.msglog (facility, severity, hostname, timestamp, application, message) VALUES ('USER', 'INFO', 'SERVER1-D1-CL', 'NULL', 'tp_msg', 'Р?нициализация СЏРґСЂР° после перезапуска')". ERROR: invalid input syntax for type timestamp: "NULL";LINE 1: ... VALUES ('USER', 'INFO', 'ABACUS-D1-CL', 'NULL', 't...; ^;[errorcode: 0] ... 2024-09-12 16:54:59 ERROR [expr.c:189/nx_expr_statement_execute()] assignment failed at line 61, character 70 in /etc/nxlog/nxlog.conf. statement execution has been aborted;[expr.c:90/nx_expr_statement_assignment_execute()] -;[expr.c:509/nx_expr_evaluate()] function 'convert' failed at line 61, character 69 in /etc/nxlog/nxlog.conf. expression evaluation has been aborted;[expr.c:279/nx_expr_eval_func()] -;[xm_charconv_funcproc_cb.c:283/nx_expr_func__convert()] -;[str.c:106/_nx_string_new_size()] oversized string (1688710), limit is 1048576 bytes 2024-09-12 16:54:59 DEBUG logdata missing or undef 'EventTime', setting to NULL 2024-09-12 16:54:59 DEBUG om_dbi SQL: INSERT INTO msglog.msglog (facility, severity, hostname, timestamp, application, message) VALUES ('USER', 'INFO', 'SERVER1-D1-CL', 'NULL', 'tp_msg', 'Р В Р’В РВ<C2> 2024-09-12 16:54:59 ERROR [om_dbi.c:256/om_dbi_write()] -;[om_dbi.c:85/om_dbi_error()] om_dbi failed to execute SQL statement "INSERT INTO msglog.msglog (facility, severity, hostname, timestamp, application, message) VALUES ('USER', 'INFO', 'SERVER1-D1-CL', 'NULL', 'tp_msg', 'Р В Р’ВРР<E2><80> ...As you can see, logdata variable $EventTime just disappear and $Message get corrupted after some retries (note: om_dbi retries crazy fast - about 50000 time every few seconds). If I change om_dbi to simple om_file then all works fine. Another note: this configuration worked fine some time since initial setup, but now error appear every time (maybe postgresql query time affects it?)  

Please give me some advice. When I used nxlog to receive DHCP server logs, I found that the logs were missing and the nxlog logs showed error messages. INFO inode changed for 'E:\DHCP\DhcpSrvLog-Fri.log' (56→56): reopening possibly rotated file  

We currently have NXLog running on Windows 2019 with the file integrity module that monitors files in sub directories under a main directly.  I.E.  Say we have about 20 subdirectories for files under a directory called c:\code. If any file is changed/deleted/added under that directory, an alert message is sent out via the OM_UDP module to our siem. The problem I am looking to resolve, is that I will be copying over about 10 new directories with hundreds of files in each directory, and I don't want NXLog to generate hundreds of alerts because it found new files and directories.  Is there a way to “reset” or “refresh” the baseline after the new files/folders are copied over so it will know those files should be there?