Hi,
we've configured im_maculs and have noticed, that it does not handle expected ULS logs (which are seen with log stream command).
We then configured im_exec module, to run log stream and have compared configurations head-to-head, the input with im_exec receives expected logs, while im_maculs does not.
Here is configuration:
<Input m_uls>
Module im_maculs
<Exec>
# Filter
if ($subsystem == 'com.apple.launchservices' and $category == 'open')
{
$Hostname = hostname();
} else
{
drop();
}
to_json();
</Exec>
</Input>
<Input m_logstream>
Module im_exec
Command /usr/bin/log
Arg stream
Arg --style=ndjson
Arg --type=log
<Exec>
if $raw_event =~ /^{/
{
# Filter
if ($subsystem == 'com.apple.launchservices' and $category == 'open')
{
$Hostname = hostname();
} else
{
drop();
}
to_json();
} else
{
# Fix ERROR [im_exec|m_logstream] failed to parse json string, lexical error: invalid char in json text.; Filtering the log data using "t; (right here) ------^; [Filtering the log data using "type == 1024"]
# Since first log stream output line is not a json log entry, but informational message
drop();
}
</Exec>
</Input>
The m_logstream Input produces log message every time a graphical application is openned in macOS, while the m_uls - does not.
simtom created
Hello,
I'm deploying nxlog-ce docker container in order to collect logs from several servers.
My container is running and stores logs in files. I would like store logs in a postgresql database but the om_dbi module is missing in the container.
How can I add this module?
b.aucher created
Hi folks,
Is NXLog CE compatible with Windows Server 2003? I am getting “The installation is not supported by this processor type” error. Works fine on other OS's.
Regards
Ben
ben.patrick created
Is anyone else having problems trying to download NXLog CE? I select the file I want to download and click the “Download” button, and it just hangs.
John Shaw created
Hello, i used option AllowUntrusted TRUE with the ssl output module but i have still error ssl verification failed
ERROR SSL certificate verification failed: unable to get local issuer certificate
this option is not supposed to avoid this error ?
https://docs.nxlog.co/ce/current/index.html#om_ssl
My output conf:
<Output ssl>
Module om_ssl
Host mysyslogserverPort 514
AllowUntrusted TRUE
OutputType Syslog_TLS
Exec to_syslog_ietf();CAFile
</Output>
regards
Guillaume
Guillaume Morin created
hello. I have nxlog working oh so well sending Windows Events to Graylog. Works perfectly, couldnt be happier.
I however wanted to start sending some logs that an application creates. Seems to be configured properly to send. I can see the in the message section of graylog the lines of the log and they come into graylog as they're created. However the line gets cut off after 64 characters per each line. How can I get the full line of the log?
jmaics created
Hi,
we are using nxlog-ce-3.0.2272 on Linux (CentOS 7), after a server reboot nxlog is not started. The error message in nxlog logfile is: ERROR: couldn't open pidfile /run/nxlog/nxlog.pid.
After the reboot of the server the directory /run/nxlog is missing, which seems to cause the error.
The directory is created when nxlog-ce is installed on the server and nxlog is started OK.
If I manually create the directory /run/nxlog and then start nxlog it also works.
Is this a known error or have missed something ?
BR Joakim
joakim created
Hi
We are not receiving the Windows security logs via the nxlog agent. We noticed that nxlog.log successfully connected the destination IP, and port.
2023-04-12 08:18:57 INFO [CORE|main] nxlog-5.7.7898 (68bb24e7e@REL_v5.7) started on Windows2023-04-12 08:18:57 INFO [om_udp|syslogout] connecting to 10.129.5.20:5212023-04-12 08:18:57 INFO [om_udp|syslogout] successfully connected to 10.129.5.20:521
Nxlog. conf file configuration
## Please set the ROOT to the folder your nxlog was installed into,## otherwise it will not start.define ROOT C:\Program Files (x86)\nxlog
define CERTDIR %ROOT%\certdefine CONFDIR %ROOT%\confdefine LOGDIR %ROOT%\datadefine LOGFILE %LOGDIR%\nxlog.logLogFile %LOGFILE%Moduledir %ROOT%\modulesCacheDir %ROOT%\dataPidfile %ROOT%\data\nxlog.pidSpoolDir %ROOT%\data<Extension syslog>Module xm_syslog</Extension><Input eventlog>Module im_msvistalog# ReadFromLast True<QueryXML><QueryList><Query Id="0"><Select Path="ForwardedEvents">*</Select></Query></QueryList></QueryXML></Input><Processor eventlog_transformer>Module pm_transformer# OutputFormat syslog_rfc5424</Processor><Processor buffer>Module pm_buffer# 100 MB disk bufferMaxSize 102400Type disk</Processor>########################OUTPUTS##########################<Output syslogout>Module om_udpHost 10.129.5.20:521########################PUTS EVENT IN IETF FORMAT############Exec to_syslog_snare();</Output><Route 1>Path eventlog => eventlog_transformer => syslogout</Route>
Note: We are using the nxlog agent version 5.7.7898 and security events are available in Forwarded events
jilin created
I used a configuration right out of the documentation. This error is logged at startup. I have to remove all fields referenced in curly braces ${}
2023-03-31 13:45:13 ERROR Couldn't parse Exec block at C:\Program Files\nxlog\conf\nxlog.conf:60; couldn't parse statement at line 62, character 13 in C:\Program Files\nxlog\conf\nxlog.conf; invalid character: '$' (0x24)
Config:
# Convert integer type fields if (${Framed-MTU}) ${Framed-MTU} = integer(${Framed-MTU});
jmorrison created
Hi,
I had this situation whereby when I unplugged the cable of the outgoing interface of the nxlog server, the logs went into the cache folder. When I connected back to the interface, the new logs continues to flow to the external logger, but the cached logs remain inside the cache folder and do not forward out to the external logger.
below is my configuration. Looking forward to any help, please. thank you
User rootGroup rootPanic Soft##modify change# default values:# PidFile /opt/nxlog/var/run/nxlog/nxlog.pid# CacheDir /opt/nxlog/var/spool/nxlog# ModuleDir /opt/nxlog/lib/nxlog/modules# SpoolDir /opt/nxlog/var/spool/nxlogCacheDir /data/nxlogcache
define CERTDIR /opt/nxlog/var/lib/nxlog/certdefine CONFDIR /opt/nxlog/var/lib/nxlog
# Note that these two lines define constants only; the log file location# is ultimately set by the `LogFile` directive (see below). The# `MYLOGFILE` define is also used to rotate the log file automatically# (see the `_fileop` block).define LOGDIR /opt/nxlog/var/log/nxlogdefine MYLOGFILE %LOGDIR%/nxlog.log
# By default, `LogFile %MYLOGFILE%` is set in log4ensics.conf. This# allows the log file location to be modified via NXLog Manager. If you# are not using NXLog Manager, you can instead set `LogFile` below and# disable the `include` line.LogFile %MYLOGFILE%#include %CONFDIR%/log4ensics.conf
<Extension _syslog> Module xm_syslog</Extension>
<Extension exec> Module xm_exec</Extension>
<Extension _leef>Module xm_leef</Extension>
<Output Egress_To_xxx> Module om_tcp Port 1524 Host xxx.xxx.xxx.xxx</Output>
# Set port and Ip Address to listen to traffic<Input External_Ingress> Module im_tcp Port 1524 Host 0.0.0.0</Input><Output External_Log_Locally> Module om_file File '/data/store/External/External_logs.txt'
#Set log rotation to run daily and keep 90 days worth of logs <Schedule> when @daily <Exec> # Create year/month directories if necessary dir_make('/data/store/External/' + strftime(now() - 86400, '%Y-%m'));
# Rotate current file into the correct directory rotate_to('/data/store/External/' + strftime(now() - 86400, '%Y-%m/External_logs_%Y-%m-%d.txt'));
#Remove files older than 90 days exec_async("/usr/bin/find", "/data/store/External", "-mtime", "+90", "-type", "f", "-delete"); </Exec> </Schedule>
</Output>
<Processor External_To_xxx_Buffer> Module pm_buffer Type Disk #130MiB buffer MaxSize 130000000 WarnLimit 100000000</Processor>
<Route External_To_xxx_Path> Path External_Ingress => External_To_xxx_Buffer => Egress_To_xxx</Route>
<Route External_To_Local> Path External_Ingress => External_Log_Locally</Route>
<Input Internal_Ingress> Module im_tcp Port 2524 Host 0.0.0.0
<Exec> # edit syslog header for xxxx if $raw_event =~ /\d\d:\d\d:\d\d\s+(xxxx)\s+/ { $Hostname = $1; to_syslog_bsd(); }
</Exec>
</Input>
<Output Internal_Log_Locally> Module om_file File '/data/store/Internal/Internal_logs.txt'
#Set log rotation to run daily and keep 90 days worth of logs <Schedule> when @daily <Exec> # Create year/month directories if necessary dir_make('/data/store/Internal/' + strftime(now() - 86400, '%Y-%m'));
# Rotate current file into the correct directory rotate_to('/data/store/Internal/' + strftime(now() - 86400, '%Y-%m/Internal_logs_%Y-%m-%d.txt'));
#Remove files older than 90 days exec_async("/usr/bin/find", "/data/store/Internal", "-mtime", "+90", "-type", "f", "-delete");
</Exec>
</Schedule></Output>
<Processor Internal_To_xxx_Buffer> Module pm_buffer Type Disk #130MiB buffer MaxSize 130000000 WarnLimit 100000000</Processor>
<Route Internal_To_xxx_Path> Path Internal_Ingress => Internal_To_xxx_Buffer => Egress_To_xxx</Route>
<Route Internal_To_Local> Path Internal_Ingress => Internal_Log_Locally</Route>
<Input xxx_UDP_Ingress> Module im_udp Port 514 Host 0.0.0.0
<Exec> # edit syslog header for xxx else if $MessageSourceAddress == 'xx.xxx.x.x' { $Hostname = 'xxx'; to_syslog_bsd(); }
# edit syslog header for xxx else if $MessageSourceAddress == 'xx.xxx.xx.xx' { $Hostname = 'xxx'; to_syslog_bsd(); }
# edit syslog header for xxx else if $MessageSourceAddress == 'xx.xx.xx' { $Hostname = 'xxx'; to_syslog_bsd(); }
</Exec>
</Input>
<Output xxx_UDP_Log_Locally> Module om_file File '/data/store/UDP/xxx_UDP_logs.txt'
#Set log rotation to run daily and keep 90 days worth of logs <Schedule> when @daily <Exec> # Create year/month directories if necessary dir_make('/data/store/UDP/' + strftime(now() - 86400, '%Y-%m'));
# Rotate current file into the correct directory rotate_to('/data/store/UDP/' + strftime(now() - 86400, '%Y-%m/xxxx_UDP_logs_%Y-%m-%d.txt'));
#Remove files older than 90 days exec_async("/usr/bin/find", "/data/store/UDP", "-mtime", "+90", "-type", "f", "-delete");
</Exec> </Schedule></Output>
<Processor xxx_UDP_To_xxx_Buffer> Module pm_buffer Type Disk #130MiB buffer MaxSize 130000000 WarnLimit 100000000</Processor>
<Route xxx_UDP_To_xxx_Path> Path xxx_UDP_Ingress => xxx_UDP_To_xxx_Buffer => Egress_To_xxx</Route>
<Route xxx_UDP_To_Local> Path xxx_UDP_Ingress => xxx_UDP_Log_Locally</Route>
<Input audit_log> Module im_file File '/var/log/secure' <Exec> to_syslog_bsd(); </Exec></Input>
<Processor xxx_Audit_To_xxx_Buffer> Module pm_buffer Type Disk #130MiB buffer MaxSize 130000000 WarnLimit 100000000</Processor>
<Route xx_Audit_To_xx_Path> Path audit_log => xxx_Audit_To_xxx_Buffer => Egress_To_xxx</Route>
<Input audit_log_2> Module im_file File '/var/log/audit/audit.log' <Exec> to_syslog_bsd(); </Exec></Input>
<Processor xxx_Audit_2_To_xxx_Buffer> Module pm_buffer Type Disk #130MiB buffer MaxSize 130000000 WarnLimit 100000000</Processor>
<Route xxx_Audit2_To_xxxx_Path> Path audit_log_2 => xxxx_Audit_2_To_xxxx_Buffer => Egress_To_xxxx</Route>
# This block rotates `%MYLOGFILE%` on a schedule. Note that if `LogFile`# is changed in log4ensics.conf via NXLog Manager, rotation of the new# file should also be configured there.<Extension _fileop> Module xm_fileop
# Check the size of our log file hourly, rotate if larger than 5MB <Schedule> Every 1 hour <Exec> if ( file_exists('%MYLOGFILE%') and (file_size('%MYLOGFILE%') >= 5M) ) { file_cycle('%MYLOGFILE%', 8); } </Exec> </Schedule>
# Rotate our log file every week on Sunday at midnight <Schedule> When @weekly Exec if file_exists('%MYLOGFILE%') file_cycle('%MYLOGFILE%', 8); </Schedule></Extension>
billychua created
nxlog-ce-3.1.2319
- add custom conf to ` /etc/nxlog/nxlog.d`;
- systemctl restart nxlog;
- systemctl status nxlog;
- got message:
- how to enable the im_dbi module ?
- the config looks like:
config file: `/etc/nxlog/nxlog.d/icslog.conf`:
<Input dbi>
Module im_dbi
Driver mysql
Option host 127.0.0.1
Option username root
Option password pp
Option dbname logs
SQL SELECT * FROM ics_alarm_log
</Input>
<Output file>
Module om_file
File '/tmp/ics_alarm_log.csv'
</Output>yang server created
Hello,
I am trying to send my custom application Windows logs to GrayLog. I am using GELF TCP. Our event text is getting cut off. It looks like our custom events write all of the data to the section GrayLog calls “short message” I did see the article that said there is a 64 character limit and we did change that with Exec $short_message = $raw_message; in the output section. This did increase the number of characters shown but there are still plenty more characters getting cut off. We have NXlog writing to a local file right now so were able to verify that it is NXlog that is cutting off the data. Is there a way to fix this? Here is my conf file:
Panic Soft#NoFreeOnExit TRUE
define ROOT C:\Program Files\nxlogdefine CERTDIR %ROOT%\certdefine CONFDIR %ROOT%\conf\nxlog.ddefine LOGDIR %ROOT%\data
include %CONFDIR%\\*.confdefine LOGFILE %LOGDIR%\nxlog.logLogFile %LOGFILE%
Moduledir %ROOT%\modulesCacheDir %ROOT%\dataPidfile %ROOT%\data\nxlog.pidSpoolDir %ROOT%\data
<Extension _syslog> Module xm_syslog</Extension>
<Extension _charconv> Module xm_charconv AutodetectCharsets iso8859-2, utf-8, utf-16, utf-32</Extension>
<Extension _exec> Module xm_exec</Extension>
<Extension _gelf> Module xm_gelf ShortMessageLength 1024 UseNullDelimiter false</Extension>
<Extension _fileop> Module xm_fileop
# Check the size of our log file hourly, rotate if larger than 5MB <Schedule> Every 1 hour Exec if (file_exists('%LOGFILE%') and \ (file_size('%LOGFILE%') >= 5M)) \ file_cycle('%LOGFILE%', 8); </Schedule>
# Rotate our log file every week on Sunday at midnight <Schedule> When @weekly Exec if file_exists('%LOGFILE%') file_cycle('%LOGFILE%', 8); </Schedule></Extension>
<Input in> Module im_msvistalog</Input>
<Output file> Module om_file File 'c:/temp/nxlog.txt'</Output>
<Output out> Module om_tcp Host xxxxxxxxxxxxxxxxx Port 5555 OutPutType GELF_TCP Exec $short_message = $raw_message;</Output>
<Route 1> Path in => file</Route>
This is what the event data shows:
[The description for EventID 0 from source xxxxxxx cannot be found: The parameter is incorrect, ]
There are several more lines after this that are just not coming over. I am very inexperienced with this software. Thank you for your help.
alarosa created
I have the following config forwarding all events to siem
Panic Soft#NoFreeOnExit TRUE
define ROOT C:\Program Files\nxlogdefine WINDNS_OUTPUT_DESTINATION_ADDRESS 1.1.1.5define WINDNS_OUTPUT_DESTINATION_PORT 10518define CERTDIR %ROOT%\certdefine CONFDIR %ROOT%\conf\nxlog.ddefine LOGDIR %ROOT%\data
define LOGFILE %LOGDIR%\nxlog.logLogFile %LOGFILE%
Moduledir %ROOT%\modulesCacheDir %ROOT%\dataPidfile %ROOT%\data\nxlog.pidSpoolDir %ROOT%\data
<Extension _json> Module xm_json </Extension> <Input windows_security_eventlog> Module im_msvistalog <QueryXML> <QueryList> <Query Id="0"> <Select Path="Application">*</Select> <Select Path="System">*</Select> <Select Path="Security">*</Select> <Select Path="ForwardedEvents">*</Select> </Query> </QueryList> </QueryXML> ReadFromLast False SavePos False </Input>
<Output out_siem_windevents> Module om_udp Host %WINDNS_OUTPUT_DESTINATION_ADDRESS% Port %WINDNS_OUTPUT_DESTINATION_PORT% Exec $EventTime = integer($EventTime) / 1000; Exec $EventReceivedTime = integer($EventReceivedTime) / 1000; Exec to_json(); </Output> <Route r2> Path windows_security_eventlog => out_siem_windevents </Route>
I need to push sysmon events to a seperate feed within Siem, would the below be an aceptable config to use? Not sure if the route statements need route r3 or if they can all exist within route r2
define ROOT C:\Program Files\nxlogdefine WINDNS_OUTPUT_DESTINATION_ADDRESS 1.1.1.5define WINDNS_OUTPUT_DESTINATION_PORT 10518define CERTDIR %ROOT%\certdefine CONFDIR %ROOT%\conf\nxlog.ddefine LOGDIR %ROOT%\data
define WINDNS_OUTPUT_DESTINATION_ADDRESS2 1.1.1.5define WINDNS_OUTPUT_DESTINATION_PORT2 10519
define LOGFILE %LOGDIR%\nxlog.logLogFile %LOGFILE%
Moduledir %ROOT%\modulesCacheDir %ROOT%\dataPidfile %ROOT%\data\nxlog.pidSpoolDir %ROOT%\data
<Extension _json> Module xm_json</Extension><Input windows_security_eventlog> Module im_msvistalog <QueryXML> <QueryList> <Query Id="0"> <Select Path="Application">*</Select> <Select Path="System">*</Select> <Select Path="Security">*</Select> <Select Path="ForwardedEvents">*</Select> </Query> </QueryList> </QueryXML> ReadFromLast False SavePos False</Input>
<Output out_siem_windevents> Module om_udp Host %WINDNS_OUTPUT_DESTINATION_ADDRESS% Port %WINDNS_OUTPUT_DESTINATION_PORT% Exec $EventTime = integer($EventTime) / 1000; Exec $EventReceivedTime = integer($EventReceivedTime) / 1000; Exec to_json();</Output><Output out_siem_windevents2> Module om_udp Host %WINDNS_OUTPUT_DESTINATION_ADDRESS2% Port %WINDNS_OUTPUT_DESTINATION_PORT2% Exec $EventTime = integer($EventTime) / 1000; Exec $EventReceivedTime = integer($EventReceivedTime) / 1000; Exec to_json();</Output><Route r2> Path windows_security_eventlog => out_siem_windevents Path windows_security_eventlog => out_siem_windevents2</Route>
gavin.lacey@telegraph.co.uk created
Running nxlog-ce-3.1.2319 on Windows.
2023-01-19 08:12:46 ERROR Failed to load module from C:\xxxxx\nxlog\modules\extension\xm_python.dll, The specified module could not be found. ; The specified module could not be found.
The NXLog Python DLL is on disk so I am wondering if this is complaining because I don't have the nxlog Python module (which I don't see in pip).I looked around for some setup instructions but I don't see any extra setup steps required for Python (aside from writing the script).
Config:<Extension python> Module xm_python PythonCode "C:\xxx\NXLogDev\modules\convert_to_splunk_hec.py"</Extension>
PythonCode
import nxlog
def get_splunk_hec_format(event): nxlog.log_warning('in get_splunk_hec_format()')
for field in event.field_names(): nxlog.log_debug('Received field:' + field)
hukel created
Hello,
I have a problem with a nxlog collector for a SIEM Graylog. On the Graylog page the nxlog appears to be Failing. But on the collector the service looks like running :
root@:/var/run/nxlog# systemctl status nxlog
● nxlog.service - LSB: logging daemon
Loaded: loaded (/etc/init.d/nxlog; generated; vendor preset: enabled)
Active: active (running) since Tue 2021-12-21 15:33:07 CET; 1 day 19h ago
Docs: man:systemd-sysv-generator(8)
Process: 26310 ExecStop=/etc/init.d/nxlog stop (code=exited, status=0/SUCCESS)
Process: 26314 ExecStart=/etc/init.d/nxlog start (code=exited, status=0/SUCCESS)
CGroup: /system.slice/nxlog.service
└─26320 /usr/bin/nxlog
When I look into the internal logs for troubleshooting I have this :
root@:/var/run/nxlog# tail /var/log/nxlog/nxlog.log
2021-12-23 10:17:32 INFO configuration OK
2021-12-23 10:17:32 ERROR Another instance is already running (pid 26320);Resource temporarily unavailable
2021-12-23 10:17:33 ERROR Another instance is already running (pid 26320);Resource temporarily unavailable
2021-12-23 10:17:34 ERROR Another instance is already running (pid 26320);Resource temporarily unavailable
2021-12-23 10:17:35 ERROR Another instance is already running (pid 26320);Resource temporarily unavailable
2021-12-23 11:17:32 INFO configuration OK
2021-12-23 11:17:32 ERROR Another instance is already running (pid 26320);Resource temporarily unavailable
2021-12-23 11:17:33 ERROR Another instance is already running (pid 26320);Resource temporarily unavailable
2021-12-23 11:17:34 ERROR Another instance is already running (pid 26320);Resource temporarily unavailable
2021-12-23 11:17:35 ERROR Another instance is already running (pid 26320);Resource temporarily unavailable
root@:/var/run/nxlog# cat /var/run/nxlog/nxlog.pid
26320
root@BDXSVLG01:/var/run/nxlog# ps -aux |grep nxlog
root 4008 0.0 0.0 12776 980 pts/6 D+ 11:21 0:00 grep --color=auto nxlog
nxlog 26320 0.0 0.0 275248 224 ? Ssl déc.21 1:03 /usr/bin/nxlog
The service that is already running is the one with the right pid so I don't get where my problem comes from.
Thank you in advance for your help.
BR, Paul
PaulAPS created
Why do I always receive the message "WARNING nxlog-ce received a termination request signal, exiting..." and I don't receive any message using GELF UDP in Graylog input, unless I use Raw/Plaintext UDP?
Panic Soft
#NoFreeOnExit TRUE
define ROOT C:\Program Files\nxlog
define CERTDIR %ROOT%\cert
define CONFDIR %ROOT%\conf\nxlog.d
define LOGDIR %ROOT%\data
include %CONFDIR%\\*.conf
define LOGFILE %LOGDIR%\nxlog.log
LogFile %LOGFILE%
Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
# define IISLOG "C:\\inetpub\\logs\\LogFiles\\W3SVC2\\u_ex*.log"
#######################################################################
#### EXTENTIONS #####
#######################################################################
<Extension _gelf>
Module xm_gelf
</Extension>
<Extension _json>
Module xm_json
</Extension>
<Extension fileop>
Module xm_fileop
</Extension>
#######################################################################
#### IIS NXLOG ######
#######################################################################
<Extension w3c>
Module xm_csv
Fields $date, $time, $s_ip
FieldTypes string, string, string
Delimiter ' '
QuoteChar '"'
EscapeControl FALSE
UndefValue -
</Extension>
<Input iis>
Module im_file
File "C:\\inetpub\\logs\\LogFiles\\W3SVC2\\u_ex*.log"
SavePos TRUE
Exec if $raw_event =~/^#/ drop();\
else\
{\
w3c->parse_csv();\
$EventTime = parsedate($date + " " + $time);\
$EventTime = parsedate($date + " " + $time + "Z");\
$SourceName = "IIS";\
$raw_event = to_json();\
}
</Input>
<Output graylog>
Module om_udp
Host 192.168.3.250
Port 1322
OutputType GELF
Exec $Hostname = hostname_fqdn();
Exec $raw_event =$Hostname + ' IIS-NXLOG ' + $raw_event;
#Use the following line for debugging (uncomment the fileop extension above as well)
# exec file_write("C:\\Program Files\\nxlog\\data\\nxlog_output.log", $raw_event);
</Output>
<Route iis-to-graylog>
Path iis => graylog
</Route>
vic chen created
I am having issues where when I try to parse a big file ~1500 lines the regexp doesn't find any lines that match and then eventually “gives up”. There is no ERROR log in the nxlog.log saying that it essentially stopped but it never writes anything to my output file. In the input file I have 6 lines that match my parser though.
<Exec>
if $raw_event =~ /srv_name="([^"]+).+?user_auth_entr="([^"]+)/
{
$event_type = 'VPN_SESSION_IP_ASSIGNED';
$version = 'v1';
$time = 'test';
$account = $2;
$account_domain = 'null';
$assigned_ip = 'null';
$source_ip = 'null';
$authentication_result = 'FAILURE';
$authentication_target = $1;
}
</Exec>I was confused at first and thought there was an issue with my statement but it checks out.
When I specifically grabbed only those 6 log lines that would match and ran the service against ONLY those logs, everything parsed and worked just fine giving my the 6 new log lines in my output file.
So this leads me to believe that the only reason it didn't work the first time is because the first time it actually matches those log lines is the 112th line. So my assumption is that it tries to process the incoming log lines against my regex and after so many not matching it just stops.
Can anyone confirm if this is accurate and if so, how can I increase the threshold and/or remove this dependency?
jhartman created
Hello everyone,
We should log on Windows server some IIS and SQL Server logs via agent in Community Edition. Through documentation I have examples that produce as results logs in csv and/or json format.Could you give me a hand in transforming the logs from json and/or csv format to key-value (kvp)
Thank you very much for the support.
giuseppe created
I am trying to parse some logs coming in and trying to figure out the proper way to build the nxlog.conf when specifying what to do depending on the conditions met in the parser. I have a working conf file that can read all the logs from a file and parse them into one set of conditions but how do I add multiple IF statements in a single Exec block?
<Input NetMotion>
Module im_file
File "C:\Testing-logs\NetMotion.txt"
<Exec>
if $raw_event =~ /m_user="([^"]+).+?pop_ip_srv="([^"]+).+?ses_start="([^"]+).+?ses_state="([^"]+).+?vip="([^"]+)/
{
if $4 = 'Connected' $event_type = 'VPN_SESSION_IP_ASSIGNED';
{
$version = 'v1';
$time = $3;
$account = $1;
$assigned_ip = $6;
$source_ip = $2;
$authentication_result = 'SUCCESS';
$authentication_target = $5;
}
if $3 == 'Disconnected' $event_type = 'VPN_SESSION_IP_TERMINATION';
{
$version = 'v1';
$time = $2;
$account = $1;
}
}
</Exec>
</Input>The above code works in the fact that it doesn't give me any errors in the NXLog log file however the actual log lines are mixed up. If the events match 'VPN_SESSION_IP_ASSIGNED' then it all works just fine. This gives me the order of event_type, version, time, account, assigned_ip, source_ip, authentication_result, authentication_target which is exactly what I need.
However for the lines where it doesn't match, it messes up the order and puts version first, and then tacks on the event_type = 'VPN_SESSION_IP_TERMINATION' at the end.
How do I get it so that when the $event_type = 'VPN_SESSION_IP_TERMINATION' the log format only shows the fields I want e.g. event_type, version, time, account.
I tried a different method where I put the second if statement directly after the first and it worked to keep the order but I still don't know how to drop the unnecessary fields from the termination events.
<Input NetMotion>
Module im_file
File "C:\Testing-logs\NetMotion.txt"
<Exec>
if $raw_event =~ /m_user="([^"]+).+?pop_ip_srv="([^"]+).+?ses_start="([^"]+).+?ses_state="([^"]+).+?vip="([^"]+)/
{
if $4 = 'Connected' $event_type = 'VPN_SESSION_IP_ASSIGNED';
if $3 == 'Disconnected' $event_type = 'VPN_SESSION_IP_TERMINATION';
$version = 'v1';
$time = $3;
$account = $1;
$assigned_ip = $6;
$source_ip = $2;
$authentication_result = 'SUCCESS';
$authentication_target = $5;
}
</Exec>
</Input>Would I do an additional if statement after that to basically say,
if $event_type == 'VPN_SESSION_IP_TERMINATION'
{
delete($assigned_ip);
delete($source_ip);
delete($authentication_result);
delete($authentication_target);
}Any and all help is appreciated!
jhartman created
Hi all, Anybody here already using NXLog on Windows 2022? I seems to work pretty fine but I just figure that certain events just are not caught, for instance, EventID 4625, for Login failures. Despite I can see lots of occurrences on Event Viewer, they are not sent to my log server.
DaniloMussolini created
