Syslog message not getting written to file

I set up a new port (2514/TCP) to receive logs messages from VMware Workspace One. I notice this new port was not writing the log messages to the file specified. (Other ports for other products work fine) I can see the messages come in when I run tcpdump, but they do not end up getting written to a file. When I looked at the /var/log/nxlog/nxlog.log file, I noticed a LOT of messages like this: ```2021-07-07 22:44:31 INFO connection accepted from 10.42.32.149:60599``` ```2021-07-07 22:44:31 WARNING TCP connection closed from 10.42.32.149:60599: End of file found``` I tried to google the error, but could not fine anything about it. My config file includes this: ``` Module xm_syslog Module im_tcp Port 2514 Host 0.0.0.0 Exec parse_syslog(); Module om_file File "/log/SYSLOG_vmwarelog/VMWARELOG_" + $Hostname + ".log" CreateDir TRUE Exec to_syslog_bsd(); Path vmwarelogtcp => vmwareout ``` Any ideas? Thank you

horstp created 4 years ago

Replies: 1

View post »
last updated 4 years ago

Credintials in pliantext (odbc,wmi)

Hello,
Is it possible to use password hash in config files instead plaintext while connecting to odbc

Module              om_odbc
    ConnectionString    Driver={ODBC Driver 13 for SQL Server}; Server=MSSQL-HOST; UID=test; PWD=testpass; Database=TESTDB`

or windows

Module im_msvistalog
    RemoteUser nxlog
    RemotePassword nxlogpass1234

RAZR created 4 years ago

Replies: 1

View post »
last updated 4 years ago

password | plaintext

Fortigate Logs to CSV

Ok so im trying to export the logs from our FortiGate to a CSV file. Thats actually allready working fine. But the problem is, lots of information is stored in the $Message. So what i want to do is to get all "variables" in the $Message in seperate fields. Im having a really hard time with this right now. Ive never really done something with regex and nxlog. I´d be really happy if you guys could help me out here.

Thats how nxlog writes an event into the csv file: 2021-06-28 00:00:05;"INFO";2;"XXX.XXX.XXX.XXX";;"date=2021-06-28,time=00:00:05,devname="XXXX",devid="XXX",logid="0000000013",type="traffic",subtype="forward",level="notice",vd="root",eventtime=1624831205715391871,tz="+0200",srcip=XXX.XXX.XXX.XXX,srcport=33084,srcintf="port1",srcintfrole="lan",dstip=XXX.XXX.XXX.XXX,dstport=80,dstintf="wan1",dstintfrole="wan",sessionid=24018243,proto=6,action="close",policyid=3,policytype="policy",poluuid="7f09e0e6-c026-51ea-ccf3-27ba9a95d742",service="HTTP",dstcountry="France",srccountry="Reserved",trandisp="snat",transip=XXX.XXX.XXX.XXX,transport=33084,appid=16648,app="Kaspersky.Update",appcat="Update",apprisk="low",applist="Std-Appcontrol",duration=5,sentbyte=836,rcvdbyte=1036,sentpkt=6,rcvdpkt=4,shapingpolicyid=7,shapersentname="A1_Outgoing",shaperdropsentbyte=0,shaperrcvdname="Incoming",shaperdroprcvdbyte=0,vwlid=0,utmaction="allow",countapp=1 mastersrcmac="XXX",srcmac="XXX",srcserver=0"

I guess replacing all the ',' with ';' would work. But i have no Idea how to do that. If you need more infos im going to send them asap.

Mrkasali created 4 years ago

Replies: 1

View post »
last updated 4 years ago

mx_multiline - Seems to still only target 1 line

Good Morning,

Hoping someone can help me find out where I have gone wrong here. I'm trying to get these xml files through nxlog using multiline. I'm targeting each section between <row></row> tags each as an event but nxlog seems to still be grabbing individual lines instead (checked via log_info and what comes up in logz.io).

config: define ROOT C:\Program Files (x86)\nxlog define ROOT_STRING C:\Program Files (x86)\nxlog define CERTDIR %ROOT%\cert Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data LogFile %ROOT%\data\nxlog.log <Extension charconv> Module xm_charconv AutodetectCharsets utf-8, euc-jp, utf-16, utf-32, iso8859-2 </Extension> <Extension xml> Module xm_xml </Extension> <Extension json> Module xm_json </Extension> <Extension multiline> Module xm_multiline HeaderLine /<row>/
</Extension> <Input OBLogs> Module im_file SavePos TRUE #ReadFromLast False File "D:\SERVER1\LogFiles\Hyland\error-profile\error-profile*.xml" InputType multiline Exec log_info("Reading log file: [" + file_name() + "]"); #Exec if $raw_event !~ /^<row>/ drop(); Exec parse_xml(); #Exec log_info("ParsedxmlRaw: [" + $raw_event + "]"); Exec $Hostname = hostname();
Exec to_json(); Exec log_info("Sending: [" + $raw_event + "]"); Exec $raw_event = '[Redact][type=onbase-xml]' + $raw_event; </Input> <Output out> Module om_tcp Host listener-wa.logz.io Port 8010 </Output> <Route OB> Path OBLogs => out </Route>

Sample log file: <?xml version="1.0" encoding="utf-8"?> <diagnosticsLog type="error-profile" startDate="06/26/2021 08:04:55">  <columns> <column friendlyName="time" name="time" /> <column friendlyName="Result" name="Result" /> <column friendlyName="Module" name="Module" /> <column friendlyName="Class" name="Class" /> <column friendlyName="SourceFile" name="SourceFile" /> <column friendlyName="Method" name="Method" /> <column friendlyName="SourceLine" name="SourceLine" /> <column friendlyName="Severity" name="Severity" /> <column friendlyName="MachineName" name="MachineName" /> <column friendlyName="IpAddress" name="IpAddress" /> <column friendlyName="ErrorId" name="ErrorId" /> <column friendlyName="ProcessID" name="ProcessID" /> <column friendlyName="ThreadID" name="ThreadID" /> <column friendlyName="TimeSpan" name="TimeSpan" /> <column friendlyName="User" name="User" /> <column friendlyName="HTTPSessionID" name="HTTPSessionID" /> <column friendlyName="HTTPForward" name="HTTPForward" /> <column friendlyName="SessionID" name="SessionID" /> <column friendlyName="SessionGUID" name="SessionGUID" /> <column friendlyName="Datasource" name="Datasource" /> <column friendlyName="Sequence" name="Sequence" /> <column friendlyName="LocalSequence" name="LocalSequence" /> <column friendlyName="Message" name="Message" /> <column friendlyName="AppPoolName" name="AppPoolName" /> </columns> <rows> <row> <col name="time">06/26/2021 08:04:55</col> <col name="TimeSpan">N/A</col> <col name="ThreadID">0x0000000B</col> <col name="User">UNITYSCHEDSERVICE</col> <col name="HTTPSessionID"></col> <col name="HTTPForward"></col> <col name="SessionGUID">cb5ce31f-641e-4eba-aaab-906691d26dec</col> <col name="SessionID">251185112</col> <col name="Datasource">Prod</col> <col name="AppPoolName"></col> <col name="IpAddress"></col> <col name="MachineName"></col> <col name="Result">0xFFFFFFFF80131500</col> <col name="Message">'Doc - Create PDF/TIFF File for This Document' was not executed because the document with ID: (123) has invalid extension 'rtf'</col> <col name="Module">Hyland.Core.Workflow.Tasks</col> <col name="Class">Hyland.Core.Workflow.Tasks.Foundation.Actions.CreatePdfTiffFile.ExecutionService</col> <col name="Method">ExecuteImpl</col> <col name="SourceFile"></col> <col name="SourceLine">0</col> <col name="Severity">Error</col> <col name="ErrorId">48c99a2c-facf-4a4e-948e-5f933ad2eae2</col> </row> <row> <col name="time">06/26/2021 08:04:55</col> <col name="TimeSpan">N/A</col> <col name="ThreadID">0x0000000B</col> <col name="User">UNITYSCHEDSERVICE</col> <col name="HTTPSessionID"></col> <col name="HTTPForward"></col> <col name="SessionGUID">cb5ce31f-641e-4eba-aaab-906691d26dec</col> <col name="SessionID">251185112</col> <col name="Datasource">Prod</col> <col name="AppPoolName"></col> <col name="IpAddress"></col> <col name="MachineName"></col> <col name="Result">0xFFFFFFFF80131500</col> <col name="Message"></col> <col name="Module">Hyland.Core.Workflow.Tasks</col> <col name="Class">Hyland.Core.Workflow.Tasks.ExecutionServiceBase</col> <col name="Method">Execute</col> <col name="SourceFile"></col> <col name="SourceLine">0</col> <col name="Severity">Error</col> <col name="ErrorId">48c99a2c-facf-4a4e-948e-5f933ad2eae2</col> </row> <row> <col name="time">06/26/2021 08:04:55</col> <col name="TimeSpan">N/A</col> <col name="ThreadID">0x0000000B</col> <col name="User">UNITYSCHEDSERVICE</col> <col name="HTTPSessionID"></col> <col name="HTTPForward"></col> <col name="SessionGUID">cb5ce31f-641e-4eba-aaab-906691d26dec</col> <col name="SessionID">251185112</col> <col name="Datasource">Prod</col> <col name="AppPoolName"></col> <col name="IpAddress"></col> <col name="MachineName"></col> <col name="Result">0xFFFFFFFF80131500</col> <col name="Message"></col> <col name="Module">Hyland.Core.Workflow</col> <col name="Class">Hyland.Core.Workflow.Action</col> <col name="Method">Execute</col> <col name="SourceFile"></col> <col name="SourceLine">0</col> <col name="Severity">Error</col> <col name="ErrorId">48c99a2c-facf-4a4e-948e-5f933ad2eae2</col> </row> </diagnosticsLog>

Thank you so much for any help you can offer!

IS-CS created 4 years ago

Nxlog gets a TCP timeout, then the next line it sends is garbage

I am using nxlog to send logs to another nxlog server set up as a concentrator. The TCP connection is going via an Amazon Load Balancer. The nxlog sender gets tcp connection error if it hasn't sent for a while. When it goes to send another message, it reconnects ok. The problem is that the line it sends ends up appearing as garbage on the concentrator.

It's not really an option to remove the load balancer, we need to have the concentrator server autorestart if it crashes so this is the easiest way.

Any ideas?

wallet created 4 years ago

Replies: 1

View post »
last updated 4 years ago

NXlog CE stops sending logs

Hello,

I've been using the Community edition of NXlog, I've read there is a Bug related to the 256 windows channel error, has there been a fix or does anybody have a workaround for this. The agent on some of my machines stop work after a couple days. Any Advice would be appreciated.

user1 created 4 years ago

NXlog CE stops sending logs

Hello,

I've been using the Community edition of NXlog, I've read there is a Bug related to the 256 windows channel error, has there been a fix or does anybody have a workaround for this. The agent on some of my machines stop work after a couple days. Any Advice would be appreciated.

user1 created 4 years ago

Replies: 1

View post »
last updated 4 years ago

NXLog EE Trial Limitations

Hello.

We are testing the NXLog EE Trial version on Windows and want to know what are its limitations?

Will be expire after some time?

Are some modules not working?

How many days can we try it?

Is there a FAQ to explain further the Trial limitations? because we did not find any.

Thank you.

Moulos created 4 years ago

Replies: 1

View post »
last updated 4 years ago

trial limitations

NX .conf - Drop Windows events based on hostname

Hello everyone

I have the following EXEC IF statement in my configuration file to drop events if username fields are equal to the computer account name. As you know Windows computer account names always end in $. host1$ host2$ etc.

if $EventID == 4624 AND ($TargetUserName == 'DESKTOP-XY43$' OR $SubjectUserName == 'DESKTOP-XY43$') drop();

Above IF statement works perfectly, however, I have several other IF statements for various event ID's AND several hundred Windows hosts. You can imagine the time and effort required to customize several hundred .conf files for each Windows desktop.

Does anyone know if I could use REGEX or some other technique like wildcard to simply say if TargetUserName or SubjectUserName string ends in $, then drop the event.

IF $EventID == 4624 AND ($TargetUserName == *$ OR $SubjectUserName == *$) drop();

Thank you

Dingofest2 created 4 years ago

Replies: 1

View post »
last updated 4 years ago

'parse_csv' failed - statement execution has been aborted; Invalid CSV input:

Hello Team,

I have configured nxlog.conf for .csv file in my production server.

.csv file content :

nxlog Conf file content for csv part :

<Extension csv> Module xm_csv Fields date1, date2, mailid, name, result Delimiter | #EscapeControl TRUE </Extension>

<Input Security_processedlog> Module im_file Exec $type = 'Security_processedlog'; File 'D:\RPAMain\Logs\General\Security\ProcessedLog*.csv' InputType LineBased
PollInterval 1 SavePos TRUE Exec csv->parse_csv(); Exec $Message = $raw_event; </Input>

I can see only the output for this entry

(10/5/2021 9:00:45 AM) | 05/10/2021 08:04:08 | lflee@moh.gov.my | FW: JBS/021886/19 | Wrong or No request form

Nxlog log says error for rest entries

2021-06-11 19:44:39 ERROR procedure 'parse_csv' failed at line 376, character 22 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; Invalid CSV input: '(20/5/2021 5:20:19 PM) | 05/20/2021 16:21:12 | "Analisis Risikan Maklumat Telefon JSJN Bukit Aman" <telconaco@rmp.gov.my> | LMAF/JSJN/56/2021 | Success'

2021-06-11 19:44:39 ERROR procedure 'parse_csv' failed at line 376, character 22 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; Invalid CSV input: '(20/5/2021 6:07:09 PM) | 05/20/2021 17:21:40 | "SI Mohamad Azizi B Yunus" <aziziyunus@rmp.gov.my> | KUANTAN REPOT 7090/21, SEK 302 KK | Success'

Please suggest

Thanks

Raji created 4 years ago

WARNING Module ##### has no input files to read

Hello Team,

I have added nxlog.conf for our windows application server.

See the nxlog reference manual at

http://nxlog.org/nxlog-docs/en/nxlog-reference-manual.html

Please set the ROOT to the folder your nxlog was installed into,

otherwise it will not start.

#define ROOT C:\Program Files\nxlog define ROOT C:\Program Files (x86)\nxlog define CERT %ROOT%\cert

Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data LogFile %ROOT%\data\nxlog.log

Include fileop while debugging, also enable in the output module below

#<Extension fileop>

Module xm_fileop

#</Extension>

<Extension json> Module xm_json </Extension>

<Extension syslog> Module xm_syslog </Extension>

<Extension csv> Module xm_csv Fields date1, date2, mailid, name, result Delimiter | #EscapeControl TRUE </Extension>

<Input internal> Module im_internal </Input>

Watch your own files

<Input file1> Module im_file File '%ROOT%\data\nxlog.log' SavePos TRUE Exec $Message = $raw_event; </Input>

<Input COPNewMNPAR_general> Module im_file Exec $type = 'COPNewMNPAR_general'; File 'D:\RPAMain\Logs\General\COPNewMNPAR\General*.txt' SavePos TRUE Exec $Message = $raw_event; </Input>

<Input COPNewMNPAR_approcessed> Module im_file Exec $type = 'COPNewMNPAR_approcessed'; File 'D:\RPAMain\Logs\General\COPNewMNPAR\AppsProcessed*.txt' SavePos TRUE Exec $Message = $raw_event; </Input>

<Input COPNewMNPAR_processing> Module im_file Exec $type = 'COPNewMNPAR_processing'; File 'D:\RPAMain\Logs\General\COPNewMNPAR\AppsProcessing*.txt' SavePos TRUE Exec $Message = $raw_event; </Input>

<Input COPNewMNPAR_exception> Module im_file Exec $type = 'COPNewMNPAR_exception'; File 'D:\RPAMain\Logs\Exception\COPNewMNPAR\Exception*.txt' SavePos TRUE Exec $Message = $raw_event; </Input>

<Input igniteserver> Module im_file Exec $type = 'igniteserver'; File 'C:\ProgramData\AutomationAnywhere\Logs\IgniteServer*.txt' SavePos TRUE Exec $Message = $raw_event; </Input>

<Input HR Process1_security> Module im_file Exec $type = 'HR Process1_security'; File 'D:\RPAMain\Logs\General\HR Process1\Security\Security*.txt' SavePos TRUE Exec $Message = $raw_event; </Input>

<Input HR Process1_general> Module im_file Exec $type = 'HR Process1_general'; File 'D:\RPAMain\Logs\General\HR Process1\Genaral*.txt' SavePos TRUE Exec $Message = $raw_event; </Input>

<Input HR Process1_exception> Module im_file Exec $type = 'HR Process1_exception'; File 'D:\RPAMain\Logs\Exception\HR Process1\Exception*.txt' SavePos TRUE Exec $Message = $raw_event; </Input>

<Input HR Process2_general> Module im_file Exec $type = 'HR Process2_general'; File 'D:\RPAMain\Logs\General\HR Process2\Genaral*.txt' SavePos TRUE Exec $Message = $raw_event; </Input>

<Input HR Process2_exception> Module im_file Exec $type = 'HR Process2_exception'; File 'D:\RPAMain\Logs\Exception\HR Process2\Exception*.txt' SavePos TRUE Exec $Message = $raw_event; </Input>

<Input Bot1_general> Module im_file Exec $type = 'Bot1_general'; File 'D:\RPAMain\Logs\General\Bot1\General*.txt' SavePos TRUE Exec $Message = $raw_event; </Input>

<Input Bot1_exception> Module im_file Exec $type = 'Bot1_exception'; File 'D:\RPAMain\Logs\Exception\Bot1\Exception*.txt' SavePos TRUE Exec $Message = $raw_event; </Input>

<Input Bot1_approcessed> Module im_file Exec $type = 'Bot1_approcessed'; File 'D:\RPAMain\Logs\App Process\Bot1\AppProcessed*.txt' SavePos TRUE Exec $Message = $raw_event; </Input>

<Input Bot1_processing> Module im_file Exec $type = 'Bot1_processing'; File 'D:\RPAMain\Logs\AppProcess\Bot1\AppProcessing*.txt' SavePos TRUE Exec $Message = $raw_event; </Input>

<Input Bot1_security> Module im_file Exec $type = 'Bot1_security'; File 'D:\RPAMain\Logs\Security\Bot1\Security*.txt' SavePos TRUE Exec $Message = $raw_event; </Input>

<Input Bot3_general> Module im_file Exec $type = 'Bot3_general'; File 'D:\RPAMain\Logs\General\Bot3\General*.txt' SavePos TRUE Exec $Message = $raw_event; </Input>

<Input Bot3_exception> Module im_file Exec $type = 'Bot3_exception'; File 'D:\RPAMain\Logs\Exception\Bot3\Exception*.txt' SavePos TRUE Exec $Message = $raw_event; </Input>

<Input Bot3_approcessed> Module im_file Exec $type = 'Bot3_approcessed'; File 'D:\RPAMain\Logs\App Process\Bot3\AppProcessed*.txt' SavePos TRUE Exec $Message = $raw_event; </Input>

<Input Bot3_processing> Module im_file Exec $type = 'Bot3_processing'; File 'D:\RPAMain\Logs\AppProcess\Bot3\AppProcessing*.txt' SavePos TRUE Exec $Message = $raw_event; </Input>

<Input Bot3_security> Module im_file Exec $type = 'Bot3_security'; File 'D:\RPAMain\Logs\Security\Bot3\Security*.txt' SavePos TRUE Exec $Message = $raw_event; </Input>

<Input MNPSAC_general> Module im_file Exec $type = 'MNPSAC_general'; File 'D:\RPAMain\Logs\General\MNPSAC\General*.txt' SavePos TRUE Exec $Message = $raw_event; </Input>

<Input MNPSAC_exception> Module im_file Exec $type = 'MNPSAC_exception'; File 'D:\RPAMain\Logs\Exception\MNPSAC\Exception*.txt' SavePos TRUE Exec $Message = $raw_event; </Input>

<Input CPARefund_general> Module im_file Exec $type = 'CPARefund_general'; File 'D:\RPAMain\Logs\General\CPARefund\General*.txt' SavePos TRUE Exec $Message = $raw_event; </Input>

<Input CPARefund_exception> Module im_file Exec $type = 'CPARefund_exception'; File 'D:\RPAMain\Logs\Exception\CPARefund\Exception*.txt' SavePos TRUE Exec $Message = $raw_event; </Input>

<Input CPARefund_processed> Module im_file Exec $type = 'CPARefund_processed'; File 'D:\RPAMain\Logs\General\CPARefund\processed*.txt' SavePos TRUE Exec $Message = $raw_event; </Input>

<Input CPARefund_processing> Module im_file Exec $type = 'CPARefund_processing'; File 'D:\RPAMain\Logs\General\CPARefund\processing*.txt' SavePos TRUE Exec $Message = $raw_event; </Input>

<Input CPA_general> Module im_file Exec $type = 'CPA_general'; File 'D:\RPAMain\Logs\General\CPA\General*.txt' SavePos TRUE Exec $Message = $raw_event; </Input>

<Input CPA_exception> Module im_file Exec $type = 'CPA_exception'; File 'D:\RPAMain\Logs\Exception\CPA\Exception*.txt' SavePos TRUE Exec $Message = $raw_event; </Input>

<Input CPA_processed> Module im_file Exec $type = 'CPA_processed'; File 'D:\RPAMain\Logs\RefundMasterList\ProcessedLog*.txt' SavePos TRUE Exec $Message = $raw_event; </Input>

<Input Bot2_general> Module im_file Exec $type = 'Bot2_general'; File 'D:\RPAMain\Logs\General\Bot2\General*.txt' SavePos TRUE Exec $Message = $raw_event; </Input>

<Input Bot2_exception> Module im_file Exec $type = 'Bot2_exception'; File 'D:\RPAMain\Logs\Exception\Bot2\Exception*.txt' SavePos TRUE Exec $Message = $raw_event; </Input>

<Input Bot2_approcessed> Module im_file Exec $type = 'Bot2_approcessed'; File 'D:\RPAMain\Logs\App Process\Bot2\AppProcessed*.txt' SavePos TRUE Exec $Message = $raw_event; </Input>

<Input Bot2_processing> Module im_file Exec $type = 'Bot2_processing'; File 'D:\RPAMain\Logs\AppProcess\Bot2\AppProcessing*.txt' SavePos TRUE Exec $Message = $raw_event; </Input>

<Input Bot2_security> Module im_file Exec $type = 'Bot1_security'; File 'D:\RPAMain\Logs\Security\Bot2\Security*.txt' SavePos TRUE Exec $Message = $raw_event; </Input>

<Input Security_general> Module im_file Exec $type = 'Security_general'; File 'D:\RPAMain\Logs\General\Security\General*.txt' SavePos TRUE Exec $Message = $raw_event; </Input>

<Input Security_processedlog> Module im_file Exec $type = 'Security_processedlog'; File 'D:\RPAMain\Logs\General\Security\ProcessedLog*.csv' InputType LineBased
PollInterval 1 SavePos TRUE Exec csv->parse_csv(); Exec $Message = $raw_event; </Input>

Windows Event Log

Uncomment im_msvistalog for Windows Vista/2008 and later

Module im_msvistalog

Uncomment im_mseventlog for Windows XP/2000/2003

Module im_mseventlog

</Input>

<Output out> Module om_tcp Host 172.31.244.219 Port 3515

Exec  $tmpmessage = $Message; delete($Message); rename_field(&quot;tmpmessage&quot;,&quot;message&quot;);
Exec  $raw_event = to_json();

# Uncomment for debug output
# Exec file_write('%ROOT%\data\nxlog_output.log', $raw_event + &quot;\n&quot;);

</Output>

<Route 1> Path internal, file1, eventlog, COPNewMNPAR_general, OPNewMNPAR_approcessed, COPNewMNPAR_processing, COPNewMNPAR_exception, igniteserver, HR Process1_security, HR Process1_general, HR Process1_exception, HR Process2_general, HR Process2_exception, Bot1_general, Bot1_exception, Bot1_approcessed, Bot1_processing, Bot1_security, Bot3_general, Bot3_exception, Bot3_approcessed, Bot3_processing, Bot3_security, MNPSAC_general, MNPSAC_exception, CPARefund_general, CPARefund_exception, CPARefund_processed, CPARefund_processing, CPA_general, CPA_exception, CPA_processed, Bot2_general, Bot2_exception, Bot2_approcessed, Bot2_processing, Bot2_security, Security_general, Security_processedlog => out </Route>

I see below warning for all application log files , nxlog is not able to read the logs from application log files. Please suggest.

021-06-11 13:22:59 WARNING Module Bot1_general has no input files to read Module Bot1_exception has no input files to read Module Bot1_security has no input files to read

I also see this log , nxlog is making an attempt but not able to read. These application log files are written every minute.

An attempt was made to access an object.

Subject: Security ID: S-1-5-18 Account Name: SHTVRPACTRLP01$ Account Domain: DIGICR Logon ID: 0x3E7

Object: Object Server: Security Object Type: File Object Name: D:\RPAMain\Logs\Exception\Bot3 Handle ID: 0xcd4 Resource Attributes:

Process Information: Process ID: 0x66ec Process Name: C:\Program Files (x86)\nxlog\nxlog.exe

Access Request Information: Accesses: ReadData (or ListDirectory)

Access Mask:		0x1

Thanks

Raji created 4 years ago

Linux to Alien Vault formatting

I'm relatively new to NXLog and to Alien Vault Log ingestion. I have followed their set up guide here, https://cybersecurity.att.com/documentation/usm-anywhere/deployment-guide/setup/linux-logs-nxlog.htm?Highlight=linux%20logs%20nxlog.

We are using NxLog EE and the nxlog manager to push configs.

I have two modules I'm looking to collect logs with and I feel that they are pretty straight forward. I'm trying to capture /var/log/messages and the audit.log. Also wanting to capture the FIM.

Using the im_file module. I'm able to capture these logs and they get to AlienVault, they are even parsed in Json which looks nice. My issue is that you can really report on anything as they don't generate any useful flags in which Alien Vault can use to trigger alarms and such. A lot of the logs get flagged as Alien Vault Generic Results, which means that the format isn't triggering their Alien Vault Data Source plugins. Some logs are getting recognized by Alien Vault and triggering the appropriate data source, but they are also not getting any useful information to report on. When I compare these logs to the Windows logs that we are capturing, the Linux logs have significantly less metadata within them. Something as simple as eventoutcome would be nice, like if I wanted to see failed attempts to elevate to sudo and see that event fail and generate and alert.

I'm wondering if anyone would mind sharing their module configs for Linux if you use Alien Vault (ATT USM) as your SIEM.

Here is my current config I am using.

LogLevel INFO Logfile %LOGDIR%/nxlog.log

<Extension agent_managment> Module xm_soapadmin Connect 1.1.1.1 Port 4041 SocketType SSL CAFile %CERTDIR%/agent-ca.pem AllowUntrusted FALSE RequireCert TRUE <ACL conf> Directory %CONFDIR% AllowRead TRUE AllowWrite TRUE </ACL> <ACL cert> Directory %CERTDIR% AllowRead TRUE AllowWrite TRUE </ACL> </Extension>

<Extension json> Module xm_json </Extension>

<Extension syslog> Module xm_syslog </Extension>

<Input var_messages_in> Module im_file File '/var/log/messages' InputType LineBased SavePos TRUE Exec parse_syslog(); </Input>

<Input fim_linux> Module im_fim File '/bin/' File '/etc/' File '/lib/' File '/opt/nxlog/bin/' File '/opt/nxlog/lib/' File '/sbin/' File '/usr/bin/' File '/usr/sbin/' Exclude '/etc/hots.deny' Exclude '/etc/mtab' </Input>

<Input var_audit_in> Module im_file File '/var/log/audit/audit.log' InputType LineBased SavePos FALSE Exec parse_syslog();
$Hostname = hostname();
$FQDN = hostname_fqdn();
$Tag = "audit";
$SourceName = "selinux";
$Message = $Raw_Event; </Input>

<Output out_syslog_ssl_br> Module om_ssl Host 1.1.1.1 Port 6514 OutputType LineBased CAFile %CERTDIR%/agent-ca.pem CertFile %CERTDIR%/agent-cert.pem CertKeyFile %CERTDIR%/agent-key.pem Exec $EventTime = $EventReceivedTime; Exec $EventReceivedTime = integer($EventReceivedTime) / 1000000; Exec $Message = to_json(); to_syslog_bsd(); </Output>

<Output out_fim_linux_ssl_BR> Module om_ssl Host 1.1.1.1 Port 6514 OutputType Binary CAFile %CERTDIR%/agent-ca.pem CertFile %CERTDIR%/agent-cert.pem CertKeyFile %CERTDIR%/agent-key.pem Exec $SourceName = "FIM-LINUX-NXLOG-EE"; Exec $EventTime = strftime($EventTime, '%Y-%m-%d %H:%M:%S'); Exec $EventReceivedTime = strftime($EventReceivedTime, '%Y-%m-%d %H:%M:%S'); Exec to_json(); to_syslog_bsd(); </Output> <Route route_syslog> Priority 1 Path var_audit_in, var_messages_in => out_syslog_ssl_br </Route> <Route FIM> Priority 2 Path fim_linux => out_fim_linux_ssl_BR </Route>

rwebster created 4 years ago

Replies: 1

View post »
last updated 4 years ago

LINUX

Log forwarding to Cloud VM

Hello Everyone,
I am trying to forward Windows Server logs to Azure VM. We have a config for on premises forwarding. I have tried to make changes in config according to what i found in documentation.

New Config: (WIth Changes)
When @weekly
Exec if file_exists('%MYLOGFILE%') file_cycle('%MYLOGFILE%', 8);
</Schedule>
</Extension>

<Extension json>
Module xm_json
</Extension>

<Extension resolver>
Module xm_resolver
</Extension>

########################INPUTS##########################
<Input eventlog>
Module im_msvistalog
# ReadFromLast and SavePos control when we read data.
# False is good for troubleshooting, but not likely what you want in Prod
#ReadFromLast False
#SavePos False
TolerateQueryErrors True
<QueryXML>
<QueryList>
<Query Id="0">
<Select Path="Microsoft-Windows-Sysmon/Operational">*</Select>
<Select Path="Security">*</Select>
</Query>
</QueryList>
</QueryXML>
<Exec>
# This `if` is just so I only see 4662 events
#if $EventID != 4662 drop();
# These fields are set, but you will need to add the values to your output at some point.
# This is done automatically in Syslog IETF and JSON, but not Snare
$TypeR = ad_guid_to_name($ObjectType);
$NameR = ad_guid_to_name($ObjectName);
# These lines just log the values to the internal log file. Not needed for production, only troubleshooting.
log_info($raw_event);
log_info("TypeR: " + $TypeR);
log_info("NameR: " + $NameR);
</Exec>
</Input>
########################OUTPUTS##########################
<Output out>
Module om_udp
Host 52.165.172.76
Port 518
Exec to_syslog_bsd();
########################PUTS EVENT IN IETF FORMAT########
Exec to_syslog_snare();
</Output>
<Route 1>
Path eventlog => syslogout
</Route>

On-premises config
Panic Soft
#NoFreeOnExit TRUE

define ROOT C:\Program Files (x86)\nxlog
define CERTDIR %ROOT%\cert
define CONFDIR %ROOT%\conf
define LOGDIR %ROOT%\data
define LOGFILE %LOGDIR%\nxlog.log
LogFile %LOGFILE%

Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data

<Extension _syslog>
Module xm_syslog
</Extension>

<Extension _resolver>
Module xm_resolver
</Extension>

# This block rotates `%MYLOGFILE%` on a schedule. Note that if `LogFile`
# is changed in log4ensics.conf via NXLog Manager, rotation of the new
# file should also be configured there.
<Extension _fileop>
Module xm_fileop

# Check the size of our log file hourly, rotate if larger than 5MB
<Schedule>
Every 1 hour
<Exec>
if ( file_exists('%MYLOGFILE%') and (file_size('%MYLOGFILE%') >= 5M) )
{
file_cycle('%MYLOGFILE%', 8);
}
</Exec>
</Schedule>

# Rotate our log file every week on Sunday at midnight
<Schedule>
When @weekly
Exec if file_exists('%MYLOGFILE%') file_cycle('%MYLOGFILE%', 8);
</Schedule>
</Extension>

<Extension json>
Module xm_json
</Extension>

<Extension resolver>
Module xm_resolver
</Extension>

########################INPUTS##########################
<Input eventlog>
Module im_msvistalog
# ReadFromLast and SavePos control when we read data.
# False is good for troubleshooting, but not likely what you want in Prod
#ReadFromLast False
#SavePos False
TolerateQueryErrors True
<QueryXML>
<QueryList>
<Query Id="0">
<Select Path="Microsoft-Windows-Sysmon/Operational">*</Select>
<Select Path="Security">*</Select>
</Query>
</QueryList>
</QueryXML>
<Exec>
# This `if` is just so I only see 4662 events
#if $EventID != 4662 drop();
# These fields are set, but you will need to add the values to your output at some point.
# This is done automatically in Syslog IETF and JSON, but not Snare
$TypeR = ad_guid_to_name($ObjectType);
$NameR = ad_guid_to_name($ObjectName);
# These lines just log the values to the internal log file. Not needed for production, only troubleshooting.
log_info($raw_event);
log_info("TypeR: " + $TypeR);
log_info("NameR: " + $NameR);
</Exec>
</Input>
########################OUTPUTS##########################
<Output syslogout>
Module om_udp
#Module om_tcp
Host 10.0.0.129
Port 515
########################PUTS EVENT IN IETF FORMAT########
Exec to_syslog_snare();
</Output>
<Route 1>
Path eventlog => syslogout
</Route>
</Output>

If anyone can provide help it will be helpfull

Thank you.

hkatlia created 4 years ago

Can NXLOG module im_dbi use AD Service account to authenticate to DB read?

I am trying to read from a MSSQL DB with an AD service account. I know the im_dbi module supports local DB account for auth but not sure about how to leverage AD service account credentials.

bane created 4 years ago

nxlog im_dbi MSSQL

is it possible to negate a match using pm_match? (nxlog ce version for now)

I am trying to us pm_pattern to filter messages that are approved for a 'limited' feed, and still have the full feed go to the admin feed. I have set up the config to send the input to two feeds, that works fine. I've then tried to use pm_pattern to match certain strings and DROP them from the restricted feed. So far no luck. I'm sure I'm missing something really simple here, and would really appreciate if anyone had the time to check the configs for me... in nxlog.conf (edited for brevity)... Module pm_pattern PatternFile "/data/conf/nxlog-patternmatch.yaml" Module om_http URL https:// destination string #Batchmode none HTTPSAllowUntrusted TRUE HTTPSCADir /etc/ssl/certs/ HTTPSCertFile /etc/ssl/certs/ca-certificates.crt Exec if defined $PatternID drop(); Path client2001 => client_filter => out2001,client-test and this is the patternmatch yaml (some matchfields removed)... 2021-06-03 01:02:03 1 Client 1 client input raw_event regexp rsyslogd raw_event regexp NetworkManager raw_event regexp Systemd raw_event regexp dnf raw_event regexp dbus raw_event regexp chrony raw_event regexp CRON raw_event regexp motd raw_event regexp snapd raw_event regexp promtail raw_event regexp nxlog raw_event regexp kernel raw_event regexp loki

wallet created 4 years ago

nxlog | pattern match | pm_match

.xlsx extension file NXLog configuration

Hello Team,

I have a .xlsx file , I need to add NXLog configuration to send this .xlsx file contents to my Nagios Log server. can we send .xlsx file ? please help with this extension module and input configuration.

Thanks

Raji created 4 years ago

Replies: 1

View post »
last updated 4 years ago

Using to_syslog_snare() but with ISO8601 date format (need timezone attached to time)

Hello. I would like to use the "to_syslog_snare()" procedure but with the use of ISO8601 timeformat so that the date and time would be formatted as "2021-05-28T07:35:49+00:00" instead of "May 28 07:35:49". How would I achieve this? Thank you!

heikis created 4 years ago

Replies: 1

View post »
last updated 4 years ago

snare | ISO8601 | timezone

100% CPU Usage in Windows Server 2016

Any suggestions on where to look for more information? Server Manager shows performance alerts and has nxlog using 100% CPU. This happens several times per day. Running nxlog-ce-2.10.2150 on Windows Server 2016 Standard.

robvas created 4 years ago

Replies: 2

View post »
last updated 4 years ago

A certain windows event log has not been sent

Hi, everyone. I haven’t overcome the problem above. Could anyone please share idea of ; -The possible methods of determining the root cause of the problem -The possible methods of overcoming this problem As soon as you can, please! Thank you.

<The problem> When sending event logs from NXlog, a certain event log has never been sent to the windows log collection server. E.g. Event ID: 4624（Successful Logon）->Has not been sent Event ID: 4634（Logout）->have been sent

<The methods already tested > 1.Debugging; The following debug log was configured in order to test that target event log (ID: 4624) was recognized by NXlog.

Exec if ($EventID == 4624) log_info("EventID = 4624 | " + $EventID + " | " + $EventTime);

The test shows that the event log was recognized by NXlog, as it was output to NXlog as follows.

2021-05-14 19:22:17 INFO EventID = 4624 | 4624 | 2021-05-14 19:22:17

Explicit output of the target event log (ID4624); The test shows that the expected event log has not been sent, though the following event logs were sent after specified event logs explicitly.

　　#In Windows Event Log (Event ID：4624 or 4625) 　　<Input In_eventlog_logon> 　　Module im_msvistalog Exec if ($EventID == 5156) drop(); Exec if ($EventID == 4624) log_info("EventID = 4624 | " + $EventID + " | " + $EventTime + " | " + $Hostname); 　 <QueryXML> <QueryList> <Query Id='0'> <Select Path="Security">*[System[(EventID=4624 or EventID=4625)]]</Select> </Query> </QueryList> </QueryXML> </Input>

AyakoFukumoto created 4 years ago

Replies: 1

View post »
last updated 4 years ago

Windows event log service and nxlog service dependancies

Hi everyone,

I would like to know if some standard installation of nxlog on a AD could lead to the nxlog service becoming dependant of the windows event log service ?

It not supposed to be, but for some reason (that I didn't understand yet) it happens in my case.

Thanks.

Regards

unda.verse created 4 years ago

Replies: 1

View post »
last updated 4 years ago

Windows event log service 1100 nxlog service dependancy dependancies