I have file log, it has like 2000 lines, when i try to send it through HTTP POST, it is only receive half data. It start from the middle line. Its start from the last 4 hour updated line/ row which the 1000+ rows/line. How to setup/ config the nxlog so it will start from the first line/ the whole file. Please help me... below is my config nxlog :

<Extension multiline> Module xm_multiline FixedLineCount 32 </Extension>

<Extension multiline_action> Module xm_multiline FixedLineCount 32 </Extension>

<Input filein> Module im_file

#Wincor
File            "C:\\DataSend\\" + "data_1.log"
InputType       multiline
SavePos 		TRUE
ReadFromLast 	TRUE
CloseWhenIdle	TRUE
PollInterval 	300	

</Input>

<Output http> Module om_http URL http://localhost/log-monitor/public/index.php/receive-log #HTTPSAllowUntrusted TRUE </Output>

<Route filein_to_http> Path filein => http </Route>

Thankyou in advance

Hi all! Actually i have this config in a server to get windows event and send to a graylog: #define ROOT C:\Program Files\nxlog define ROOT C:\Program Files (x86)\nxlog Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data LogFile %ROOT%\data\nxlog.log #Module xm_syslog Module xm_gelf #Module xm_json Module im_msvistalog # Uncomment the following to collect specific event logs only Query \ \ *\ *\ *\ \ if $Channel == 'Security' and $ObjectName =~ /.tmp/ drop(); Module om_udp Host 11x.11x.11x.11x Port 12201 #Exec to_syslog_snare(); OutputType GELF Path in1 => out But i want to know if exist any method to read a lot a .evtx files in a folder and send to graylog. I need help with this because i dont know what is the best "input modules" i need Thanks!

Hi, I monitor file which is overwritten (not appended) with the same numberof logs (3 lines). I am not able to force nxlog to send this change. Example: file.txt before change: 1123 3256 2546 2342 file.txt after overwrite: 8888 3256 2461 2342 There is the same number of lines so size did not changed of file. What I have checked so far: Every 5 sec Exec log_info("scheduled execution at " + now()); module_restart(); SavePos FALSE ReadFromLast FALSE I use as a output file and tcp steam - the same result for both. Any advice how to force nxlog to read file again?

Hi I am trying to post events from my logs files to gelf_tcp (Graylog). My log records are in flat json format. I can push logs to gelf, however with few issues; I appreciate if you can help.

Issue 1: My log record has a field called "level". When the record is sent to Graylog, level does not match the one that I have in log file. I do parse_json() first and also I tried to explicitly set the value of level based on NXLog documentation https://nxlog.co/documentation/nxlog-user-guide/xm_gelf.html There is also something wrong with documentation. It says gelf understand field "SeverityLevel" but in the example in the same page it is using "SyslogSeverityLevel". I tried to explicitly set both fields with $level field but nothing changes in graylog.

Issue 2: I have timestamp filed in my log record which is ISO 8601 format. I could not find an easy way to parse it. the parsedata() function does not help there. I end up extracting date and time part from my field and then send it to parsedate($1 + " " + $2). If you know a better way, please let me know.

Thanks

Kev

Hello to All!

I have an issue using the nxlog agent community edition. My output part of the config file looks like this:

Module      om_udp
Host        graylog.domain.local
Port        12201
OutputType  GELF

As failover mechanism I use the DNS record for graylog.domain.local. When the primary graylog node is unreachable, the DNS record is updated to the secondary node.

For some log sources, the OS is picking up the change almost immediately (there is no cache enable) but the agent doesn't deliver logs to the secondary node until the nxlog service is restarted.

Couple of questions:

1. Does the nxlog service keep any DNS cache?
2. If yes, is there any way to bypass it?
3. Any advice on how to fix the issue?

Many thanks in advance

Hi,

I want to log from my nxlog.conf to my LogFile.

Example: nxlog.conf:

include additonal/*.conf

additonal/module.conf

Log DEBUG "module.conf included"

How can i do this?

Dear team, I was currently using nxLog agent for windows (community edition) to forward logs to a centOS server. And I was looking for a documentation on how to build nxLog agent for windows plateform but could not find any. I hope the build documentation with some insight on how to prepare development environment will bring more people with project. Since many of us would love to add some of features to personalize it for personal use rather than opening issues with feature request.

I am a Linux user with some c development background so I hope to have some support on how to create development environment to build windows agent. My goal would be to add latest openSSL version support.

Thanks

Hi,

Here's my case :

I'm trying to create Syslog Server Configuration, based on CentOS. Server must :
-- Receive log from tcp -- store log into a NFS Volume automaticaly mounted by fstab on server startup ---- in case of fail of the NFS Volume, create a buffer on local disk.

Here's my configuration :

<Extension _syslog> Module xm_syslog </Extension>

<Extension _xm_file> Module xm_fileop </Extension>

<Input in> Module im_tcp Host 0.0.0.0 Port 39458 <Exec> $SERVER = hostname(); parse_syslog_ietf(); $DATE = strftime($EventTime, "%Y%m%d"); </Exec> </Input>

<Processor buffer> Module pm_buffer maxSize 204800 Type Disk Directory %LOGDIR% WarnLimit 190000 </Processor>

<Processor blocker> Module pm_blocker <Exec> $CHEMIN = "/var/partage/"+$SERVER; $test = dir_exists($CHEMIN); if $test == TRUE { blocker->block(FALSE); } else blocker->block(TRUE); </Exec> </Processor>

<Output out> Module om_file File "/var/partage/"+$SERVER+"/"+$Hostname+"/"+$DATE+"_"+$Hostname+".log" CreateDir TRUE Exec to_syslog_ietf(); </Output>

<Route tcp_to_file> Path in => buffer => blocker => out </Route>

My Problem :

• When NFS Volume is down, pm_blocker write " can't check condition "dir_exists(<Path>)" : input/outpur Error"
• When i try without pm_blocker, om_file fails with same error and no buffer created....

Question :

• What is the best solution to implement a disk buffer system for om_file module ?
• How can i catch NFS volume exceptions ?

Thanks in advance, best regards, Julien

I have a im_file to om_tcp route on the community edition.
My applications starts generating logs at 5am and runs every 15 minutes.
I have observed that it essentially skips the first X lines and syncs things only after the x lines starting 5:15 and works fine throughout the day.
There is nothing in logs at 5am.
I have tried routing it to another om_file on the same machine, it works ok.

Can you please help me fix it?

Here is the config:

########################################
# Global directives #
########################################
User nxlog
Group nxlog

LogFile /var/log/nxlog/nxlog.log
LogLevel INFO

########################################
# Modules #
########################################
<Extension _syslog>
Module xm_syslog
</Extension>

<Extension _fileop>
Module xm_fileop
</Extension>

########################################
# Input #
########################################
<Input in-my-important-app>
Module im_file
File "/var/log/tomcat/my-important-app-*.log"
ReadFromLast True
SavePos True
Exec $Filename = "myApp/" +file_basename(file_name());
Exec $Hostname = hostname_fqdn();
</Input>

########################################
# Output #
########################################
# Output to destination
<Output out-destination>
Module om_tcp
Host [destination host]
Port [destination port]
OutputType Binary
</Output>

########################################
# Routes #
########################################
<Route to_destination>
Path in-my-important-app => out-destination
</Route>

I am trying to send parse each of these XML fields into a field for graylog to handle, any ideas would help. I've added Module xm_xml and `Exec parse_windows_eventlog_xml(); to_xml();` but I'm not sure what else to do, I'm trying to work with this in the 'message' field The Federation Service validated a new credential. See XML for details. Activity ID: 494a36f8-9b89-4477-8676-0080000000e1 Additional Data XML: FreshCredentials Success None N/A https://xxxxxx.xxxxxxx.edu/adfs/services/trust AD AUTHORITY UNIVERSITY\xxxxxxxxxxxxxx N/A false N/A false N/A false false NotSet N/A N/A https://xxxxx.xxxxxx.edu/adfs/services/trust WSFederation Intranet x.x.94.22 x.x.128.226 N/A N/A N/A Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:77.0) Gecko/20100101 Firefox/77.0 /adfs/ls/

Hello,

I am collecting syslog from many different firewalls and I want to extract only a single field from these messages and output that to a file separate of the output file I am currently writing. Basically I want to match the field devname= and extract it and output the results into a list. Is there a way to avoid duplication? So I have firewall1, firewall2, firewall3 etc. and they are in the devname= field so in the log file it has devname="firewall1", devname="firewall2", devname="firewall3" etc. I am wondering how can I look for all these names and create a list of them. Thank you.

I have tried multiple ways to get the wildcard includes to work but just cant get it to work in CE. If I put the path in then it works.

define ROOT C:\Program Files (x86)\nxlog define PLUGIN %ROOT%\additional

works: include C:\Program Files (x86)\nxlog\additional\nxlog_exchange.conf

doesn't work: include additional*.conf include additional/.conf include %PLUGIN%*.conf include %PLUGIN%/.conf

Hi,

My working nxlog.conf relies on Query directives aimed at explicitly named Channels together with the im_msvistalog Module.

<Input blu_eventlog_iis> Module im_msvistalog

Query       <QueryList>\
                    <Query Id="0">\
                        <Select Path="Microsoft-IIS-Configuration/Administrative">*</Select>\
                        <Select Path="Microsoft-IIS-Configuration/Operational">*</Select>\
                        <Select Path="Microsoft-IIS-Logging/Logs">*</Select>\
                    </Query>\
                </QueryList> 
    
# Filter noise from IIS schema issues
Exec if ($Message =~ /Unable to find schema/) drop();

# Workaround for local time so as to standardize to absolute microseconds since epoch
Exec $EventTime = integer($EventTime);

# JSON is required because some Windows logs contain new-line characters.
Exec $Message = to_json();

</Input>

<Output blu_out_eventlog_iis> Module om_tcp Host %SIEM% Port %PORT%

Exec to_syslog_bsd();

</Output>

<Route route_eventlog_iis> Path blu_eventlog_iis => blu_out_eventlog_iis </Route>

I hoped to clean up some 15007 errors in the nxlog.log files included below:

2020-07-06 14:42:55 ERROR failed to subscribe to msvistalog events,the channel was not found [error code: 15007]; The specified channel could not be found.

I suspect the 15007 errors are being generated because IIS isn't setup. In other words, the named IIS Channels are queried but not found.

Is it possible to add an If/Else statement as part of those Query directives? Looking over the NXlog CE Manual I only see If/Else statement used with Exec. Any guidance would be greatly appreciated!

Hi all,

I must send the event logs (only event ID 4626 and 4625) in the log collector Linux. can you help me with configuration?

Thank you.

I'm very new to NXLog and have no clue how to fix this. I was setting up an nxlog.conf file to one of our servers and we keep getting this error. It starts with the 'not enough' fields error but then after a few lines it says 'Too many fields'. There are 21 fields on the W3c fields, but I'm not sure why it seems like it's adding Fields as an input? I added an additional string in the Fieldtypes and also combined date-time instead of 'date, time' but didn't work. Is there a way to fix this in the Conf file? Or should I be looking in the IIS?

Log expected 20, got 6 in input '#Software: Microsoft Internet Information Services 8.5' expected 20, got 2 in input '#Version: 1.0' expected 20, got 3 in input '#Date: 2020-06-30 08:35:19' Too many fields in CSV input, expected 20, got 21 in input '#Fields: date time c-ip c-port cs-username s-sitename s-computername cs-host s-ip s-port cs-method cs-uri-stem sc-status sc-win32-status sc-substatus sc-bytes cs-bytes time-taken x-session x-fullpath x-debug'

Conf <Extension w3c_ftp> Module xm_csv Fields date, time, c-ip, cs-username, s-sitename, s-computername, s-ip, cs-method, cs-uri-stem, sc-status, sc-win32-status, sc-bytes, cs-bytes, time-taken, s-port, cs-host, sc-substatus, x-session, x-fullpath, x-debug, c-port FieldTypes string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string Delimiter ' ' </Extension>

TIA!

Hi, I'm trying to send my custom jsotn to GrayLog. My json file is: { "Faxes": "68", "Last30DaysEmails": "9728", "TodayEmails": "66" } This is my nxlog config file: define ROOT C:\Program Files (x86)\nxlog Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data LogFile %ROOT%\data\nxlog.log Module xm_gelf Module xm_multiline HeaderLine /\d{4}-\d{2}-\d{2}\s\d*:\d\d:\d*,\d{3}\s[a-zA-Z]*\s\s\S\s\[Begin Lead\]/ EndLine /\d{4}-\d{2}-\d{2}\s\d*:\d\d:\d*,\d{3}\s[a-zA-Z]*\s\s\S\s\[End Lead\]/ Module xm_json Module im_file File "C:\\logs\\myfile.json" InputType multiline SavePos FALSE ReadFromLast FALSE Module om_udp Host 192.168.1.100 Port 3514 OutputType GELF Path im_file => om_udp Now my problems is: When I send the file to GrayLog server I see multi line with different messages (as json variable), and not all in a single message. How can do to configure correctly my system? Thanks for the support. Marco

Hi,

I've want to include nxlog to send all Windows logs to syslog plus Microsoft-Windows-Windows Defender/Operational, however as "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\EventLog" doesn't seem to include "Microsoft-Windows-Windows Defender" it doesn't read that config in at start and send those Windows Defender logs.

How can I get nxlog to send everything it does now, plus Microsoft-Windows-Windows Defender logs, without having to define everything? The only way I can see to do this is with two im_msvistalog defines, i.e.

define ROOT C:\Program Files (x86)\nxlog define CERTDIR %ROOT%\cert

Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data LogFile %ROOT%\data\nxlog.log

<Extension syslog> Module xm_syslog </Extension>

<Input eventlog> Module im_msvistalog </Input>

<Input eventlog_defender> Module im_msvistalog

Query <QueryList>
<Query Id="0">
<Select Path="Microsoft-Windows-Windows Defender/Operational">*</Select>
</Query>
</QueryList>

</Input>

<Processor eventlog_transformer> Module pm_transformer Exec $Hostname = hostname(); OutputFormat syslog_rfc3164 </Processor>

<Output syslogout> Module om_ssl [removed, but its to Papertrail] </Output>

<Route 1> Path eventlog => eventlog_transformer => syslogout </Route>

<Route 2> Path eventlog_defender => eventlog_transformer => syslogout </Route>

That seems to work except there are three problems compared to when I only have the Microsoft-Windows-Windows Defender going and the other blocks deleted:

1. The Windows Defender logging is a little inconsistent, testing with an EICAR test virus doesn't always appear, probably 50% chance it appears in the syslog (all events appear in Windows log)
2. The severity and source is copied from the previous log line (usually info rather than warn severity, and not Microsoft-Windows-Windows Defender), maybe a threading issue?
3. Each rows of text is in its own line rather than all in a single message which happens when its the only logging to send

Is there a better way to do 'everything' plus Microsoft-Windows-Windows Defender?

I am trying to reduce the size of the message sent from my Windows event logs to graylog and I cannot for the life of me figuring out how tell it to drop certain fields

The only thing I can figure out is that i'm supposed to use delete() but how to use and where to place it in my config is very frustrating.

I'm currently using nxlog to filter and forward syslog: Source => Filter Logs on intermediate server with nxlog installed => forward udp 514 (syslog). Config looks like the following: Module xm_syslog #syslog to Nxlog01 Module im_udp Port 514 if (($Message =~ /REGEX/) or ($Message =~ /REGEX/)) drop(); Exec if $MessageSourceAddress != "1.1.1.1" drop(); Module pm_norepeat CheckFields Hostname, Message, SourceName #Syslog Output to DST server Module om_udp Host dstserver_ip Port 514 Exec to_syslog_bsd(); Path _sys_in => _sys_norepeat => _sys_out Firstly is this possible with the CE agent? I already performed a tcpdump on the intermediate server where nxlog is installed and i can see the syslog being received but not sent after filtered. There is udp 514 connectivity between nxlog server and destination server. I tried: * Removing all regex filtering (since i thought my regex wasn't good enough), no results. * Changed im_udp to im_tcp (i thought maybe you can't used the same port in both input and output modules). * Checked whether there actually is network connectivity between nxlog server and destination server using nc by sending a udp 514 packet to dst server. Can anyone help with this?

Hi All, Needed help with parsing/modify. Would greatly appreciate some direction. At the moment, I am parsing a plain-test log-file and sending to a remote server: Jun 19 16:29:28 server12345 [...] Jun 19 16:29:28 server12345 --- Jun 19 16:29:27 server12345 [program.state :3371] The above is what I get with parsing. I was hoping to make every line transform like this Jun 19 16:29:28 server12345 **programName1** [...] Jun 19 16:29:28 server12345 **programName1** --- Jun 19 16:29:27 server12345 **programName1** [program.state :3371] Can someone help me understand, how I can get programName1 appended to each file, after the server hostname? Also, is it possible to parse the server-name, and replace it as follows: Jun 19 16:29:28 **server12345--NA** programName1 [...] Jun 19 16:29:28 **server12345--NA** programName1 --- Jun 19 16:29:27 **server12345--NA** programName1 [program.state :3371] Would appreciate some guidance on how to change this via config. I currently, am reading in a textfile via the om_file method. **Update**: Relevant snippets of config: Module im_file File 'C:\program\var\log\file.log' #InputType multiline_parser Exec parse_syslog(); Module pm_norepeat CheckFields Hostname, Message, SourceName OutputFormat syslog_rfc3164 Path log_file => norepeat => log_output Module om_udp Host x.x.x.x Port 514