8
responses
responses
Hi Folks,
I have following nxlog.conf which works fine.
define ROOT C:\Program Files (x86)\nxlog
Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log
LogLevel INFO
<Extension logrotate>
Module xm_fileop
<Schedule>
When @daily
Exec file_cycle('%ROOT%\data\nxlog.log', 7);
</Schedule>
</Extension>
<Processor process-buffer>
Module pm_buffer
Type Mem
MaxSize 16384
</Processor>
<Extension gelfExt>
Module xm_gelf
# Avoid truncation of the short_message field to 64 characters.
ShortMessageLength 65536
</Extension>
<Extension multiline>
Module xm_multiline
HeaderLine /^\d+\/\d\d\/\d\d\d\d+\s\d+:\d\d:\d\d+\s+[A-Z]+/
</Extension>
<Input filein>
Module im_file
File 'C:\DNSLogs\DNSIN.log'
# SavePos TRUE
ReadFromLast FALSE
InputType multiline
</Input>
<Output fileout>
Module om_file
File 'C:\DNSLogs\DNSOUT.log'
#Exec $raw_event = "-------------------------------------\n" + $raw_event;
</Output>
<Route parse_multiline>
Path filein => process-buffer => fileout
</Route>
Now, I am trying to send the data to our log collector GrayLog & added output & route
define ROOT C:\Program Files (x86)\nxlog
Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log
LogLevel INFO
<Extension logrotate>
Module xm_fileop
<Schedule>
When @daily
Exec file_cycle('%ROOT%\data\nxlog.log', 7);
</Schedule>
</Extension>
<Processor process-buffer>
Module pm_buffer
Type Mem
MaxSize 16384
</Processor>
<Extension gelfExt>
Module xm_gelf
# Avoid truncation of the short_message field to 64 characters.
ShortMessageLength 65536
</Extension>
<Extension multiline>
Module xm_multiline
HeaderLine /^\d+\/\d\d\/\d\d\d\d+\s\d+:\d\d:\d\d+\s+[A-Z]+/
</Extension>
<Input filein>
Module im_file
File 'C:\DNSLogs\MGTDCP03_DN.log'
# SavePos TRUE
ReadFromLast FALSE
InputType multiline
</Input>
<Output fileout>
Module om_file
File 'C:\DNSLogs\MGTDCP03_DNSOUT-2.log'
#Exec $raw_event = "-------------------------------------\n" + $raw_event;
</Output>
<Route parse_multiline>
Path filein => process-buffer => fileout
</Route>
<Output gelf>
Module om_tcp
Host 172.17.1.87
Port 5044
OutputType GELF_TCP
<Exec>
$gl2_source_collector = '${sidecar.nodeId}';
$collector_node_id = '${sidecar.nodeName}';
</Exec>
</Output>
<Route to_graylog>
Path filein => process-buffer => gelf
</Route>
However, I start getting error msgs in nxlog.log. I have tried some troubleshooting steps but so far I am not able to get around the problem. Appreciate if you can help me resolve this issue.
2020-05-15 05:43:17 WARNING stopping nxlog service
2020-05-15 05:43:17 WARNING nxlog-ce received a termination request signal, exiting...
2020-05-15 05:43:18 INFO nxlog-ce-2.10.2150 started
nxlog failed to start: Couldn't parse Exec block at C:\Program Files\Graylog\sidecar\generated\nxlog.conf:58
couldn't parse statement at line 59, character 67 in C:\Program Files\Graylog\sidecar\generated\nxlog.conf
invalid character: '
' (0xd)
2020-05-15 05:44:17 ERROR Couldn't parse Exec block at C:\Program Files\Graylog\sidecar\generated\nxlog.conf:58; couldn't parse statement at line 59, character 67 in C:\Program Files\Graylog\sidecar\generated\nxlog.conf; invalid character: ';' (0xd)
2020-05-15 05:44:17 WARNING stopping nxlog service
2020-05-15 05:44:17 WARNING nxlog-ce received a termination request signal, exiting...
2020-05-15 05:44:18 ERROR Couldn't parse Exec block at C:\Program Files\Graylog\sidecar\generated\nxlog.conf:58; couldn't parse statement at line 59, character 67 in C:\Program Files\Graylog\sidecar\generated\nxlog.conf; invalid character: ';' (0xd)
2020-05-15 05:44:18 ERROR cannot add processor module 'process-buffer' to route 'to_graylog' because it is already added to route 'parse_multiline', you should define another instance at C:\Program Files\Graylog\sidecar\generated\nxlog.conf:65
2020-05-15 05:44:18 ERROR module 'gelf' has configuration errors, not adding to route 'to_graylog' at C:\Program Files\Graylog\sidecar\generated\nxlog.conf:65
2020-05-15 05:44:18 ERROR route to_graylog is not functional without output modules, ignored at C:\Program Files\Graylog\sidecar\generated\nxlog.conf:65
2020-05-15 05:44:18 WARNING not starting unused module gelf
2020-05-15 05:44:18 INFO nxlog-ce-2.10.2150 started
Comments (7)
Arch, i have sorted the issue with pm_buffer. Actually these configurations are pushed via graylog side car, and the variable populates hostname & collector node id automatically.
Basically, I have 2 nxlog.conf file and they both work fine. One is for collection specific windows security event, and another one is for ingesting windows dns debug log with detailed option enabled.
Now, when i try to combine both these files into 1 file, i receive several errors.
Here is file 1 [working fine]
Here is file 2[working fine]
Here is merge of file 1 & 2 [not working]
Here is the error log from nxlog
Here is the nxlog.conf generated via graylog sidecar
Maybe you could try splitting im_msvistalog in two smaller modules.
Thanks manuel for you input, I used , and the problem was resolved.
Now the only problem that is left that I am getting error when use these exec statements.
Exec gl2_source_collector = 32fe8b2d-5c52-4ecd-9e6f-bf99d93cd0f0; Exec collector_node_id = Server01;
If change Exec block to something like this. This is mentioned in graylog
I get the following error
Not sure what is going on. I have the same block in other Graylog NXlog templates and it works fine. however it's not working in the new merged template.
Hello,
Could you please drop your current config?
I've tried to find those errors using previously posted but not sure is this is the right one.
Regards, Arch
I was able to sort it out, however 2 issues remain.
when using graylog 3.4.4 with variables to populate nodeId and hostname, nxlog fails to start. $gl2_source_collector = '${sidecar.nodeId}'; $collector_node_id = '${sidecar.nodeName}';
i have separated the output for dns & windows event, however duplicate logs are being received.
define ROOT C:\Program Files (x86)\nxlog define NOT_STARTING_WITH_DATE_REGEX /^(?!\d+\/\d+\/\d+).+/ define EMPTY_EVENT_REGEX /(^$|^\s+$)/ define DROP_DNSQUERYEVENT_REGEX /((NXDOMAIN)|(YXDOMAIN))/ define PUBLICIPV4 /\b(?!(10)|192.168|172.(2[0-9]|1[6-9]|3[0-2]))[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}/ define DIRECTION /(Snd)/
I would say you need to separate the routes...
Also you may need to remove the empty
Exec
blocks...