Unable to send logs to GrayLog - Multiple Route

Tags:

#1 navdeepsingh83

Hi Folks,

I have following nxlog.conf which works fine.

define ROOT C:\Program Files (x86)\nxlog

Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log
LogLevel INFO

<Extension logrotate>
    Module  xm_fileop
    <Schedule>
        When    @daily
        Exec    file_cycle('%ROOT%\data\nxlog.log', 7);
     </Schedule>
</Extension>

<Processor process-buffer>
  Module pm_buffer
  Type Mem
  MaxSize 16384
</Processor>

<Extension gelfExt>
  Module xm_gelf
  # Avoid truncation of the short_message field to 64 characters.
  ShortMessageLength 65536
</Extension>

<Extension multiline>
    Module          xm_multiline
    HeaderLine      /^\d+\/\d\d\/\d\d\d\d+\s\d+:\d\d:\d\d+\s+[A-Z]+/
</Extension>

<Input filein>
    Module          im_file
    File            'C:\DNSLogs\DNSIN.log'
#   SavePos         TRUE
    ReadFromLast    FALSE
    InputType       multiline
</Input>

<Output fileout>
    Module          om_file
    File            'C:\DNSLogs\DNSOUT.log'
    #Exec    $raw_event = "-------------------------------------\n" + $raw_event;
</Output>

<Route parse_multiline>
    Path filein => process-buffer => fileout
</Route>

Now, I am trying to send the data to our log collector GrayLog & added output & route

define ROOT C:\Program Files (x86)\nxlog

Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log
LogLevel INFO

<Extension logrotate>
    Module  xm_fileop
    <Schedule>
        When    @daily
        Exec    file_cycle('%ROOT%\data\nxlog.log', 7);
     </Schedule>
</Extension>

<Processor process-buffer>
  Module pm_buffer
  Type Mem
  MaxSize 16384
</Processor>

<Extension gelfExt>
  Module xm_gelf
  # Avoid truncation of the short_message field to 64 characters.
  ShortMessageLength 65536
</Extension>

<Extension multiline>
    Module          xm_multiline
    HeaderLine      /^\d+\/\d\d\/\d\d\d\d+\s\d+:\d\d:\d\d+\s+[A-Z]+/
</Extension>

<Input filein>
    Module          im_file
    File            'C:\DNSLogs\MGTDCP03_DN.log'
#   SavePos         TRUE
    ReadFromLast    FALSE
    InputType       multiline
</Input>

<Output fileout>
    Module          om_file
    File            'C:\DNSLogs\MGTDCP03_DNSOUT-2.log'
    #Exec    $raw_event = "-------------------------------------\n" + $raw_event;
</Output>

<Route parse_multiline>
    Path filein => process-buffer => fileout
</Route>

<Output gelf>
	Module om_tcp
	Host 172.17.1.87
	Port 5044
	OutputType  GELF_TCP
	<Exec>
	  $gl2_source_collector = '${sidecar.nodeId}';
	  $collector_node_id = '${sidecar.nodeName}';
	</Exec>
</Output>

<Route to_graylog>
  Path filein => process-buffer => gelf
</Route>

However, I start getting error msgs in nxlog.log. I have tried some troubleshooting steps but so far I am not able to get around the problem. Appreciate if you can help me resolve this issue.

2020-05-15 05:43:17 WARNING stopping nxlog service
2020-05-15 05:43:17 WARNING nxlog-ce received a termination request signal, exiting...
2020-05-15 05:43:18 INFO nxlog-ce-2.10.2150 started
nxlog failed to start: Couldn't parse Exec block at C:\Program Files\Graylog\sidecar\generated\nxlog.conf:58
couldn't parse statement at line 59, character 67 in C:\Program Files\Graylog\sidecar\generated\nxlog.conf
invalid character: '
' (0xd)

2020-05-15 05:44:17 ERROR Couldn't parse Exec block at C:\Program Files\Graylog\sidecar\generated\nxlog.conf:58; couldn't parse statement at line 59, character 67 in C:\Program Files\Graylog\sidecar\generated\nxlog.conf; invalid character: ';' (0xd)
2020-05-15 05:44:17 WARNING stopping nxlog service
2020-05-15 05:44:17 WARNING nxlog-ce received a termination request signal, exiting...
2020-05-15 05:44:18 ERROR Couldn't parse Exec block at C:\Program Files\Graylog\sidecar\generated\nxlog.conf:58; couldn't parse statement at line 59, character 67 in C:\Program Files\Graylog\sidecar\generated\nxlog.conf; invalid character: ';' (0xd)
2020-05-15 05:44:18 ERROR cannot add processor module 'process-buffer' to route 'to_graylog' because it is already added to route 'parse_multiline', you should define another instance at C:\Program Files\Graylog\sidecar\generated\nxlog.conf:65
2020-05-15 05:44:18 ERROR module 'gelf' has configuration errors, not adding to route 'to_graylog' at C:\Program Files\Graylog\sidecar\generated\nxlog.conf:65
2020-05-15 05:44:18 ERROR route to_graylog is not functional without output modules, ignored at C:\Program Files\Graylog\sidecar\generated\nxlog.conf:65
2020-05-15 05:44:18 WARNING not starting unused module gelf
2020-05-15 05:44:18 INFO nxlog-ce-2.10.2150 started
#2 ArkadiyDeactivated Nxlog ✓
#1 navdeepsingh83
Hi Folks, I have following nxlog.conf which works fine. define ROOT C:\Program Files (x86)\nxlog Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data LogFile %ROOT%\data\nxlog.log LogLevel INFO <Extension logrotate> Module xm_fileop <Schedule> When @daily Exec file_cycle('%ROOT%\data\nxlog.log', 7); </Schedule> </Extension> <Processor process-buffer> Module pm_buffer Type Mem MaxSize 16384 </Processor> <Extension gelfExt> Module xm_gelf # Avoid truncation of the short_message field to 64 characters. ShortMessageLength 65536 </Extension> <Extension multiline> Module xm_multiline HeaderLine /^\d+\/\d\d\/\d\d\d\d+\s\d+:\d\d:\d\d+\s+[A-Z]+/ </Extension> <Input filein> Module im_file File 'C:\DNSLogs\DNSIN.log' # SavePos TRUE ReadFromLast FALSE InputType multiline </Input> <Output fileout> Module om_file File 'C:\DNSLogs\DNSOUT.log' #Exec $raw_event = "-------------------------------------\n" + $raw_event; </Output> <Route parse_multiline> Path filein => process-buffer => fileout </Route> Now, I am trying to send the data to our log collector GrayLog & added output & route define ROOT C:\Program Files (x86)\nxlog Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data LogFile %ROOT%\data\nxlog.log LogLevel INFO <Extension logrotate> Module xm_fileop <Schedule> When @daily Exec file_cycle('%ROOT%\data\nxlog.log', 7); </Schedule> </Extension> <Processor process-buffer> Module pm_buffer Type Mem MaxSize 16384 </Processor> <Extension gelfExt> Module xm_gelf # Avoid truncation of the short_message field to 64 characters. ShortMessageLength 65536 </Extension> <Extension multiline> Module xm_multiline HeaderLine /^\d+\/\d\d\/\d\d\d\d+\s\d+:\d\d:\d\d+\s+[A-Z]+/ </Extension> <Input filein> Module im_file File 'C:\DNSLogs\MGTDCP03_DN.log' # SavePos TRUE ReadFromLast FALSE InputType multiline </Input> <Output fileout> Module om_file File 'C:\DNSLogs\MGTDCP03_DNSOUT-2.log' #Exec $raw_event = "-------------------------------------\n" + $raw_event; </Output> <Route parse_multiline> Path filein => process-buffer => fileout </Route> <Output gelf> Module om_tcp Host 172.17.1.87 Port 5044 OutputType GELF_TCP <Exec> $gl2_source_collector = '${sidecar.nodeId}'; $collector_node_id = '${sidecar.nodeName}'; </Exec> </Output> <Route to_graylog> Path filein => process-buffer => gelf </Route> However, I start getting error msgs in nxlog.log. I have tried some troubleshooting steps but so far I am not able to get around the problem. Appreciate if you can help me resolve this issue. 2020-05-15 05:43:17 WARNING stopping nxlog service 2020-05-15 05:43:17 WARNING nxlog-ce received a termination request signal, exiting... 2020-05-15 05:43:18 INFO nxlog-ce-2.10.2150 started nxlog failed to start: Couldn't parse Exec block at C:\Program Files\Graylog\sidecar\generated\nxlog.conf:58 couldn't parse statement at line 59, character 67 in C:\Program Files\Graylog\sidecar\generated\nxlog.conf invalid character: ' ' (0xd) 2020-05-15 05:44:17 ERROR Couldn't parse Exec block at C:\Program Files\Graylog\sidecar\generated\nxlog.conf:58; couldn't parse statement at line 59, character 67 in C:\Program Files\Graylog\sidecar\generated\nxlog.conf; invalid character: ';' (0xd) 2020-05-15 05:44:17 WARNING stopping nxlog service 2020-05-15 05:44:17 WARNING nxlog-ce received a termination request signal, exiting... 2020-05-15 05:44:18 ERROR Couldn't parse Exec block at C:\Program Files\Graylog\sidecar\generated\nxlog.conf:58; couldn't parse statement at line 59, character 67 in C:\Program Files\Graylog\sidecar\generated\nxlog.conf; invalid character: ';' (0xd) 2020-05-15 05:44:18 ERROR cannot add processor module 'process-buffer' to route 'to_graylog' because it is already added to route 'parse_multiline', you should define another instance at C:\Program Files\Graylog\sidecar\generated\nxlog.conf:65 2020-05-15 05:44:18 ERROR module 'gelf' has configuration errors, not adding to route 'to_graylog' at C:\Program Files\Graylog\sidecar\generated\nxlog.conf:65 2020-05-15 05:44:18 ERROR route to_graylog is not functional without output modules, ignored at C:\Program Files\Graylog\sidecar\generated\nxlog.conf:65 2020-05-15 05:44:18 WARNING not starting unused module gelf 2020-05-15 05:44:18 INFO nxlog-ce-2.10.2150 started

Hello,

Two points to look at:

  • you need to define one more pm_buffer module because first one already used by other route;
  • something wrong with your Exec directive in gelf module, I'm not sure what you are trying to achieve using variables like this.

Regards, Arch