Unable to send csv logs to graylog udp using nxlog


#1 navdeepsingh83

Hi Everyone, I have copied jira access logs, filter it using csv and put under a directory as csv file. I am using below nxconf which gets loaded and nxlog service gets started, however no logs are send to graylog server.

#define ROOT C:\Program Files\nxlog define ROOT C:\Program Files (x86)\nxlog

Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data LogFile %ROOT%\data\nxlog.log

<Extension json> Module xm_json </Extension>

<Extension fileop> Module xm_fileop </Extension>

<Extension _syslog> Module xm_syslog </Extension>

<Extension gelf> Module xm_gelf </Extension>

<Extension jira> Module xm_csv Fields $IPAddress,$UserName,$DateTime,$HTTPAction,$ResponseCode,$Column10,$Column11 FieldTypes string,string,string,string,string,string,string Delimiter"," </Extension>

<Input in> Module im_file File "C:\Users\jira\Documents\TempOut\JiraAccessLogs\accessLog.csv" #ReadFromLast False #Recursive True #SavePos True

<Exec> if $raw_event =~ /^#/ drop(); else { jira->parse_csv();
to_json(); } </Exec>

</Input>

<Output out>

Module om_udp
Host 172.17.1.87
Port 5046
OutputType  GELF_UDP
Exec $short_message = $raw_event; # Avoids truncation of the short_message field.
Exec $collector_node_id = 'SINNB0094';
Exec $Hostname = hostname_fqdn();
#Use the following line for debugging (uncomment the fileop extension above as well)
Exec file_write(&quot;C:\Users\\jira\\Documents\\TempOut\\JiraAccessLogs\\nxlog-debug.log&quot;, $raw_event);

</Output>

<Route 1> Path in => out </Route>

if i replace, output section with below and send the data to txt/log file, i can see the logs converted to json and written.

Module om_file File "C:\nxlog-debug.txt"

Ii suspect there is some issue with out with om_udp but since i don’t see any error or warning, it’s difficult to troubleshoot. can you spot what is wrong with the out?

#2 Zhengshi Nxlog ✓
#1 navdeepsingh83
Hi Everyone, I have copied jira access logs, filter it using csv and put under a directory as csv file. I am using below nxconf which gets loaded and nxlog service gets started, however no logs are send to graylog server. #define ROOT C:\Program Files\nxlog define ROOT C:\Program Files (x86)\nxlog Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data LogFile %ROOT%\data\nxlog.log <Extension json> Module xm_json </Extension> <Extension fileop> Module xm_fileop </Extension> <Extension _syslog> Module xm_syslog </Extension> <Extension gelf> Module xm_gelf </Extension> <Extension jira> Module xm_csv Fields $IPAddress,$UserName,$DateTime,$HTTPAction,$ResponseCode,$Column10,$Column11 FieldTypes string,string,string,string,string,string,string Delimiter"," </Extension> <Input in> Module im_file File "C:\Users\jira\Documents\TempOut\JiraAccessLogs\accessLog.csv" #ReadFromLast False #Recursive True #SavePos True <Exec> if $raw_event =~ /^#/ drop(); else { jira->parse_csv(); to_json(); } </Exec> </Input> <Output out> Module om_udp Host 172.17.1.87 Port 5046 OutputType GELF_UDP Exec $short_message = $raw_event; # Avoids truncation of the short_message field. Exec $collector_node_id = 'SINNB0094'; Exec $Hostname = hostname_fqdn(); #Use the following line for debugging (uncomment the fileop extension above as well) Exec file_write(&quot;C:\Users\\jira\\Documents\\TempOut\\JiraAccessLogs\\nxlog-debug.log&quot;, $raw_event); </Output> <Route 1> Path in => out </Route> if i replace, output section with below and send the data to txt/log file, i can see the logs converted to json and written. Module om_file File "C:\nxlog-debug.txt" Ii suspect there is some issue with out with om_udp but since i don’t see any error or warning, it’s difficult to troubleshoot. can you spot what is wrong with the out?

I don't see anything glaring in your config.

i don’t see any error or warning

Assuming you mean in the NXLog log file?
In that case, you could check on the graylog side to see if it is rejected for some reason. Wrong fields, doesn't like date format, etc.
If you aren't seeing logs on the graylog side showing your events, check the network to make sure it is arriving with tcpdump or wireshark, etc.