Parsing Problems | View post | Log collection solutions

abasha 5 years ago

Hello All,

I have a huge .csv file, this contains logs from Service Now instance. I have the following nxlog configuration file. But when i run the parser, error file i generate exceeds more than 1 GB. The source file itself is only about 225 MB.

Please set the ROOT to the folder your nxlog was installed into,

otherwise it will not start.

#define ROOT C:\Program Files\nxlog define ROOT C:\Program Files (x86)\nxlog

Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data LogFile %ROOT%\data\nxlog.log

<Extension multiline> Module xm_multiline HeaderLine /^\d{1,2}/\d{1,2}/\d{4}\s/ </Extension>

<Extension json> Module xm_json </Extension>

<Extension csv> Module xm_csv Fields $Created,$Level,$Message,$Source,$CreatedBy FieldTypes string, string, string, string, string </Extension>

<Extension syslog> Module xm_syslog </Extension>

<Input eventlog> Module im_msvistalog ReadFromLast TRUE SavePos TRUE Query <QueryList>
<Query Id="0">
<Select Path="Security">[System[(EventID=4768)]]</Select>
<Select Path="Security">[System[(EventID=4769)]]</Select>
<Select Path="Security">[System[(EventID=4771)]]</Select>
<Select Path="Security">[System[(EventID=4624)]]</Select>
<Select Path="Security">[System[(EventID=4625)]]</Select>
<Select Path="Security">[System[(EventID=4634)]]</Select>
<Select Path="Security">[System[(EventID=4647)]]</Select>
<Select Path="Security">[System[(EventID=4648)]]</Select>
<Select Path="Security">[System[(EventID=4656)]]</Select>
<Select Path="Security">[System[(EventID=4719)]]</Select>
<Select Path="Security">[System[(EventID=4720)]]</Select>
<Select Path="Security">[System[(EventID=4722)]]</Select>
<Select Path="Security">[System[(EventID=4723)]]</Select>
<Select Path="Security">[System[(EventID=4724)]]</Select>
<Select Path="Security">[System[(EventID=4725)]]</Select>
<Select Path="Security">[System[(EventID=4726)]]</Select>
<Select Path="Security">[System[(EventID=4727)]]</Select>
<Select Path="Security">[System[(EventID=4728)]]</Select>
<Select Path="Security">[System[(EventID=4729)]]</Select>
<Select Path="Security">[System[(EventID=4730)]]</Select>
<Select Path="Security">[System[(EventID=4731)]]</Select>
<Select Path="Security">[System[(EventID=4732)]]</Select>
<Select Path="Security">[System[(EventID=4733)]]</Select>
<Select Path="Security">[System[(EventID=4734)]]</Select>
<Select Path="Security">[System[(EventID=4735)]]</Select>
<Select Path="Security">[System[(EventID=4737)]]</Select>
<Select Path="Security">[System[(EventID=4738)]]</Select>
<Select Path="Security">[System[(EventID=4739)]]</Select>
<Select Path="Security">[System[(EventID=4741)]]</Select>
<Select Path="Security">[System[(EventID=4742)]]</Select>
<Select Path="Security">[System[(EventID=4743)]]</Select>
<Select Path="System">[System[(EventID=7036)]]</Select>
<Select Path="Application">[System[(EventID=18454)]]</Select>
<Select Path="Application">[System[(EventID=18456)]]</Select>
</Query>
</QueryList> Exec to_json(); </Input>

<Input filein> Module im_file File 'e:\ServiceNow\agent\export\snow_log.csv'

InputType       multiline
ReadFromLast    FALSE
SavePos         FALSE
&lt;Exec&gt;
    # Ignore top line
    if $raw_event =~ /Created,Level,Message,Source,Created by/ drop();
	if $raw_event =~ /Warning/ drop();
	if $raw_event =~ /Information/ drop();
	
    # Convert Newline and Tab to printed character
    #$raw_event =~ s/\R/\\r\\n/g;
    #$raw_event =~ s/\t/\\t/g;
	
	$raw_event = replace($raw_event,&quot;\n&quot;, &quot; &quot;);
	$raw_event = replace($raw_event,&quot;\r&quot;, &quot; &quot;);
	$raw_event = replace($raw_event,&quot;\t&quot;, &quot; &quot;);
			
	$SourceName = 'SNOWLogs';
    # Parse $raw_event as CSV
    csv-&gt;parse_csv();

    # Convert to JSON
    to_json();
&lt;/Exec&gt;

</Input>

<Output fileout> Module om_tcp Host logger Port 5140 #Exec to_syslog_bsd(); </Output>

<Output out> Module om_tcp Host logger Port 5140 </Output>

<Route r1> Path eventlog => out </Route>

<Route parse_xml> Path filein => fileout </Route>

For few lines it reads the data properly, but in some lines, it does not read the complete data. I am also trying to drop off unwanted data like information or warning, just to ensure i collect only Error information. But still it does not help. Error information from the file is very limited, so that I can reduce the amount of data to be ingested into ELK.

Sample of Error messages as follows: Created Level Message 9/10/2019 3:00 Error java.lang.NullPointerException: java.lang.NullPointerException: 9/10/2019 1:07 Error java.lang.NullPointerException: java.lang.NullPointerException: 9/10/2019 1:04 Error [code]Failed Exploring CI Pattern, Pattern name: <b>Docker Pattern</b>, Process ID: <b>12887</b>, To Check Pattern Log Press <a href="$sw_horizontal_discovery_log.do?discoLogId=aeeb6a6d1b33fb40db5e43b4bd4bcb5a&ipAddress=10.144.112.51&pid=12887&preExecution=&host_sys_id=d3fd5bff87e04504065e00f509434dc2&host_name=dm01db02.ga.ssga.root&patternId=dd15665a7fe022004e83e2065f2a0c57&patternName=Docker Pattern&patternType=1&isCloud=false"><u><b>Here</b></u></a>[/code] 9/10/2019 1:04 Error [code]Failed Exploring CI Pattern, Pattern name: <b>Docker Pattern</b>, Process ID: <b>12841</b>, To Check Pattern Log Press <a href="$sw_horizontal_discovery_log.do?discoLogId=aeeb6a6d1b33fb40db5e43b4bd4bcb11&ipAddress=10.145.112.57&pid=12841&preExecution=&host_sys_id=9ac8ef3887bc0904065e00f509434d22&host_name=dm02db08.ga.ssga.root&patternId=dd15665a7fe022004e83e2065f2a0c57&patternName=Docker Pattern&patternType=1&isCloud=false"><u><b>Here</b></u></a>[/code] 9/10/2019 1:04 Error [code]Failed Exploring CI Pattern, Pattern name: <b>Docker Pattern</b>, Process ID: <b>13373</b>, To Check Pattern Log Press <a href="$sw_horizontal_discovery_log.do?discoLogId=eeeb6a6d1b33fb40db5e43b4bd4bcb41&ipAddress=10.145.112.51&pid=13373&preExecution=&host_sys_id=ca716bb387244504065e00f509434dd6&host_name=dm02db02.ga.ssga.root&patternId=dd15665a7fe022004e83e2065f2a0c57&patternName=Docker Pattern&patternType=1&isCloud=false"><u><b>Here</b></u></a>[/code] 9/10/2019 1:04 Error [code]Failed Exploring CI Pattern, Pattern name: <b>Docker Pattern</b>, Process ID: <b>13328</b>, To Check Pattern Log Press <a href="$sw_horizontal_discovery_log.do?discoLogId=acebe6ad1bff7f404d41dd7edd4bcb1f&ipAddress=10.145.112.54&pid=13328&preExecution=&host_sys_id=7e912fb387244504065e00f509434d8c&host_name=dm02db05.ga.ssga.root&patternId=dd15665a7fe022004e83e2065f2a0c57&patternName=Docker Pattern&patternType=1&isCloud=false"><u><b>Here</b></u></a>[/code] 9/10/2019 1:04 Error [code]Failed Exploring CI Pattern, Pattern name: <b>Docker Pattern</b>, Process ID: <b>12911</b>, To Check Pattern Log Press <a href="$sw_horizontal_discovery_log.do?discoLogId=80eb2a6d1b33fb40db5e43b4bd4bcb88&ipAddress=10.144.112.56&pid=12911&preExecution=&host_sys_id=964e9fff87e04504065e00f509434d5f&host_name=dm01db07.ga.ssga.root&patternId=dd15665a7fe022004e83e2065f2a0c57&patternName=Docker Pattern&patternType=1&isCloud=false"><u><b>Here</b></u></a>[/code] 9/10/2019 1:04 Error [code]Failed Exploring CI Pattern, Pattern name: <b>Docker Pattern</b>, Process ID: <b>12899</b>, To Check Pattern Log Press <a href="$sw_horizontal_discovery_log.do?discoLogId=40eb2a6d1b33fb40db5e43b4bd4bcbc2&ipAddress=10.144.112.53&pid=12899&preExecution=&host_sys_id=391e5bff87e04504065e00f509434d3e&host_name=dm01db04.ga.ssga.root&patternId=dd15665a7fe022004e83e2065f2a0c57&patternName=Docker Pattern&patternType=1&isCloud=false"><u><b>Here</b></u></a>[/code] 9/10/2019 1:04 Error [code]Failed Exploring CI Pattern, Pattern name: <b>Docker Pattern</b>, Process ID: <b>13264</b>, To Check Pattern Log Press <a href="$sw_horizontal_discovery_log.do?discoLogId=48eb2a6d1b33fb40db5e43b4bd4bcb6a&ipAddress=10.145.112.56&pid=13264&preExecution=&host_sys_id=f0b1afb387244504065e00f509434df6&host_name=dm02db07.ga.ssga.root&patternId=dd15665a7fe022004e83e2065f2a0c57&patternName=Docker Pattern&patternType=1&isCloud=false"><u><b>Here</b></u></a>[/code] 9/10/2019 1:04 Error [code]Failed Exploring CI Pattern, Pattern name: <b>Docker Pattern</b>, Process ID: <b>12879</b>, To Check Pattern Log Press <a href="$sw_horizontal_discovery_log.do?discoLogId=44eb2a6d1b33fb40db5e43b4bd4bcbf4&ipAddress=10.144.112.50&pid=12879&preExecution=&host_sys_id=6cfddfbb87e04504065e00f509434d75&host_name=dm01db01.ga.ssga.root&patternId=dd15665a7fe022004e83e2065f2a0c57&patternName=Docker Pattern&patternType=1&isCloud=false"><u><b>Here</b></u></a>[/code] 9/10/2019 1:04 Error [code]Failed Exploring CI Pattern, Pattern name: <b>Docker Pattern</b>, Process ID: <b>13267</b>, To Check Pattern Log Press <a href="$sw_horizontal_discovery_log.do?discoLogId=4adba2ad1bff7f404d41dd7edd4bcbb1&ipAddress=10.145.112.55&pid=13267&preExecution=&host_sys_id=19a12fb387244504065e00f509434d28&host_name=dm02db06.ga.ssga.root&patternId=dd15665a7fe022004e83e2065f2a0c57&patternName=Docker Pattern&patternType=1&isCloud=false"><u><b>Here</b></u></a>[/code] 9/10/2019 1:04 Error [code]Failed Exploring CI Pattern, Pattern name: <b>Docker Pattern</b>, Process ID: <b>12901</b>, To Check Pattern Log Press <a href="$sw_horizontal_discovery_log.do?discoLogId=cedba2ad1bff7f404d41dd7edd4bcb90&ipAddress=10.144.112.57&pid=12901&preExecution=&host_sys_id=665edfbf87e04504065e00f509434d29&host_name=dm01db08.ga.ssga.root&patternId=dd15665a7fe022004e83e2065f2a0c57&patternName=Docker Pattern&patternType=1&isCloud=false"><u><b>Here</b></u></a>[/code] 9/10/2019 1:04 Error [code]Failed Exploring CI Pattern, Pattern name: <b>Docker Pattern</b>, Process ID: <b>13323</b>, To Check Pattern Log Press <a href="$sw_horizontal_discovery_log.do?discoLogId=cadbae6d1bff7f404d41dd7edd4bcb7b&ipAddress=10.145.112.53&pid=13323&preExecution=&host_sys_id=10916b7387244504065e00f509434d22&host_name=dm02db04.ga.ssga.root&patternId=dd15665a7fe022004e83e2065f2a0c57&patternName=Docker Pattern&patternType=1&isCloud=false"><u><b>Here</b></u></a>[/code] 9/10/2019 1:04 Error [code]Failed Exploring CI Pattern, Pattern name: <b>Docker Pattern</b>, Process ID: <b>13312</b>, To Check Pattern Log Press <a href="$sw_horizontal_discovery_log.do?discoLogId=fbcbeead1b377f40276510e4bd4bcbd2&ipAddress=10.145.112.50&pid=13312&preExecution=&host_sys_id=d7616bb387244504065e00f509434dd3&host_name=dm02db01.ga.ssga.root&patternId=dd15665a7fe022004e83e2065f2a0c57&patternName=Docker Pattern&patternType=1&isCloud=false"><u><b>Here</b></u></a>[/code] 9/10/2019 1:04 Error [code]Failed Exploring CI Pattern, Pattern name: <b>Docker Pattern</b>, Process ID: <b>12891</b>, To Check Pattern Log Press <a href="$sw_horizontal_discovery_log.do?discoLogId=b7cbeead1b377f40276510e4bd4bcb97&ipAddress=10.144.112.54&pid=12891&preExecution=&host_sys_id=642edbff87e04504065e00f509434dd6&host_name=dm01db05.ga.ssga.root&patternId=dd15665a7fe022004e83e2065f2a0c57&patternName=Docker Pattern&patternType=1&isCloud=false"><u><b>Here</b></u></a>[/code] 9/10/2019 1:04 Error [code]Failed Exploring CI Pattern, Pattern name: <b>Docker Pattern</b>, Process ID: <b>13255</b>, To Check Pattern Log Press <a href="$sw_horizontal_discovery_log.do?discoLogId=23cbae6d1bff7f404d41dd7edd4bcb6c&ipAddress=10.145.112.52&pid=13255&preExecution=&host_sys_id=d581ebb387244504065e00f509434da2&host_name=dm02db03.ga.ssga.root&patternId=dd15665a7fe022004e83e2065f2a0c57&patternName=Docker Pattern&patternType=1&isCloud=false"><u><b>Here</b></u></a>[/code] 9/10/2019 1:04 Error [code]Failed Exploring CI Pattern, Pattern name: <b>Docker Pattern</b>, Process ID: <b>13008</b>, To Check Pattern Log Press <a href="$sw_horizontal_discovery_log.do?discoLogId=47cb266d1b33fb40db5e43b4bd4bcb6c&ipAddress=10.144.112.52&pid=13008&preExecution=&host_sys_id=fe0ed7ff87e04504065e00f509434dd8&host_name=dm01db03.ga.ssga.root&patternId=dd15665a7fe022004e83e2065f2a0c57&patternName=Docker Pattern&patternType=1&isCloud=false"><u><b>Here</b></u></a>[/code] 9/10/2019 1:04 Error [code]Failed Exploring CI Pattern, Pattern name: <b>Docker Pattern</b>, Process ID: <b>12885</b>, To Check Pattern Log Press <a href="$sw_horizontal_discovery_log.do?discoLogId=c7cb266d1b33fb40db5e43b4bd4bcb8c&ipAddress=10.144.112.55&pid=12885&preExecution=&host_sys_id=a03e1fff87e04504065e00f509434d97&host_name=dm01db06.ga.ssga.root&patternId=dd15665a7fe022004e83e2065f2a0c57&patternName=Docker Pattern&patternType=1&isCloud=false"><u><b>Here</b></u></a>[/code] 9/10/2019 1:03 Error java.lang.NullPointerException: java.lang.NullPointerException: 9/10/2019 1:03 Error java.lang.NullPointerException: java.lang.NullPointerException: 9/10/2019 1:02 Error java.lang.NullPointerException: java.lang.NullPointerException: 9/10/2019 1:01 Error java.lang.NullPointerException: java.lang.NullPointerException: 9/10/2019 1:00 Error cmdb_metadata : Found duplicate cmdb_rel_type records with name: Master of::Stack Member of having sys_ids: 357afff213a21300f39f721a6144b076, c8c685710b22130005d90d2835673aa8: no thrown error 9/10/2019 1:00 Error java.lang.NullPointerException: java.lang.NullPointerException: 9/10/2019 1:00 Error LICENSE_DETAILS.ALLOCATED ua_stats_defn Calculation: DEF1000115 not found: no thrown error 9/10/2019 0:34 Error java.lang.NullPointerException: java.lang.NullPointerException: 9/10/2019 0:30 Error cmdb_metadata : Found duplicate cmdb_rel_type records with name: Master of::Stack Member of having sys_ids: 357afff213a21300f39f721a6144b076, c8c685710b22130005d90d2835673aa8: no thrown error 9/10/2019 0:30 Error cmdb_metadata : Found duplicate cmdb_rel_type records with name: Master of::Stack Member of having sys_ids: 357afff213a21300f39f721a6144b076, c8c685710b22130005d90d2835673aa8: no thrown error 9/10/2019 0:03 Error UATablePkgOverrideHandler: Could not find the package with source com.snc.problem: no thrown error 9/10/2019 0:03 Error UATablePkgOverrideHandler: Could not find the package with source com.snc.incident: no thrown error 9/10/2019 0:00 Error [code]Canceled discovery of <a href="discovery_schedule.do?sys_id=71c932b1db5aa3403f737afc0f96195a"><u>SSGA Windows Active Servers</u></a>. Already at maximum number of active 'Scheduled' invocations (3) for a given schedule[/code]

Can someone please help me achieve or rectify my config file ?

Thanks a million in advance.