Parsing Problems

Tags:

#1 abasha 4 years ago

Hello All,

I have a huge .csv file, this contains logs from Service Now instance. I have the following nxlog configuration file. But when i run the parser, error file i generate exceeds more than 1 GB. The source file itself is only about 225 MB.

Please set the ROOT to the folder your nxlog was installed into,

otherwise it will not start.

#define ROOT C:\Program Files\nxlog define ROOT C:\Program Files (x86)\nxlog

Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data LogFile %ROOT%\data\nxlog.log

<Extension multiline> Module xm_multiline HeaderLine /^\d{1,2}/\d{1,2}/\d{4}\s/ </Extension>

<Extension json> Module xm_json </Extension>

<Extension csv> Module xm_csv Fields $Created,$Level,$Message,$Source,$CreatedBy FieldTypes string, string, string, string, string </Extension>

<Extension syslog> Module xm_syslog </Extension>

<Input eventlog> Module im_msvistalog ReadFromLast TRUE SavePos TRUE Query <QueryList>
<Query Id="0">
<Select Path="Security">[System[(EventID=4768)]]</Select>
<Select Path="Security">[System[(EventID=4769)]]</Select>
<Select Path="Security">[System[(EventID=4771)]]</Select>
<Select Path="Security">[System[(EventID=4624)]]</Select>
<Select Path="Security">[System[(EventID=4625)]]</Select>
<Select Path="Security">[System[(EventID=4634)]]</Select>
<Select Path="Security">[System[(EventID=4647)]]</Select>
<Select Path="Security">[System[(EventID=4648)]]</Select>
<Select Path="Security">[System[(EventID=4656)]]</Select>
<Select Path="Security">[System[(EventID=4719)]]</Select>
<Select Path="Security">[System[(EventID=4720)]]</Select>
<Select Path="Security">[System[(EventID=4722)]]</Select>
<Select Path="Security">[System[(EventID=4723)]]</Select>
<Select Path="Security">[System[(EventID=4724)]]</Select>
<Select Path="Security">[System[(EventID=4725)]]</Select>
<Select Path="Security">[System[(EventID=4726)]]</Select>
<Select Path="Security">[System[(EventID=4727)]]</Select>
<Select Path="Security">[System[(EventID=4728)]]</Select>
<Select Path="Security">[System[(EventID=4729)]]</Select>
<Select Path="Security">[System[(EventID=4730)]]</Select>
<Select Path="Security">[System[(EventID=4731)]]</Select>
<Select Path="Security">[System[(EventID=4732)]]</Select>
<Select Path="Security">[System[(EventID=4733)]]</Select>
<Select Path="Security">[System[(EventID=4734)]]</Select>
<Select Path="Security">[System[(EventID=4735)]]</Select>
<Select Path="Security">[System[(EventID=4737)]]</Select>
<Select Path="Security">[System[(EventID=4738)]]</Select>
<Select Path="Security">[System[(EventID=4739)]]</Select>
<Select Path="Security">[System[(EventID=4741)]]</Select>
<Select Path="Security">[System[(EventID=4742)]]</Select>
<Select Path="Security">[System[(EventID=4743)]]</Select>
<Select Path="System">[System[(EventID=7036)]]</Select>
<Select Path="Application">[System[(EventID=18454)]]</Select>
<Select Path="Application">[System[(EventID=18456)]]</Select>
</Query>
</QueryList> Exec to_json(); </Input>

<Input filein> Module im_file File 'e:\ServiceNow\agent\export\snow_log.csv'

InputType       multiline
ReadFromLast    FALSE
SavePos         FALSE
&lt;Exec&gt;
    # Ignore top line
    if $raw_event =~ /Created,Level,Message,Source,Created by/ drop();
	if $raw_event =~ /Warning/ drop();
	if $raw_event =~ /Information/ drop();
	
    # Convert Newline and Tab to printed character
    #$raw_event =~ s/\R/\\r\\n/g;
    #$raw_event =~ s/\t/\\t/g;
	
	$raw_event = replace($raw_event,&quot;\n&quot;, &quot; &quot;);
	$raw_event = replace($raw_event,&quot;\r&quot;, &quot; &quot;);
	$raw_event = replace($raw_event,&quot;\t&quot;, &quot; &quot;);
			
	$SourceName = 'SNOWLogs';
    # Parse $raw_event as CSV
    csv-&gt;parse_csv();

    # Convert to JSON
    to_json();
&lt;/Exec&gt;

</Input>

<Output fileout> Module om_tcp Host logger Port 5140 #Exec to_syslog_bsd(); </Output>

<Output out> Module om_tcp Host logger Port 5140 </Output>

<Route r1> Path eventlog => out </Route>

<Route parse_xml> Path filein => fileout </Route>

For few lines it reads the data properly, but in some lines, it does not read the complete data. I am also trying to drop off unwanted data like information or warning, just to ensure i collect only Error information. But still it does not help. Error information from the file is very limited, so that I can reduce the amount of data to be ingested into ELK.

Sample of Error messages as follows: Created Level Message 9/10/2019 3:00 Error java.lang.NullPointerException: java.lang.NullPointerException: 9/10/2019 1:07 Error java.lang.NullPointerException: java.lang.NullPointerException: 9/10/2019 1:04 Error [code]Failed Exploring CI Pattern, Pattern name: <b>Docker Pattern</b>, Process ID: <b>12887</b>, To Check Pattern Log Press <a href="$sw_horizontal_discovery_log.do?discoLogId=aeeb6a6d1b33fb40db5e43b4bd4bcb5a&ipAddress=10.144.112.51&pid=12887&preExecution=&host_sys_id=d3fd5bff87e04504065e00f509434dc2&host_name=dm01db02.ga.ssga.root&patternId=dd15665a7fe022004e83e2065f2a0c57&patternName=Docker Pattern&patternType=1&isCloud=false"><u><b>Here</b></u></a>[/code] 9/10/2019 1:04 Error [code]Failed Exploring CI Pattern, Pattern name: <b>Docker Pattern</b>, Process ID: <b>12841</b>, To Check Pattern Log Press <a href="$sw_horizontal_discovery_log.do?discoLogId=aeeb6a6d1b33fb40db5e43b4bd4bcb11&ipAddress=10.145.112.57&pid=12841&preExecution=&host_sys_id=9ac8ef3887bc0904065e00f509434d22&host_name=dm02db08.ga.ssga.root&patternId=dd15665a7fe022004e83e2065f2a0c57&patternName=Docker Pattern&patternType=1&isCloud=false"><u><b>Here</b></u></a>[/code] 9/10/2019 1:04 Error [code]Failed Exploring CI Pattern, Pattern name: <b>Docker Pattern</b>, Process ID: <b>13373</b>, To Check Pattern Log Press <a href="$sw_horizontal_discovery_log.do?discoLogId=eeeb6a6d1b33fb40db5e43b4bd4bcb41&ipAddress=10.145.112.51&pid=13373&preExecution=&host_sys_id=ca716bb387244504065e00f509434dd6&host_name=dm02db02.ga.ssga.root&patternId=dd15665a7fe022004e83e2065f2a0c57&patternName=Docker Pattern&patternType=1&isCloud=false"><u><b>Here</b></u></a>[/code] 9/10/2019 1:04 Error [code]Failed Exploring CI Pattern, Pattern name: <b>Docker Pattern</b>, Process ID: <b>13328</b>, To Check Pattern Log Press <a href="$sw_horizontal_discovery_log.do?discoLogId=acebe6ad1bff7f404d41dd7edd4bcb1f&ipAddress=10.145.112.54&pid=13328&preExecution=&host_sys_id=7e912fb387244504065e00f509434d8c&host_name=dm02db05.ga.ssga.root&patternId=dd15665a7fe022004e83e2065f2a0c57&patternName=Docker Pattern&patternType=1&isCloud=false"><u><b>Here</b></u></a>[/code] 9/10/2019 1:04 Error [code]Failed Exploring CI Pattern, Pattern name: <b>Docker Pattern</b>, Process ID: <b>12911</b>, To Check Pattern Log Press <a href="$sw_horizontal_discovery_log.do?discoLogId=80eb2a6d1b33fb40db5e43b4bd4bcb88&ipAddress=10.144.112.56&pid=12911&preExecution=&host_sys_id=964e9fff87e04504065e00f509434d5f&host_name=dm01db07.ga.ssga.root&patternId=dd15665a7fe022004e83e2065f2a0c57&patternName=Docker Pattern&patternType=1&isCloud=false"><u><b>Here</b></u></a>[/code] 9/10/2019 1:04 Error [code]Failed Exploring CI Pattern, Pattern name: <b>Docker Pattern</b>, Process ID: <b>12899</b>, To Check Pattern Log Press <a href="$sw_horizontal_discovery_log.do?discoLogId=40eb2a6d1b33fb40db5e43b4bd4bcbc2&ipAddress=10.144.112.53&pid=12899&preExecution=&host_sys_id=391e5bff87e04504065e00f509434d3e&host_name=dm01db04.ga.ssga.root&patternId=dd15665a7fe022004e83e2065f2a0c57&patternName=Docker Pattern&patternType=1&isCloud=false"><u><b>Here</b></u></a>[/code] 9/10/2019 1:04 Error [code]Failed Exploring CI Pattern, Pattern name: <b>Docker Pattern</b>, Process ID: <b>13264</b>, To Check Pattern Log Press <a href="$sw_horizontal_discovery_log.do?discoLogId=48eb2a6d1b33fb40db5e43b4bd4bcb6a&ipAddress=10.145.112.56&pid=13264&preExecution=&host_sys_id=f0b1afb387244504065e00f509434df6&host_name=dm02db07.ga.ssga.root&patternId=dd15665a7fe022004e83e2065f2a0c57&patternName=Docker Pattern&patternType=1&isCloud=false"><u><b>Here</b></u></a>[/code] 9/10/2019 1:04 Error [code]Failed Exploring CI Pattern, Pattern name: <b>Docker Pattern</b>, Process ID: <b>12879</b>, To Check Pattern Log Press <a href="$sw_horizontal_discovery_log.do?discoLogId=44eb2a6d1b33fb40db5e43b4bd4bcbf4&ipAddress=10.144.112.50&pid=12879&preExecution=&host_sys_id=6cfddfbb87e04504065e00f509434d75&host_name=dm01db01.ga.ssga.root&patternId=dd15665a7fe022004e83e2065f2a0c57&patternName=Docker Pattern&patternType=1&isCloud=false"><u><b>Here</b></u></a>[/code] 9/10/2019 1:04 Error [code]Failed Exploring CI Pattern, Pattern name: <b>Docker Pattern</b>, Process ID: <b>13267</b>, To Check Pattern Log Press <a href="$sw_horizontal_discovery_log.do?discoLogId=4adba2ad1bff7f404d41dd7edd4bcbb1&ipAddress=10.145.112.55&pid=13267&preExecution=&host_sys_id=19a12fb387244504065e00f509434d28&host_name=dm02db06.ga.ssga.root&patternId=dd15665a7fe022004e83e2065f2a0c57&patternName=Docker Pattern&patternType=1&isCloud=false"><u><b>Here</b></u></a>[/code] 9/10/2019 1:04 Error [code]Failed Exploring CI Pattern, Pattern name: <b>Docker Pattern</b>, Process ID: <b>12901</b>, To Check Pattern Log Press <a href="$sw_horizontal_discovery_log.do?discoLogId=cedba2ad1bff7f404d41dd7edd4bcb90&ipAddress=10.144.112.57&pid=12901&preExecution=&host_sys_id=665edfbf87e04504065e00f509434d29&host_name=dm01db08.ga.ssga.root&patternId=dd15665a7fe022004e83e2065f2a0c57&patternName=Docker Pattern&patternType=1&isCloud=false"><u><b>Here</b></u></a>[/code] 9/10/2019 1:04 Error [code]Failed Exploring CI Pattern, Pattern name: <b>Docker Pattern</b>, Process ID: <b>13323</b>, To Check Pattern Log Press <a href="$sw_horizontal_discovery_log.do?discoLogId=cadbae6d1bff7f404d41dd7edd4bcb7b&ipAddress=10.145.112.53&pid=13323&preExecution=&host_sys_id=10916b7387244504065e00f509434d22&host_name=dm02db04.ga.ssga.root&patternId=dd15665a7fe022004e83e2065f2a0c57&patternName=Docker Pattern&patternType=1&isCloud=false"><u><b>Here</b></u></a>[/code] 9/10/2019 1:04 Error [code]Failed Exploring CI Pattern, Pattern name: <b>Docker Pattern</b>, Process ID: <b>13312</b>, To Check Pattern Log Press <a href="$sw_horizontal_discovery_log.do?discoLogId=fbcbeead1b377f40276510e4bd4bcbd2&ipAddress=10.145.112.50&pid=13312&preExecution=&host_sys_id=d7616bb387244504065e00f509434dd3&host_name=dm02db01.ga.ssga.root&patternId=dd15665a7fe022004e83e2065f2a0c57&patternName=Docker Pattern&patternType=1&isCloud=false"><u><b>Here</b></u></a>[/code] 9/10/2019 1:04 Error [code]Failed Exploring CI Pattern, Pattern name: <b>Docker Pattern</b>, Process ID: <b>12891</b>, To Check Pattern Log Press <a href="$sw_horizontal_discovery_log.do?discoLogId=b7cbeead1b377f40276510e4bd4bcb97&ipAddress=10.144.112.54&pid=12891&preExecution=&host_sys_id=642edbff87e04504065e00f509434dd6&host_name=dm01db05.ga.ssga.root&patternId=dd15665a7fe022004e83e2065f2a0c57&patternName=Docker Pattern&patternType=1&isCloud=false"><u><b>Here</b></u></a>[/code] 9/10/2019 1:04 Error [code]Failed Exploring CI Pattern, Pattern name: <b>Docker Pattern</b>, Process ID: <b>13255</b>, To Check Pattern Log Press <a href="$sw_horizontal_discovery_log.do?discoLogId=23cbae6d1bff7f404d41dd7edd4bcb6c&ipAddress=10.145.112.52&pid=13255&preExecution=&host_sys_id=d581ebb387244504065e00f509434da2&host_name=dm02db03.ga.ssga.root&patternId=dd15665a7fe022004e83e2065f2a0c57&patternName=Docker Pattern&patternType=1&isCloud=false"><u><b>Here</b></u></a>[/code] 9/10/2019 1:04 Error [code]Failed Exploring CI Pattern, Pattern name: <b>Docker Pattern</b>, Process ID: <b>13008</b>, To Check Pattern Log Press <a href="$sw_horizontal_discovery_log.do?discoLogId=47cb266d1b33fb40db5e43b4bd4bcb6c&ipAddress=10.144.112.52&pid=13008&preExecution=&host_sys_id=fe0ed7ff87e04504065e00f509434dd8&host_name=dm01db03.ga.ssga.root&patternId=dd15665a7fe022004e83e2065f2a0c57&patternName=Docker Pattern&patternType=1&isCloud=false"><u><b>Here</b></u></a>[/code] 9/10/2019 1:04 Error [code]Failed Exploring CI Pattern, Pattern name: <b>Docker Pattern</b>, Process ID: <b>12885</b>, To Check Pattern Log Press <a href="$sw_horizontal_discovery_log.do?discoLogId=c7cb266d1b33fb40db5e43b4bd4bcb8c&ipAddress=10.144.112.55&pid=12885&preExecution=&host_sys_id=a03e1fff87e04504065e00f509434d97&host_name=dm01db06.ga.ssga.root&patternId=dd15665a7fe022004e83e2065f2a0c57&patternName=Docker Pattern&patternType=1&isCloud=false"><u><b>Here</b></u></a>[/code] 9/10/2019 1:03 Error java.lang.NullPointerException: java.lang.NullPointerException: 9/10/2019 1:03 Error java.lang.NullPointerException: java.lang.NullPointerException: 9/10/2019 1:02 Error java.lang.NullPointerException: java.lang.NullPointerException: 9/10/2019 1:01 Error java.lang.NullPointerException: java.lang.NullPointerException: 9/10/2019 1:00 Error cmdb_metadata : Found duplicate cmdb_rel_type records with name: Master of::Stack Member of having sys_ids: 357afff213a21300f39f721a6144b076, c8c685710b22130005d90d2835673aa8: no thrown error 9/10/2019 1:00 Error java.lang.NullPointerException: java.lang.NullPointerException: 9/10/2019 1:00 Error LICENSE_DETAILS.ALLOCATED ua_stats_defn Calculation: DEF1000115 not found: no thrown error 9/10/2019 0:34 Error java.lang.NullPointerException: java.lang.NullPointerException: 9/10/2019 0:30 Error cmdb_metadata : Found duplicate cmdb_rel_type records with name: Master of::Stack Member of having sys_ids: 357afff213a21300f39f721a6144b076, c8c685710b22130005d90d2835673aa8: no thrown error 9/10/2019 0:30 Error cmdb_metadata : Found duplicate cmdb_rel_type records with name: Master of::Stack Member of having sys_ids: 357afff213a21300f39f721a6144b076, c8c685710b22130005d90d2835673aa8: no thrown error 9/10/2019 0:03 Error UATablePkgOverrideHandler: Could not find the package with source com.snc.problem: no thrown error 9/10/2019 0:03 Error UATablePkgOverrideHandler: Could not find the package with source com.snc.incident: no thrown error 9/10/2019 0:00 Error [code]Canceled discovery of <a href="discovery_schedule.do?sys_id=71c932b1db5aa3403f737afc0f96195a"><u>SSGA Windows Active Servers</u></a>. Already at maximum number of active 'Scheduled' invocations (3) for a given schedule[/code]

Can someone please help me achieve or rectify my config file ?

Thanks a million in advance.

Permalink

#2 Zhengshi Nxlog ✓ 4 years ago (Last updated 4 years ago )

#1 abasha

Hello All, I have a huge .csv file, this contains logs from Service Now instance. I have the following nxlog configuration file. But when i run the parser, error file i generate exceeds more than 1 GB. The source file itself is only about 225 MB. Please set the ROOT to the folder your nxlog was installed into, otherwise it will not start. #define ROOT C:\Program Files\nxlog define ROOT C:\Program Files (x86)\nxlog Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data LogFile %ROOT%\data\nxlog.log <Extension multiline> Module xm_multiline HeaderLine /^\d{1,2}/\d{1,2}/\d{4}\s/ </Extension> <Extension json> Module xm_json </Extension> <Extension csv> Module xm_csv Fields $Created,$Level,$Message,$Source,$CreatedBy FieldTypes string, string, string, string, string </Extension> <Extension syslog> Module xm_syslog </Extension> <Input eventlog> Module im_msvistalog ReadFromLast TRUE SavePos TRUE Query <QueryList> <Query Id="0"> <Select Path="Security">[System[(EventID=4768)]]</Select> <Select Path="Security">[System[(EventID=4769)]]</Select> <Select Path="Security">[System[(EventID=4771)]]</Select> <Select Path="Security">[System[(EventID=4624)]]</Select> <Select Path="Security">[System[(EventID=4625)]]</Select> <Select Path="Security">[System[(EventID=4634)]]</Select> <Select Path="Security">[System[(EventID=4647)]]</Select> <Select Path="Security">[System[(EventID=4648)]]</Select> <Select Path="Security">[System[(EventID=4656)]]</Select> <Select Path="Security">[System[(EventID=4719)]]</Select> <Select Path="Security">[System[(EventID=4720)]]</Select> <Select Path="Security">[System[(EventID=4722)]]</Select> <Select Path="Security">[System[(EventID=4723)]]</Select> <Select Path="Security">[System[(EventID=4724)]]</Select> <Select Path="Security">[System[(EventID=4725)]]</Select> <Select Path="Security">[System[(EventID=4726)]]</Select> <Select Path="Security">[System[(EventID=4727)]]</Select> <Select Path="Security">[System[(EventID=4728)]]</Select> <Select Path="Security">[System[(EventID=4729)]]</Select> <Select Path="Security">[System[(EventID=4730)]]</Select> <Select Path="Security">[System[(EventID=4731)]]</Select> <Select Path="Security">[System[(EventID=4732)]]</Select> <Select Path="Security">[System[(EventID=4733)]]</Select> <Select Path="Security">[System[(EventID=4734)]]</Select> <Select Path="Security">[System[(EventID=4735)]]</Select> <Select Path="Security">[System[(EventID=4737)]]</Select> <Select Path="Security">[System[(EventID=4738)]]</Select> <Select Path="Security">[System[(EventID=4739)]]</Select> <Select Path="Security">[System[(EventID=4741)]]</Select> <Select Path="Security">[System[(EventID=4742)]]</Select> <Select Path="Security">[System[(EventID=4743)]]</Select> <Select Path="System">[System[(EventID=7036)]]</Select> <Select Path="Application">[System[(EventID=18454)]]</Select> <Select Path="Application">[System[(EventID=18456)]]</Select> </Query> </QueryList> Exec to_json(); </Input> <Input filein> Module im_file File 'e:\ServiceNow\agent\export\snow_log.csv' InputType multiline ReadFromLast FALSE SavePos FALSE <Exec> # Ignore top line if $raw_event =~ /Created,Level,Message,Source,Created by/ drop(); if $raw_event =~ /Warning/ drop(); if $raw_event =~ /Information/ drop(); # Convert Newline and Tab to printed character #$raw_event =~ s/\R/\\r\\n/g; #$raw_event =~ s/\t/\\t/g; $raw_event = replace($raw_event,"\n", " "); $raw_event = replace($raw_event,"\r", " "); $raw_event = replace($raw_event,"\t", " "); $SourceName = 'SNOWLogs'; # Parse $raw_event as CSV csv->parse_csv(); # Convert to JSON to_json(); </Exec> </Input> <Output fileout> Module om_tcp Host logger Port 5140 #Exec to_syslog_bsd(); </Output> <Output out> Module om_tcp Host logger Port 5140 </Output> <Route r1> Path eventlog => out </Route> <Route parse_xml> Path filein => fileout </Route> For few lines it reads the data properly, but in some lines, it does not read the complete data. I am also trying to drop off unwanted data like information or warning, just to ensure i collect only Error information. But still it does not help. Error information from the file is very limited, so that I can reduce the amount of data to be ingested into ELK. Sample of Error messages as follows: Created Level Message 9/10/2019 3:00 Error java.lang.NullPointerException: java.lang.NullPointerException: 9/10/2019 1:07 Error java.lang.NullPointerException: java.lang.NullPointerException: 9/10/2019 1:04 Error [code]Failed Exploring CI Pattern, Pattern name: <b>Docker Pattern</b>, Process ID: <b>12887</b>, To Check Pattern Log Press <a href="$sw_horizontal_discovery_log.do?discoLogId=aeeb6a6d1b33fb40db5e43b4bd4bcb5a&ipAddress=10.144.112.51&pid=12887&preExecution=&host_sys_id=d3fd5bff87e04504065e00f509434dc2&host_name=dm01db02.ga.ssga.root&patternId=dd15665a7fe022004e83e2065f2a0c57&patternName=Docker Pattern&patternType=1&isCloud=false"><u><b>Here</b></u></a>[/code] 9/10/2019 1:04 Error [code]Failed Exploring CI Pattern, Pattern name: <b>Docker Pattern</b>, Process ID: <b>12841</b>, To Check Pattern Log Press <a href="$sw_horizontal_discovery_log.do?discoLogId=aeeb6a6d1b33fb40db5e43b4bd4bcb11&ipAddress=10.145.112.57&pid=12841&preExecution=&host_sys_id=9ac8ef3887bc0904065e00f509434d22&host_name=dm02db08.ga.ssga.root&patternId=dd15665a7fe022004e83e2065f2a0c57&patternName=Docker Pattern&patternType=1&isCloud=false"><u><b>Here</b></u></a>[/code] 9/10/2019 1:04 Error [code]Failed Exploring CI Pattern, Pattern name: <b>Docker Pattern</b>, Process ID: <b>13373</b>, To Check Pattern Log Press <a href="$sw_horizontal_discovery_log.do?discoLogId=eeeb6a6d1b33fb40db5e43b4bd4bcb41&ipAddress=10.145.112.51&pid=13373&preExecution=&host_sys_id=ca716bb387244504065e00f509434dd6&host_name=dm02db02.ga.ssga.root&patternId=dd15665a7fe022004e83e2065f2a0c57&patternName=Docker Pattern&patternType=1&isCloud=false"><u><b>Here</b></u></a>[/code] 9/10/2019 1:04 Error [code]Failed Exploring CI Pattern, Pattern name: <b>Docker Pattern</b>, Process ID: <b>13328</b>, To Check Pattern Log Press <a href="$sw_horizontal_discovery_log.do?discoLogId=acebe6ad1bff7f404d41dd7edd4bcb1f&ipAddress=10.145.112.54&pid=13328&preExecution=&host_sys_id=7e912fb387244504065e00f509434d8c&host_name=dm02db05.ga.ssga.root&patternId=dd15665a7fe022004e83e2065f2a0c57&patternName=Docker Pattern&patternType=1&isCloud=false"><u><b>Here</b></u></a>[/code] 9/10/2019 1:04 Error [code]Failed Exploring CI Pattern, Pattern name: <b>Docker Pattern</b>, Process ID: <b>12911</b>, To Check Pattern Log Press <a href="$sw_horizontal_discovery_log.do?discoLogId=80eb2a6d1b33fb40db5e43b4bd4bcb88&ipAddress=10.144.112.56&pid=12911&preExecution=&host_sys_id=964e9fff87e04504065e00f509434d5f&host_name=dm01db07.ga.ssga.root&patternId=dd15665a7fe022004e83e2065f2a0c57&patternName=Docker Pattern&patternType=1&isCloud=false"><u><b>Here</b></u></a>[/code] 9/10/2019 1:04 Error [code]Failed Exploring CI Pattern, Pattern name: <b>Docker Pattern</b>, Process ID: <b>12899</b>, To Check Pattern Log Press <a href="$sw_horizontal_discovery_log.do?discoLogId=40eb2a6d1b33fb40db5e43b4bd4bcbc2&ipAddress=10.144.112.53&pid=12899&preExecution=&host_sys_id=391e5bff87e04504065e00f509434d3e&host_name=dm01db04.ga.ssga.root&patternId=dd15665a7fe022004e83e2065f2a0c57&patternName=Docker Pattern&patternType=1&isCloud=false"><u><b>Here</b></u></a>[/code] 9/10/2019 1:04 Error [code]Failed Exploring CI Pattern, Pattern name: <b>Docker Pattern</b>, Process ID: <b>13264</b>, To Check Pattern Log Press <a href="$sw_horizontal_discovery_log.do?discoLogId=48eb2a6d1b33fb40db5e43b4bd4bcb6a&ipAddress=10.145.112.56&pid=13264&preExecution=&host_sys_id=f0b1afb387244504065e00f509434df6&host_name=dm02db07.ga.ssga.root&patternId=dd15665a7fe022004e83e2065f2a0c57&patternName=Docker Pattern&patternType=1&isCloud=false"><u><b>Here</b></u></a>[/code] 9/10/2019 1:04 Error [code]Failed Exploring CI Pattern, Pattern name: <b>Docker Pattern</b>, Process ID: <b>12879</b>, To Check Pattern Log Press <a href="$sw_horizontal_discovery_log.do?discoLogId=44eb2a6d1b33fb40db5e43b4bd4bcbf4&ipAddress=10.144.112.50&pid=12879&preExecution=&host_sys_id=6cfddfbb87e04504065e00f509434d75&host_name=dm01db01.ga.ssga.root&patternId=dd15665a7fe022004e83e2065f2a0c57&patternName=Docker Pattern&patternType=1&isCloud=false"><u><b>Here</b></u></a>[/code] 9/10/2019 1:04 Error [code]Failed Exploring CI Pattern, Pattern name: <b>Docker Pattern</b>, Process ID: <b>13267</b>, To Check Pattern Log Press <a href="$sw_horizontal_discovery_log.do?discoLogId=4adba2ad1bff7f404d41dd7edd4bcbb1&ipAddress=10.145.112.55&pid=13267&preExecution=&host_sys_id=19a12fb387244504065e00f509434d28&host_name=dm02db06.ga.ssga.root&patternId=dd15665a7fe022004e83e2065f2a0c57&patternName=Docker Pattern&patternType=1&isCloud=false"><u><b>Here</b></u></a>[/code] 9/10/2019 1:04 Error [code]Failed Exploring CI Pattern, Pattern name: <b>Docker Pattern</b>, Process ID: <b>12901</b>, To Check Pattern Log Press <a href="$sw_horizontal_discovery_log.do?discoLogId=cedba2ad1bff7f404d41dd7edd4bcb90&ipAddress=10.144.112.57&pid=12901&preExecution=&host_sys_id=665edfbf87e04504065e00f509434d29&host_name=dm01db08.ga.ssga.root&patternId=dd15665a7fe022004e83e2065f2a0c57&patternName=Docker Pattern&patternType=1&isCloud=false"><u><b>Here</b></u></a>[/code] 9/10/2019 1:04 Error [code]Failed Exploring CI Pattern, Pattern name: <b>Docker Pattern</b>, Process ID: <b>13323</b>, To Check Pattern Log Press <a href="$sw_horizontal_discovery_log.do?discoLogId=cadbae6d1bff7f404d41dd7edd4bcb7b&ipAddress=10.145.112.53&pid=13323&preExecution=&host_sys_id=10916b7387244504065e00f509434d22&host_name=dm02db04.ga.ssga.root&patternId=dd15665a7fe022004e83e2065f2a0c57&patternName=Docker Pattern&patternType=1&isCloud=false"><u><b>Here</b></u></a>[/code] 9/10/2019 1:04 Error [code]Failed Exploring CI Pattern, Pattern name: <b>Docker Pattern</b>, Process ID: <b>13312</b>, To Check Pattern Log Press <a href="$sw_horizontal_discovery_log.do?discoLogId=fbcbeead1b377f40276510e4bd4bcbd2&ipAddress=10.145.112.50&pid=13312&preExecution=&host_sys_id=d7616bb387244504065e00f509434dd3&host_name=dm02db01.ga.ssga.root&patternId=dd15665a7fe022004e83e2065f2a0c57&patternName=Docker Pattern&patternType=1&isCloud=false"><u><b>Here</b></u></a>[/code] 9/10/2019 1:04 Error [code]Failed Exploring CI Pattern, Pattern name: <b>Docker Pattern</b>, Process ID: <b>12891</b>, To Check Pattern Log Press <a href="$sw_horizontal_discovery_log.do?discoLogId=b7cbeead1b377f40276510e4bd4bcb97&ipAddress=10.144.112.54&pid=12891&preExecution=&host_sys_id=642edbff87e04504065e00f509434dd6&host_name=dm01db05.ga.ssga.root&patternId=dd15665a7fe022004e83e2065f2a0c57&patternName=Docker Pattern&patternType=1&isCloud=false"><u><b>Here</b></u></a>[/code] 9/10/2019 1:04 Error [code]Failed Exploring CI Pattern, Pattern name: <b>Docker Pattern</b>, Process ID: <b>13255</b>, To Check Pattern Log Press <a href="$sw_horizontal_discovery_log.do?discoLogId=23cbae6d1bff7f404d41dd7edd4bcb6c&ipAddress=10.145.112.52&pid=13255&preExecution=&host_sys_id=d581ebb387244504065e00f509434da2&host_name=dm02db03.ga.ssga.root&patternId=dd15665a7fe022004e83e2065f2a0c57&patternName=Docker Pattern&patternType=1&isCloud=false"><u><b>Here</b></u></a>[/code] 9/10/2019 1:04 Error [code]Failed Exploring CI Pattern, Pattern name: <b>Docker Pattern</b>, Process ID: <b>13008</b>, To Check Pattern Log Press <a href="$sw_horizontal_discovery_log.do?discoLogId=47cb266d1b33fb40db5e43b4bd4bcb6c&ipAddress=10.144.112.52&pid=13008&preExecution=&host_sys_id=fe0ed7ff87e04504065e00f509434dd8&host_name=dm01db03.ga.ssga.root&patternId=dd15665a7fe022004e83e2065f2a0c57&patternName=Docker Pattern&patternType=1&isCloud=false"><u><b>Here</b></u></a>[/code] 9/10/2019 1:04 Error [code]Failed Exploring CI Pattern, Pattern name: <b>Docker Pattern</b>, Process ID: <b>12885</b>, To Check Pattern Log Press <a href="$sw_horizontal_discovery_log.do?discoLogId=c7cb266d1b33fb40db5e43b4bd4bcb8c&ipAddress=10.144.112.55&pid=12885&preExecution=&host_sys_id=a03e1fff87e04504065e00f509434d97&host_name=dm01db06.ga.ssga.root&patternId=dd15665a7fe022004e83e2065f2a0c57&patternName=Docker Pattern&patternType=1&isCloud=false"><u><b>Here</b></u></a>[/code] 9/10/2019 1:03 Error java.lang.NullPointerException: java.lang.NullPointerException: 9/10/2019 1:03 Error java.lang.NullPointerException: java.lang.NullPointerException: 9/10/2019 1:02 Error java.lang.NullPointerException: java.lang.NullPointerException: 9/10/2019 1:01 Error java.lang.NullPointerException: java.lang.NullPointerException: 9/10/2019 1:00 Error cmdb_metadata : Found duplicate cmdb_rel_type records with name: Master of::Stack Member of having sys_ids: 357afff213a21300f39f721a6144b076, c8c685710b22130005d90d2835673aa8: no thrown error 9/10/2019 1:00 Error java.lang.NullPointerException: java.lang.NullPointerException: 9/10/2019 1:00 Error LICENSE_DETAILS.ALLOCATED ua_stats_defn Calculation: DEF1000115 not found: no thrown error 9/10/2019 0:34 Error java.lang.NullPointerException: java.lang.NullPointerException: 9/10/2019 0:30 Error cmdb_metadata : Found duplicate cmdb_rel_type records with name: Master of::Stack Member of having sys_ids: 357afff213a21300f39f721a6144b076, c8c685710b22130005d90d2835673aa8: no thrown error 9/10/2019 0:30 Error cmdb_metadata : Found duplicate cmdb_rel_type records with name: Master of::Stack Member of having sys_ids: 357afff213a21300f39f721a6144b076, c8c685710b22130005d90d2835673aa8: no thrown error 9/10/2019 0:03 Error UATablePkgOverrideHandler: Could not find the package with source com.snc.problem: no thrown error 9/10/2019 0:03 Error UATablePkgOverrideHandler: Could not find the package with source com.snc.incident: no thrown error 9/10/2019 0:00 Error [code]Canceled discovery of <a href="discovery_schedule.do?sys_id=71c932b1db5aa3403f737afc0f96195a"><u>SSGA Windows Active Servers</u></a>. Already at maximum number of active 'Scheduled' invocations (3) for a given schedule[/code] Can someone please help me achieve or rectify my config file ? Thanks a million in advance.

It is sometimes easier to trim down config and tests to verify everything is working the way you are expecting to begin with.

Config for CSV:
Fields $Created,$Level,$Message,$Source,$CreatedBy
Top of file:
Created Level Message

Just to verify, are all of these events supposed to have $Source and $CreatedBy as well? I am unsure where the break is supposed to be.
By default, the xm_csv will use commas as the field separator.

Example Log Entry:

9/10/2019 1:04 Error [code]Failed Exploring CI Pattern, Pattern name: <b>Docker Pattern</b>, Process ID: <b>12879</b>, To Check Pattern Log Press <a href="$sw_horizontal_discovery_log.do?discoLogId=44eb2a6d1b33fb40db5e43b4bd4bcbf4&amp;ipAddress=10.144.112.50&amp;pid=12879&amp;preExecution=&amp;host_sys_id=6cfddfbb87e04504065e00f509434d75&amp;host_name=dm01db01.ga.ssga.root&amp;patternId=dd15665a7fe022004e83e2065f2a0c57&amp;patternName=Docker Pattern&amp;patternType=1&amp;isCloud=false"><u><b>Here</b></u></a>[/code]

JSON Output with PrettyPrint True for ease of reading:

{
  "EventReceivedTime": "2019-09-10T17:16:21.120782-05:00",
  "SourceModuleName": "filein",
  "SourceModuleType": "im_file",
  "SourceName": "SNOWLogs",
  "Created": "9/10/2019 1:04 Error [code]Failed Exploring CI Pattern",
  "Level": "Pattern name: <b>Docker Pattern</b>",
  "Message": "Process ID: <b>12879</b>",
  "Source": "To Check Pattern Log Press <a href=\"$sw_horizontal_discovery_log.do?discoLogId=44eb2a6d1b33fb40db5e43b4bd4bcbf4&amp;ipAddress=10.144.112.50&amp;pid=12879&amp;preExecution=&amp;host_sys_id=6cfddfbb87e04504065e00f509434d75&amp;host_name=dm01db01.ga.ssga.root&amp;patternId=dd15665a7fe022004e83e2065f2a0c57&amp;patternName=Docker Pattern&amp;patternType=1&amp;isCloud=false\"><u><b>Here</b></u></a>[/code]"
}

Secondary question, the $Created field, is that just supposed to be the date and time? Type may be better with datetime.

As far as the size of the document goes, I see you are sending to an external server for fileout instance. You also have SavePos and ReadFromLast set to False. With these settings, are you running the NXLog service multiple times? i.e. restarting the service. If so, are you clearing the output file in between? If not, you will be appending to that log the entire source file repeatedly.