0
answers

Hello All,

I have a huge .csv file, this contains logs from Service Now instance. I have the following nxlog configuration file. But when i run the parser, error file i generate exceeds more than 1 GB. The source file itself is only about 225 MB.

## Please set the ROOT to the folder your nxlog was installed into,
## otherwise it will not start.

#define ROOT C:\Program Files\nxlog
define ROOT C:\Program Files (x86)\nxlog

Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log

<Extension multiline>
Module xm_multiline
HeaderLine /^\d{1,2}\/\d{1,2}\/\d{4}\s/
</Extension>

<Extension json>
Module xm_json
</Extension>

<Extension csv>
Module xm_csv
Fields $Created,$Level,$Message,$Source,$CreatedBy
FieldTypes string, string, string, string, string
</Extension>

<Extension syslog>
Module xm_syslog
</Extension>

<Input eventlog>
Module im_msvistalog
ReadFromLast TRUE
SavePos TRUE
Query <QueryList> \
<Query Id="0"> \
<Select Path="Security">*[System[(EventID=4768)]]</Select> \
<Select Path="Security">*[System[(EventID=4769)]]</Select> \
<Select Path="Security">*[System[(EventID=4771)]]</Select> \
<Select Path="Security">*[System[(EventID=4624)]]</Select> \
<Select Path="Security">*[System[(EventID=4625)]]</Select> \
<Select Path="Security">*[System[(EventID=4634)]]</Select> \
<Select Path="Security">*[System[(EventID=4647)]]</Select> \
<Select Path="Security">*[System[(EventID=4648)]]</Select> \
<Select Path="Security">*[System[(EventID=4656)]]</Select> \
<Select Path="Security">*[System[(EventID=4719)]]</Select> \
<Select Path="Security">*[System[(EventID=4720)]]</Select> \
<Select Path="Security">*[System[(EventID=4722)]]</Select> \
<Select Path="Security">*[System[(EventID=4723)]]</Select> \
<Select Path="Security">*[System[(EventID=4724)]]</Select> \
<Select Path="Security">*[System[(EventID=4725)]]</Select> \
<Select Path="Security">*[System[(EventID=4726)]]</Select> \
<Select Path="Security">*[System[(EventID=4727)]]</Select> \
<Select Path="Security">*[System[(EventID=4728)]]</Select> \
<Select Path="Security">*[System[(EventID=4729)]]</Select> \
<Select Path="Security">*[System[(EventID=4730)]]</Select> \
<Select Path="Security">*[System[(EventID=4731)]]</Select> \
<Select Path="Security">*[System[(EventID=4732)]]</Select> \
<Select Path="Security">*[System[(EventID=4733)]]</Select> \
<Select Path="Security">*[System[(EventID=4734)]]</Select> \
<Select Path="Security">*[System[(EventID=4735)]]</Select> \
<Select Path="Security">*[System[(EventID=4737)]]</Select> \
<Select Path="Security">*[System[(EventID=4738)]]</Select> \
<Select Path="Security">*[System[(EventID=4739)]]</Select> \
<Select Path="Security">*[System[(EventID=4741)]]</Select> \
<Select Path="Security">*[System[(EventID=4742)]]</Select> \
<Select Path="Security">*[System[(EventID=4743)]]</Select> \
<Select Path="System">*[System[(EventID=7036)]]</Select> \
<Select Path="Application">*[System[(EventID=18454)]]</Select> \
<Select Path="Application">*[System[(EventID=18456)]]</Select> \
</Query> \
</QueryList>
Exec to_json();
</Input>

<Input filein>
Module im_file
File 'e:\ServiceNow\agent\export\snow_log.csv'

InputType multiline
ReadFromLast FALSE
SavePos FALSE
<Exec>
# Ignore top line
if $raw_event =~ /Created,Level,Message,Source,Created by/ drop();
if $raw_event =~ /Warning/ drop();
if $raw_event =~ /Information/ drop();

# Convert Newline and Tab to printed character
#$raw_event =~ s/\R/\\r\\n/g;
#$raw_event =~ s/\t/\\t/g;

$raw_event = replace($raw_event,"\n", " ");
$raw_event = replace($raw_event,"\r", " ");
$raw_event = replace($raw_event,"\t", " ");

$SourceName = 'SNOWLogs';
# Parse $raw_event as CSV
csv->parse_csv();

# Convert to JSON
to_json();
</Exec>
</Input>

<Output fileout>
Module om_tcp
Host logger
Port 5140
#Exec to_syslog_bsd();
</Output>

<Output out>
Module om_tcp
Host logger
Port 5140
</Output>

<Route r1>
Path eventlog => out
</Route>

<Route parse_xml>
Path filein => fileout
</Route>

For few lines it reads the data properly, but in some lines, it does not read the complete data. I am also trying to drop off unwanted data like information or warning, just to ensure i collect only Error information. But still it does not help. Error information from the file is very limited, so that I can reduce the amount of data to be ingested into ELK.

Sample of Error messages as follows:
Created Level Message
9/10/2019 3:00 Error java.lang.NullPointerException: java.lang.NullPointerException:
9/10/2019 1:07 Error java.lang.NullPointerException: java.lang.NullPointerException:
9/10/2019 1:04 Error [code]Failed Exploring CI Pattern, Pattern name: <b>Docker Pattern</b>, Process ID: <b>12887</b>, To Check Pattern Log Press <a href="$sw_horizontal_discovery_log.do?discoLogId=aeeb6a6d1b33fb40db5e43b4bd4bcb5a&amp;ipAddress=10.144.112.51&amp;pid=12887&amp;preExecution=&amp;host_sys_id=d3fd5bff87e04504065e00f509434dc2&amp;host_name=dm01db02.ga.ssga.root&amp;patternId=dd15665a7fe022004e83e2065f2a0c57&amp;patternName=Docker Pattern&amp;patternType=1&amp;isCloud=false"><u><b>Here</b></u></a>[/code]
9/10/2019 1:04 Error [code]Failed Exploring CI Pattern, Pattern name: <b>Docker Pattern</b>, Process ID: <b>12841</b>, To Check Pattern Log Press <a href="$sw_horizontal_discovery_log.do?discoLogId=aeeb6a6d1b33fb40db5e43b4bd4bcb11&amp;ipAddress=10.145.112.57&amp;pid=12841&amp;preExecution=&amp;host_sys_id=9ac8ef3887bc0904065e00f509434d22&amp;host_name=dm02db08.ga.ssga.root&amp;patternId=dd15665a7fe022004e83e2065f2a0c57&amp;patternName=Docker Pattern&amp;patternType=1&amp;isCloud=false"><u><b>Here</b></u></a>[/code]
9/10/2019 1:04 Error [code]Failed Exploring CI Pattern, Pattern name: <b>Docker Pattern</b>, Process ID: <b>13373</b>, To Check Pattern Log Press <a href="$sw_horizontal_discovery_log.do?discoLogId=eeeb6a6d1b33fb40db5e43b4bd4bcb41&amp;ipAddress=10.145.112.51&amp;pid=13373&amp;preExecution=&amp;host_sys_id=ca716bb387244504065e00f509434dd6&amp;host_name=dm02db02.ga.ssga.root&amp;patternId=dd15665a7fe022004e83e2065f2a0c57&amp;patternName=Docker Pattern&amp;patternType=1&amp;isCloud=false"><u><b>Here</b></u></a>[/code]
9/10/2019 1:04 Error [code]Failed Exploring CI Pattern, Pattern name: <b>Docker Pattern</b>, Process ID: <b>13328</b>, To Check Pattern Log Press <a href="$sw_horizontal_discovery_log.do?discoLogId=acebe6ad1bff7f404d41dd7edd4bcb1f&amp;ipAddress=10.145.112.54&amp;pid=13328&amp;preExecution=&amp;host_sys_id=7e912fb387244504065e00f509434d8c&amp;host_name=dm02db05.ga.ssga.root&amp;patternId=dd15665a7fe022004e83e2065f2a0c57&amp;patternName=Docker Pattern&amp;patternType=1&amp;isCloud=false"><u><b>Here</b></u></a>[/code]
9/10/2019 1:04 Error [code]Failed Exploring CI Pattern, Pattern name: <b>Docker Pattern</b>, Process ID: <b>12911</b>, To Check Pattern Log Press <a href="$sw_horizontal_discovery_log.do?discoLogId=80eb2a6d1b33fb40db5e43b4bd4bcb88&amp;ipAddress=10.144.112.56&amp;pid=12911&amp;preExecution=&amp;host_sys_id=964e9fff87e04504065e00f509434d5f&amp;host_name=dm01db07.ga.ssga.root&amp;patternId=dd15665a7fe022004e83e2065f2a0c57&amp;patternName=Docker Pattern&amp;patternType=1&amp;isCloud=false"><u><b>Here</b></u></a>[/code]
9/10/2019 1:04 Error [code]Failed Exploring CI Pattern, Pattern name: <b>Docker Pattern</b>, Process ID: <b>12899</b>, To Check Pattern Log Press <a href="$sw_horizontal_discovery_log.do?discoLogId=40eb2a6d1b33fb40db5e43b4bd4bcbc2&amp;ipAddress=10.144.112.53&amp;pid=12899&amp;preExecution=&amp;host_sys_id=391e5bff87e04504065e00f509434d3e&amp;host_name=dm01db04.ga.ssga.root&amp;patternId=dd15665a7fe022004e83e2065f2a0c57&amp;patternName=Docker Pattern&amp;patternType=1&amp;isCloud=false"><u><b>Here</b></u></a>[/code]
9/10/2019 1:04 Error [code]Failed Exploring CI Pattern, Pattern name: <b>Docker Pattern</b>, Process ID: <b>13264</b>, To Check Pattern Log Press <a href="$sw_horizontal_discovery_log.do?discoLogId=48eb2a6d1b33fb40db5e43b4bd4bcb6a&amp;ipAddress=10.145.112.56&amp;pid=13264&amp;preExecution=&amp;host_sys_id=f0b1afb387244504065e00f509434df6&amp;host_name=dm02db07.ga.ssga.root&amp;patternId=dd15665a7fe022004e83e2065f2a0c57&amp;patternName=Docker Pattern&amp;patternType=1&amp;isCloud=false"><u><b>Here</b></u></a>[/code]
9/10/2019 1:04 Error [code]Failed Exploring CI Pattern, Pattern name: <b>Docker Pattern</b>, Process ID: <b>12879</b>, To Check Pattern Log Press <a href="$sw_horizontal_discovery_log.do?discoLogId=44eb2a6d1b33fb40db5e43b4bd4bcbf4&amp;ipAddress=10.144.112.50&amp;pid=12879&amp;preExecution=&amp;host_sys_id=6cfddfbb87e04504065e00f509434d75&amp;host_name=dm01db01.ga.ssga.root&amp;patternId=dd15665a7fe022004e83e2065f2a0c57&amp;patternName=Docker Pattern&amp;patternType=1&amp;isCloud=false"><u><b>Here</b></u></a>[/code]
9/10/2019 1:04 Error [code]Failed Exploring CI Pattern, Pattern name: <b>Docker Pattern</b>, Process ID: <b>13267</b>, To Check Pattern Log Press <a href="$sw_horizontal_discovery_log.do?discoLogId=4adba2ad1bff7f404d41dd7edd4bcbb1&amp;ipAddress=10.145.112.55&amp;pid=13267&amp;preExecution=&amp;host_sys_id=19a12fb387244504065e00f509434d28&amp;host_name=dm02db06.ga.ssga.root&amp;patternId=dd15665a7fe022004e83e2065f2a0c57&amp;patternName=Docker Pattern&amp;patternType=1&amp;isCloud=false"><u><b>Here</b></u></a>[/code]
9/10/2019 1:04 Error [code]Failed Exploring CI Pattern, Pattern name: <b>Docker Pattern</b>, Process ID: <b>12901</b>, To Check Pattern Log Press <a href="$sw_horizontal_discovery_log.do?discoLogId=cedba2ad1bff7f404d41dd7edd4bcb90&amp;ipAddress=10.144.112.57&amp;pid=12901&amp;preExecution=&amp;host_sys_id=665edfbf87e04504065e00f509434d29&amp;host_name=dm01db08.ga.ssga.root&amp;patternId=dd15665a7fe022004e83e2065f2a0c57&amp;patternName=Docker Pattern&amp;patternType=1&amp;isCloud=false"><u><b>Here</b></u></a>[/code]
9/10/2019 1:04 Error [code]Failed Exploring CI Pattern, Pattern name: <b>Docker Pattern</b>, Process ID: <b>13323</b>, To Check Pattern Log Press <a href="$sw_horizontal_discovery_log.do?discoLogId=cadbae6d1bff7f404d41dd7edd4bcb7b&amp;ipAddress=10.145.112.53&amp;pid=13323&amp;preExecution=&amp;host_sys_id=10916b7387244504065e00f509434d22&amp;host_name=dm02db04.ga.ssga.root&amp;patternId=dd15665a7fe022004e83e2065f2a0c57&amp;patternName=Docker Pattern&amp;patternType=1&amp;isCloud=false"><u><b>Here</b></u></a>[/code]
9/10/2019 1:04 Error [code]Failed Exploring CI Pattern, Pattern name: <b>Docker Pattern</b>, Process ID: <b>13312</b>, To Check Pattern Log Press <a href="$sw_horizontal_discovery_log.do?discoLogId=fbcbeead1b377f40276510e4bd4bcbd2&amp;ipAddress=10.145.112.50&amp;pid=13312&amp;preExecution=&amp;host_sys_id=d7616bb387244504065e00f509434dd3&amp;host_name=dm02db01.ga.ssga.root&amp;patternId=dd15665a7fe022004e83e2065f2a0c57&amp;patternName=Docker Pattern&amp;patternType=1&amp;isCloud=false"><u><b>Here</b></u></a>[/code]
9/10/2019 1:04 Error [code]Failed Exploring CI Pattern, Pattern name: <b>Docker Pattern</b>, Process ID: <b>12891</b>, To Check Pattern Log Press <a href="$sw_horizontal_discovery_log.do?discoLogId=b7cbeead1b377f40276510e4bd4bcb97&amp;ipAddress=10.144.112.54&amp;pid=12891&amp;preExecution=&amp;host_sys_id=642edbff87e04504065e00f509434dd6&amp;host_name=dm01db05.ga.ssga.root&amp;patternId=dd15665a7fe022004e83e2065f2a0c57&amp;patternName=Docker Pattern&amp;patternType=1&amp;isCloud=false"><u><b>Here</b></u></a>[/code]
9/10/2019 1:04 Error [code]Failed Exploring CI Pattern, Pattern name: <b>Docker Pattern</b>, Process ID: <b>13255</b>, To Check Pattern Log Press <a href="$sw_horizontal_discovery_log.do?discoLogId=23cbae6d1bff7f404d41dd7edd4bcb6c&amp;ipAddress=10.145.112.52&amp;pid=13255&amp;preExecution=&amp;host_sys_id=d581ebb387244504065e00f509434da2&amp;host_name=dm02db03.ga.ssga.root&amp;patternId=dd15665a7fe022004e83e2065f2a0c57&amp;patternName=Docker Pattern&amp;patternType=1&amp;isCloud=false"><u><b>Here</b></u></a>[/code]
9/10/2019 1:04 Error [code]Failed Exploring CI Pattern, Pattern name: <b>Docker Pattern</b>, Process ID: <b>13008</b>, To Check Pattern Log Press <a href="$sw_horizontal_discovery_log.do?discoLogId=47cb266d1b33fb40db5e43b4bd4bcb6c&amp;ipAddress=10.144.112.52&amp;pid=13008&amp;preExecution=&amp;host_sys_id=fe0ed7ff87e04504065e00f509434dd8&amp;host_name=dm01db03.ga.ssga.root&amp;patternId=dd15665a7fe022004e83e2065f2a0c57&amp;patternName=Docker Pattern&amp;patternType=1&amp;isCloud=false"><u><b>Here</b></u></a>[/code]
9/10/2019 1:04 Error [code]Failed Exploring CI Pattern, Pattern name: <b>Docker Pattern</b>, Process ID: <b>12885</b>, To Check Pattern Log Press <a href="$sw_horizontal_discovery_log.do?discoLogId=c7cb266d1b33fb40db5e43b4bd4bcb8c&amp;ipAddress=10.144.112.55&amp;pid=12885&amp;preExecution=&amp;host_sys_id=a03e1fff87e04504065e00f509434d97&amp;host_name=dm01db06.ga.ssga.root&amp;patternId=dd15665a7fe022004e83e2065f2a0c57&amp;patternName=Docker Pattern&amp;patternType=1&amp;isCloud=false"><u><b>Here</b></u></a>[/code]
9/10/2019 1:03 Error java.lang.NullPointerException: java.lang.NullPointerException:
9/10/2019 1:03 Error java.lang.NullPointerException: java.lang.NullPointerException:
9/10/2019 1:02 Error java.lang.NullPointerException: java.lang.NullPointerException:
9/10/2019 1:01 Error java.lang.NullPointerException: java.lang.NullPointerException:
9/10/2019 1:00 Error cmdb_metadata : Found duplicate cmdb_rel_type records with name: Master of::Stack Member of having sys_ids: 357afff213a21300f39f721a6144b076, c8c685710b22130005d90d2835673aa8: no thrown error
9/10/2019 1:00 Error java.lang.NullPointerException: java.lang.NullPointerException:
9/10/2019 1:00 Error LICENSE_DETAILS.ALLOCATED ua_stats_defn Calculation: DEF1000115 not found: no thrown error
9/10/2019 0:34 Error java.lang.NullPointerException: java.lang.NullPointerException:
9/10/2019 0:30 Error cmdb_metadata : Found duplicate cmdb_rel_type records with name: Master of::Stack Member of having sys_ids: 357afff213a21300f39f721a6144b076, c8c685710b22130005d90d2835673aa8: no thrown error
9/10/2019 0:30 Error cmdb_metadata : Found duplicate cmdb_rel_type records with name: Master of::Stack Member of having sys_ids: 357afff213a21300f39f721a6144b076, c8c685710b22130005d90d2835673aa8: no thrown error
9/10/2019 0:03 Error UATablePkgOverrideHandler: Could not find the package with source com.snc.problem: no thrown error
9/10/2019 0:03 Error UATablePkgOverrideHandler: Could not find the package with source com.snc.incident: no thrown error
9/10/2019 0:00 Error [code]Canceled discovery of <a href="discovery_schedule.do?sys_id=71c932b1db5aa3403f737afc0f96195a"><u>SSGA Windows Active Servers</u></a>. Already at maximum number of active 'Scheduled' invocations (3) for a given schedule[/code]

Can someone please help me achieve or rectify my config file ?

Thanks a million in advance.

AskedSeptember 10, 2019 - 11:31pm

Comments (3)

  • Zhengshi's picture
    (NXLog)

    It is sometimes easier to trim down config and tests to verify everything is working the way you are expecting to begin with.

    Config for CSV:
    Fields $Created,$Level,$Message,$Source,$CreatedBy
    Top of file:
    Created Level Message

    Just to verify, are all of these events supposed to have $Source and $CreatedBy as well? I am unsure where the break is supposed to be.
    By default, the xm_csv will use commas as the field separator.

    Example Log Entry:

    9/10/2019 1:04 Error [code]Failed Exploring CI Pattern, Pattern name: <b>Docker Pattern</b>, Process ID: <b>12879</b>, To Check Pattern Log Press <a href="$sw_horizontal_discovery_log.do?discoLogId=44eb2a6d1b33fb40db5e43b4bd4bcbf4&amp;ipAddress=10.144.112.50&amp;pid=12879&amp;preExecution=&amp;host_sys_id=6cfddfbb87e04504065e00f509434d75&amp;host_name=dm01db01.ga.ssga.root&amp;patternId=dd15665a7fe022004e83e2065f2a0c57&amp;patternName=Docker Pattern&amp;patternType=1&amp;isCloud=false"><u><b>Here</b></u></a>[/code]
    

    JSON Output with PrettyPrint True for ease of reading:

    {
      "EventReceivedTime": "2019-09-10T17:16:21.120782-05:00",
      "SourceModuleName": "filein",
      "SourceModuleType": "im_file",
      "SourceName": "SNOWLogs",
      "Created": "9/10/2019 1:04 Error [code]Failed Exploring CI Pattern",
      "Level": "Pattern name: <b>Docker Pattern</b>",
      "Message": "Process ID: <b>12879</b>",
      "Source": "To Check Pattern Log Press <a href=\"$sw_horizontal_discovery_log.do?discoLogId=44eb2a6d1b33fb40db5e43b4bd4bcbf4&amp;ipAddress=10.144.112.50&amp;pid=12879&amp;preExecution=&amp;host_sys_id=6cfddfbb87e04504065e00f509434d75&amp;host_name=dm01db01.ga.ssga.root&amp;patternId=dd15665a7fe022004e83e2065f2a0c57&amp;patternName=Docker Pattern&amp;patternType=1&amp;isCloud=false\"><u><b>Here</b></u></a>[/code]"
    }
    

    Secondary question, the $Created field, is that just supposed to be the date and time? Type may be better with datetime.

    As far as the size of the document goes, I see you are sending to an external server for fileout instance. You also have SavePos and ReadFromLast set to False. With these settings, are you running the NXLog service multiple times? i.e. restarting the service. If so, are you clearing the output file in between? If not, you will be appending to that log the entire source file repeatedly.

    September 11, 2019 - 12:31am
  • abasha's picture

    Hello Zhengshi,

    First of all thanks for your time to look into my query, yes all messages have $Source and $CreatedBy fields,
    the biggest problem is $Message - this sometimes its singular line and sometimes it multiple lines
    Plus the delimiter is comma and even $Message may contain the same delimiter which is causing the parser to fail.

    Below are some sample messages:

    "Created","Level","Message","Source","Created by"
    "2019-09-10 03:00:04","Information","Export started","ScheduleExportJob","admin"
    "2019-09-10 03:00:04","Information","*** Done synchronizing svc_ci_assoc ***","*** Script","system"
    "2019-09-10 03:00:04","Information","No Discovery Failures Today on :2019-09-10 03:00:04","*** Script","admin"
    "2019-09-10 03:00:04","Information","Syncing subscription Service Management Suite v2 With Platform Runtime - Fulfiller User","*** Script","system"
    "2019-09-10 03:00:04","Information","*** About to synchronize svc_ci_assoc ***","*** Script","system"
    "2019-09-10 03:00:04","Error","java.lang.NullPointerException: java.lang.NullPointerException: ","com.glide.ui.ServletErrorListener","guest"
    "2019-09-10 02:59:45","Information","approved--12","change Task","814612"
    "2019-09-10 02:59:45","Information","Inside ACL-->The Approval is :::approved and the State is:::3The stage is-->12","*** Script","814612"
    "2019-09-10 02:59:23","Warning","org.mozilla.javascript.EcmaError: ""ServerStats"" is not defined.
    Caused by error in Schedule Item: 'Stats: Servlet' at line 1

    ==> 1: gs.include('ServerStats'); new ServerStats().run();
    ","Evaluator","system"
    "2019-09-10 02:59:11","Information","approved--9","change Task","641357"
    "2019-09-10 02:59:11","Information","Inside ACL-->The Approval is :::approved and the State is:::1The stage is-->9","*** Script","641357"
    "2019-09-10 02:59:06","Information","Operation is : insert","*** Script","discoveryadmin"
    "2019-09-10 02:59:05","Information","Operation is : insert","*** Script","discoveryadmin"
    "2019-09-10 02:59:05","Information","Operation is : insert","*** Script","discoveryadmin"
    "2019-09-10 02:59:05","Information","Operation is : insert","*** Script","discoveryadmin"
    "2019-09-10 02:59:05","Information","Operation is : insert","*** Script","discoveryadmin"
    "2019-09-10 02:59:05","Information","Operation is : insert","*** Script","discoveryadmin"
    "2019-09-10 02:59:05","Information","Operation is : insert","*** Script","discoveryadmin"
    "2019-09-10 02:59:04","Information","Operation is : insert","*** Script","discoveryadmin"
    "2019-09-10 02:59:04","Information","Operation is : insert","*** Script","discoveryadmin"
    "2019-09-10 02:59:04","Information","Operation is : insert","*** Script","discoveryadmin"
    "2019-09-10 02:59:04","Information","Operation is : insert","*** Script","discoveryadmin"
    "2019-09-10 02:59:04","Information","Operation is : insert","*** Script","discoveryadmin"
    "2019-09-10 02:58:23","Warning","org.mozilla.javascript.EcmaError: ""ServerStats"" is not defined.
    Caused by error in Schedule Item: 'Stats: Servlet' at line 1

    ==> 1: gs.include('ServerStats'); new ServerStats().run();
    ","Evaluator","system"
    "2019-09-10 02:58:19","Information","approved--9","change Task","631867"
    "2019-09-10 02:58:19","Information","Inside ACL-->The Approval is :::approved and the State is:::1The stage is-->9","*** Script","631867"
    "2019-09-10 02:58:05","Information","Inside ACL-->The Approval is :::approved and the State is:::1The stage is-->9","*** Script","631867"
    "2019-09-10 02:58:05","Information","approved--9","change Task","631867"
    "2019-09-10 02:58:03","Information","Operation is : update","*** Script","discoveryadmin"
    "2019-09-10 02:58:02","Information","Operation is : insert","*** Script","discoveryadmin"
    "2019-09-10 02:57:44","Information","Inside ACL-->The Approval is :::approved and the State is:::3The stage is-->12","*** Script","814612"
    "2019-09-10 02:57:44","Information","approved--12","change Task","814612"
    "2019-09-10 02:57:27","Warning","org.mozilla.javascript.EcmaError: ""ServerStats"" is not defined.
    Caused by error in Schedule Item: 'Stats: Servlet' at line 1

    ==> 1: gs.include('ServerStats'); new ServerStats().run();
    ","Evaluator","system"
    "2019-09-10 02:57:10","Information","Inside ACL-->The Approval is :::approved and the State is:::1The stage is-->9","*** Script","641357"
    "2019-09-10 02:57:10","Information","approved--9","change Task","641357"
    "2019-09-10 02:57:03","Information","Table: sys_audit_delete, time: 0:00:00.029 {attachments=0:00:00.001, total rows=203061, started=2019-09-10 02:57:03, bulk operations time=0:00:00.002, estimated delete count=0, estimate time=0:00:00.027, deleteMethod=chunkDelete}","TableCleaner","system"
    "2019-09-10 02:57:03","Information","Table: sys_user_preference, time: 0:00:00.006 {total rows=38473, started=2019-09-10 02:57:03, bulk operations time=0:00:00.000, estimated delete count=0, estimate time=0:00:00.006, deleteMethod=chunkDelete}","TableCleaner","system"
    "2019-09-10 02:57:03","Information","Table: sys_ui_mobile_offline_result, time: 0:00:00.001 {attachments=0:00:00.000, total rows=0, started=2019-09-10 02:57:03, bulk operations time=0:00:00.000, estimated delete count=0, estimate time=0:00:00.001, deleteMethod=chunkDelete}","TableCleaner","system"
    "2019-09-10 02:57:03","Information","Table: sys_replication_queue, time: 0:00:00.002 {total rows=0, started=2019-09-10 02:57:03, bulk operations time=0:00:00.001, estimated delete count=0, estimate time=0:00:00.001, deleteMethod=chunkDelete}","TableCleaner","system"
    "2019-09-10 02:57:03","Information","Table: credential_test_run, time: 0:00:00.001 {attachments=0:00:00.000, total rows=0, started=2019-09-10 02:57:03, bulk operations time=0:00:00.001, estimated delete count=0, estimate time=0:00:00.000, deleteMethod=chunkDelete}","TableCleaner","system"
    "2019-09-10 02:57:03","Information","Table: usageanalytics_count, time: 0:00:00.018 {attachments=0:00:00.001, total rows=114901, started=2019-09-10 02:57:03, bulk operations time=0:00:00.002, estimated delete count=0, estimate time=0:00:00.016, deleteMethod=chunkDelete}","TableCleaner","system"
    "2019-09-10 02:57:03","Information","Table: sys_user_session, time: 0:00:00.045 {total rows=25479, started=2019-09-10 02:57:03, bulk operations time=0:00:00.040, estimated delete count=1144, estimate time=0:00:00.005, deleteMethod=chunkDelete}","TableCleaner","system"
    "2019-09-10 02:57:03","Information","Table: plan_execution, time: 0:00:00.001 {total rows=0, started=2019-09-10 02:57:03, bulk operations time=0:00:00.001, estimated delete count=0, estimate time=0:00:00.000, deleteMethod=chunkDelete}","TableCleaner","system"
    "2019-09-10 02:57:03","Information","Table: sys_cache_flush, time: 0:00:00.002 {total rows=612, started=2019-09-10 02:57:03, bulk operations time=0:00:00.002, estimated delete count=14, estimate time=0:00:00.000, deleteMethod=chunkDelete}","TableCleaner","system"
    "2019-09-10 02:57:03","Information","Table: sys_querystat, time: 0:00:00.001 {total rows=0, started=2019-09-10 02:57:03, bulk operations time=0:00:00.000, estimated delete count=0, estimate time=0:00:00.001, deleteMethod=chunkDelete}","TableCleaner","system"
    "2019-09-10 02:57:03","Information","Table: discovery_net_base, time: 0:00:00.035 {total rows=55386, started=2019-09-10 02:57:03, bulk operations time=0:00:00.014, estimated delete count=0, estimate time=0:00:00.021, deleteMethod=chunkDelete}","TableCleaner","system"
    "2019-09-10 02:57:03","Information","Table: cmdb_health_scorecard, time: 0:00:00.001 {total rows=0, started=2019-09-10 02:57:03, bulk operations time=0:00:00.000, estimated delete count=0, estimate time=0:00:00.001, deleteMethod=chunkDelete}","TableCleaner","system"
    "2019-09-10 02:57:03","Information","Table: sys_execution_tracker, time: 0:00:00.002 {attachments=0:00:00.000, total rows=777, started=2019-09-10 02:57:03, bulk operations time=0:00:00.001, estimated delete count=0, estimate time=0:00:00.001, deleteMethod=chunkDelete}","TableCleaner","system"
    "2019-09-10 02:57:02","Information","Table: sys_api_stats, time: 0:00:00.001 {started=2019-09-10 02:57:02, bulk operations time=0:00:00.000, estimated delete count=0, estimate time=0:00:00.001, deleteMethod=chunkDelete}","TableCleaner","system"
    "2019-09-10 02:57:02","Information","Table: ecc_agent_metric, time: 0:00:00.131 {attachments=0:00:00.000, total rows=441405, started=2019-09-10 02:57:02, bulk operations time=0:00:00.073, estimated delete count=612, estimate time=0:00:00.058, deleteMethod=chunkDelete}","TableCleaner","system"
    "2019-09-10 02:57:02","Information","Table: pwd_activity_monitor, time: 0:00:00.001 {attachments=0:00:00.000, total rows=0, started=2019-09-10 02:57:02, bulk operations time=0:00:00.001, estimated delete count=0, estimate time=0:00:00.000, deleteMethod=chunkDelete}","TableCleaner","system"
    "2019-09-10 02:57:02","Information","Table: sp_log, time: 0:00:00.003 {attachments=0:00:00.000, total rows=5808, started=2019-09-10 02:57:02, bulk operations time=0:00:00.002, estimated delete count=2, estimate time=0:00:00.001, deleteMethod=chunkDelete}","TableCleaner","system"
    "2019-09-10 02:56:43","Warning","org.mozilla.javascript.EcmaError: ""getEditLink"" is not defined.
    Caused by error in <refname> at line 12

    9: gf.setRenderProperties(renderer.getRenderProperties());
    10: return gf.getRenderedPage();
    11: }
    ==> 12: getEditLink();
    ","Evaluator","613468"
    "2019-09-10 02:56:23","Warning","org.mozilla.javascript.EcmaError: ""ServerStats"" is not defined.
    Caused by error in Schedule Item: 'Stats: Servlet' at line 1

    ==> 1: gs.include('ServerStats'); new ServerStats().run();
    ","Evaluator","system"
    "2019-09-10 02:56:18","Information","Inside ACL-->The Approval is :::approved and the State is:::1The stage is-->9","*** Script","631867"
    "2019-09-10 02:56:18","Information","approved--9","change Task","631867"
    "2019-09-10 02:56:07","Information","Operation is : update","*** Script","discoveryadmin"
    "2019-09-10 02:56:07","Information","Operation is : update","*** Script","discoveryadmin"
    "2019-09-10 02:56:06","Information","Operation is : insert","*** Script","discoveryadmin"
    "2019-09-10 02:56:04","Information","approved--9","change Task","631867"
    "2019-09-10 02:56:04","Information","Inside ACL-->The Approval is :::approved and the State is:::1The stage is-->9","*** Script","631867"
    "2019-09-10 02:56:03","Information","Operation is : insert","*** Script","discoveryadmin"
    "2019-09-10 02:55:57","Information","Operation is : update","*** Script","discoveryadmin"
    "2019-09-10 02:55:57","Information","Operation is : update","*** Script","discoveryadmin"
    "2019-09-10 02:55:56","Information","Operation is : insert","*** Script","discoveryadmin"
    "2019-09-10 02:55:53","Information","Operation is : insert","*** Script","discoveryadmin"
    "2019-09-10 02:55:43","Information","Inside ACL-->The Approval is :::approved and the State is:::3The stage is-->12","*** Script","814612"
    "2019-09-10 02:55:43","Information","approved--12","change Task","814612"
    "2019-09-10 02:55:43","Information","Operation is : update","*** Script","discoveryadmin"
    "2019-09-10 02:55:37","Information","Operation is : insert","*** Script","discoveryadmin"
    "2019-09-10 02:55:34","Warning","org.mozilla.javascript.EcmaError: ""getEditLink"" is not defined.
    Caused by error in <refname> at line 12

    9: gf.setRenderProperties(renderer.getRenderProperties());
    10: return gf.getRenderedPage();
    11: }
    ==> 12: getEditLink();
    ","Evaluator","663035"

    Below are the error log generated while parsing,

    2019-09-10 18:09:27 INFO nxlog-ce-2.10.2150 started
    2019-09-10 18:09:27 ERROR procedure 'parse_csv' failed at line 97, character 25 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; Invalid CSV input: '"2019-09-10 02:59:23","Warning","org.mozilla.javascript.EcmaError: ""ServerStats"" is not defined.'
    2019-09-10 18:09:27 ERROR procedure 'parse_csv' failed at line 97, character 25 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; Not enough fields in CSV input, expected 5, got 1 in input ' Caused by error in Schedule Item: 'Stats: Servlet' at line 1'
    2019-09-10 18:09:27 ERROR procedure 'parse_csv' failed at line 97, character 25 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; Not enough fields in CSV input, expected 5, got 0 in input ''
    2019-09-10 18:09:27 ERROR procedure 'parse_csv' failed at line 97, character 25 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; Not enough fields in CSV input, expected 5, got 1 in input '==> 1: gs.include('ServerStats'); new ServerStats().run();'
    2019-09-10 18:09:27 ERROR procedure 'parse_csv' failed at line 97, character 25 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; Invalid CSV input: '","Evaluator","system"'
    2019-09-10 18:09:27 ERROR procedure 'parse_csv' failed at line 97, character 25 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; Invalid CSV input: '"2019-09-10 02:58:23","Warning","org.mozilla.javascript.EcmaError: ""ServerStats"" is not defined.'
    2019-09-10 18:09:27 ERROR procedure 'parse_csv' failed at line 97, character 25 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; Not enough fields in CSV input, expected 5, got 1 in input ' Caused by error in Schedule Item: 'Stats: Servlet' at line 1'
    2019-09-10 18:09:27 ERROR procedure 'parse_csv' failed at line 97, character 25 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; Not enough fields in CSV input, expected 5, got 0 in input ''
    2019-09-10 18:09:27 ERROR procedure 'parse_csv' failed at line 97, character 25 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; Not enough fields in CSV input, expected 5, got 1 in input '==> 1: gs.include('ServerStats'); new ServerStats().run();'
    2019-09-10 18:09:27 ERROR procedure 'parse_csv' failed at line 97, character 25 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; Invalid CSV input: '","Evaluator","system"'
    2019-09-10 18:09:27 ERROR procedure 'parse_csv' failed at line 97, character 25 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; Invalid CSV input: '"2019-09-10 02:57:27","Warning","org.mozilla.javascript.EcmaError: ""ServerStats"" is not defined.'
    2019-09-10 18:09:27 ERROR procedure 'parse_csv' failed at line 97, character 25 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; Not enough fields in CSV input, expected 5, got 1 in input ' Caused by error in Schedule Item: 'Stats: Servlet' at line 1'
    2019-09-10 18:09:27 ERROR procedure 'parse_csv' failed at line 97, character 25 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; Not enough fields in CSV input, expected 5, got 0 in input ''
    2019-09-10 18:09:27 ERROR procedure 'parse_csv' failed at line 97, character 25 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; Not enough fields in CSV input, expected 5, got 1 in input '==> 1: gs.include('ServerStats'); new ServerStats().run();'
    2019-09-10 18:09:27 ERROR procedure 'parse_csv' failed at line 97, character 25 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; Invalid CSV input: '","Evaluator","system"'
    2019-09-10 18:09:27 INFO connecting to logger.ga.ssga.root:5140
    2019-09-10 18:09:27 INFO connecting to logger.ga.ssga.root:5140
    2019-09-10 18:09:30 ERROR procedure 'parse_csv' failed at line 97, character 25 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; Invalid CSV input: '"2019-09-10 02:56:43","Warning","org.mozilla.javascript.EcmaError: ""getEditLink"" is not defined.'
    2019-09-10 18:09:30 ERROR procedure 'parse_csv' failed at line 97, character 25 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; Not enough fields in CSV input, expected 5, got 1 in input ' Caused by error in <refname> at line 12'
    2019-09-10 18:09:30 ERROR procedure 'parse_csv' failed at line 97, character 25 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; Not enough fields in CSV input, expected 5, got 0 in input ''
    2019-09-10 18:09:30 ERROR procedure 'parse_csv' failed at line 97, character 25 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; Not enough fields in CSV input, expected 5, got 1 in input ' 9: gf.setRenderProperties(renderer.getRenderProperties());'
    2019-09-10 18:09:30 ERROR procedure 'parse_csv' failed at line 97, character 25 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; Not enough fields in CSV input, expected 5, got 1 in input ' 10: return gf.getRenderedPage();'
    2019-09-10 18:09:30 ERROR procedure 'parse_csv' failed at line 97, character 25 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; Not enough fields in CSV input, expected 5, got 1 in input ' 11: }'
    2019-09-10 18:09:30 ERROR procedure 'parse_csv' failed at line 97, character 25 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; Not enough fields in CSV input, expected 5, got 1 in input '==> 12: getEditLink();'
    2019-09-10 18:09:30 ERROR procedure 'parse_csv' failed at line 97, character 25 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; Invalid CSV input: '","Evaluator","613468"'

    The reason i removed SavePos and ReadFromLast as False, because i am trying to troubleshoot and debug the script. As I do not get proper messages in my log server, I have my service stopped at this moment until I get any fix for it.

    If the source file can help, I can upload the same to some common drive and share you the path for analysis. The complete input file.

    Thanks, Ameer

    September 11, 2019 - 10:27pm
  • Zhengshi's picture
    (NXLog)

    Ameer,

    I would normally say source would help, but this part throws me off:

    Plus the delimiter is comma and even $Message may contain the same delimiter which is causing the parser to fail.

    If that is the case, it is hard for any parser to distinguish between the fields. Can the source create logs with a different delimiter? Ideally one that would not appear in the data.
    https://nxlog.co/documentation/nxlog-user-guide/xm_csv.html#xm_csv_config_char

    statement execution has been aborted; Not enough fields in CSV input, expected 5, got 1
    These types of errors are likely where a multiline was not detected properly. I would look into this and take this in a couple steps.
    First, make sure that your multiline is working properly.
    Second, make sure that there are the proper number of fields available with a distinguishable delimiter.

    September 13, 2019 - 2:08am

Answers (0)