8
responses

Hello,

I have some IIS logs that contain a single " and I am getting errors when I try to use parse_csv saying the data is invalid csv input. As soon as I take out the single ", the log sends fine.

What can I do to resolve this issue?

AskedSeptember 19, 2019 - 10:42pm

Comments (7)

  • Zhengshi's picture
    (NXLog)

    How is your config set up? The default W3C format used on IIS should be able to be parsed with parse_csv() as showing in the manual.
    https://nxlog.co/documentation/nxlog-user-guide/iis.html

    If you could link the config and the offending event line, it may help.

  • motts's picture

    I have the parse_csv exec set.

    With the single " in the log, it's saying this error message:

    2019-09-20 08:41:28 ERROR procedure 'parse_csv' failed at line 49, character 26 in C:\Program Files\nxlog\conf\nxlog.conf. statement execution has been aborted
    Invalid CSV input: %logdata%

    If I take out the single " then the log parses fine. Here is a sample log:

    2019-09-1912:44:32 IIS host-name 1.1.1.1 GET /FALSE_QUERY.html - 443 - 2.2.2.2 HTTP/1.1 - - "></a><script>alert('Qualys+XSS+test')</script> 3.3.3.3 302 0 0 757 152 46

  • Zhengshi's picture
    (NXLog)

    In lieu of not having the config you are using, I just took the example out of the manual and modified it to include the additional fields.
    I tried under NXLog EE 4.5.4503 and NXLog CE 2.10.2150 . EscapeChar is likely what is missing for you.
    Config:

    <Extension w3c_parser>
        Module          xm_csv
        Fields          date, time, s-ip, cs-method, cs-uri-stem, cs-uri-query, s-port, cs-username, c-ip, cs(User-Agent), cs(Referer), sc-status, sc-substatus, sc-win32-status, time-taken, last, one, two, three, four, five
        FieldTypes      string, string, string, string, string, string, integer, string, string, string, string, integer, integer, string, string, integer, integer, integer, integer, integer, integer
        Delimiter       ' '
        EscapeChar      '"'
        QuoteChar       '"'
        EscapeControl   FALSE
        UndefValue      -
    </Extension>
    
    <Extension json>
        Module          xm_json
    </Extension>
    <Input iis_w3c>
        Module          im_file
        File            '/root/w3c.log'
        ReadFromLast    False
        SavePos         False
        <Exec>
            if $raw_event =~ /^#/ drop();
            else
            {
                w3c_parser->parse_csv();
                $EventTime = parsedate($date + "T" + $time + ".000Z");
            }
            to_json();
        </Exec>
    </Input>
    <Output out>
        Module      om_file
        File        '/tmp/w3c.out'
    </Output>
    <Route r1>
        Path        iis_w3c => out
    </Route>
    

    Output expanded:

    {
      "EventReceivedTime": "2019-09-20 18:50:34",
      "SourceModuleName": "iis_w3c",
      "SourceModuleType": "im_file",
      "date": "2019-09-1912:44:32",
      "time": "IIS",
      "s-ip": "host-name",
      "cs-method": "1.1.1.1",
      "cs-uri-stem": "GET",
      "cs-uri-query": "/FALSE_QUERY.html",
      "s-port": null,
      "cs-username": "443",
      "c-ip": null,
      "cs(User-Agent)": "2.2.2.2",
      "cs(Referer)": "HTTP/1.1",
      "sc-status": null,
      "sc-substatus": null,
      "sc-win32-status": "\"></a><script>alert('Qualys+XSS+test')</script>",
      "time-taken": "3.3.3.3",
      "last": 302,
      "one": 0,
      "two": 0,
      "three": 757,
      "four": 152,
      "five": 46,
      "EventTime": null
    }
    

  • motts's picture

    Hmm I seem to be having issues. I seem to have everything you do, I think, but I am not getting anything from the sc-win32-status still. Here is my return data:

    {"EventReceivedTime":"2019-09-24T09:33:40.556688-04:00","SourceModuleName":"in","SourceModuleType":"im_file","Number":"2019-09-1912:44:32","Others":"W3SVC8","sname":"test-server","sip":"1.1.1.1","method":"GET","stem":"/FALSE_QUERY.html","nothing":null,"port":"443","dest":"2.2.2.2","http":"HTTP/1.1","status":null}
    

    Here is my NxLog conf. Maybe you can see something wrong here.

    define ROOT C:\Program Files\nxlog
     #define ROOT C:\Program Files (x86)\nxlog
    
     Moduledir %ROOT%\modules
     CacheDir %ROOT%\data
     Pidfile %ROOT%\data\nxlog.pid
     SpoolDir %ROOT%\data
     LogFile %ROOT%\data\nxlog.log
     #LogLevel DEBUG
    
    <Extension w3c>
    
        Module  xm_csv
        Fields $Number, $Others, $sname, $sip, $method, $stem, $nothing, $port, $nothing, $dest, $http, $nothing, $status
        FieldTypes string, string, string, string, string, string, string, string, string, string, string, string, string
        Delimiter \t
        EscapeChar '"'
        QuoteChar '"'
        EscapeControl FALSE
        UndefValue -
    
    </Extension>
    
    <Extension json>
        Module xm_json
    </Extension>
    
    
    <Input in>
    
        Module im_file
    
            File   "C:\\Users\\Administrator\\Desktop\\test.csv"
            SavePos         false
            ReadFromLast    false
            Exec            w3c->parse_csv();
            Exec            if $raw_event =~ /^#/ drop();
            Exec            to_json();
    
    </Input>
    
    <Output out>
    
        Module  om_file
        File    "C:\\users\\administrator\\desktop\\results.txt"
    
    </Output>
    
    
    <Route 1>
    
        Path    in => out
    
    </Route>
    

  • Zhengshi's picture
    (NXLog)

    I think it could be helpful if you could provide an actual source file. I noticed your config uses Delimiter of \t. Since the example was copied in, my resulting copy out didn't have a tab in it, only spaces. That makes our number of fields not line up as well.
    With the source file it could be easier to work through. As it is, everything looks ok from my tests.

    Please upload the source file to a file sharing service.

  • motts's picture

    Ok so I found my particular issue.

    I was trying to make my work less by not having the full number of columns and fields defined in the config, just up to the column with the single double quote.

    Adding in all the extra columns and fields for the file resolved the issue.

    Thanks a bunch.

Answer (1)