im_msvistalog EventTime being sent as String to ElasticSearch

Post a different question
4
responses

I'm attempting to demo xnlog and running into a problem where the Windows Server 2016 event logs are being sent to AWS ElasticSearch Service with the EventTime being a string. This basically renders it impossible to index the logs, as the Kibana board requires a time-field name and is not recongizing the string as a datetime.  Any suggestions on this, or is this a potential bug with Server 2016?

AskedJanuary 6, 2017 - 7:08pm

Answer (1)

It's very unlikely that this is due to Windows 2016. Perhaps there is something in your config that converts it to a string but it's hard to tell the cause without that.

AnsweredJanuary 8, 2017 - 5:26pm

Comments (3)

  • chris.bowen's picture

    Here's my current configuration:

    <Extension syslog>
    Module      xm_syslog
</Extension>

<Extension json>
    Module      xm_json
</Extension>

<Input EventIn>
    Module im_msvistalog
    ReadFromLast TRUE
    <QueryXML>
        <QueryList>
            <Query Id='1'>
                <Select Path='Application'>*</Select>
                <Select Path='Security'>*</Select>
                <Select Path='System'>*</Select>
            </Query>
        </QueryList>
    </QueryXML>
</Input>

<Output EventOut>
    Module      om_http
    URL         http://endpoint.io:443
    ContentType application/json
    Exec        set_http_request_path(strftime($EventTime, "/nxlog-%Y%m%d/" + $SourceModuleName)); to_json();
</Output>

<Route 1>
    Path    EventIn => EventOut
</Route>

    This is the output that ElsaticSearch Service is receiving for the EventReceivedTime and EventTime:

    • EventReceivedTime string
    • EventTime string

    January 9, 2017 - 10:45pm
  • sa's picture

    Note: before proceeding, check if you have the right mapping in place (i.e. EventTime is mapped as a date)

     curl -XGET 'localhost:9200/_template'

    The output should contain something like this:

    mappings":{"_default_":{"properties":{"EventTime":{"format":"YYYY-MM-dd HH:mm:ss","type":"date"}}

     

    If your mapping looks OK, then read on...

    I tried to run a quick test with your config, but unfortunately my Elasticsearch is running in developer mode, so I cannot address it directly from an outside source (long story short: it's running in a non-privileged container) and thus I cannot access it directly from a Windows 2012 box.

    But I can and do use an Nxlog instance in the same container to accept events from the Windows server. What I have is basically a KEN stack (Kibana, Elasticsearch, Nxlog) in a container receiving om_tcp (binary) data and feeding it to ES. This way the EventTime field is treated properly as a date data type.

    Nxlog conf on the KEN server:

    <Input in> 
            Module im_tcp
            Host 0.0.0.0
            Port 1514
            InputType Binary
    </Input>

    <Output es>
            Module      om_http
                URL         http://localhost:9200
                ContentType application/json
                Exec        set_http_request_path(strftime($EventTime, "/nxlog-%Y%m%d/" + $SourceModuleName)); to_json();
    </Output>

    <Route r>
            Path in => es
    </Route>

    I'm not suggesting that this is the right (let alone only) way to go, but you can use a similar config to check if Windows 2016 events are received correctly and if you are in a time crunch wrt your demo, it can be a workaround for your problem.

     

    January 10, 2017 - 1:44pm