Hi,

I use module om_http for send events to host via https, but after start NXlog shows error:

ERROR SSL certificate verification failed: self signed certificate in certificate chain (err: 19)

Thanks for your ideas!

Hello,

is the any way to set custom timeout in om_http? or custom retry mechanism?

Thanks

Hi everyone,

The user guide section for the om_http module explains:

Each event is transferred in a single POST request.
https://nxlog.co/documentation/nxlog-user-guide#om_http

When https is used, is the TLS handshake also performed for each event? This seems like it would be quite inefficient.

Are there any plans to support bulk / batch exporting of many events in a single http post, specifically for exporting data to the Splunk HEC?

rsyslog and syslog-ng support batch exporting events to Splunk using the external python script omsplunkhec.py, as explained here:
https://www.splunk.com/blog/2017/03/30/syslog-ng-and-hec-scalable-aggregated-data-collection-in-splunk.html

NXLog supports batch exporting events to Elasticsearch using their bulk API:
https://nxlog.co/documentation/nxlog-user-guide#om_elasticsearch

With reference to the following guide for sending NXLog output to Splunk using HEC:
https://nxlog.co/documentation/nxlog-user-guide#splunk-hec-collection

The NXLog community edition om_http is missing the add_http_header() procedure:
https://nxlog.co/docs/nxlog-ce/nxlog-reference-manual.html#om_http

This means it is not possible to output nxlog-ce data to http where a header needs to be added to the http post, such as the Splunk HEC:
https://docs.splunk.com/Documentation/Splunk/7.0.3/Data/HECWalkthrough

Is this functionality deliberately missing from the community edition, or can it please be added?

I have the usual error "Unexpected data from server ..." using om_http to send Windows event logs through Nginx to Redis.

The problem is that when randomly this error happens ( once every 5 days) the service unexpectently stops !

From the source code I can understand that NXLog receives a response which cannot map to a correnponding request.

It has a response with an appropriate length but the corresponding request is null! Possibly is a bad logic implementation, or even a network problem that could cause request/response mismatch.I really don't know.

But I also really can't understand why NxLog preferres to die instead of  just return from the function in case this happens and not execute the code following the condition that drives the service to crash at the end.

Stelios

Hi,

I'm using nxlog community edition nxlog-ce-2.9.1347, I have few questions related om_http module.
We have a centralized log server(Log Insight) and 10 application servers. Log Insight server exposes a REST API to post the logs data.
I'm using NXLOG as a log forwarder to Log Insight Server from all my application servers. Please clarify the following questions.

POST URL : http://loginsight:9000/api/v1/messages/ingest/4C4C4544-0037-5910-805A-C4C04F585831

Request Body:

{"messages": [{

 "fields": [
 {"name": "Channel", "content": "Security"},
 {"name": "EventID", "content": "4688"},
 {"name": "EventRecordID", "content": "33311266"},
 {"name": "Keywords", "content": "Audit Success"},
 {"name": "Level", "content": "Information"},
 {"name": "OpCode","content": "Info"},
 {"name": "ProcessID", "content": "4"},
 {"name": "ProviderName", "content": "Microsoft-Windows-Security-Auditing"},
 {"name": "Task", "content": "Process Creation"},
 {"name": "ThreadID", "content": "64"}
 ],
 "text": "A new process has been created.",
 "timestamp": 1396622879241
 }
 ]
}

1. How do I format my log data into the request as mentioned above in NXLOG. Request should be formated into json with fields and the data accordingly.
   I'm able to parse IIS logs,eventlogs and logs from files into JSON but got stuck with calling REST API with request body. Please find my nxlog.conf below.

2. Is it a good idea to directly send the log data to log server via REST API? if not what are the disadvantages. 
3. Does om_http module support retrying logic\buffering just in case REST API is down or doesn't respond.
4. What is the best architecture for sending the logs to centralized server, I see lot of people online follow NXLOG=>LOG STASH=>ELASTICSEARCH or Some Centralized server(Log Insight in my case).
5. Should I use pm_buffer, as my log files will be rotated after certain memory limit is reached in case if the REST API is down or om_http module handles this automatically.
 

 Following configuration reads IIS, event logs and logs from files.

## This is a sample configuration file. See the nxlog reference manual about the
## configuration options. It should be installed locally and is also available
## online at http://nxlog.org/nxlog-docs/en/nxlog-reference-manual.html

## Please set the ROOT to the folder your nxlog was installed into,
## otherwise it will not start.

#define ROOT C:\Program Files\nxlog
define ROOT C:\Program Files (x86)\nxlog

Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log

<Extension json>
    Module      xm_json
</Extension>
  
<Extension w3c>
  # Map the fields from the IIS log file (you can open the IIS log file to see the header and know what fields to map)
    Module      xm_csv
    Fields   $date, $time, $s-ip, $cs-method, $cs-uri-stem, $cs-uri-query, $s-port, $cs-username, $c-ip, $cs(User-Agent), $sc-status, $sc-substatus, $sc-win32-status, $time-taken
    FieldTypes  string, string, string, string, string, string, integer, string, string, string, integer, integer, integer, integer
    Delimiter   ' '
</Extension>

<Extension multiline>
    Module        xm_multiline
    HeaderLine    /^\d{4}\-\d{2}\-\d{2} \d{2}\:\d{2}\:\d{2}.\d{3}/
</Extension>

<Extension charconv>
    Module    xm_charconv
    AutodetectCharsets utf-8, utf-16, utf-32, iso8859-2
</Extension>

<Input eventlog>
    Module      im_msvistalog
    ReadFromLast    True
    Module      im_msvistalog
        Query       <QueryList>\
                        <Query Id="0">\
                            <Select Path="Application">*</Select>\
                            <Select Path="System">*</Select>\
                            <Select Path="Security">*</Select>\
                        </Query>\
                    </QueryList>
</Input>
 
<Input iis>
    Module      im_file
    File 'C:\inetpub\logs\LogFiles\W3SVC1\u_ex*.log'
    ReadFromLast TRUE
    Exec        if $raw_event =~ /^#/ drop();                \
                else                                             \
                {                                                \
                    w3c->parse_csv();                            \
                    $EventTime = parsedate($date + " " + $time); \
                    to_json ();                                  \
                }
</Input>

<Input webconsole>
    Module im_file
    File 'C:\Stash\WebConsole.log' 
    InputType multiline
    SavePos TRUE
    # The call to convert_fields automatically converts the input to utf-8
    Exec        convert_fields("AUTO","utf-8"); \
            if $raw_event =~ /^(\d{4}\-\d{2}\-\d{2} \d{2}\:\d{2}\:\d{2}.\d{3})    \[(\S+)\]    \[(\S+)\]    \[(\S+)\]    \[(\S+)\]    \[(.*)\]    \[(.*)\]    (.*)/s \
                    { \
                        $time = $1; \
                        $hostname = $2; \
                        $activityId = $3; \
                        $userIddeviceId = $4; \
                        $threadId = $5; \
                        $level = $6; \
                        $logger = $7; \
                        $message = $8; \
                        to_json(); \
                    } \
                    else \
                    { \
                        drop(); \
                }
</Input>

<Output eventlog-out>
    Module      om_tcp
    Host        127.0.0.1
    Port        3515
    Exec        $EventReceivedTime = integer($EventReceivedTime) / 1000000; \
                to_json();
</Output>
 
<Output iis-out>
    Module      om_tcp
    Host        127.0.0.1
    Port        3516
</Output>

<Output general-out>
    Module      om_tcp
    Host        127.0.0.1
    Port        3517
</Output>

<Route 1>

    Path eventlog => eventlog-out
</Route>
 
<Route 2>
    Path iis => iis-out
</Route>

<Route 3>
    Path webconsole => general-out
</Route>

Sample logs(Webconsole.log):

2015-07-10 10:24:17.424    [20EX15736]    [00000000-0000-0000-0000-000000000000]    [0000000-0000000]    [00008]    [Info ]    [TestModule]    Testing log stash3    
2015-07-10 10:24:17.425    [20EX15736]    [00000000-0000-0000-0000-000000000000]    [0000000-0000000]    [00008]    [Info ]    [TestModule]    Testing log stash4    
2015-07-10 10:24:17.448    [20EX15736]    [00000000-0000-0000-0000-000000000000]    [0000000-0000000]    [00008]    [Error]    [TestModule]    *** EXCEPTION ***
System.DivideByZeroException: Attempted to divide by zero.
   at TCPPublisher.Program.Main(String[] args) in c:\Users\test\Documents\Visual Studio 2013\Projects\TCPDemo\TCPPublisher\Program.cs:line 26

Thanks in advance!
Mohan G