Hi,

my ODBC import won't work, checked everything but always this error occurs:

ERROR im_odbc couldn't connect to the database, IM014:1:0:[Microsoft][ODBC Driver Manager] The specified DSN contains an architecture mismatch between the Driver and Application

Config:

<Input ODBC>
Module im_odbc
ConnectionString DSN=S_ODBC;database=SophosSecurity;

SavePos TRUE
PollInterval 5
IdIsTimestamp FALSE
</Input>

best regards

I'm using NXLog to read log files and send to to Logstash. Normally this works fine, but I'm now trying to send logs from a file, where the new events gets added at the top of the file, not the bottom. Now it's not sending anything.

This is from my NXLog config.

<Input file>
Module  im_file
File "C:\\TEMP\\export.txt"
InputType LineBased
Exec $Message = $raw_event;
SavePos TRUE    
ReadFromLast TRUE
Exec if $raw_event =~ /^#/ drop();                    

Is it possible to read from bottom to top?

Are there any numbers about how nxLog performs when it is processing a high rate of messages being placed into a log file?

Right now we have a couple of incidents which resulted in a few thousand messages being logged per second.

I assume this is more than nxLog can handle but am wondering about any performance testing that has been run

Hi,

I need to use im_dbi import module of nxlog.

This module isn't part of actual setup, so where can I get it?

regards

Christian

Hi all,

after installd nxlog-ce-2.9.1347-1_rhe6.x86_64.rpm on Red Hat I tried to start the service "/etc/init.d/nxlog start" and I'm get the following error:

" Starting nxlog deamon...

/usr/bin/nxlog: symbol lookup error: /usr/bin/nxlog: undefiend symbol: apr_pool_create_unmanaged_ex   "

Does anyone have an idea?

Hello, I'm having trouble centralize logs because my storage system performs poorly.

I have several nxlog-ce agents sending logs with om_tcp module and server I have a nxlog-ce recording these logs with the module om_file in a shared directory on my NAS.

What recommendation for improving the performance of my solution as a whole? Use another distributed file system? Using an unstructured database? Exsitem more options?

Obs .: I need to be able to retrieve the log in its original format, I need to be fault-tolerant and I need to have high write performance.

I'm using NXLog to read a log file and send it to Logstash.  This works fine, but some of the log lines are duplicates.  They're in separate lines, but the content is exactly the same.  I can't change the way the logs are written to the log file, so the only way is to fix it either with NXLog before it gets send, or in Logstash when it arrives, which I prefer not to do.

I see NXLog does have a function for this, but it's not working for me.  I've tried this in my config file.

<Processor norepeat>
Module  pm_norepeat
</Processor>

<Route 1>
Path in => norepeat => out
</Route>

This is abviously not working for me, am I maybe missing something here?

I've seen some posts from about a year ago that NXLog is unable to parse attributes using xm_xml, I just wanted to check if this is still true?

I am running NXLog as a service on Windows machines and want to be able to parse the following message, is it possible?

<log4j:event logger="com.sentry.test.LogContextListener" timestamp="1437661699866" level="TRACE" thread="localhost-startStop-1"> <log4j:message><![CDATA[This is a trace message about how we should use C#]]></log4j:message> </log4j:event>

Hi guys,

could You please share experience on filtering windows event logs.

I have the folowing configuration in input tag:

Query <QueryList>\
<Query Id="0">\
<Select Path="Application">*</Select>\
<Select Path="System">*</Select>\
<Select Path="Security">*</Select>\
</Query>\
</QueryList>

# For windows 2003 and earlier use the following:
#   Module      im_mseventlog
#Exec if ($raw_event =~ /INFO\s+62464/) drop();
# Query za event logove (uzima samo definirane)
#Query <QueryList>\
#<Query Id='1'>\
#<Select Path='Application'>*[System[(EventID='32068')]]</Select>\
#<Select Path='System'>*[System[(EventID='7001')]]</Select>\
#</Query>\
#</QueryList>
## Level 1 (ID=30  Critical)     severity level events
# Level 2 (ID=40  Error)        severity level events
# Level 3 (ID=50  Warning)      severity level events
# Level 4 (ID=80  Information)  severity level events
# Level 5 (ID=100 Verbose)      severity level events
# Drop 4, i 5 level severity logs

Exec if ($EventType == 'VERBOSE') OR ($EventType == 'Verbose') drop();
Exec if ($EventType == 'INFORMATION') OR ($EventType == 'Information') drop();
Exec if $raw_event =~ /INFO\s+4648/ drop();

..

I planed to filter specific events by entering name of 'eventtype' as in the example above. I would like to filter all Verbose,Information, and Warning levels. This unfortunately doesnt filter security INFORMATION level.

Could You please please give proper example of filtering logs by severity  and by specific event_ID?

Thank you very much.

Appreciate the help.

I am working in Product development company in india. I have downloaded nxlog community edition and setup to transfer our logs into logstash server.

Our environment details as below.

OS - Windows 7 Professional.

nxLog installed as service.  

Everything working as we expected.

Now what i want to ask is, can we take this setup into our production environment? 

Is Community edition fully free of cost or is there any license has to be purchase for this (nxLog Community edition)?

Please give us license details about nxLog community edtion in Production environment. 

Thanks

Sugumar J

Hi,

is it possiable to access IIS logs located on a network share? 

for example, right now I'm using:

    Module    im_file
    File    "D:\\Logs\\W3SVC1\\u_ex*"
    SavePos  TRUE

and it works.

But if i change it to:

    Module    im_file
    File    "\\netapp-ams-cifs\\IISlogs\\Trader1\\W3SVC1\\u_ex*"
    SavePos  TRUE
 

i see an error on the nxlog saying: ERROR failed to open directory: \netapp-ams-cifs\IISlogs\Trader1\W3SVC1: The system cannot find the path specified. 

Can anyone help please?

Thanks

Hey all,

I'm trying to install nxlog rpm on redhat 6.

Once I run the rpm I get error message :

 " Failed dependencies:

libdbi >= 0.8.1 is needed by nxlog-ce-2.9.1347-1.x86_64"

When I'm trying install libdbi 0.8.1 I get error on   dependencies collision with a more advanced version of libdbi.
The only way to continue the installation is to delete the newer version - and it is very problematic.

Someone experienced the problem?

Thanks a lot.

On the site only source zip for version 2.8.1248 is available

Thanks,

  Angelo.

Hi,

I am evaluating nxlog on Windows but I've run into a snag.

I have logs organized like this:

D:\ServerLogs\2015-07-16\*.log

Where the directory is rotated based on today's date. Now, if I use the recursive parameter set to true for the file watcher, including this

D:\ServerLogs\*.log

works. However, I don't want to include all files dues to size and/or relevance. I only want to include some of them. As far as I can gather, the recursive parameter only works when there's a wildcard on the filename.

What I need is to be able to include the following:

D:\ServerLogs\*\Log1.log

so it follows the directory rotation on that file alone. I can't figure out how to accomplish this with nxlog.

Please advise.

Thank you.

Hi,

I'm using nxlog community edition nxlog-ce-2.9.1347, I have few questions related om_http module.
We have a centralized log server(Log Insight) and 10 application servers. Log Insight server exposes a REST API to post the logs data.
I'm using NXLOG as a log forwarder to Log Insight Server from all my application servers. Please clarify the following questions.

POST URL : http://loginsight:9000/api/v1/messages/ingest/4C4C4544-0037-5910-805A-C4C04F585831

Request Body:

{"messages": [{

 "fields": [
 {"name": "Channel", "content": "Security"},
 {"name": "EventID", "content": "4688"},
 {"name": "EventRecordID", "content": "33311266"},
 {"name": "Keywords", "content": "Audit Success"},
 {"name": "Level", "content": "Information"},
 {"name": "OpCode","content": "Info"},
 {"name": "ProcessID", "content": "4"},
 {"name": "ProviderName", "content": "Microsoft-Windows-Security-Auditing"},
 {"name": "Task", "content": "Process Creation"},
 {"name": "ThreadID", "content": "64"}
 ],
 "text": "A new process has been created.",
 "timestamp": 1396622879241
 }
 ]
}

1. How do I format my log data into the request as mentioned above in NXLOG. Request should be formated into json with fields and the data accordingly.
   I'm able to parse IIS logs,eventlogs and logs from files into JSON but got stuck with calling REST API with request body. Please find my nxlog.conf below.

2. Is it a good idea to directly send the log data to log server via REST API? if not what are the disadvantages. 
3. Does om_http module support retrying logic\buffering just in case REST API is down or doesn't respond.
4. What is the best architecture for sending the logs to centralized server, I see lot of people online follow NXLOG=>LOG STASH=>ELASTICSEARCH or Some Centralized server(Log Insight in my case).
5. Should I use pm_buffer, as my log files will be rotated after certain memory limit is reached in case if the REST API is down or om_http module handles this automatically.
 

 Following configuration reads IIS, event logs and logs from files.

## This is a sample configuration file. See the nxlog reference manual about the
## configuration options. It should be installed locally and is also available
## online at http://nxlog.org/nxlog-docs/en/nxlog-reference-manual.html

## Please set the ROOT to the folder your nxlog was installed into,
## otherwise it will not start.

#define ROOT C:\Program Files\nxlog
define ROOT C:\Program Files (x86)\nxlog

Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log

<Extension json>
    Module      xm_json
</Extension>
  
<Extension w3c>
  # Map the fields from the IIS log file (you can open the IIS log file to see the header and know what fields to map)
    Module      xm_csv
    Fields   $date, $time, $s-ip, $cs-method, $cs-uri-stem, $cs-uri-query, $s-port, $cs-username, $c-ip, $cs(User-Agent), $sc-status, $sc-substatus, $sc-win32-status, $time-taken
    FieldTypes  string, string, string, string, string, string, integer, string, string, string, integer, integer, integer, integer
    Delimiter   ' '
</Extension>

<Extension multiline>
    Module        xm_multiline
    HeaderLine    /^\d{4}\-\d{2}\-\d{2} \d{2}\:\d{2}\:\d{2}.\d{3}/
</Extension>

<Extension charconv>
    Module    xm_charconv
    AutodetectCharsets utf-8, utf-16, utf-32, iso8859-2
</Extension>

<Input eventlog>
    Module      im_msvistalog
    ReadFromLast    True
    Module      im_msvistalog
        Query       <QueryList>\
                        <Query Id="0">\
                            <Select Path="Application">*</Select>\
                            <Select Path="System">*</Select>\
                            <Select Path="Security">*</Select>\
                        </Query>\
                    </QueryList>
</Input>
 
<Input iis>
    Module      im_file
    File 'C:\inetpub\logs\LogFiles\W3SVC1\u_ex*.log'
    ReadFromLast TRUE
    Exec        if $raw_event =~ /^#/ drop();                \
                else                                             \
                {                                                \
                    w3c->parse_csv();                            \
                    $EventTime = parsedate($date + " " + $time); \
                    to_json ();                                  \
                }
</Input>

<Input webconsole>
    Module im_file
    File 'C:\Stash\WebConsole.log' 
    InputType multiline
    SavePos TRUE
    # The call to convert_fields automatically converts the input to utf-8
    Exec        convert_fields("AUTO","utf-8"); \
            if $raw_event =~ /^(\d{4}\-\d{2}\-\d{2} \d{2}\:\d{2}\:\d{2}.\d{3})    \[(\S+)\]    \[(\S+)\]    \[(\S+)\]    \[(\S+)\]    \[(.*)\]    \[(.*)\]    (.*)/s \
                    { \
                        $time = $1; \
                        $hostname = $2; \
                        $activityId = $3; \
                        $userIddeviceId = $4; \
                        $threadId = $5; \
                        $level = $6; \
                        $logger = $7; \
                        $message = $8; \
                        to_json(); \
                    } \
                    else \
                    { \
                        drop(); \
                }
</Input>

<Output eventlog-out>
    Module      om_tcp
    Host        127.0.0.1
    Port        3515
    Exec        $EventReceivedTime = integer($EventReceivedTime) / 1000000; \
                to_json();
</Output>
 
<Output iis-out>
    Module      om_tcp
    Host        127.0.0.1
    Port        3516
</Output>

<Output general-out>
    Module      om_tcp
    Host        127.0.0.1
    Port        3517
</Output>

<Route 1>

    Path eventlog => eventlog-out
</Route>
 
<Route 2>
    Path iis => iis-out
</Route>

<Route 3>
    Path webconsole => general-out
</Route>

Sample logs(Webconsole.log):

2015-07-10 10:24:17.424    [20EX15736]    [00000000-0000-0000-0000-000000000000]    [0000000-0000000]    [00008]    [Info ]    [TestModule]    Testing log stash3    
2015-07-10 10:24:17.425    [20EX15736]    [00000000-0000-0000-0000-000000000000]    [0000000-0000000]    [00008]    [Info ]    [TestModule]    Testing log stash4    
2015-07-10 10:24:17.448    [20EX15736]    [00000000-0000-0000-0000-000000000000]    [0000000-0000000]    [00008]    [Error]    [TestModule]    *** EXCEPTION ***
System.DivideByZeroException: Attempted to divide by zero.
   at TCPPublisher.Program.Main(String[] args) in c:\Users\test\Documents\Visual Studio 2013\Projects\TCPDemo\TCPPublisher\Program.cs:line 26

Thanks in advance!
Mohan G

Hey,

nxlog.log dosent create.     the folder /var/log/nxlog/  is empty.  

My environmet is RedHat 6.

What can be the reason ? 

Thanks.

Hi,

i have follwing logentry:

I have the following sql_exec command outline in my config:

<Output out>
    Module      om_odbc
    ConnectionString    DSN=nxlog;
    <Exec>
      sql_exec("INSERT INTO eventlog (hostname) VALUES (?)", '$Hostname');
    </Exec>
</Output>

This is exactly how the 2 examples in the documentation have it set up, I cannot seem to get it to work though I get the following error in my error log:

procedure 'sql_exec()' does not exist or takes different arguments

Can anyone shed any light on why this would be happening or exactly what arguements it is looking for?  Thanks