Basic Configuration from syslog flat file to IBM QRadar - Connectivity seems to work, can't tell if I'm sending data.

Post a different question
8
responses

We're using NX Log (CE) as a test to see if it will work for our purposes. The overall idea is to use it as a forwarder of syslog flat files to any brand of SIEM.

My config looks like this:

Panic Soft
#NoFreeOnExit TRUE

define ROOT C:\Program Files (x86)\nxlog
define CERTDIR %ROOT%\cert
define CONFDIR %ROOT%\conf
define LOGDIR %ROOT%\data
define LOGFILE %LOGDIR%\nxlog.log
LogFile %LOGFILE%

Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data

<Extension syslog>
Module xm_syslog
</Extension>

#<Input Alerts>
# Module im_file
# File 'E:\\DGQradarExports\\ForwarderCust\\Alerts\\*'
# ReadFromLast True
# Exec parse_syslog();
#</Input>

<Input Events>
Module im_file
File 'E:\\DGQradarExports\\ForwarderCust\\Events\\*'
ReadFromLast True
Exec parse_syslog();
</Input>

<Input Process>
Module im_file
File 'E:\\DGQradarExports\\ForwarderCust\\Process\\*'
ReadFromLast True
Exec parse_syslog();
</Input>

<Output Customer>
Module om_tcp
Host 192.168.160.141
Port 514
</Output>

<Route customer_siem>
Path Events,Process => Customer
</Route>

<Extension _charconv>
Module xm_charconv
AutodetectCharsets iso8859-2, utf-8, utf-16, utf-32
</Extension>

<Extension _exec>
Module xm_exec
</Extension>

<Extension _fileop>
Module xm_fileop

# Check the size of our log file hourly, rotate if larger than 5MB
<Schedule>
Every 1 hour
Exec if (file_exists('%LOGFILE%') and \
(file_size('%LOGFILE%') >= 5M)) \
file_cycle('%LOGFILE%', 8);
</Schedule>

# Rotate our log file every week on Sunday at midnight
<Schedule>
When @weekly
Exec if file_exists('%LOGFILE%') file_cycle('%LOGFILE%', 8);
</Schedule>
</Extension>

My log just shows this:
2018-11-19 08:50:43 INFO nxlog-ce-2.10.2102 started
2018-11-19 08:50:43 INFO connecting to 192.168.160.141:514

QRadar shows an information source has registered but no data ever flows.

I should see a 'connection was successful' message shouldn't I?
Is there any way to up the logging so I can tell if NX Log is even reading the files and attempting to send them? I really can't tell what it's doing currently.

Multiple files exist in the input directories, I'm trying to have NX Log work through all of them, send them to SIEM and then wait for more files. Config examples seem straightforward, I just can't tell what it's doing.

Any help is appreciated.

AskedNovember 19, 2018 - 3:08pm

Answers (2)

One tip I love for working out config issues in foreground mode and using log_info() to show me events in real time.
nxlog -f to run in foreground mode.

Something like the following to add log_info() to the modules.

<Input Process>
    Module im_file
    File 'E:\\DGQradarExports\\ForwarderCust\\Process\\*'
    ReadFromLast True
    Exec parse_syslog();
    Exec log_info("Input Event: " + $raw_event);
</Input>

<Output Customer>
    Module om_tcp
    Host 192.168.160.141
    Port 514
    Exec log_info("Output Event: " + $raw_event);
</Output>

Once you see if NXLog is processing events how you would like, you could use something like wireshark or tcpdump to make sure the events are making it to your end server. Port not open all the way is a common issue we see.

AnsweredNovember 19, 2018 - 3:55pm

Comments (5)

  • nhart's picture

    I added the logging sections as per the below.

    <Input Events>
    Module im_file
    File 'E:\\DGQradarExports\\ForwarderCust\\Events\\*'
    ReadFromLast False
    Exec parse_syslog();
    Exec log_info("Input Event: " + $raw_event);
    </Input>

    <Input Process>
    Module im_file
    File 'E:\\DGQradarExports\\ForwarderCust\\Process\\*'
    ReadFromLast False
    Exec parse_syslog();
    Exec log_info("Input Event: " + $raw_event);
    </Input>

    <Output Customer>
    Module om_udp
    Host 192.168.160.141
    Port 514
    Exec log_info("Input Event: " + $raw_event);
    </Output>

    Running nxlog.exe in the foreground doesn't show me any more info.
    CMD Output:
    C:\Program Files (x86)\nxlog>nxlog.exe -f
    2018-11-20 04:21:49 INFO nxlog-ce-2.10.2102 started

    I can however see all the files being touched by the nxlog.exe process in procmon.

    November 20, 2018 - 10:26am
  • b0ti's picture
    (NXLog)

    For testing you should also add SavePos FALSE or remove configcache.dat manually to make sure the files are read from the beginning every time.
    Also Exec parse_syslog(); doesn't make much sense in this scenario if the input file isn't syslog and the parsed fields are not used.

    November 20, 2018 - 10:45am
  • Zhengshi's picture
    (NXLog)

    The flat files are syslog, I assumed NX Log needed to Parse them to send them over as Syslog via UDP?

    If you aren't modifying the underlying fields, then it isn't important. NXLog will forward as is, and your receiver will see them as Syslog if they are Syslog in your Input.

    When you use parse_syslog, it will populate the individual fields from their values in $raw_event. The manual goes into detail about which fields are created. This is useful if you need to modify a field to change or insert a new value, do calculations based on a field, make decisions on the event based on a field, etc.

    November 20, 2018 - 4:07pm
  • nhart's picture

    Using SavPos = False seemed to shunt data over. I can only assume this is because connectivity was finally right and it reprocessed the files.
    I'm leaving it running for 24hrs to ensure it works with fresh flat files that are added and that my 'cycling' scripts work for the old stuff.

    I've installed the enterprise trial for now so I can test leef also.

    November 20, 2018 - 3:58pm

It can be configured to send logs in the FileForwarder format. We are currently working on documentation that will have the details on sending some common formats to QRadar.
As an MSP you should also consider the following:

AnsweredNovember 19, 2018 - 5:16pm

Comments (1)

  • nhart's picture

    I'm currently waiting for a call with NXLog on the enterprise edition and if it'll work for us.
    I was hoping to test this out in a basic sense before I invested much more time into it. Sadly so far it's been pretty unsuccessful testing.

    As far as I can tell the enterprise edition doesn't offer any more logging potential so for this particular issue it doesn't strike me as helpful to install the trial instead.

    November 20, 2018 - 10:32am