Determine NXLog Agent Health Status

Post a different question
7
responses

Hi,

We are planning to deploy NXLog to thousands of endpoints and need to know when an agent is no longer sending data regularly.

Is there an established method for determining NXLog is working normally at scale?

Thanks!

AskedFebruary 5, 2020 - 5:15pm

Answer (1)

Hey!

You can use statistical counters to help see this.
Check out the Detecting a Dead Agent or Log Source chapter in the manual.

Basically it is this config:

<Input in>
    Module  im_tcp
    Port    2345
    <Exec>
        create_stat("msgrate", "RATE", 3600);
        add_stat("msgrate", 1);
    </Exec>
    <Schedule>
        Every   3600 sec
        <Exec>
            create_stat("msgrate", "RATE", 10);
            add_stat("msgrate", 0);
            if defined get_stat("msgrate") and get_stat("msgrate") <= 1
                log_error("No messages received from the source!");
        </Exec>
    </Schedule>
</Input>
AnsweredFebruary 5, 2020 - 6:59pm

Comments (6)

  • casey1234's picture

    Hi Zhengshi!

    This is great but we're trying to find out if NXLog itself or the server it resides on is down.

    What most of our applications do is send in a periodic REST request into an API at a fifteen minute interval to check in with various agent details.
    We were hoping we could leverage NXLog to send this same kind of heartbeat and then after a certain interval it would be flagged.

    I've been simulating this by sending data from script to an http output module that then forwards the request but it looks like this is creating excess logging.

    It looks like your above method you're using NXLog to listen to other sources, however we're hoping monitor NXLog itself and to continue using our API gateway to receive agent polling.

    Does that make sense?

    Suggestions?

    Thanks buddy!

    February 5, 2020 - 7:27pm
  • casey1234's picture

    Hi Zhengshi!

    This is great but we're trying to find out if NXLog itself or the server it resides on is down.

    What most of our applications do is send in a periodic REST request into an API at a fifteen minute interval to check in with various agent details.
    We were hoping we could leverage NXLog to send this same kind of heartbeat and then after a certain interval it would be flagged.

    I've been simulating this by sending data from script to an http output module that then forwards the request but it looks like this is creating excess logging.

    It looks like your above method you're using NXLog to listen to other sources, however we're hoping monitor NXLog itself and to continue using our API gateway to receive agent polling.

    Does that make sense?

    Suggestions?

    Thanks buddy!

    February 5, 2020 - 10:17pm
  • Arkadiy's picture

    Hello.

    Looks like modules im_mark and om_http can help you. First one is create a message in chosen interval and the other are simply sending stuff to selected address. So in your case it will looks like this:

    <Input mark>
    Module  im_mark
    MarkInterval    5
    Mark    YOUR ADVERTISING HERE
</Input>

<Output http>
        Module om_http
        URL http://server:port
</Output>


<Route ping>
        Path mark => http
</Route>

    More info about modules are here:

    • https://nxlog.co/documentation/nxlog-user-guide/im_mark.html

    • https://nxlog.co/documentation/nxlog-user-guide/om_http.html

    Let us now is this solves your problem.

    Best regards, Arch

    February 6, 2020 - 8:23pm
  • casey1234's picture

    Ok great!

    If I wanted to include server a specific attribute that would be generated from script, can I dynamically add that to the Mark property?

    Is it possible to execute the script and then set it as a global constant that I can use in the Mark property?

    Thanks for the great feedback!

    February 7, 2020 - 7:55pm
  • Arkadiy's picture

    Hi,

    This way NXLog will parse any incoming message - right now they comes only from im_mark - and invoke perl function. This function add environmental variable to $raw_message as $Message field and you can do anything with it now.

    Config:

    Panic Soft

define CERTDIR /opt/nxlog/var/lib/nxlog/cert
define CONFDIR /opt/nxlog/var/lib/nxlog
define LOGDIR /opt/nxlog/var/log/nxlog
define MYLOGFILE %LOGDIR%/nxlog.log
include %CONFDIR%/log4ensics.conf

<Extension perl>
    Module xm_perl
    PerlCode /opt/perl.pl
</Extension>

<Extension>
    Module xm_json
</Extension>

<Input in>
    Module im_mark
    MarkInterval 5
    Exec log_info("Before:" + $raw_event);
    <Exec>
    if not 
        perl_call("process");
        to_json();
    </Exec>
</Input>

<Output out>
    #Module om_http    
    #URL http://server:port
    Module om_null
    Exec log_info("After:" + $raw_event);
</Output>

<Route py>
    Path in => out
</Route>

    and Perl file for example:


    use strict;
use warnings;

#module for working with event data from NxLog
use Log::Nxlog;
# use Env;


sub process
{
    my ( $event ) = @_;
    my $mark = Log::Nxlog::get_field($event, 'Message');
    my $userName = $ENV{'LOGNAME'};
    my $answer = "Important message from Perl: Hello, $userName";
    if ( defined($mark))
    {   

        Log::Nxlog::set_field_string($event, 'Message', $answer);
        Log::Nxlog::log_info('message changed');
    }
}

    Other way is to use directives like envvar, which allows you to include Enviromental Variables directly into nxlog.conf along with include_stdout which could execute some external script and get it's STDOUT in the nxlog.conf. This one is much easier but works only in Enterprise Edition. :)

    Hope it helps.

    Best regards, Arch

    February 11, 2020 - 12:22pm