NXLog Agent Minder is a hyper scalable API first agent management solution for NXLog Enterprise Edition. It comes in the form of a server, a command line management tool and a set of well documented public APIs.

Getting started with NXLog Agent Minder

Installing NXLog Agent Minder

The product is delivered as a generic Linux package in DEB and RPM package formats. The binary is currently statically linked.

Installing on Debian and Ubuntu systems

apt install ./nxlog-minder.deb

Installing on Redhat, SUSE and other RPM based systems

zypper install ./nxlog-minder.rpm
yum install ./nxlog-minder.rpm
dnf install ./nxlog-minder.rpm

Starting and stopping NXLog Agent Minder

systemctl start minder.service

Checking the logs of NXLog Agent Minder

systemctl status minder.service
journalctl --unit minder.service

Initial Configuration

NXLog Agent Minder will generate a CA key and certificate and a server key and certificate for the agent connection port and the API port. These are stored in /opt/minder/conf/cert and /opt/minder/conf/PKI

Using your own keys and certificates

Overwrite the existing keys and certificates as needed and they will be loaded on next start.

/opt/minder/conf/cert holds the files used for external communication.

  • agent communication

    • minder-cert.pem

    • minder-key.pem

  • API port communication

    • api-cert.pem

    • api-key.pem

/opt/minder/conf/PKI holds the files of the PKI

  • ca-cert.pem

  • ca-key.pem

Connecting an agent

Agents will need to be set up with a simple configuration file.

This configuration file must be placed in /opt/nxlog/etc/nxlog.d and named managed.conf

The file must be included from the main nxlog.conf file located in /opt/nxlog/etc.

define NXLOG_MANAGER_ADDRESS 192.168.1.1
define NXLOG_MANAGER_PORT 4041

LogLevel    INFO
LogFile     %MYLOGFILE%

<Extension agent_managment>
    Module          xm_admin
    Connect         %NXLOG_MANAGER_ADDRESS%
    Port            %NXLOG_MANAGER_PORT%
    SocketType      SSL
    AllowUntrusted  TRUE
    RequireCert     FALSE
    <ACL conf>
        Directory   %CONFDIR%
        AllowRead   TRUE
        AllowWrite  TRUE
    </ACL>
    <ACL cert>
        Directory   %CERTDIR%
        AllowRead   TRUE
        AllowWrite  TRUE
    </ACL>
    <labels>
        dplstate "new"
    </labels>
</Extension>
Note
nxlog.conf is set up with the required include when it is installed.
Important
If you are running NXLog Enterprise Edition version 4.x the configuration file location is /opt/nxlog/var/lib/nxlog/log4ensics.conf

Configuring the new agent

First the agent needs to be enrolled. The IP address 192.68.1.1 belongs to the NXLog Agent Minder server, and 4041 is the agent management port.

./cli.sh enroll agent-1 192.168.1.1:4041
Note
Specifying the wrong IP address here will take the agent offline. Make sure you use the address that will be visible for the agent. Network and port address translation techniques, load balancers may occlude the actual IP address of the agent-manager.

Then the configuration can be edited:

./cli.sh edit-agent agent-1

This will load the current agent configuration in an editor:

LogLevel  INFO
LogFile   %MYLOGFILE%

<Extension admin>
    Module       xm_admin

    Connect      192.168.1.1
    Port         4041
    SocketType   SSL
    CAFile       %CERTDIR%/agent-ca.pem
    CertFile     %CERTDIR%/agent-cert.pem
    CertKeyFile  %CERTDIR%/agent-key.pem

    <ACL conf>
         Directory   %CONFDIR%
         AllowRead   TRUE
         AllowWrite  TRUE
    </ACL>

    <ACL cert>
         Directory   %CERTDIR%
         AllowRead   TRUE
         AllowWrite  TRUE
    </ACL>

    <labels>
       dplstate  "enrolled"
    </labels>
</Extension>

It is recommended to change the label dplstate to configured, to mark the agent as not needing additional configuration.

im_mark can be applied to generate heartbeat messages:

<Input mark>
    Module          im_mark
    MarkInterval    1
    Mark            NXLog heartbeat
</Input>

im_internal can collect the agent logs so they can also be sent to a central location:

<Input nxlog>
   Module im_internal
</Input>

Then an output module and a route can be set up so the messages are sent to a destination:

<Output tcp_out>
   Module om_tcp
   Host   your-syslog-server:1514
</Output>

<Route r_n>
   Path mark,nxlog => tcp_out
</Route>
Warning
Please take care to create a valid configuration. There is no configuration validation implemented at this point. If the new configuration breaks the agent, manual intervention on the agent will be necessary to recover.

Running a simple query

curl -k -sS -X GET 'https://minder-server:8080/agents/ids'

Prometheus quickstart

Prometheus can be deployed in a matter of minutes using docker and the following script.

Make sure you update the Prometheus configuration with the correct address of NXLog Agent Minder

#!/bin/bash

PROM_IMG=minder-prometheus-test
PROM_CONTAINER=minder-prometheus-2
GRAF_CONTAINER=minder-grafana-2
NET=minder-bridge
MINDER_SERVER=192.168.1.1:8080

# Create prometheus configuration
cat > prometheus.yml << EOF
global:
  scrape_interval:     15s # By default, scrape targets every 15 seconds.
  external_labels:
    monitor: 'codelab-monitor'
scrape_configs:
  - job_name: 'prometheus'
    scrape_interval: 5s
    static_configs:
      - targets: ['localhost:9090']
  - job_name: 'nxlog'
    scrape_interval: 60s
    scheme: https
    static_configs:
      - targets: ['$MINDER_SERVER']
    tls_config:
      insecure_skip_verify: true
EOF

cat > Dockerfile << EOF
FROM quay.io/prometheus/prometheus
ADD prometheus.yml /etc/prometheus
EOF
docker build -t $PROM_IMG .
docker create --name $PROM_CONTAINER --hostname $PROM_CONTAINER -p 9090:9090 $PROM_IMG
docker start $PROM_CONTAINER

docker run -h $GRAF_CONTAINER --name $GRAF_CONTAINER -d -p 3000:3000 grafana/grafana

docker network create -d bridge $NET
docker network connect  $NET $PROM_CONTAINER
docker network connect  $NET $GRAF_CONTAINER

NXLog Agent Minder architectural overview

NXLog Agent Minder is a server application that communicates with agents via its agent management interface (TCP port 4041), and provides a public API over its API port (TCP port 8080). It ships with a sample command line tool leveraging the API port for communication.

The NXLog Enterprise Edition agent runs a management extension module called xm_admin. This module implements the low level agent management API and it also handles the connection to NXLog Agent Minder

NXLog Agent Minder provides a Prometheus compatible metrics endpoint on its public API port.

The image below shows the connections used in a NXLog Agent Minder deployment.

NXLog EE Agent
NXLog EE Agent
NXLog EE Agent
NXLog EE Agent
NXLog EE Agent
NXLog EE Agent
NXLog EE Agent
NXLog EE Agent
NXLog EE Agent
NXLog EE Agent
NXLog EE Agent
NXLog EE Agent
NXLog Agent Minder
NXLog Agent Minder
cli.sh
cli.sh
custom client
custom client
TCP/8080
TCP/8080
TCP/4041
TCP/4041
xm_admin
xm_admin
Prometheus
Prometheus
Viewer does not support full SVG 1.1

NXLog Agent Minder command line interface

The command line interface is currently a shell script based prototype. Its functionality is expected to change. It will eventually be replaced.

cli.sh dependencies

It relies on the presence of the following command line tools:

  • curl

  • jq

  • sed

  • grep

  • cat

  • edit

The script uses the public API’s provided by the product.

Using the command line interface

cli.sh invocation takes the following general format

cli.sh command [target] [parameters]

Please refer to the built-in help for details:

> cli.sh help
Usage cli.sh [minder-url] command [target] [parameters]

 minder-url is the optional API URL of minder. Defaults to https://127.0.0.1:8080

 command is one of
   info target
     print serverinfo JSON for target
   state target
      print operational state for target
   agentstats
      print agent status grid
   edit-agent agent-name
      loads config from agent, edits it locally,
      pushes it back, then restarts agent
   restart target
      restart agents
   start target
      start agents
   stop target
      stop agents
   list [filter]
      print a list of agent names for filter
   count [filter]
      print the number of agents matching filter
   getlog target
      pull and print log. The third argument can optionally provide the number of bytes to read
   getconf target
      pull and print config from target. In case of an agent name the literal contents of the file are returned.
      When a filter is used a JSON structure is returner
   enroll target agent_manager_address
      deploy key/cert and connection configuration to agents. Only connect mode agents supported.
   getfile target ACL file-name
      get and print file contents from target
   putfile target ACL file-name
      put file on the agent's specified ACL
   putconfig filter file-name
      put the file on the specifed agents as the config file
   checkconfig agent-name file-name
      get and compare the agent's config file to the local file

 target is a
   filter or
   agent name

 filter works as follows:
   keyword=search_expression for exact matching (FASTEST)
   keyword=regex_expression for regex matching (SLOWEST)
   keyword~=search_expression for substring matching

 keyword can be the following
   name
   ip
   os
   version
   net
   module
   state
   route (currently unsupported)

NXLog Agent Minder Agent management best practices

Agent enrollment life-cycle

The recommended life-cycle consists of the following stages:

  1. Stage: new

  2. Stage: enrolled

  3. Stage: configured

Initially the agent starts up with its deployment configuration. That configuration should have the following content in /opt/minder/etc/nxlog/nxlog.d/managed.conf.

define NXLOG_MANAGER_ADDRESS 192.168.1.1
define NXLOG_MANAGER_PORT 4041

LogLevel    INFO
LogFile     %MYLOGFILE%

<Extension agent_managment>
    Module          xm_admin
    Connect         %NXLOG_MANAGER_ADDRESS%
    Port            %NXLOG_MANAGER_PORT%
    SocketType      SSL
    AllowUntrusted  TRUE
    RequireCert     FALSE
    <ACL conf>
        Directory   %CONFDIR%
        AllowRead   TRUE
        AllowWrite  TRUE
    </ACL>
    <ACL cert>
        Directory   %CERTDIR%
        AllowRead   TRUE
        AllowWrite  TRUE
    </ACL>
    <labels>
        dplstate "new"
    </labels>
</Extension>

Deployment state labels

The label dplstate: "new" represents the agents deployment state.

When the agent is enrolled that will change to dplstate: "enrolled" This can be used to filter for the agents that need attention.

When the agent receives its production configuration its label should be changed to dplstate: "configured"

Using custom labels for targeting

NXLog Agent Minder generates a simple connection configuration during enrollment. This can be later overridden by including a modified version of it in the configuration files being pushed. Labels may be defined in the configuration or on the agents based on the output of scripts.

In case a safer way is needed for adding labels the following procedure may be applied.

Deploy custom labels script

Create script /usr/local/bin/custom_labels.sh on the agent:

#!/bin/bash
LABELCONFIG=/opt/nxlog/etc/nxlog.d/local_labels
LOCAL_PREFIX=l_

[ -f $LABELCONFIG ] && awk -v LOCAL_PREFIX=${LOCAL_PREFIX} '{print LOCAL_PREFIX$1, $2}' $LABELCONFIG

exit 0
Add custom labels

The script above loads the contents of the local_labels file.

phone "+15555555"
location "DC-1"
Note
The labels in local_labels will be prefixed by the string l_ to make it easier to distinguish and protect against name clashes. This behaviour can be changed in the script above.
Load local labels

The agent configuration should be amended as follows

LogLevel  INFO
LogFile   %MYLOGFILE%

<Extension admin>
    Module       xm_admin
    Connect      192.168.1.1
    Port         4041
    SocketType   SSL
    CAFile       %CERTDIR%/agent-ca.pem
    CertFile     %CERTDIR%/agent-cert.pem
    CertKeyFile  %CERTDIR%/agent-key.pem

    <ACL conf>
         Directory   %CONFDIR%
         AllowRead   TRUE
         AllowWrite  TRUE
    </ACL>

    <ACL cert>
         Directory   %CERTDIR%
         AllowRead   TRUE
         AllowWrite  TRUE
    </ACL>

    <labels>
       dplstate  "configured"
       include_stdout /usr/local/bin/custom_labels.sh
    </labels>
</Extension>

Container metadata as labels

This example script shows how labels can be leveraged to facilitate filtering on agent local information.

Dropping the following in /usr/local/bin/get_docker_id.sh

#!/bin/bash

awk -F / '/memory/{printf "container_id \"%s\"\n",$3}' /proc/self/cgroup
awk 'END{printf "container_ip \"%s\"\n", $1}' /etc/hosts

then adding include_stdout /usr/local/sbin/get_docker_id.sh in the Labels section will add container metadata as labels.

NXLog Agent Minder Command line options

The server can be started from the command line by typing the following.

start command
MINDER_LOG=info minder --api-addr 0.0.0.0:8080 --agent-addr 0.0.0.0:4041 --connect-to 1.2.3.4:8080 --cert /path/to/certificate.pem --key /path/to/private/key.pem

The recommended method for starting and stopping minder is via systemd. The package comes with a pre-configured unit file.

systemctl start minder.service

Logging configuration is stored in the file /opt/minder/conf/MINDER_LOG The file contains an environment variable.

MINDER_LOG=info
Table 1. command line options
option description

--log LOGLEVEL

specify the desired log level for minder, as one of error, warn, info, debug or trace

--log-config configuration_file

the file specifies should hold the logging settings in log4rs configuration format

--api-addr 0.0.0.0:8080

where the public API should listen (default value: 0.0.0.0:8080)

--api-creds USER[:PASSWORD]

credentials to access public API (no authentication required if unspecified)

--api-cert FILE (default value: ROOT_DIR\conf\cert\api-cert.pem)

specify the path to the certificate for the public API

--api-key FILE (default value: ROOT_DIR\conf\cert\api-key.pem)

specify the path to the private key for the public API

--agent-addr 0.0.0.0:4041

where NXLog Agent Minder listens for incoming agent connections (default value: 0.0.0.0:4041)

--connect-to 1.2.3.4:8080

agents in listening-mode for NXLog Agent Minder to connect to

--cert FILE (default value: ROOT_DIR\conf\cert\minder-cert.pem)

specify the path to the certificate for the server

--key FILE (default value: ROOT_DIR\conf\cert\minder-key.pem)

specify the path to the private key for the server

--ca-cert FILE (default value: ROOT_DIR\conf\PKI\ca-cert.pem)

specify the path to the CA certificate for the server

--ca-key FILE (default value: ROOT_DIR\conf\PKI\ca-key.pem)

specify the path to the CA private key for the server

--ca-pass PASSPHRASE

passphrase for the CA private key for the server if it’s encrypted

--probe-period PROBE_PERIOD

period (in seconds) with which to probe connected agents; should be in range from 60 to 1200 seconds (from 1 to 20 minutes)

--root_dir DIR (default value: /opt/minder on Unix-like platforms or C:\Program Files\minder on Windows)

specify the root directory of the minder’s file hierarchy; directories like ROOT_DIR/conf/cert should exist

Certificate management

NXLog Agent Minder requires three certificates to work:

  • CA: Used to sign and verify certificates of agents and NXLog Agent Minder.

  • Minder: the main certificate that is used to secure TLS connections between NXLog Agent Minder and agents. Must be signed with CA key.

  • API: secures the public API of NXLog Agent Minder.

The certificates and corresponding private keys can be specified explicitly in the NXLog Agent Minder configuration, otherwise they’re expected at their default locations. If particular certificate and private key are not specified in the app configuration and not available at default paths then NXLog Agent Minder generates them. Both Minder and API certificates generated by NXLog Agent Minder are always signed with CA key.

Table 2. environment variables
name values description

MINDER_LOG

error, warn, info, debug, trace

defines the level of logging

Configuring Prometheus

Prometheus is a time series database. NXLog Agent Minder provides a compatible api endpoint at https://minder-server:8080/metrics.

Prometheus can be set up to regularly scrape this endpoint and collect metrics using the following configuration:

scrape_configs:
  - job_name: 'nxlog'
    scrape_interval: 60s
    scheme: https
    static_configs:
      - targets: ['192.168.1.1:8080']
    tls_config:
      insecure_skip_verify: true

Creating a data source in Grafana

Grafana is a popular data visualization tool that works well with Prometheus. A data source must be set up in Grafana to access Prometheus' time series data.

The Bearer token is specific to your local installation.

You will need to have the following

  • Working authentication for Grafana API

  • Graphana host’s name, port and protocol

  • Prometheus host’s name, port, protocol

The example below shows a way for creating a data source.

curl -H "Authorization: Bearer BASE64_TOKEN_STRING" \
     -H "content-type: application/json" -X POST \
     "http://minder-graphana-2:3000/api/datasources/" \
     -d '{
           "name": "Prometheus-import",
           "type": "prometheus",
           "url": "http://minder-prometheus-2:9090",
           "access": "proxy"
}'
Note
If you used the example docker setup provided in the guide the host name in the data source definition above should work. The host name where curl connects should be changed to match your docker host. You will need to create an API key with admin rights and use that to perform this operation.

NXLog Agent Minder Public APIs

Counting agents

Table 3. agents/count
method + URI description

GET BASE_URI/agents/count?filter=<filter expression>

return number of agents matching specified filter

request

curl -k -sS -X GET 'https://minder-server:8080/agents/count?filter=os=Linux'

response

{
  "count": 5
}


Listing agents

Table 4. agents/ids
method + URI description

GET BASE_URI/agents/ids?filter=<filter expression>

return IDs of agents matching specified filter

request

curl -k -sS -X GET 'https://minder-server:8080/agents/ids'

response

{
  "agents": [
    "agent-5",
    "agent-3",
    "agent-4",
    "agent-1",
    "agent-2"
  ]
}


Agent information

Table 5. agents
method + URI description

GET BASE_URI/agents?filter=<filter expression>

return information about agents matching specified filter

request

curl -k -sS -X GET 'https://minder-server:8080/agents?filter=name=agent-1'

response

{
  "agents": {
    "agent-1": {
      "status": "success",
      "data": {
        "started": 1607085234314904,
        "load": 0.009999999776482582,
        "pid": 1,
        "mem": 7925760,
        "os": "Linux",
        "version": "5.1.6201",
        "systeminfo": "OS: Linux, Hostname: agent-1, Release: 5.9.11-1-default, Version: #1 SMP Wed Nov 25 05:49:27 UTC 2020 (91426ef), Arch: x86_64, 8 CPU(s), 31.2Gb memory",
        "hostname": "agent-1",
        "servertime": 1607088935658667,
        "modules": {},
        "labels": {
          "dplstate": "new",
          "container_id": "91331611a670fcaad03473bc1f8c546f5ee5c18cbecefed0b2406ee3f3c7eca3",
          "container_ip": "172.21.0.2"
        }
      }
    }
  }
}


GET BASE_URI/agents/:agent-1

return information about agent-1

request

curl -k -sS -X GET 'https://minder-server:8080/agents/:agent-1'

response

{
  "started": 1607085234314904,
  "load": 0,
  "pid": 1,
  "mem": 7925760,
  "os": "Linux",
  "version": "5.1.6201",
  "systeminfo": "OS: Linux, Hostname: agent-1, Release: 5.9.11-1-default, Version: #1 SMP Wed Nov 25 05:49:27 UTC 2020 (91426ef), Arch: x86_64, 8 CPU(s), 31.2Gb memory",
  "hostname": "agent-1",
  "servertime": 1607089112947565,
  "modules": {},
  "labels": {
    "dplstate": "new",
    "container_id": "91331611a670fcaad03473bc1f8c546f5ee5c18cbecefed0b2406ee3f3c7eca3",
    "container_ip": "172.21.0.2"
  }
}


Agent commands

Table 6. agents
method + URI description

POST BASE_URI/agents?filter=<filter expression>

execute an operation (start/restart/stop) on agents matching specified filter. POST data must contain the command in json format: {"operation": "command" }

request

curl -k -sS -H "content-type: application/json"  -X POST 'https://minder-server:8080/agents?filter=name=agent-1' -d '{"operation": "restart" }'

response

{
  "agents": {
    "agent-1": {
      "status": "success",
      "data": null
    }
  }
}


POST BASE_URI/agents/:agent-1

execute an operation (start/restart/stop) on agent-1. POST data must contain the command in json format: {"operation": "command" }

request

curl -k -sS -H "content-type: application/json"  -X POST 'https://minder-server:8080/agents/:agent-1' -d '{"operation": "restart" }'

response

  • The API responds with HTTP/1.1 204 No Content on success

  • The API responds with HTTP/1.1 404 Not Found if the agent is unknown to NXLog Agent Minder

Module information

Table 7. agents/modules
method + URI description

GET BASE_URI/agents/modules/:module1?filter=<filter expression>

return info about module module1 of agents matching specified filter.

request

curl -k -sS -X GET 'https://minder-server:8080/agents/modules/:module1?filter=name=agent-1'

response

{
  "agents": {
    "agent-1": {
      "status": "success",
      "data": {
        "module_name": "module1",
        "evt-recvd": 0,
        "evt-drop": 0,
        "evt-fwd": 0,
        "queuesize": 0,
        "queuelimit": 100,
        "batchsize": 0,
        "status": 3,
        "module-type": 3,
        "module": "om_null",
        "variables": {}
      }
    }
  }
}


GET BASE_URI/agents/:agent-1/modules/:module1

return info about module module1 of agent-1

request

curl -k -sS -X GET 'https://minder-server:8080/agents/:agent-1/modules/:module1'

response

{
  "module_name": "module1",
  "evt-recvd": 0,
  "evt-drop": 0,
  "evt-fwd": 0,
  "queuesize": 0,
  "queuelimit": 100,
  "batchsize": 0,
  "status": 3,
  "module-type": 3,
  "module": "om_null",
  "variables": {}
}


Module commands

Table 8. agents/modules
method + URI description

POST BASE_URI/agents/modules/:module1?filter=<filter expression>

execute an operation (start/restart/stop) for the module module1 on agents matching specified filter. POST data must contain the command in json format: {"operation": "command" }

request

curl -k -sS -H "content-type: application/json" -X POST \
    'https://minder-server:8080/agents/modules/:module1?filter=name=agent-1' \
    -d '{"operation": "restart" }'

response

{
  "agents": {
    "agent-1": {
      "status": "success",
      "data": null
    }
  }
}


Reading files

Table 9. agents/files
method + URI description

GET BASE_URI/agents/files/:path/:file?filter=<filter expression>

retrieve the content of the file file under ACL path of agents matching specified filter in the response body. The response is a JSON structure as multipe agents may contain files named the same.

request

curl -k -sS -X GET 'https://minder-server:8080/agents/files/:conf/:managed.conf?filter=name=agent-2'

response

{
  "agents": {
    "agent-2": {
      "status": "success",
      "data": {
        "file": "LogLevel  INFO\nLogFile   %MYLOGFILE%\n\n<Extension admin>\n    Module       xm_admin\n\n    Connect      192.168.1.1\n    Port         4041\n    SocketType   SSL\n    CAFile       %CERTDIR%/agent-ca.pem\n    CertFile     %CERTDIR%/agent-cert.pem\n    CertKeyFile  %CERTDIR%/agent-key.pem\n\n    <ACL conf>\n         Directory   %CONFDIR%\n         AllowRead   TRUE\n         AllowWrite  TRUE\n    </ACL>\n\n    <ACL cert>\n         Directory   %CERTDIR%\n         AllowRead   TRUE\n         AllowWrite  TRUE\n    </ACL>\n\n    <labels>\n       dplstate  \"enrolled\"\n    </labels>\n</Extension>\n"
      }
    }
  }
}


GET BASE_URI/agents/:agent-1/files/:path/:file

retrieve the content of the file file under ACL path of agent-1 in the response body. The content is returned as plain text.

request

curl -k -sS -X GET 'https://minder-server:8080/agents/:agent-2/files/:conf/:managed.conf'

response

LogLevel  INFO
LogFile   %MYLOGFILE%

<Extension admin>
    Module       xm_admin

    Connect      192.168.1.1
    Port         4041
    SocketType   SSL
    CAFile       %CERTDIR%/agent-ca.pem
    CertFile     %CERTDIR%/agent-cert.pem
    CertKeyFile  %CERTDIR%/agent-key.pem

    <ACL conf>
         Directory   %CONFDIR%
         AllowRead   TRUE
         AllowWrite  TRUE
    </ACL>

    <ACL cert>
         Directory   %CERTDIR%
         AllowRead   TRUE
         AllowWrite  TRUE
    </ACL>

    <labels>
       dplstate  "enrolled"
    </labels>
</Extension>


Writing files

Table 10. agents/files
method + URI description

PUT BASE_URI/agents/files/:path/:file?filter=<filter expression>

put the content of the request body into the file file under ACL path of agents matching specified filter

request

curl -k -sS -X PUT 'https://minder-server:8080/agents/files/:conf/:local_labels?filter=name=agent-1' -d 'phone "+155555555"'

response

{
  "agents": {
    "agent-1": {
      "status": "success",
      "data": null
    }
  }
}


PUT BASE_URI/agents/:agent-1/files/:path/:file

put the content of the request body into the file file under ACL path of agent-1

request

curl -k -sS -X PUT 'https://minder-server:8080/agents/:agent-1/files/:conf/:local_labels' -d 'phone "+155555552"'

response

  • The API responds with HTTP/1.1 204 No Content on success

  • The API responds with HTTP/1.1 404 Not Found if the file cannot be found on the agent

Reading the internal log

Table 11. agents/log
method + URI description

GET BASE_URI/agents/log?filter=<filter expression>&maxSize=1024

retrieve the content of the log file of the agents matching specified filter limited by maxSize bytes (or 10000 bytes if not specified)

request

curl -k -sS -X GET 'https://minder-server:8080/agents/log?filter=name=agent-1&maxSize=100'

response

{
  "agents": {
    "agent-1": {
      "status": "success",
      "data": {
        "logfile": "3:04:42 INFO [xm_admin|admin] getLog called\n2020-12-09 13:05:03 INFO [xm_admin|admin] getLog called\n"
      }
    }
  }
}


GET BASE_URI/agents/:agent-1/log?maxSize=1024

retrieve the content of the log file of agent-1 limited by maxSize bytes (or 10000 bytes if not specified)

Reading the configuration

Table 12. agents/config
method + URI description

GET BASE_URI/agents/config?filter=<filter expression>

retrieve the content of the config file of agents matching specified filter

request

curl -k -sS -X GET 'https://minder-server:8080/agents/config?filter=name=agent-1'

response

{
  "agents": {
    "agent-1": {
      "status": "success",
      "data": {
        "file": "LogLevel  INFO\nLogFile   %MYLOGFILE%\n\n<Extension admin>\n    Module       xm_admin\n\n    Connect      192.168.1.1\n    Port         4041\n    SocketType   SSL\n    CAFile       %CERTDIR%/agent-ca.pem\n    CertFile     %CERTDIR%/agent-cert.pem\n    CertKeyFile  %CERTDIR%/agent-key.pem\n\n    <ACL conf>\n         Directory   %CONFDIR%\n         AllowRead   TRUE\n         AllowWrite  TRUE\n    </ACL>\n\n    <ACL cert>\n         Directory   %CERTDIR%\n         AllowRead   TRUE\n         AllowWrite  TRUE\n    </ACL>\n\n    <labels>\n       dplstate  \"configured\"\n       include_stdout /usr/local/bin/get_docker_id.sh\n       include_stdout /usr/local/bin/custom_labels.sh\n       template \"tpl1\"\n    </labels>\n</Extension>\n\n<Input i_n>\n   Module im_null\n</Input>\n\n<Output module1>\n   Module om_null\n</Output>\n\n<Route r_n>\n   Path i_n => module1\n</Route>"
      }
    }
  }
}


GET BASE_URI/agents/:agent-1/config

retrieve the content of the config file of agent-1

request

curl -k -sS -X GET 'https://minder-server:8080/agents/:agent-1/config'

response

LogLevel  INFO
LogFile   %MYLOGFILE%

<Extension admin>
    Module       xm_admin

    Connect      192.168.1.1
    Port         4041
    SocketType   SSL
    CAFile       %CERTDIR%/agent-ca.pem
    CertFile     %CERTDIR%/agent-cert.pem
    CertKeyFile  %CERTDIR%/agent-key.pem

    <ACL conf>
         Directory   %CONFDIR%
         AllowRead   TRUE
         AllowWrite  TRUE
    </ACL>

    <ACL cert>
         Directory   %CERTDIR%
         AllowRead   TRUE
         AllowWrite  TRUE
    </ACL>

    <labels>
       dplstate  "configured"
       include_stdout /usr/local/bin/get_docker_id.sh
       include_stdout /usr/local/bin/custom_labels.sh
       template "tpl1"
    </labels>
</Extension>

<Input i_n>
   Module im_null
</Input>

<Output module1>
   Module om_null
</Output>

<Route r_n>
   Path i_n => module1
</Route>


Writing the configuration

Table 13. agents/config
method + URI description

PUT BASE_URI/agents/config?filter=<filter expression>

put the content of the config file of agents matching specified filter

request

curl -k -sS -X PUT 'https://minder-server:8080/agents/config?filter=name=agent-1'

response

{
  "agents": {
    "agent-1": {
      "status": "success",
      "data": null
    }
  }
}
Important
This call performs no checks on the validity of the configuration file. Invalid configuration may make the agent unreachable after restart.

PUT BASE_URI/agents/:agent-1/config

put the content of the config file of agent-1

request

curl -k -sS -X PUT 'https://minder-server:8080/agents/:agent-1/config' -d "Better be a valid configuration file!"

response

  • The API responds with HTTP/1.1 204 No Content on success

Important
This call performs no checks on the validity of the configuration file. Invalid configuration may make the agent unreachable after restart.

Reading agent state

Table 14. agents/state
method + URI description

GET BASE_URI/agents/state?filter=<filter expression>

get the state (ok, warning or error) of agents matching specified filter

request

curl -k -sS -X GET 'https://minder-server:8080/agents/state?filter=name=agent-\[12\]'

response

{
  "agent-1": "ok",
  "agent-2": "ok"
}


GET BASE_URI/agents/:agent-1/state

get the state (ok, warning or error) of agent-1

request

curl -k -sS -X GET 'https://minder-server:8080/agents/:agent-1/state'

response

{
  "state": "ok"
}


Enrolling agents

Enrolling agents is a complex procedure. It consists of the following steps:

  • generating connection configuration

  • generating client key

  • generating client certificate

  • pushing the connection configuration to the targeted agents set

  • pushing the client key to the targeted agent set

  • pushing the client certificate to the targeted agent set

  • pushing the CA certificate to the targeted agent set

  • restarting the targeted agent set

All of these steps are handled in the background by NXLog Agent Minder.

To enroll your agents you will need to provide the apparent IP address and port of NXLog Agent Minder - 192.168.1.1:4041 in our examples.

Note
If your agents already have a valid configuration this will overwrite that, effectively resetting the agent to a blank state.
Table 15. agents/enroll

method + URI

description

POST BASE_URI/agents/enroll?filter=<filter expression>

enroll agents matching specified filter (see description of enrollment request format below)

request

curl -k -sS -H "content-type: application/json" \
	-X POST 'https://minder-server:8080/agents/enroll?filter=name=agent-2' \
	-d '{ "connection":{"mode": "connect", "address": "192.168.1.1:4041"}}'

response

{
  "agents": {
    "agent-2": {
      "status": "success"
    }
  }
}


POST BASE_URI/agents/:agent-1/enroll

enroll agent-1 (see description of enrollment request format below)

request

curl -k -sS -H "content-type: application/json" \
	-X POST 'https://minder-server:8080/agents/:agent-2/enroll' \
	-d '{ "connection":{"mode": "connect", "address": "192.168.1.1:4041"}}'

response

  • The API responds with HTTP/1.1 204 No Content on success

NXLog Agent Minder Agent Filter Syntax

Agents can be targeted individually or in sets using filters. Filters search the connected agents using the attributes provided in their serverinfo response data.

The general format of the filter expression:

attribute[!][~]=value

Where ! stands for negation, = for "equals", ~= for "almost equals".

There are three different match types supported.

Exact match

GET BASE_URI/agents/count?filter=name=agent-1 will match agent-1 and agent-1 only.

Regex match

GET BASE_URI/agents/count?filter=name=agent.* will match agents whose names begin with the string agent and continue with an arbitrary character sequence. See man 7 regex for details on regular expressions

Substring match

GET BASE_URI/agents/count?filter=name~=agent will match agents whose names contain the substring agent.

Tip
Substring matching is much less expensive computationally than the functionally equivalent regular expression.
Note
String literal in the filtering expression is case-sensitive.
Table 16. filter syntax
attribute description example

name

The name of the agent

GET BASE_URI/agents/count?filter=name=agentname

ip

The apparent IP address of the agent

GET BASE_URI/agents/count?filter=ip=192.168.0.1

os

The reported operating system of the agent

GET BASE_URI/agents/count?filter=os=linux

version

The NXLog Enterprise Edition agent’s version number

GET BASE_URI/agents/count?filter=version=5.1.6133

net

The apparent network the agent is part of (exact match only)

GET BASE_URI/agents/count?filter=net=192.168.0.0/24

module

The canonical name of the module

GET BASE_URI/agents/count?filter=module=o_udp

route

Currently unsupported, as this information is not contained in the agents' serverinfo response

GET BASE_URI/agents/count?filter=route=route_null

label

The name and value of the label provided by the agent. See the examples for details

GET BASE_URI/agents/count?filter=label=name to match on the presence of a label with name name (exact match)

GET BASE_URI/agents/count?filter=label~=labelname@labelvalue to match label with name name (exact match) and value value (substring match)

state

The state of the agent (ok, warning or error; exact match only)

GET BASE_URI/agents/count?filter=state=error

NXLog Agent Minder Enrollment Request Format

Table 17. top level object
field description format

certificate

optional parameters for agent’s certificate generation

see description of the certificate object below

connection

mandatory connection options of the agent

see description of the connection object below

confdir-path

optional path to the conf ACL for the agents that use non-standard conf ACL path

string containing path to the conf ACL

certdir-path

optional path to the cert ACL for the agents that use non-standard cert ACL path

string containing path to the cert ACL

Table 18. certificate object
field description format

common-name

optional common name (CN) of the agent’s certificate subject field

string; agent’s hostname is used if this field is unspecified or null

country

optional country (C) of the agent’s certificate subject field

string containing two-letter country code; country (C) is not set if this field is unspecified or null

state

optional state (ST) of the agent’s certificate subject field

string; state (ST) is not set if this field is unspecified or null

locality

optional locality (L) of the agent’s certificate subject field

string; locality (L) is not set if this field is unspecified or null

organization

optional organization (O) of the agent’s certificate subject field

string; organization (O) is not set if this field is unspecified or null

organization-unit

optional organization unit (OU) of the agent’s certificate subject field

string; organization unit (OU) is not set if this field is unspecified or null

not-after

optional notAfter constraint of the agent’s certificate

string containing time in ASN.1 format; notAfter constraint of 730 days (2 years) from today is used if this field is unspecified or null

not-before

optional

string containing time in ASN.1 format; notBefore constraint of (0 days from) today is used if this field is unspecified or null

serial

optional serial field of the agent’s certificate

non-negative integer; 1 is used if this field is unspecified or null

encrypt_key

optional controls whether to encrypt agent’s private key (with a randomly generated password)

boolean; agent’s private key is unencrypted if this field is unspecified

Table 19. connection object
field description format

mode

mandatory connection mode of the agent

either "connect" or "listen" string

address

mandatory address to establish outgoing connection to or listen for incoming connections on

string containing NXLog Agent Minder’s IP address and agent management port of in "IP:PORT" format

Minimal Enrollment Request Example

{
    "connection": {
        "mode": "connect",
        "address": "192.168.1.1:4041"
    }
}

Full Enrollment Request Example

{
    "certificate": {
        "common-name": "agent",
        "country": "US",
        "state": "CA",
        "locality": "San-Francisco",
        "organization": "NXLog",
        "organization-unit": "Dev",
        "not-after": "20201231235959Z",
        "not-before": "20200101000000Z",
        "serial": 128
    },
    "connection": {
        "mode": "connect",
        "address": "192.168.1.1:4041"
    }
}

Prometheus Metrics

Minder exposes Prometheus metrics on BASE_URI/metrics endpoint.

Metrics endpoint uses the same credentials as all other API endpoints!

Table 20. exposed metrics
metric labels description

minder_cpu_load

-

CPU load of the minder

minder_memory_usage

-

Memory usage of the minder

minder_requests

-

Total amount of requests to the minder’s API

minder_response_time

-

Histogram of response times of the minder’s API

minder_errors

-

Total amount of errors returned by the minder’s API

agents

-

Total amount of agents connected

agents_connect

-

Amount of agents connected in 'connect' mode

agents_listen

-

Amount of agents connected in 'listen' mode

agent_state

agent

Agent’s state: 0 is OK, 1 is WARNING and 2 is ERROR

agent_requests

agent

Total amount of requests to the agent

agent_response_time

agent

Histogram of response times of the agent

agent_transport_errors

agent

Total amount of transport errors while communicating with the agent

agent_cpu_load

agent

load of the agent as in RFC546

agent_memory_usage

agent

Memory usage of the agent

module_events_received

agent, module_name, module_type

Total amounts of the events received by the module of the agent

module_events_dropped

agent, module_name, module_type

Total amount of the events dropped by the module of the agent

module_events_forwarded

agent, module_name, module_type

Total amount of the events forwarded by the module of the agent

module_queue_size

agent, module_name, module_type

Queue size of the module of the agent