Ask questions. Get answers. Find technical product solutions from passionate experts in the NXLog community.

"module file not found" when using file->file_size() or other file functions in Exec

I am trying to use the example in https://docs.nxlog.co/ce/current/index.html#om_file for file rotation on Windows (nxlog-ce-3.1.2319).

I receive the following error

ERROR Couldn't parse Exec block at xxx.conf:104; couldn't parse statement at line 107, character 29 in xxx.conf; module file not found
ERROR module 'testfile' has configuration errors

using this configuration.   The output works fine if I don't use the functions,  so I assume om_file must be loading (by default?).

<Output testfile>
    Module  om_file
    File    "E:/nxlog_output/active/nxlog-out.txt"
    <Exec>
        # Format output
        to_json();

        # Rotate file based on size, move to staging folder
        if (file->file_size() > 10M)
        {
            $stagingFolder = 'E:/nxlog_output/staged/';
            $newfile = $stagingFolder + 'data_' + strftime(now(), '%Y%m%d%H%M%S') + '.log';
            file->rotate_to($newfile);
        }
    </Exec>
</Output>

 


hukel
Replies: 6
View post »
gahorvath
Help using this forum - searching and following Google results

Apologies if I'm being dense, but I need some help with navigation of this site.


hukel
Replies: 1
View post »
gahorvath
file_name does't work. nxlog-ce-3.1.2319.msi

Hi,

I have installed nxlog service (nxlog-ce-3.1.2319.msi) on windows core 2019 machine. I have a config:

define EVENT_REGEX /^.*(<EventData>.+<\/EventData>)$/

<Extension xml>
    Module  xm_xml
</Extension>

<Extension json>
    Module  xm_json
</Extension>

<Input k8s_containers>
    Module  im_file
    File    "c:\var\log\containers\*.log"
    <Exec>
        if $raw_event =~ %EVENT_REGEX%
        {
             parse_xml($1);
        }
        else
        {
            drop();
        }

        $log_type = "k8s_container";
        $hostname = hostname();
        $host_ip  = host_ip();
        $log_file = file_name();

        if $log_file =~ /(.+)_(.+)_(.+)-(.+).log$/
        {
            $k8s_pod = $1;
            $k8s_namespace = $2;
            $k8s_container = $3;
            $k8s_container_id = $4;
        }

        to_json();

    </Exec>
</Input>

<Output file>
    Module  om_file
    File    "c:\\k\\nxlog.log"
</Output>

<Route containerlog>
    Path k8s_containers => file
</Route>

Everythings work fine, but log line has “log_file”: “unknown”. And because of that I didn't get $k8s_* fields.

How should I debug/resolv this issue?


ARTEM A
Replies: 5
View post »
laszlofoldesi
NXLog-CE Question

Hello,

This is not a installation question.

Using wget, as I have done for past 6 years was grab a NXLog-CE installation and install on my Linux core servers.   Yesterday 11/22/2022 I was unable to do this. I also noticed the Web Site has changed for downloading community versions and now  I need to make account. I'm assuming at this point,  Steps  needed  are install NXLog on any core servers I need to make account  on NXLog  site, Download the package needed. Transfer the NXLog package to  a closed environment that we have,  Upload NXLog package to a internal repo and distribute it as needed?  

 I'm also assuming this is a security procedure taken by NXLog?   If anyone could enlighten me on the new changes  that would be great.

Thanks

-Greg

 


greg.smith
Replies: 0
View post »
greg.smith
Issues with the nxlog agent when installed on Citrix MCS VDI machines.
Hellow everyone!

I have a scenario that uses Citrix MCS where I installed the agent on the master image that provides clone images that should go with the nxlog agent installed and running. But the agent goes up with some errors as below:

2022-09-23 13:51:38 ERROR couldn't connect to udp socket on <IP:XYZ:514>; The socket operation was attempted to an unreachable network.
2022-09-23 13:51:46 WARNING Due to the limitation in the Windows EventLog subsystem, the query cannot contain more than 256 sources.
2022-09-23 13:51:46 WARNING The following sources are omitted to avoid exceeding the limit in the generated query: Setup WitnessClientAdmin
2022-09-23 13:52:14 WARNING received a system shutdown request
2022-09-23 13:52:14 WARNING stopping nxlog service
2022-09-23 13:52:14 WARNING nxlog-ce received a termination request signal, exiting...
2022-11-02 23:16:38 INFO nxlog-ce-2.11.2190 started
2022-11-02 23:16:44 WARNING Due to the limitation in the Windows EventLog subsystem, the query cannot contain more than 256 sources.
2022-11-02 23:16:44 WARNING The following sources are omitted to avoid exceeding the limit in the generated query: Setup WitnessClientAdmin
2022-11-02 23:27:15 ERROR EvtNext failed with error 15007: The specified channel could not be found. Check channel configuration.
2022-11-02 23:27:16 WARNING Due to the limitation in the Windows EventLog subsystem, the query cannot contain more than 256 sources.
2022-11-02 23:27:16 WARNING The following sources are omitted to avoid exceeding the limit in the generated query: WitnessClientAdmin
2022-11-02 23:27:16 ERROR Failed to retrieve eventlog fields; The handle is invalid.

Has anyone had a problem like this using Citrix MCS?

Thanks
James \0/

gijosgun
Replies: 1
View post »
gahorvath
NXLog Community Edition - excessive CPU consumption on Windows workstations
We are using NXLog CE's im_msvistalog module to forward Windows Event Logs from the Security log, with some filtering, to an external syslog server.  Functionally this works well and does exactly what we need it to.

The problem we are having is that nxlog.exe process often consumes rather high percentages of a workstation's CPU in bursts.  Between 25 and 35 percent every few minutes, for around a minute at a time.  This is generally too much of a performance hit and I need to find some way to resolve it.

I have already mitigated the size of the event log file that nxlog.exe is querying from by clearing the Security log entirely, so this is happening even on a system with not more than a few dozen log entries to read from.  The query itself is fairly simple, it loads all Event ID 4625 entries from the Security log (these are logon failures).  It then has a single command to drop any logon failures that were initiated for a computer account instead of a user account (this is done by reading the target account trying to logon, string parsing the account username to see if the final character is a "$", which denotes a computer account, and dropping the log if the "$" is found).

What can I do to mitigate the excessive CPU usage?

What I have tried so far: clearing the Windows event log that nxlog is reading from to reduce the size of the data it needs to read from disk, using UDP syslog forwarding instead of TCP, removed the parsing that dropped Event Log ID 4625 entries where the target account being logged in was a computer account instead of a user account.  None of this has helped.

Edit to add: I did try writing out text logging instead of syslog forwarding.  This worked but I am still experiencing the periodic excessive CPU consumption.  The problem is likely in the im_msvistalog input module, I would assume.

bp81
Replies: 6
View post »
DR_
Error with multiple Host in <Output> om_tcp
I'm using nxlog-ce-2.11.2190.msi  (Community Edition) on Windows 2016 to send Windows Logs to a syslog server.

If I put more than one Host to the <Output> section like

<Output to_splunk>
    Module      om_tcp
    Host        abc1.corp.net
    Host        abc2.corp.net
    Host        abc3.corp.net
    Port        514
    Exec        to_syslog_ietf();
</Output>

I get the following error when validating the configuration

C:\Windows\system32>"C:\Program Files (x86)\nxlog\nxlog.exe" -v
2021-10-29 10:12:03 ERROR host is already defined at C:\Program Files (x86)\nxlog\conf\nxlog.conf:67

With just one Host defined, it works.

What is false? Is the featuren not supported by the Community Edition?

Regards,
Martin


mhu1234
Replies: 1
View post »
mhu1234
How to run im_exec every 10 seconds
Hi all,

I'm using CE.

Could you tell me how to write nxlog.conf?
I want to im_exec every 10Sec.
I have no idea about what should I write in schedule.

<Input messages>
    Module   im_exec
    Command  "C:\Windows\System32\cmd.exe"
    Arg      /k
    Arg      dir
    <Schedule>
        Every   10 sec
        <Exec>
         I want to do messages(im_exec) again!
        </Exec>
    </Schedule>
</Input>

<Output file>
    Module  om_file
    File    "C:\\test_logs\\output_test1.txt"
</Output>


<Route messages_to_file>
    Path    messages => file
</Route>

shinobu
Nxlog Community Edition package for Debian Buster
Hi all,

I'm looking for a debian Buster package for nxlog-ce but i've noticed that it's not available on the download page.
Is there a reason ? Can i expect a future release soon ?

Thank you very much for your assistance.
Paul.

pboniface
Replies: 1
View post »
pboniface
Get NXLog to use a random port for each connection
Hi All,

I wonder if someone can answer this for me.

According to the documentation, it states that for a UDP client, the localport will be a random high port as per https://nxlog.co/documentation/nxlog-user-guide/om_udp.html

I have a situation where I am sending Zeek logs via UDP through a Google Seesaw load balancer see https://github.com/google/seesaw

The issue I am facing is that each separate log packet / connection from NXLog has the same client source port i.e 41460 in my case.

Tcpdump confirms this

Packet 1
15:55:10.533740 IP (tos 0x0, ttl 64, id 57228, offset 0, flags [DF], proto UDP (17), length 506) 172.16.4.10.41640 > 172.16.4.166.12210: [udp sum ok] UDP, length 478

Packet 2
15:55:10.534026 IP (tos 0x0, ttl 64, id 57229, offset 0, flags [DF], proto UDP (17), length 847)172.16.4.10.41640 > 172.16.4.166.12210: [udp sum ok] UDP, length 819

Is there a way to get NXLog to use a random client port for each connection?

It looks as if it chooses a random high port when the service is started.

Cheers

Cyberkryption



cyberkryptoin
Replies: 1
View post »
b0ti
NXLOG CE - recursing over backlog
Trialing the NXLOG CE version i forward logs for MS Windows Eventlog, now i found just over 200 events are logged while there are over 50 000.
It there a way to configured nxlog so it consumes the logs older than today ?



commandline-be
Replies: 2
View post »
commandline-be
EVP_CIPHER_CTX_init results in libcrypto not found
Dear,

Attempting to compile nxlog-ce on Arch Linux i hit a snag.

./configure 
results in 
checking for openssl/evp.h... yes
checking for EVP_CIPHER_CTX_init in -lcrypto... no
configure: error: libcrypto not found

-- the configure: error is generated on 

#define HAVE_LIBCRYPTO 1
_ACEOF

  LIBS="-lcrypto $LIBS"

else
  as_fn_error $? "libcrypto not found" "$LINENO" 5
fi

I doubt if i want to modify ./configure to skip this. What can i do to fix this ?

lib is installed

/usr/lib/libressl/libcrypto.so
/usr/lib/libressl/libcrypto.so.46
/usr/lib/libressl/libcrypto.so.46.0.1
/usr/lib/libressl/pkgconfig/libcrypto.pc
/usr/lib/libcryptopp.so
/usr/lib/libcryptopp.so.8
/usr/lib/libcryptopp.so.8.2.0
/usr/lib/libcrypto.so
/usr/lib/libcrypto.so.1.1



Br,

Joris


commandline-be
Replies: 7
View post »
AM_263121
Memory issues on NXLog
We have an application that does some multiple updates every morning between 6am and 7am. During this time, it generates massive amounts of log entries.
This in turn causes the box to run out of memory, triggering Linux's OOM daemon. Running the NxLog-ce.

I have added

PersistLogqueue TRUE
SyncLogqueue TRUE
CacheFlushInterval always
CacheSync TRUE

To the config, and will see if that makes a difference, but it seems that is more to safeguard messages from being lost.

I have looked at https://nxlog.co/question/802/nxlog-ce-memory-leak and https://nxlog.co/question/4132/cache-disk-works-not-good, but not sure those two are what I am after?

Wernervdmerwe
Replies: 4
View post »
Wernervdmerwe
Set Interval for input modules
Say I wanted to run a PowerShell script on an interval using the `im_exec` module how would I do that?

Module im_exec Command "C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe" Arg "script goes here" $output = $raw_event;

I can make the interval work by creating pauses in the code but then it appears as though the script process would run forever which is undesirable for efficiency and stability reasons. 
Is there a way to run the code on a interval, say every fifteen minutes? 

Thoughts?

Thanks in advance!!

casey1234
Replies: 2
View post »
casey1234
forwarded events
Hello, I have a WEC server receiving the logs form my network computers, in this server I have the NXLog community edition to forward this logs, but in the exabeam analytics does not see logs from the machines the login and log out, I feel that the nxlogs does not forward all events, Do I need to use other version of client or what else should I do to verify if is send the full log?

Regards

Ben


USRJJAAG3643H5DQ
Replies: 1
View post »
Zhengshi
Data vanished from files after transmission
Hi,

I transmitted IIS logs and message tracking logs from one server to the other through nxlog and saved it in a location.
After complete transmission I stopped the nxlog service.
I was able to understand that the transmission is complete and I downloaded the data and was working on it.
suddenly within 10 minutes few files are lost under IIS and message tracking. 

I do not know the reason behind this. Is it really possible?
Can the data get lost after stopping the service? or something could have caused it?
I'm trying to find out the root cause. Pls help



Sangeetha
Replies: 1
View post »
Zhengshi
#015 is appended to log data sent through CE
Hi,

I have been trying to stream data and the data transfer was successful with a #015 appended to each line in my log file.
This is happening to all the log types trasferred.
Can you let me know what could cause that?



Sangeetha
Replies: 1
View post »
b0ti
using CE and EE in a same machine
Hi,

I'm using EE trial edition now in my machine but i need to use CE edition as well for testing. Can I use both in same machine? will I lose my EE trial if i download CE now?

Sangeetha
Replies: 1
View post »
Zhengshi
syslog_tls too many open files
There is a bug in NXLog Community Edition 2.10.2150, with module `im_ssl` regarding opening CA files.

NXLog seems to create a file descriptor for the CA file each time a new connection is made. That is, NXLog opens the file again and again. If NXLog runs on Linux, this can quickly hit the OS limit of maximum number of files open. 

The following error is seen repeatedly once the OS limit of file descriptors is reached:

```ERROR SSL error, failed to load ca cert from '<path_to_file>', Too many open files```

Raising the OS limit is only a temporary solution: eventually, the next limit can be hit

How can this bug be fixed?

lmpardey
Replies: 1
View post »
Zhengshi
nxlog error when tried to use xm_w3c module
Hi,

I'm using a CE where xm_w3c module is not available. So i'm getting below error:

2019-04-09 14:59:30 ERROR Failed to load module from C:\Program Files (x86)\nxlog\modules\extension\xm_w3c.dll, The specified module could not be found.  ; The specified module could not be found.  
2019-04-09 14:59:30 ERROR Invalid InputType 'w3c_parser' at C:\Program Files (x86)\nxlog\conf\nxlog.conf:94


Is there a way to overcome this error in CE by downloading the particular module ?
Or should I try using the Enterprise edition?


Sangeetha
Replies: 1
View post »
Sangeetha