10
responses

Hello, I want to send IIS W3C logs to the Syslog server. The nxlog.conf file is as follows. If we assume that there are too many domains on the server, I want to quickly create this config file with powershell. Is this possible?

In this configuration file, I can define the website and log directories manually. This is true, but it may be easier to add website and log directories by creating a "dynamically" config file.

When I add a wildcard "*" instead of website-name and folders with site ids such as W3SVC10, W3SVC11, no log is sent to the syslog server.

# Domain list:
Get-ChildItem C:\inetpub\vhosts -Directory -Exclude .skel,default,forwarding,Servers,sitebuilder,webmail | ForEach-Object {$_.Name}

# Log Directories Path:
Get-Website domain1.org | % { Join-Path ($_.logFile.Directory -replace '%SystemDrive%', $env:SystemDrive) "W3SVC$($_.id)" }
Get-Website domain2.net | % { Join-Path ($_.logFile.Directory -replace '%SystemDrive%', $env:SystemDrive) "W3SVC$($_.id)" }

If we can do this with an alternative method, it is not necessary to do it with powershell. Do you have any suggestions?

Panic Soft
#NoFreeOnExit TRUE

define ROOT     C:\Program Files (x86)\nxlog
define CERTDIR  %ROOT%\cert
define CONFDIR  %ROOT%\conf
define LOGDIR   %ROOT%\data
define LOGFILE  %LOGDIR%\nxlog.log
LogFile %LOGFILE%

Moduledir %ROOT%\modules
CacheDir  %ROOT%\data
Pidfile   %ROOT%\data\nxlog.pid
SpoolDir  %ROOT%\data

<Extension json>
    Module      xm_json
</Extension>

<Extension syslog>
    Module xm_syslog
</Extension>

<Input internal>
    Module im_internal
</Input>

# Watch your IIS log files
<Input domain1.org>
    Module   im_file
    File     'C:\Inetpub\vhosts\domain1.org\logs\iis\W3SVC8\u_ex*.log'
    SavePos  TRUE
    Recursive TRUE
    Exec     $Message = $raw_event;
</Input>

# Watch your IIS log files
<Input domain2.net>
    Module   im_file
    File     'C:\Inetpub\vhosts\domain2.net\logs\iis\W3SVC9\u_ex*.log'
    SavePos  TRUE
    Recursive TRUE
    Exec     $Message = $raw_event;
</Input>

# Watch your IIS log files
<Input domain3.com>
    Module   im_file
    File     'C:\Inetpub\vhosts\domain3.com\logs\iis\W3SVC10\u_ex*.log'
    SavePos  TRUE
    Recursive TRUE
    Exec     $Message = $raw_event;
</Input>

# Watch your IIS log files
<Input domain4.ru>
    Module   im_file
    File     'C:\Inetpub\vhosts\domain4.ru\logs\iis\W3SVC11\u_ex*.log'
    SavePos  TRUE
    Recursive TRUE
    Exec     $Message = $raw_event;
</Input>

# Watch your IIS log files
<Input domain5.de>
    Module   im_file
    File     'C:\Inetpub\vhosts\domain5.de\logs\iis\W3SVC12\u_ex*.log'
    SavePos  TRUE
    Recursive TRUE
    Exec     $Message = $raw_event;
</Input>

# Watch your IIS log files
<Input domain6.nl>
    Module   im_file
    File     'C:\Inetpub\vhosts\domain6.nl\logs\iis\W3SVC13\u_ex*.log'
    SavePos  TRUE
    Recursive TRUE
    Exec     $Message = $raw_event;
</Input>

<Output out>
    Module om_udp
    Host 8.8.4.4
    Port 514

    Exec  $tmpmessage = $Message; delete($Message); rename_field("tmpmessage","message");
    Exec  $raw_event = to_json();

    # Uncomment for debug output
    # Exec file_write('%ROOT%\data\nxlog_output.log', $raw_event + "\n");
</Output>

<Route 1>
    Path internal, domain1.org, domain2.net, domain3.com, domain4.ru, domain5.de, domain6.nl  => out
</Route>
AskedJanuary 25, 2021 - 8:44pm

Answer (1)

Hi Adam,

Wildcarts in folder names are not supported in Community Edition, but you may try using Recursive directive. In your case, something similar to the following should work:

 Module im_file
 File 'C:\\Inetpub\vhosts\domain3.com\logs\iis\'
 Recursive TRUE

Comments (9)

  • adam's picture

    Hi Raf,

    If I cannot use wildcard characters, will it work for logs created for addresses such as "domain1.org, domain2.net, domain4.ru"?

    C:\Program Files (x86)\nxlog\conf\nxlog.conf

    if we use the nxlog config file as you suggest

    ==========================================================

    <Extension syslog>
        Module xm_syslog
    </Extension>
    
    <Input internal>
        Module im_internal
    </Input>
    
    # Watch your IIS log files
    <Input vhosts>
        Module   im_file
        File     'C:\\Inetpub\vhosts\domain1.org\logs\iis'
        Recursive TRUE
        Exec     $Message = $raw_event;
    </Input>
    
    <Output out>
        Module om_udp
        Host 8.8.4.4
        Port 514
    
        Exec  $tmpmessage = $Message; delete($Message); rename_field("tmpmessage","message");
        Exec  $raw_event = to_json();
    
        # Uncomment for debug output
        # Exec file_write('%ROOT%\data\nxlog_output.log', $raw_event + "\n");
    </Output>
    
    <Route 1>
        Path internal, vhosts => out
    </Route>
    

    ==========================================================

    *** I see errors like this in the nxlog.log file.

    C:\Program Files (x86)\nxlog\data\nxlog.log

    2021-01-28 20:46:48 WARNING stopping nxlog service
    2021-01-28 20:46:48 WARNING nxlog-ce received a termination request signal, exiting...
    2021-01-28 20:46:49 ERROR failed to open C:\\Inetpub\vhosts\domain1.org\logs\iis; Access is denied. 
    2021-01-28 20:46:49 INFO nxlog-ce-2.10.2150 started
    2021-01-28 20:46:51 ERROR failed to open C:\\Inetpub\vhosts\domain1.org\logs\iis; Access is denied. 
    

  • raf's picture
    (NXLog)

    Hi Adam,

    I'm sorry - apparently, I've posted an incomplete line. It should be something like this:

     File     'C:\\Inetpub\vhosts\domain1.org\logs\iis\u_ex*.log'
    

    Let me know if that works for you.

    Best,
    Rafal

  • adam's picture

    Hi Raf,

    I also tried the format you sent last before responding. I am currently only posting logs for the domain1.org website. There is no problem for domain1.org.

    Other domains do not send any new logs to the Log server. domain2.net, domain3.com, domain4.ru etc.

  • raf's picture
    (NXLog)

    Hmm, that's weird. Are we sure those domains actually generate domains in this root?

    Are the privileges set right?

    Do those files exist during the start of the NXLog agent process?

    Could you set your LogLevel to this and check if there's anything useful?

    LogLevel DEBUG

    Regards,
    Rafal

  • adam's picture

    Hi, These files and folders exist from the very beginning. We do not give any special permissions for any domain. You may know that we create these webpages with the PLESK control panel. So what permissions domain1.org has, other domains are domain2.net, domain3.com, etc. It is being created as a new website with the same permissions.

    NXLOG config file is as follows

    Panic Soft
    #NoFreeOnExit TRUE
    
    define ROOT     C:\Program Files (x86)\nxlog
    define CERTDIR  %ROOT%\cert
    define CONFDIR  %ROOT%\conf
    define LOGDIR   %ROOT%\data
    define LOGFILE  %LOGDIR%\nxlog.log
    LogFile %LOGFILE%
    
    Moduledir %ROOT%\modules
    CacheDir  %ROOT%\data
    Pidfile   %ROOT%\data\nxlog.pid
    SpoolDir  %ROOT%\data
    
    <Extension json>
        Module      xm_json
    </Extension>
    
    <Extension syslog>
        Module xm_syslog
    </Extension>
    
    <Input internal>
        Module im_internal
    </Input>
    
    # Watch your IIS log files
    <Input vhosts>
        Module   im_file
        File     'C:\\Inetpub\vhosts\domain1.org\logs\iis\\W3SVC8\u_ext*.log'
        Recursive TRUE
        Exec     $Message = $raw_event;
    </Input>
    
    <Output out>
        Module om_udp
        Host 8.8.4.4
        Port 514
    
        Exec  $tmpmessage = $Message; delete($Message); rename_field("tmpmessage","message");
        Exec  $raw_event = to_json();
    
        # Uncomment for debug output
        # Exec file_write('%ROOT%\data\nxlog_output.log', $raw_event + "\n");
    </Output>
    
    <Route 1>
        Path internal, vhosts => out
    </Route>
    

    The file and folder list is as follows.

    C:\inetpub\vhosts>dir /s |findstr "logs" |more
    25.01.2021  20:10    <DIR>          logs
     Directory of C:\inetpub\vhosts\domain1.org\logs
     Directory of C:\inetpub\vhosts\domain1.org\logs\FailedRequests
     Directory of C:\inetpub\vhosts\domain1.org\logs\iis
     Directory of C:\inetpub\vhosts\domain1.org\logs\iis\W3SVC8
     Directory of C:\inetpub\vhosts\domain1.org\logs\php_errors
     Directory of C:\inetpub\vhosts\domain1.org\logs\php_errors\domain1.org
    25.01.2021  20:14    <DIR>          logs
     Directory of C:\inetpub\vhosts\domain2.net\logs
     Directory of C:\inetpub\vhosts\domain2.net\logs\FailedRequests
     Directory of C:\inetpub\vhosts\domain2.net\logs\iis
     Directory of C:\inetpub\vhosts\domain2.net\logs\iis\W3SVC9
     Directory of C:\inetpub\vhosts\domain2.net\logs\php_errors
     Directory of C:\inetpub\vhosts\domain2.net\logs\php_errors\domain2.net
    25.01.2021  20:14    <DIR>          logs
     Directory of C:\inetpub\vhosts\domain3.com\logs
     Directory of C:\inetpub\vhosts\domain3.com\logs\FailedRequests
     Directory of C:\inetpub\vhosts\domain3.com\logs\iis
     Directory of C:\inetpub\vhosts\domain3.com\logs\iis\W3SVC10
     Directory of C:\inetpub\vhosts\domain3.com\logs\php_errors
     Directory of C:\inetpub\vhosts\domain3.com\logs\php_errors\domain3.com
    25.01.2021  20:23    <DIR>          logs
     Directory of C:\inetpub\vhosts\domain4.ru\logs
     Directory of C:\inetpub\vhosts\domain4.ru\logs\FailedRequests
     Directory of C:\inetpub\vhosts\domain4.ru\logs\iis
     Directory of C:\inetpub\vhosts\domain4.ru\logs\iis\W3SVC11
     Directory of C:\inetpub\vhosts\domain4.ru\logs\php_errors
     Directory of C:\inetpub\vhosts\domain4.ru\logs\php_errors\domain4.ru
    25.01.2021  20:25    <DIR>          logs
     Directory of C:\inetpub\vhosts\domain5.de\logs
     Directory of C:\inetpub\vhosts\domain5.de\logs\FailedRequests
     Directory of C:\inetpub\vhosts\domain5.de\logs\iis
     Directory of C:\inetpub\vhosts\domain5.de\logs\iis\W3SVC12
     Directory of C:\inetpub\vhosts\domain5.de\logs\php_errors
     Directory of C:\inetpub\vhosts\domain5.de\logs\php_errors\domain5.de
    25.01.2021  20:26    <DIR>          logs
     Directory of C:\inetpub\vhosts\domain6.nl\logs
     Directory of C:\inetpub\vhosts\domain6.nl\logs\FailedRequests
     Directory of C:\inetpub\vhosts\domain6.nl\logs\iis
     Directory of C:\inetpub\vhosts\domain6.nl\logs\iis\W3SVC13
     Directory of C:\inetpub\vhosts\domain6.nl\logs\php_errors
     Directory of C:\inetpub\vhosts\domain6.nl\logs\php_errors\domain6.nl
    

  • adam's picture

    I added the "LogLevel DEBUG" line to the nxlog.conf file. Can you please check?

    I am sending a log for domain1.org. There is no problem, but unfortunately no new log is sent for other domains (such as domain2.net, domain3.com etc.)

    nxlog.conf file 
    
    Panic Soft
    #NoFreeOnExit TRUE
    
    define ROOT     C:\Program Files (x86)\nxlog
    define CERTDIR  %ROOT%\cert
    define CONFDIR  %ROOT%\conf
    define LOGDIR   %ROOT%\data
    define LOGFILE  %LOGDIR%\nxlog.log
    LogFile %LOGFILE%
    **LogLevel DEBUG**
    
    Moduledir %ROOT%\modules
    CacheDir  %ROOT%\data
    Pidfile   %ROOT%\data\nxlog.pid
    SpoolDir  %ROOT%\data
    
    <Extension json>
        Module      xm_json
    </Extension>
    
    <Extension syslog>
        Module xm_syslog
    </Extension>
    
    <Input internal>
        Module im_internal
    </Input>
    
    # Watch your IIS log files
    <Input vhosts>
        Module   im_file
        File     'C:\\Inetpub\vhosts\domain1.org\logs\iis\\W3SVC8\u_ext*.log'
        Recursive TRUE
        Exec     $Message = $raw_event;
    </Input>
    
    
    
    2021-01-29 21:17:35 DEBUG new event in event_thread [vhosts:MODULE_SPECIFIC]
    2021-01-29 21:17:35 DEBUG new event in event_thread [vhosts:READ]
    2021-01-29 21:17:35 DEBUG nx_event_to_jobqueue: READ (vhosts)
    2021-01-29 21:17:35 DEBUG event added to jobqueue
    2021-01-29 21:17:35 DEBUG future event, event thread sleeping 132842ms in cond_timedwait
    2021-01-29 21:17:35 DEBUG worker 0 got signal for new job
    2021-01-29 21:17:35 DEBUG worker 0 processing event 0x10dd5b0
    2021-01-29 21:17:35 DEBUG PROCESS_EVENT: READ (vhosts)
    2021-01-29 21:17:35 DEBUG Module vhosts got EOF from C:\\Inetpub\vhosts\domain1.org\logs\iis\\W3SVC8\u_extend1.log
    2021-01-29 21:17:35 DEBUG got EOF for C:\\Inetpub\vhosts\domain1.org\logs\iis\\W3SVC8\u_extend1.log
    2021-01-29 21:17:35 DEBUG worker 0 waiting for new event
    2021-01-29 21:17:35 DEBUG new event in event_thread [vhosts:MODULE_SPECIFIC]
    2021-01-29 21:17:35 DEBUG new event in event_thread [vhosts:READ]
    2021-01-29 21:17:35 DEBUG future event, event thread sleeping 132842ms in cond_timedwait
    2021-01-29 21:17:35 DEBUG new event in event_thread [vhosts:MODULE_SPECIFIC]
    2021-01-29 21:17:35 DEBUG nx_event_to_jobqueue: MODULE_SPECIFIC (vhosts)
    2021-01-29 21:17:35 DEBUG event added to jobqueue
    2021-01-29 21:17:35 DEBUG new event in event_thread [vhosts:READ]
    2021-01-29 21:17:35 DEBUG future event, event thread sleeping 859365ms in cond_timedwait
    2021-01-29 21:17:35 DEBUG worker 1 got signal for new job
    2021-01-29 21:17:35 DEBUG worker 1 processing event 0x10dd460
    2021-01-29 21:17:35 DEBUG PROCESS_EVENT: MODULE_SPECIFIC (vhosts)
    2021-01-29 21:17:35 DEBUG Value specified for File parameter contains wildcards: 'C:\\Inetpub\vhosts\domain1.org\logs\iis\\W3SVC8\u_ext*.log'
    2021-01-29 21:17:35 DEBUG reading directory entries under 'C:\\Inetpub\vhosts\domain1.org\logs\iis\\W3SVC8' to check for matching files
    2021-01-29 21:17:35 DEBUG checking '.' against wildcard 'u_ext*.log':
    2021-01-29 21:17:35 DEBUG ignoring directory entry '.'
    2021-01-29 21:17:35 DEBUG checking '..' against wildcard 'u_ext*.log':
    2021-01-29 21:17:35 DEBUG ignoring directory entry '..'
    2021-01-29 21:17:35 DEBUG checking 'u_extend1.log' against wildcard 'u_ext*.log':
    2021-01-29 21:17:35 DEBUG 'u_extend1.log' matches wildcard 'u_ext*.log'
    2021-01-29 21:17:35 DEBUG im_file_add_file: C:\\Inetpub\vhosts\domain1.org\logs\iis\\W3SVC8\u_extend1.log
    2021-01-29 21:17:35 DEBUG checking 'web_stat_executor.conf' against wildcard 'u_ext*.log':
    2021-01-29 21:17:35 DEBUG 'web_stat_executor.conf' does not match wildcard 'u_ext*.log'
    2021-01-29 21:17:35 DEBUG check file: C:\\Inetpub\vhosts\domain1.org\logs\iis\\W3SVC8\u_extend1.log
    2021-01-29 21:17:35 DEBUG worker 1 waiting for new event
    2021-01-29 21:17:35 DEBUG new event in event_thread [vhosts:READ]
    2021-01-29 21:17:35 DEBUG new event in event_thread [vhosts:MODULE_SPECIFIC]
    2021-01-29 21:17:35 DEBUG future event, event thread sleeping 859365ms in cond_timedwait
    2021-01-29 21:17:36 DEBUG new event in event_thread [vhosts:READ]
    2021-01-29 21:17:36 DEBUG nx_event_to_jobqueue: READ (vhosts)
    2021-01-29 21:17:36 DEBUG event added to jobqueue
    

  • raf's picture
    (NXLog)

    Oh, sure, I've just given an example, obviously, this won't change the directory.

    More precisely, you may try this:

    File 'C:\\Inetpub\vhosts\u_ext*.log'

    Best regards,
    Rafal

  • adam's picture

    Hello Raf, In this way, domain1.org, domain2.net, domain3.com etc. I was able to post the logs of all sites to the Graylog server. Thanks dude. If it wasn't for you, I would have given up on this job. Thank you very much for your help. You are great!