When collecting windows exchange tracking log, I encountered a strange problem

Tags: NXLog Community Edition

#1 Chung Wang 10 months ago (Last updated 10 months ago )

======================================================================

error log

2023-02-15 10:39:04 INFO nxlog-ce-3.1.2319 started2023-02-15 10:44:31 ERROR apr_stat() failed on file E:\Log\IIS\W3SVC2\u_ex230207.log; 存取被拒。 2023-02-15 10:44:31 WARNING input file was deleted: E:\Log\IIS\W3SVC1\u_ex230207.log2023-02-15 10:44:33 WARNING input file was deleted: E:\Log\IIS\W3SVC2\u_ex230207.log2023-02-15 11:00:01 ERROR apr_stat() failed on file E:\Log\MessageTracking\MSGTRK2023021005-4.LOG; 存取被拒。 2023-02-15 11:00:03 WARNING input file was deleted: E:\Log\MessageTracking\MSGTRK2023021005-4.LOG2023-02-15 11:32:02 ERROR apr_stat() failed on file E:\Log\MessageTracking\MSGTRK2023021005-5.LOG; 存取被拒。 2023-02-15 11:32:04 WARNING input file was deleted: E:\Log\MessageTracking\MSGTRK2023021005-5.LOG2023-02-15 12:00:01 ERROR apr_stat() failed on file E:\Log\MessageTracking\MSGTRK2023021005-6.LOG; 存取被拒。 2023-02-15 12:00:03 WARNING input file was deleted: E:\Log\MessageTracking\MSGTRK2023021005-6.LOG2023-02-15 13:43:11 ERROR apr_stat() failed on file E:\Log\MessageTracking\MSGTRK2023021005-7.LOG; 存取被拒。 2023-02-15 13:43:13 WARNING input file was deleted: E:\Log\MessageTracking\MSGTRK2023021005-7.LOG2023-02-15 14:00:02 ERROR apr_stat() failed on file E:\Log\MessageTracking\MSGTRK2023021005-8.LOG; 存取被拒。 2023-02-15 14:00:04 WARNING input file was deleted: E:\Log\MessageTracking\MSGTRK2023021005-8.LOG2023-02-15 14:32:56 ERROR apr_stat() failed on file E:\Log\MessageTracking\MSGTRK2023021005-9.LOG; 存取被拒。 2023-02-15 14:32:58 WARNING input file was deleted: E:\Log\MessageTracking\MSGTRK2023021005-9.LOG2023-02-15 15:00:02 WARNING input file was deleted: E:\Log\MessageTracking\MSGTRK2023021005-10.LOG2023-02-15 15:31:04 WARNING input file was deleted: E:\Log\MessageTracking\MSGTRK2023021005-11.LOG2023-02-15 16:00:02 WARNING input file was deleted: E:\Log\MessageTracking\MSGTRK2023021005-12.LOG2023-02-15 16:33:08 WARNING input file was deleted: E:\Log\MessageTracking\MSGTRK2023021006-1.LOG2023-02-15 17:00:02 WARNING input file was deleted: E:\Log\MessageTracking\MSGTRK2023021006-2.LOG2023-02-15 17:12:10 WARNING stopping nxlog service2023-02-15 17:12:10 WARNING nxlog-ce received a termination request signal, exiting...

============================================================================

nxlog config

## Please set the ROOT to the folder your nxlog was installed into, otherwise it will not start.define NCloud 172.21.30.1define MailLog E:\Log\MessageTrackingdefine IISLog E:\Log\IISdefine ROOT C:\Program Files\nxlogdefine CERTDIR %ROOT%\certdefine CONFDIR %ROOT%\confdefine LOGDIR %ROOT%\datadefine LOGFILE %LOGDIR%\nxlog.logLogFile %LOGFILE%

Moduledir %ROOT%\modulesCacheDir %ROOT%\dataPidfile %ROOT%\data\nxlog.pidSpoolDir %ROOT%\data

## Load the modules needed by the outputs<Extension syslog> Module xm_syslog</Extension>

## For Exchange Message Tracking log file use the following:<Input in_maillog> Module im_file File '%MailLog%\MSGTRK*.LOG' ReadFromLast TRUE SavePos TRUE</Input>

<Output out_maillog> Module om_udp Host %NCloud% Port 514 Exec $SyslogFacilityValue = 2; Exec $SourceName = 'Exchange'; Exec to_syslog_bsd();</Output>

<Route maillog> Path in_maillog => out_maillog</Route>

## For Windows Event log use the following:<Input in_eventlog> Module im_msvistalog ReadFromLast TRUE SavePos TRUE Query <QueryList> \ <Query Id="0"> \ <Select Path="Security">*[System[(EventID=4624 or EventID=4625 or EventID=4626 or EventID=4627 or EventID=4634 or EventID=4646 or EventID=4647 or EventID=4648 or EventID=4649 or EventID=4672 or EventID=4675)]]</Select> \ <Select Path="Security">*[System[(EventID=4778 or EventID=4779 or EventID=4800 or EventID=4801 or EventID=4802 or EventID=4803 or EventID=4964 or EventID=4976 or EventID=5378 or EventID=5632 or EventID=5633)]]</Select> \ <Select Path="Security">*[System[(EventID=4768 or EventID=4769 or EventID=4770 or EventID=4771 or EventID=4772 or EventID=4773 or EventID=4774 or EventID=4775 or EventID=4776 or EventID=4777 or EventID=4820)]]</Select> \ <Select Path="Security">*[System[(EventID=4720 or EventID=4722 or EventID=4723 or EventID=4724 or EventID=4725 or EventID=4726 or EventID=4727 or EventID=4731 or EventID=4732 or EventID=4733 or EventID=4734)]]</Select> \ <Select Path="Security">*[System[(EventID=4735 or EventID=4738 or EventID=4739 or EventID=4740 or EventID=4749 or EventID=4750 or EventID=4751 or EventID=4752 or EventID=4753 or EventID=4764 or EventID=4765)]]</Select> \ <Select Path="Security">*[System[(EventID=4766 or EventID=4767 or EventID=4780 or EventID=4781 or EventID=4782 or EventID=4793 or EventID=4794 or EventID=4797 or EventID=4798 or EventID=4799 or EventID=5376 or EventID=5377)]]</Select> \ <Select Path="Security">*[System[(EventID=4608 or EventID=4610 or EventID=4611 or EventID=4612 or EventID=4614 or EventID=4615 or EventID=4616 or EventID=4618 or EventID=4621 or EventID=4622 or EventID=4697)]]</Select> \ <Select Path="Security">*[System[(EventID=5024 or EventID=5025 or EventID=5027 or EventID=5028 or EventID=5029 or EventID=5030 or EventID=5032 or EventID=5033 or EventID=5034 or EventID=5035 or EventID=5037)]]</Select> \ <Select Path="Security">*[System[(EventID=5038 or EventID=5056 or EventID=5058 or EventID=5059 or EventID=5061 or EventID=5890 or EventID=6281 or EventID=6400 or EventID=6401 or EventID=6402 or EventID=6403)]]</Select> \ <Select Path="Security">*[System[(EventID=6404 or EventID=6405 or EventID=6406 or EventID=6407 or EventID=6408 or EventID=6409 or EventID=6410)]]</Select> \ </Query> \ </QueryList> </Input>

<Output out_eventlog> Module om_udp Host %NCloud% Port 514 Exec $SyslogFacilityValue = 17; Exec $Message = string($SourceName) + ": " + string($EventID) + ": " + $Message; Exec if ($EventType == 'ERROR' or $EventType == 'AUDIT_FAILURE') { $SyslogSeverityValue = 3; } \ else if ($EventType == 'WARNING') { $SyslogSeverityValue = 4; } \ else if ($EventType == 'INFO' or $EventType == 'AUDIT_SUCCESS') { $SyslogSeverityValue = 5; } Exec to_syslog_bsd();</Output>

<Route eventlog> Path in_eventlog => out_eventlog</Route>

## For Microsoft IIS(Internet Information Server) log file use the following:<Input in_iislog> Module im_file File '%IISLog%\u_ex*.log' ReadFromLast TRUE Recursive TRUE SavePos TRUE</Input>

<Output out_iislog> Module om_udp Host %NCloud% Port 514 Exec $SyslogFacilityValue = 22; Exec $raw_event = "IIS [info]: " + $raw_event ; Exec to_syslog_bsd();</Output>

<Route iislog> Path in_iislog => out_iislog</Route>

======================================================================

Please how can I deal with this problem?

Permalink

#2 konstantinos Nxlog ✓ 10 months ago

#1 Chung Wang

Hi, I have a very strange question to askWhen collecting windows exchange tracking log, I encountered a strange problemIf you delete files older than 7 days in the tracking log folder, nxlog will generate an error, and at the same time the exchange system will also be affected and cannot operate======================================================================error log2023-02-15 10:39:04 INFO nxlog-ce-3.1.2319 started2023-02-15 10:44:31 ERROR apr_stat() failed on file E:\Log\IIS\W3SVC2\u_ex230207.log; 存取被拒。 2023-02-15 10:44:31 WARNING input file was deleted: E:\Log\IIS\W3SVC1\u_ex230207.log2023-02-15 10:44:33 WARNING input file was deleted: E:\Log\IIS\W3SVC2\u_ex230207.log2023-02-15 11:00:01 ERROR apr_stat() failed on file E:\Log\MessageTracking\MSGTRK2023021005-4.LOG; 存取被拒。 2023-02-15 11:00:03 WARNING input file was deleted: E:\Log\MessageTracking\MSGTRK2023021005-4.LOG2023-02-15 11:32:02 ERROR apr_stat() failed on file E:\Log\MessageTracking\MSGTRK2023021005-5.LOG; 存取被拒。 2023-02-15 11:32:04 WARNING input file was deleted: E:\Log\MessageTracking\MSGTRK2023021005-5.LOG2023-02-15 12:00:01 ERROR apr_stat() failed on file E:\Log\MessageTracking\MSGTRK2023021005-6.LOG; 存取被拒。 2023-02-15 12:00:03 WARNING input file was deleted: E:\Log\MessageTracking\MSGTRK2023021005-6.LOG2023-02-15 13:43:11 ERROR apr_stat() failed on file E:\Log\MessageTracking\MSGTRK2023021005-7.LOG; 存取被拒。 2023-02-15 13:43:13 WARNING input file was deleted: E:\Log\MessageTracking\MSGTRK2023021005-7.LOG2023-02-15 14:00:02 ERROR apr_stat() failed on file E:\Log\MessageTracking\MSGTRK2023021005-8.LOG; 存取被拒。 2023-02-15 14:00:04 WARNING input file was deleted: E:\Log\MessageTracking\MSGTRK2023021005-8.LOG2023-02-15 14:32:56 ERROR apr_stat() failed on file E:\Log\MessageTracking\MSGTRK2023021005-9.LOG; 存取被拒。 2023-02-15 14:32:58 WARNING input file was deleted: E:\Log\MessageTracking\MSGTRK2023021005-9.LOG2023-02-15 15:00:02 WARNING input file was deleted: E:\Log\MessageTracking\MSGTRK2023021005-10.LOG2023-02-15 15:31:04 WARNING input file was deleted: E:\Log\MessageTracking\MSGTRK2023021005-11.LOG2023-02-15 16:00:02 WARNING input file was deleted: E:\Log\MessageTracking\MSGTRK2023021005-12.LOG2023-02-15 16:33:08 WARNING input file was deleted: E:\Log\MessageTracking\MSGTRK2023021006-1.LOG2023-02-15 17:00:02 WARNING input file was deleted: E:\Log\MessageTracking\MSGTRK2023021006-2.LOG2023-02-15 17:12:10 WARNING stopping nxlog service2023-02-15 17:12:10 WARNING nxlog-ce received a termination request signal, exiting...============================================================================nxlog config## Please set the ROOT to the folder your nxlog was installed into, otherwise it will not start.define NCloud 172.21.30.1define MailLog E:\Log\MessageTrackingdefine IISLog E:\Log\IISdefine ROOT C:\Program Files\nxlogdefine CERTDIR %ROOT%\certdefine CONFDIR %ROOT%\confdefine LOGDIR %ROOT%\datadefine LOGFILE %LOGDIR%\nxlog.logLogFile %LOGFILE%Moduledir %ROOT%\modulesCacheDir %ROOT%\dataPidfile %ROOT%\data\nxlog.pidSpoolDir %ROOT%\data## Load the modules needed by the outputs<Extension syslog> Module xm_syslog</Extension>## For Exchange Message Tracking log file use the following:<Input in_maillog> Module im_file File '%MailLog%\MSGTRK*.LOG' ReadFromLast TRUE SavePos TRUE</Input><Output out_maillog> Module om_udp Host %NCloud% Port 514 Exec $SyslogFacilityValue = 2; Exec $SourceName = 'Exchange'; Exec to_syslog_bsd();</Output><Route maillog> Path in_maillog => out_maillog</Route>## For Windows Event log use the following:<Input in_eventlog> Module im_msvistalog ReadFromLast TRUE SavePos TRUE Query <QueryList> \ <Query Id="0"> \ <Select Path="Security">*[System[(EventID=4624 or EventID=4625 or EventID=4626 or EventID=4627 or EventID=4634 or EventID=4646 or EventID=4647 or EventID=4648 or EventID=4649 or EventID=4672 or EventID=4675)]]</Select> \ <Select Path="Security">*[System[(EventID=4778 or EventID=4779 or EventID=4800 or EventID=4801 or EventID=4802 or EventID=4803 or EventID=4964 or EventID=4976 or EventID=5378 or EventID=5632 or EventID=5633)]]</Select> \ <Select Path="Security">*[System[(EventID=4768 or EventID=4769 or EventID=4770 or EventID=4771 or EventID=4772 or EventID=4773 or EventID=4774 or EventID=4775 or EventID=4776 or EventID=4777 or EventID=4820)]]</Select> \ <Select Path="Security">*[System[(EventID=4720 or EventID=4722 or EventID=4723 or EventID=4724 or EventID=4725 or EventID=4726 or EventID=4727 or EventID=4731 or EventID=4732 or EventID=4733 or EventID=4734)]]</Select> \ <Select Path="Security">*[System[(EventID=4735 or EventID=4738 or EventID=4739 or EventID=4740 or EventID=4749 or EventID=4750 or EventID=4751 or EventID=4752 or EventID=4753 or EventID=4764 or EventID=4765)]]</Select> \ <Select Path="Security">*[System[(EventID=4766 or EventID=4767 or EventID=4780 or EventID=4781 or EventID=4782 or EventID=4793 or EventID=4794 or EventID=4797 or EventID=4798 or EventID=4799 or EventID=5376 or EventID=5377)]]</Select> \ <Select Path="Security">*[System[(EventID=4608 or EventID=4610 or EventID=4611 or EventID=4612 or EventID=4614 or EventID=4615 or EventID=4616 or EventID=4618 or EventID=4621 or EventID=4622 or EventID=4697)]]</Select> \ <Select Path="Security">*[System[(EventID=5024 or EventID=5025 or EventID=5027 or EventID=5028 or EventID=5029 or EventID=5030 or EventID=5032 or EventID=5033 or EventID=5034 or EventID=5035 or EventID=5037)]]</Select> \ <Select Path="Security">*[System[(EventID=5038 or EventID=5056 or EventID=5058 or EventID=5059 or EventID=5061 or EventID=5890 or EventID=6281 or EventID=6400 or EventID=6401 or EventID=6402 or EventID=6403)]]</Select> \ <Select Path="Security">*[System[(EventID=6404 or EventID=6405 or EventID=6406 or EventID=6407 or EventID=6408 or EventID=6409 or EventID=6410)]]</Select> \ </Query> \ </QueryList> </Input><Output out_eventlog> Module om_udp Host %NCloud% Port 514 Exec $SyslogFacilityValue = 17; Exec $Message = string($SourceName) + ": " + string($EventID) + ": " + $Message; Exec if ($EventType == 'ERROR' or $EventType == 'AUDIT_FAILURE') { $SyslogSeverityValue = 3; } \ else if ($EventType == 'WARNING') { $SyslogSeverityValue = 4; } \ else if ($EventType == 'INFO' or $EventType == 'AUDIT_SUCCESS') { $SyslogSeverityValue = 5; } Exec to_syslog_bsd();</Output><Route eventlog> Path in_eventlog => out_eventlog</Route>## For Microsoft IIS(Internet Information Server) log file use the following:<Input in_iislog> Module im_file File '%IISLog%\u_ex*.log' ReadFromLast TRUE Recursive TRUE SavePos TRUE</Input><Output out_iislog> Module om_udp Host %NCloud% Port 514 Exec $SyslogFacilityValue = 22; Exec $raw_event = "IIS [info]: " + $raw_event ; Exec to_syslog_bsd();</Output><Route iislog> Path in_iislog => out_iislog</Route> ======================================================================Please how can I deal with this problem?

HI Chung,

The agent is complaining that one of the input files that is monitoring has been deleted externally. That's because even if the file is “old” it is still a match for your wildcard expression:
File '%MailLog%\MSGTRK*.LOG'

The way around this is to rename the file after it's fully read by the agent (on EOF) so that it no longer matches *.LOG. The way to do this is shown here: https://docs.nxlog.co/userguide/configure/rotation.html#input

I'd also set CloseWhenIdle TRUE https://docs.nxlog.co/ce/current/index.html#im_file_config as the external application might be trying an exclusive lock.

If set to TRUE, this boolean directive specifies that open input files should be closed as soon as possible after there is no more data to read. Some applications request an exclusive lock on the log file when written or rotated, and this directive can possibly help if the application tries again to acquire the lock. The default is FALSE.

Cheers,

Konstantinos