Using a regex to create a new field for syslog

Tags:

#1 JP_357786

Hi,

I have a working configuration for sending Windows DHCP server logs to a remote syslog server, where we are combining the logs with FreeRADIUS logs for auditing and troubleshooting WiFi logon events.

As the Windows servers use MAC addresses without colons, and our other logs use MAC addresses with colons, in order to more easily correlate events I would like to use NXLog to take the client MAC address ($MACAddress) and create a new variable ($ColonMAC) which will be appended to the end of the messages from the DCHP server.

I have tried to use some of the regex from here: https://www.perlmonks.org/?node_id=947757

Having no Perl experience, I don't know how to properly format the code or where in the config file is the most appropriate place to add it in. All attempts so far have resulted in NXLog finding syntax errors in the following line.

Can anyone suggest what needs to be added to the config below?

Panic Soft
#NoFreeOnExit TRUE

#GLOBAL CONFIG

define ROOT     C:\Program Files (x86)\nxlog
define CERTDIR  %ROOT%\cert
define CONFDIR  %ROOT%\conf
define LOGDIR   %ROOT%\data
define LOGFILE  %LOGDIR%\nxlog.log
LogFile %LOGFILE%

Moduledir %ROOT%\modules
CacheDir  %ROOT%\data
Pidfile   %ROOT%\data\nxlog.pid
SpoolDir  %ROOT%\data

<Extension _syslog>
    Module      xm_syslog
</Extension>

<Extension _charconv>
    Module      xm_charconv
    AutodetectCharsets iso8859-2, utf-8, utf-16, utf-32
</Extension>

<Extension _exec>
    Module      xm_exec
</Extension>

<Extension _fileop>
    Module      xm_fileop

    # Check the size of our log file hourly, rotate if larger than 5MB
    <Schedule>
        Every   1 hour
        Exec    if (file_exists('%LOGFILE%') and \
                   (file_size('%LOGFILE%') >= 5M)) \
                    file_cycle('%LOGFILE%', 8);
    </Schedule>

    # Rotate our log file every week on Sunday at midnight
    <Schedule>
        When    @weekly
        Exec    if file_exists('%LOGFILE%') file_cycle('%LOGFILE%', 8);
    </Schedule>
</Extension>

<Extension dhcp_csv_parser>
    Module      xm_csv
    Fields      ID, Date, Time, Description, IPAddress, ClientHostname, MACAddress, \
                UserName, TransactionID, QResult, ProbationTime, CorrelationID, \
                DHCID, VendorClassHex, VendorClassASCII, UserClassHex, \
                UserClassASCII, RelayAgentInformation, DnsRegError
</Extension>

<Extension dhcpv6_csv_parser>
    Module      xm_csv
    Fields      ID, Date, Time, Description, IPv6Address, Hostname, ErrorCode, \
                DuidLength, DuidBytesHex, UserName, Dhcid, SubnetPrefix
</Extension>


#INPUT

<Input dhcp_server_audit>
    Module          im_file
    File "C:\\Windows\\Sysnative\\dhcp\\DhcpSrvLog-*.log"
	    <Exec>
        # Only process lines that begin with an event ID
        if $raw_event =~ /^\d+,/
        {
            $FileName = file_name();
            if $FileName =~ /DhcpSrvLog-/
            {
                dhcp_csv_parser->parse_csv();
                $QResult = integer($QResult);
                if $QResult == 0 $QMessage = "NoQuarantine";
                else if $QResult == 1 $QMessage = "Quarantine";
                else if $QResult == 2 $QMessage = "Drop Packet";
                else if $QResult == 3 $QMessage = "Probation";
                else if $QResult == 6 $QMessage = "No Quarantine Information";
            }
            else
            {
                dhcpv6_csv_parser->parse_csv();
            }
			$EventTime = strptime($Date + ' ' + $Time, '%m/%d/%y %H:%M:%S');
            $ID = integer($ID);
            # DHCP Event IDs
            if $ID == 0 $Message = "The log was started.";
			            else if $ID == 1 $Message = "The log was stopped.";
            else if $ID == 2 $Message = "The log was temporarily paused due to low disk space.";
			else if ($ID >= 10 and $ID <= 16) $Message = $Description + " |" + $IPAddress + " |" + $ClientHostname + " |" + $MACAddress + " |" + $UserName + " |" + $ColonMac;
            else if $ID == 17 drop();
            else if $ID == 18 drop();
            else if ($ID >= 20 and $ID <= 23) $Message = $Description + " |" + $IPAddress + " |" + $ClientHostname + " |" + $MACAddress + " |" + $UserName;
            else if $ID == 24 $Message = "IP address cleanup operation has began.";
            else if $ID == 25 $Message = "IP address cleanup statistics.";
            else if $ID == 30 drop();
            else if $ID == 31 drop();
            else if $ID == 32 drop();
            else if $ID == 33 $Message = $Description + " |" + $IPAddress + " |" + $ClientHostname + " |" + $MACAddress + " |" + $UserName;
            else if $ID == 34 drop();
            else if $ID == 35 drop();
            else if $ID == 36 drop();
            else if ($ID >= 50 and $ID < 1000)
                $Message = "Codes above 50 are used for Rogue Server Detection " +
                           "information.";
            # DHCPv6 Event IDs
            else if $ID == 11000 $Message = "DHCPv6 Solicit.";
            else if $ID == 11001 $Message = "DHCPv6 Advertise.";
            else if $ID == 11002 $Message = "DHCPv6 Request.";
            else if $ID == 11003 $Message = "DHCPv6 Confirm.";
            else if $ID == 11004 $Message = "DHCPv6 Renew.";
            else if $ID == 11005 $Message = "DHCPv6 Rebind.";
            else if $ID == 11006 $Message = "DHCPv6 Decline.";
            else if $ID == 11007 $Message = "DHCPv6 Release.";
            else if $ID == 11008 $Message = "DHCPv6 Information Request.";
            else if $ID == 11009 $Message = "DHCPv6 Scope Full.";
            else if $ID == 11010 $Message = "DHCPv6 Started.";
            else if $ID == 11011 $Message = "DHCPv6 Stopped.";
            else if $ID == 11012 $Message = "DHCPv6 Audit log paused.";
            else if $ID == 11013 $Message = "DHCPv6 Log File.";
            else if $ID == 11014 $Message = "DHCPv6 Bad Address.";
            else if $ID == 11015 $Message = "DHCPv6 Address is already in use.";
            else if $ID == 11016 $Message = "DHCPv6 Client deleted.";
            else if $ID == 11017 $Message = "DHCPv6 DNS record not deleted.";
            else if $ID == 11018 $Message = "DHCPv6 Expired.";
            else if $ID == 11019
                $Message = "DHCPv6 Leases Expired and Leases Deleted.";
            else if $ID == 11020 $Message = "DHCPv6 Database cleanup begin.";
            else if $ID == 11021 $Message = "DHCPv6 Database cleanup end.";
            else if $ID == 11022 $Message = "DNS IPv6 Update Request.";
            else if $ID == 11023 $Message = "DNS IPv6 Update Failed.";
            else if $ID == 11024 $Message = "DNS IPv6 Update Successful.";
            else if $ID == 11028
                $Message = "DNS IPv6 update request failed as the DNS update " +
                           "request queue limit exceeded.";
            else if $ID == 11029 $Message = "DNS IPv6 update request failed.";
            else if $ID == 11030
                $Message = "DHCPv6 stateless client records purged.";
            else if $ID == 11031
                $Message = "DHCPv6 stateless client record is purged as the " +
                           "purge interval has expired for this client record.";
            else if $ID == 11032
                $Message = "DHCPV6 Information Request from IPV6 Stateless Client.";
            else drop();
        }
        # Discard header lines (which do not begin with an event ID)
        else drop();
    </Exec>
</Input>


#OUTPUT

<Output out_syslog>
    Module  om_udp
    Host    10.0.0.200
    Port    514
    Exec    to_syslog_bsd();
</Output>

#ROUTES

<Route 1>
Path	dhcp_server_audit => out_syslog
</Route>
#2 manuel.munozDeactivated Nxlog ✓
#1 JP_357786
Hi, I have a working configuration for sending Windows DHCP server logs to a remote syslog server, where we are combining the logs with FreeRADIUS logs for auditing and troubleshooting WiFi logon events. As the Windows servers use MAC addresses without colons, and our other logs use MAC addresses with colons, in order to more easily correlate events I would like to use NXLog to take the client MAC address ($MACAddress) and create a new variable ($ColonMAC) which will be appended to the end of the messages from the DCHP server. I have tried to use some of the regex from here: https://www.perlmonks.org/?node_id=947757 Having no Perl experience, I don't know how to properly format the code or where in the config file is the most appropriate place to add it in. All attempts so far have resulted in NXLog finding syntax errors in the following line. Can anyone suggest what needs to be added to the config below? Panic Soft #NoFreeOnExit TRUE #GLOBAL CONFIG define ROOT C:\Program Files (x86)\nxlog define CERTDIR %ROOT%\cert define CONFDIR %ROOT%\conf define LOGDIR %ROOT%\data define LOGFILE %LOGDIR%\nxlog.log LogFile %LOGFILE% Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data <Extension _syslog> Module xm_syslog </Extension> <Extension _charconv> Module xm_charconv AutodetectCharsets iso8859-2, utf-8, utf-16, utf-32 </Extension> <Extension _exec> Module xm_exec </Extension> <Extension _fileop> Module xm_fileop # Check the size of our log file hourly, rotate if larger than 5MB <Schedule> Every 1 hour Exec if (file_exists('%LOGFILE%') and \ (file_size('%LOGFILE%') >= 5M)) \ file_cycle('%LOGFILE%', 8); </Schedule> # Rotate our log file every week on Sunday at midnight <Schedule> When @weekly Exec if file_exists('%LOGFILE%') file_cycle('%LOGFILE%', 8); </Schedule> </Extension> <Extension dhcp_csv_parser> Module xm_csv Fields ID, Date, Time, Description, IPAddress, ClientHostname, MACAddress, \ UserName, TransactionID, QResult, ProbationTime, CorrelationID, \ DHCID, VendorClassHex, VendorClassASCII, UserClassHex, \ UserClassASCII, RelayAgentInformation, DnsRegError </Extension> <Extension dhcpv6_csv_parser> Module xm_csv Fields ID, Date, Time, Description, IPv6Address, Hostname, ErrorCode, \ DuidLength, DuidBytesHex, UserName, Dhcid, SubnetPrefix </Extension> #INPUT <Input dhcp_server_audit> Module im_file File "C:\\Windows\\Sysnative\\dhcp\\DhcpSrvLog-*.log" <Exec> # Only process lines that begin with an event ID if $raw_event =~ /^\d+,/ { $FileName = file_name(); if $FileName =~ /DhcpSrvLog-/ { dhcp_csv_parser->parse_csv(); $QResult = integer($QResult); if $QResult == 0 $QMessage = "NoQuarantine"; else if $QResult == 1 $QMessage = "Quarantine"; else if $QResult == 2 $QMessage = "Drop Packet"; else if $QResult == 3 $QMessage = "Probation"; else if $QResult == 6 $QMessage = "No Quarantine Information"; } else { dhcpv6_csv_parser->parse_csv(); } $EventTime = strptime($Date + ' ' + $Time, '%m/%d/%y %H:%M:%S'); $ID = integer($ID); # DHCP Event IDs if $ID == 0 $Message = "The log was started."; else if $ID == 1 $Message = "The log was stopped."; else if $ID == 2 $Message = "The log was temporarily paused due to low disk space."; else if ($ID >= 10 and $ID <= 16) $Message = $Description + " |" + $IPAddress + " |" + $ClientHostname + " |" + $MACAddress + " |" + $UserName + " |" + $ColonMac; else if $ID == 17 drop(); else if $ID == 18 drop(); else if ($ID >= 20 and $ID <= 23) $Message = $Description + " |" + $IPAddress + " |" + $ClientHostname + " |" + $MACAddress + " |" + $UserName; else if $ID == 24 $Message = "IP address cleanup operation has began."; else if $ID == 25 $Message = "IP address cleanup statistics."; else if $ID == 30 drop(); else if $ID == 31 drop(); else if $ID == 32 drop(); else if $ID == 33 $Message = $Description + " |" + $IPAddress + " |" + $ClientHostname + " |" + $MACAddress + " |" + $UserName; else if $ID == 34 drop(); else if $ID == 35 drop(); else if $ID == 36 drop(); else if ($ID >= 50 and $ID < 1000) $Message = "Codes above 50 are used for Rogue Server Detection " + "information."; # DHCPv6 Event IDs else if $ID == 11000 $Message = "DHCPv6 Solicit."; else if $ID == 11001 $Message = "DHCPv6 Advertise."; else if $ID == 11002 $Message = "DHCPv6 Request."; else if $ID == 11003 $Message = "DHCPv6 Confirm."; else if $ID == 11004 $Message = "DHCPv6 Renew."; else if $ID == 11005 $Message = "DHCPv6 Rebind."; else if $ID == 11006 $Message = "DHCPv6 Decline."; else if $ID == 11007 $Message = "DHCPv6 Release."; else if $ID == 11008 $Message = "DHCPv6 Information Request."; else if $ID == 11009 $Message = "DHCPv6 Scope Full."; else if $ID == 11010 $Message = "DHCPv6 Started."; else if $ID == 11011 $Message = "DHCPv6 Stopped."; else if $ID == 11012 $Message = "DHCPv6 Audit log paused."; else if $ID == 11013 $Message = "DHCPv6 Log File."; else if $ID == 11014 $Message = "DHCPv6 Bad Address."; else if $ID == 11015 $Message = "DHCPv6 Address is already in use."; else if $ID == 11016 $Message = "DHCPv6 Client deleted."; else if $ID == 11017 $Message = "DHCPv6 DNS record not deleted."; else if $ID == 11018 $Message = "DHCPv6 Expired."; else if $ID == 11019 $Message = "DHCPv6 Leases Expired and Leases Deleted."; else if $ID == 11020 $Message = "DHCPv6 Database cleanup begin."; else if $ID == 11021 $Message = "DHCPv6 Database cleanup end."; else if $ID == 11022 $Message = "DNS IPv6 Update Request."; else if $ID == 11023 $Message = "DNS IPv6 Update Failed."; else if $ID == 11024 $Message = "DNS IPv6 Update Successful."; else if $ID == 11028 $Message = "DNS IPv6 update request failed as the DNS update " + "request queue limit exceeded."; else if $ID == 11029 $Message = "DNS IPv6 update request failed."; else if $ID == 11030 $Message = "DHCPv6 stateless client records purged."; else if $ID == 11031 $Message = "DHCPv6 stateless client record is purged as the " + "purge interval has expired for this client record."; else if $ID == 11032 $Message = "DHCPV6 Information Request from IPV6 Stateless Client."; else drop(); } # Discard header lines (which do not begin with an event ID) else drop(); </Exec> </Input> #OUTPUT <Output out_syslog> Module om_udp Host 10.0.0.200 Port 514 Exec to_syslog_bsd(); </Output> #ROUTES <Route 1> Path dhcp_server_audit => out_syslog </Route>

Why don't you add sothething like the following right after the last drop() in the input module?

$ColonMAC = $MACAddress;
if $ColonMAC =~ /^([0-9a-fA-F]{2})([0-9a-fA-F]{2})([0-9a-fA-F]{2})([0-9a-fA-F]{2})([0-9a-fA-F]{2})([0-9a-fA-F]{2})$/ {
	$ColonMAC = $1 + ":" + $2 + ":" + $3 + ":" + $4 + ":" + $5 + ":" + $6;
}