Handling Escape Characters in Log Data
I'm collecting log data using im_file that includes “domain\username” in a field. I run it through xm_csv to parse the csv formatted data into fields, then I extract that data with regex into a domain field and a username field, which works for all but those usernames that start with escape characters, n r t and b. I had been tweaking the regex at first thinking it was failing for some reason, but it really seems like the field is double quoted and is allowing escape characters to exist, even though I have it set to use a single quote when I parse the csv data.
Windows Server 2019 Standard, NXLog CE version 3.2.2329.
Input/Output example
Input
2025/04/14,10:21:18,Disconnect,01DB045CCE64165800505699187E001,CD-PL-22-0030,domain\raymondc,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,50,0,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,1,,
Output
{"version":"v1","authentication_target":"vpn01","time":"2025-04-14 10:21:18","DeviceName":"CD-PL-22-0030","account":"domain\raymondc","event_type":"VPN_SESSION_TERMINATION"}
Config
Panic Soft
#NoFreeOnExit TRUE
define ROOT C:\Program Files\nxlog
define CERTDIR %ROOT%\cert
define CONFDIR %ROOT%\conf\nxlog.d
define LOGDIR %ROOT%\data
include %CONFDIR%\*.conf
define LOGFILE %LOGDIR%\nxlog.log
LogFile %LOGFILE%
Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
<Extension _json>
Module xm_json
</Extension>
<Extension _charconv>
Module xm_charconv
AutodetectCharsets iso8859-2, utf-8, utf-16, utf-32
</Extension>
<Extension _exec>
Module xm_exec
</Extension>
<Extension _fileop>
Module xm_fileop
# Check the size of our log file hourly, rotate if larger than 5MB
<Schedule>
Every 1 hour
Exec if (file_exists('%LOGFILE%') and \
(file_size('%LOGFILE%') >= 5M)) \
file_cycle('%LOGFILE%', 8);
</Schedule>
# Rotate our log file every week on Sunday at midnight
<Schedule>
When @weekly
Exec if file_exists('%LOGFILE%') file_cycle('%LOGFILE%', 8);
</Schedule>
</Extension>
<Extension csv>
Module xm_csv
Fields Date,Time,Event,DeviceID,DeviceName,UserName,POPAddress,VirtualAddress,Status_TimeConnected(sec),Status_BatteryRemaining(%),Status_CompressionState,Interface_Name,Interface_MACAddress,Interface_Reserved,Interface_Speed,Interface_SignalStrength,AP_NetworkName(SSID),AP_MACAddress(BSSID),AP_Vendor,AP_SecurityType,AP_Subnet,AP_RadioChannel,Location_Latitude,Location_Longitude,Location_Altitude,Status_EncryptionState,Reserved,Flags_NetMotion,Flags_Custom,Networking_TotalInTCPBytes,Networking_TotalOutTCPBytes,Networking_TotalInUDPBytes,Networking_TotalOutUDPBytes,Networking_TotalInBytes,Networking_TotalOutBytes,Networking_IMPReceives,Networking_IMPDelivers,Networking_IMPInDuplicateFrames,Networking_IMPOutPeerRetransmittedFrames,Networking_IMPInDuplicateBytes,Networking_IMPOutRetransmittedBytes,Networking_IMPOutRequests,Networking_IMPOutRetransmittedFrames,Status_TerminateReason,Status_TerminateStatus,Networking_IMPInCompressedFrames,Networking_IMPInDecompressibleBytes,Networking_IMPInDecompressedBytes,Networking_IMPOutCompressedFrames,Networking_IMPOutCompressibleBytes,Networking_IMPOutCompressedBytes,Networking_IMPInDataFrames,Networking_IMPInFragmentedFrames,Networking_IMPFragmentedFramesReassemblyReq'd,Networking_IMPFragmentedFramesReassemblyOK,Networking_IMPFragmentedFramesReassemblyTimeouts,Networking_IMPFragmentedFramesReassemblyFailures,Networking_IMPInSyncFrames,Networking_IMPInEstablishedFrames,Networking_IMPInAbortFrames,Networking_IMPInMortisFrames,Networking_IMPInPostMortemFrames,Networking_IMPInEchoRequests,Networking_IMPInEchoResponses,Networking_IMPInBroadcastFrames,Networking_IMPOutDataFrames,Networking_IMPOutFragmentedFrames,Networking_IMPOutFragmentCreates,Networking_IMPOutFragmentOKs,Networking_IMPOutFragmentFailures,Networking_IMPOutSyncFrames,Networking_IMPOutEstablishedFrames,Networking_IMPOutAbortFrames,Networking_IMPOutMortisFrames,Networking_IMPOutPostMortemFrames,Networking_IMPOutEchoRequests,Networking_IMPOutEchoResponses,Networking_IMPOutBroadcastFrames,Errors_InFrameHeader,Errors_InFrameAddress,Errors_InUnknownProtocols,Errors_InFramesDiscarded,Errors_OutFramesDiscarded,Errors_OutFramesNoRoute,EntityID,AuthenticationMode,DeviceAuthIdentity
QuoteMethod String
QuoteChar '
</Extension>
<Input in>
Module im_file
File 'C:\Program Files\NetMotion Server\logs\nmact*.log'
ActiveFiles 2
ReadOrder MTimeNewestFirst
InputType LineBased
CloseWhenIdle True
PollInterval 30
<Exec>
Add basic fields and drop unwanted lines.
if (
($raw_event =~ /Connect,/i)
)
{
$version = ("v1");
$authentication_target = "vpn01";
}
else
{
drop();
}
Regex to pull out the date and time and parse it into a datetime field.
if (
($raw_event =~ /^(\S{4})/(\S{2})/(\S{2}),(\S{2}):(\S{2}):(\S{2})/)
)
{
$EventTime = ($1 + "-" + $2 + "-" + $3 + " " + $4 + ":" + $5 + ":" + $6 + "-07:00");
$EventTime = parsedate($EventTime);
}
csv->parse_csv();
Extract domain+username and normalize to lower case.
if ($UserName =~ /([\w-]+)\b\{1}\b([\w-]+)/)
{
log_info("captured: " + $1 + " and " + $2);
$account_domain = lc($1);
$UserName = lc($2);
}
else if ($UserName =~ /^(\w*)@(\S*)$/)
{
log_info("captured: " + $1 + " and " + $2);
$account_domain = lc($2);
$UserName = lc($1);
}
else
{
log_warning("unable to capture username: " + $UserName);
}
Set username to devicename for unattended authentication.
if ($AuthenticationMode == "8") $UserName = $DeviceName;
Update blank username to device name
if (not defined($UserName)) $UserName = $DeviceName;
Assign event type and authentication result.
if (
($Event =~ /Connect/)
)
{
$event_type = "VPN_SESSION_IP_ASSIGNED";
$authentication_result = "SUCCESS";
}
else
{
$event_type = "VPN_SESSION_TERMINATION";
}
Convert terminate reason to integer and check against list.
$Status_TerminateReason = integer($Status_TerminateReason);
if ($Status_TerminateReason IN (122, 113)) $UserName = $DeviceName;
Keep version, authentication_target, EventTime, DeviceName, UserName, event_type, account_domain, VirtualAddress, POPAddress, authentication_result
rename_field("EventTime", "time");
rename_field("UserName", "account");
rename_field("VirtualAddress", "assigned_ip");
rename_field("POPAddress", "source_ip");
delete($DeviceAuthIdentity);
delete($SourceModuleName);
delete($SourceModuleType);
delete($Date);
delete($Time);
delete($EventReceivedTime);
delete($Event);
delete($DeviceID);
delete($AuthenticationMode);
delete($EntityID);
delete($Status_TerminateStatus);
Future enhancement could be to mark logs with terminate reasons, especially for logon failures.
delete($Status_TerminateReason);
if (
($source_ip =~ /^(\S{7,15}):(\d{1,5})/)
)
{
$source_ip = $1;
}
to_json();
</Exec>
</Input>
<Output resultfile_out>
Module om_file
file 'C:\Windows\Temp\nxlog-results.json'
</Output>
<Route 1>
Path in=> resultfile_out
</Route>
My log shows the field captured, but with a semicolon instead of “raymondc”.
2025-04-14 10:19:47 WARNING stopping nxlog service
2025-04-14 10:19:47 WARNING nxlog-ce received a termination request signal, exiting...
2025-04-14 10:19:49 WARNING not starting unused module out
2025-04-14 10:19:49 INFO nxlog-ce-3.2.2329 started
2025-04-14 10:21:49 WARNING unable to capture username: domain;aymondc
2025-04-14 10:25:22 WARNING stopping nxlog service
2025-04-14 10:25:23 WARNING nxlog-ce received a termination request signal, exiting...
2025-04-14 10:25:24 WARNING not starting unused module out
2025-04-14 10:25:24 INFO nxlog-ce-3.2.2329 started