Regex to set variable | Log collection solutions

#1 nembosec 5 years ago

Hi, I’m trying to use regex in nxlog. My current configuration is to save firewall logs to a file .txt using the $Sender value to create the file name.

.......

<Input *****> Module im_tcp Host 0.0.0.0 Port 1001 <Exec> if $raw_event =~ /LEEF/ parse_leef(); else parse_syslog(); </Exec> </Input>

.......

<Output > define OUT_DIR %LOGDIR2%/ Module om_file File "%OUT_DIR%/" + $Sender + ".txt" <Schedule> Every 3600 sec <Exec> if ->file_size() > 0M { set_var('newfile', file_name() + strftime(now(), '_%Y%m%d%H%M%S') + '.log'); rotate_to(get_var('newfile')); exec_async('C:/Program Files/GnuWin32/bin/bzip2.exe', 'E:// *.log'); } </Exec> </Schedule> </Output>

.........

This is the Log: <13>Sep 4 16:07:23 Firewall: LEEF:1.0|FORCEPOINT|Firewall|1.1.1|Connection_Discarded|src=122.1.1.1 EventReceivedTime=2019-09-04 16:07:23 SourceModuleName=****** SourceModuleType=im_tcp LEEFVersion=<1> LEEF:0.0 Vendor=FORCEPOINT vSrcName=Firewall Version=1.1.1 EventID=Connection_Discarded devTimeFormat=MMM dd yyyy HH:mm:ss devTime=2019-09-04 16:07:23 proto=1 dstPort=80 srcPort=53438 dst=192.1.1.1 sender=services.fw.mi01.custom.cloud node 1 action=Discard

the system sets the value of $Sender like this: $Sender = services.fw.mi01.custom.cloud node 1 action=Discard.txt

but I need instead the system to set $Sender this way, only up to "node 1": $Sender = services.fw.mi01.custom.cloud node 1.txt

I thought about using a regex to extrapolate the value I need, but it doesn’t work. this one: <Exec> if $Sender =~ /(?<=sender=).[^\t]+/g; $Sender = $1 </Exec>

Can I do this thing? If so, what should I do?

Thank you Antonio

Permalink

#2 Zhengshi Nxlog ✓ 5 years ago

#1 nembosec

Hi, I’m trying to use regex in nxlog. My current configuration is to save firewall logs to a file .txt using the $Sender value to create the file name. ....... <Input *****> Module im_tcp Host 0.0.0.0 Port 1001 <Exec> if $raw_event =~ /LEEF/ parse_leef(); else parse_syslog(); </Exec> </Input> ....... <Output > define OUT_DIR %LOGDIR2%/ Module om_file File "%OUT_DIR%/" + $Sender + ".txt" <Schedule> Every 3600 sec <Exec> if ->file_size() > 0M { set_var('newfile', file_name() + strftime(now(), '_%Y%m%d%H%M%S') + '.log'); rotate_to(get_var('newfile')); exec_async('C:/Program Files/GnuWin32/bin/bzip2.exe', 'E:// *.log'); } </Exec> </Schedule> </Output> ......... This is the Log: <13>Sep 4 16:07:23 Firewall: LEEF:1.0|FORCEPOINT|Firewall|1.1.1|Connection_Discarded|src=122.1.1.1 EventReceivedTime=2019-09-04 16:07:23 SourceModuleName=****** SourceModuleType=im_tcp LEEFVersion=<1> LEEF:0.0 Vendor=FORCEPOINT vSrcName=Firewall Version=1.1.1 EventID=Connection_Discarded devTimeFormat=MMM dd yyyy HH:mm:ss devTime=2019-09-04 16:07:23 proto=1 dstPort=80 srcPort=53438 dst=192.1.1.1 sender=services.fw.mi01.custom.cloud node 1 action=Discard the system sets the value of $Sender like this: $Sender = services.fw.mi01.custom.cloud node 1 action=Discard.txt but I need instead the system to set $Sender this way, only up to "node 1": $Sender = services.fw.mi01.custom.cloud node 1.txt I thought about using a regex to extrapolate the value I need, but it doesn’t work. this one: <Exec> if $Sender =~ /(?<=sender=).[^\t]+/g; $Sender = $1 </Exec> Can I do this thing? If so, what should I do? Thank you Antonio

What version of NXLog are you using? I used NXLog EE v4.5.4503 to test this quickly and received the following:

2019-09-04 12:08:39 INFO nxlog-4.5.4503 started
2019-09-04 12:08:39 INFO Sender: services.fw.mi01.custom.cloud node 1
2019-09-04 12:08:39 INFO {"EventReceivedTime":"2019-09-04 16:07:23","SourceModuleName":"****** SourceModuleType=im_tcp","SourceModuleType":"im_file","Hostname":"Firewall:","LEEFVersion":"<1> LEEF:0.0","Vendor":"FORCEPOINT","SourceName":"Firewall","Version":"1.1.1","EventID":"Connection_Discarded","MessageSourceAddress":"122.1.1.1","devTimeFormat":"MMM dd yyyy HH:mm:ss","EventTime":"2019-09-04T16:07:23.000000-04:00","proto":"1","dstPort":80,"srcPort":53438,"dst":"192.1.1.1","sender":"services.fw.mi01.custom.cloud node 1","action":"Discard"}

Note that I used JSON to see the fields, and it looks like sender is set appropriately unless you were wanting to remove the node 1 part from the field?.

Conf I used:

<Input in> 
    Module im_file
    File '/opt/nxlog/etc/leef.log'
    ReadFromLast False
    SavePos False
    <Exec>
        if $raw_event =~ /LEEF/
        parse_leef();
        else
        parse_syslog();
	to_json(); log_info("Sender: " + $sender);log_info($raw_event);
    </Exec>
</Input>
<Output out> 
    define OUT_DIR /tmp/
    Module om_file
    File "%OUT_DIR%/" + $Sender + ".txt"
</Output>