Trying to transform a strange data format
I've got some data that comes in with a somewhat unusual format. It's a set of fixed fields, followed by a variable length set of keys, followed by a set of values. It looks something like this (but with more fields):
col1, col2, col3, description(key1; key2; ...;keyn), val1, val2, ..., valn
I'm trying to transform this into something more like:
a=col1, b=col2, c=col3, key1=val1, key2=val2, ..., keyn=valn
I've actually got this working by using Exec and a bit of perl that I wrote that tears apart $raw_event and writes the modified logline to a domain socket, where a second instance Route is listening and sends the log over the network to its destination. My problem is that this is not terribly performant, since it starts a perl process per log line. I've had trouble figuring out another way to do this, mostly because the number of keys/values is variable.
Any suggestions on ways this might be done that are likely to have better performance?