Trying to transform a strange data format

View thread

davidatpinger

I've got some data that comes in with a somewhat unusual format.  It's a set of fixed fields, followed by a variable length set of keys, followed by a set of values.  It looks something like this (but with more fields):

col1, col2, col3, description(key1; key2; ...;keyn), val1, val2, ..., valn

I'm trying to transform this into something more like:

a=col1, b=col2, c=col3, key1=val1, key2=val2, ..., keyn=valn

I've actually got this working by using Exec and a bit of perl that I wrote that tears apart $raw_event and writes the modified logline to a domain socket, where a second instance Route is listening and sends the log over the network to its destination.  My problem is that this is not terribly performant, since it starts a perl process per log line.  I've had trouble figuring out another way to do this, mostly because the number of keys/values is variable.

Any suggestions on ways this might be done that are likely to have better performance?