Trying to transform a strange data format

Tags:

#1 davidatpinger

I've got some data that comes in with a somewhat unusual format.  It's a set of fixed fields, followed by a variable length set of keys, followed by a set of values.  It looks something like this (but with more fields):

col1, col2, col3, description(key1; key2; ...;keyn), val1, val2, ..., valn

I'm trying to transform this into something more like:

a=col1, b=col2, c=col3, key1=val1, key2=val2, ..., keyn=valn

I've actually got this working by using Exec and a bit of perl that I wrote that tears apart $raw_event and writes the modified logline to a domain socket, where a second instance Route is listening and sends the log over the network to its destination.  My problem is that this is not terribly performant, since it starts a perl process per log line.  I've had trouble figuring out another way to do this, mostly because the number of keys/values is variable.

Any suggestions on ways this might be done that are likely to have better performance?

#2 adm Nxlog ✓
#1 davidatpinger
I've got some data that comes in with a somewhat unusual format.  It's a set of fixed fields, followed by a variable length set of keys, followed by a set of values.  It looks something like this (but with more fields): col1, col2, col3, description(key1; key2; ...;keyn), val1, val2, ..., valn I'm trying to transform this into something more like: a=col1, b=col2, c=col3, key1=val1, key2=val2, ..., keyn=valn I've actually got this working by using Exec and a bit of perl that I wrote that tears apart $raw_event and writes the modified logline to a domain socket, where a second instance Route is listening and sends the log over the network to its destination.  My problem is that this is not terribly performant, since it starts a perl process per log line.  I've had trouble figuring out another way to do this, mostly because the number of keys/values is variable. Any suggestions on ways this might be done that are likely to have better performance?

If you already have perl code then you can reuse that with xm_perl. This will be executed in the same process space so it should be a bit faster.