Hi, I have this type of input in nxlog.conf: ``` Module im_udp Host 0.0.0.0 Port 514 Exec parse_syslog(); to_json(); perl_call("process"); ``` My question is, how should I include that JSON output that I get from to_json() to my perl code? Should I write like this?: ``` my ( $event ) = @_; ``` Or it's only the output of parse_syslog_bsd() (as in example for xm_perl https://nxlog.co/documentation/nxlog-user-guide/xm_perl.html)? **More generally, my question is how to include JSON output that i get from to_json() to perl code of xm_perl module?**

It does not appear to be possible for NxLog Community to add fields to a JSON document which contain complex fields because parse_json() converts those complex fields to strings, thereby breaking the document as it is sent upstream. Alernatively, I'm doing something wrong ;).

Let's say the NxLog is reading a file with a JSON document on each line and I want to add a custom field. I understand I would do the following:

Exec parse_json();
Exec $new_field = 'too sexy';
Exec to_json();

If the original JSON line is 

{"field1":1, "field2":2, "fieldComplex":{"a":3,"b",4}}

I would expect my resuting document, after NxLog injection, to be:

{"field1":1, "field2":2, "fieldComplex":{"a":3,"b",4}, "new_field":"too sexy"}

But that's not what happens. The complex field is converted to text and the resulting document is:

{"field1":1, "field2":2, "fieldComplex":"{'a':3,'b',4}", "new_field":"too sexy"}

Can this behaviour be avoided?

Thanks, Geoff

Hi everyone,

We have an internal application on a windows box which logs in a way which is slightly mangled. All other windows logs come through nxlog but on these messages the entire message field gets dropped.

Is there a way to parse using ifs? eg:

if $service == "Homegrown app" {

}

So that I can either change the output format or parse them untouched into a file etc...

Does anyone know if there are any characters that don't get escaped properly that might mangle the json output?

Sorry for not giving a huge amount of information, I have to keep this as agnostic as I can.

Hi,

I am trying to parse a log4net file into json.

Here's my sample log4net:

----------------

2015-01-27 01:06:18,859 [7] ERROR Web.Cms.Content.Base.Taxonomy.TaxonomyDetectionProvider [(null)] - Get taxonomy Type Failed for Tools
2015-01-27 06:34:31,051 [26] ERROR www.Status404 [(null)] - ErrorId: 20150127_102b01c6-3208-48c5-8c8b-ae4f92cf2b20
    UserAgent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.99 Safari/537.36
    HostAddress: 192.168.10.2
    RequestUrl: /ErrorPages/404.aspx
    MachineName: QA01
    Raw Url:/undefined/
    Referrer: http://qa1.www.something.com/toolset.aspx

2015-01-27 06:34:33,270 [26] DEBUG Web.Caching.Core.CacheManagerBase [(null)] - Custom CacheProvider:Web.Caching.Core.AppFabricCacheManager,Web.Caching.Core Disabled

 

Now I am using xm_multiline to capture each log entries.

----------------

<Extension multiline>
    Module        xm_multiline
    HeaderLine    /^\d{4}\-\d{2}\-\d{2} \d{2}\:\d{2}\:\d{2},\d{3}/
    EndLine        /\r?\n\r?\n^\d{4}\-\d{2}\-\d{2} \d{2}\:\d{2}\:\d{2},\d{3}/
</Extension>

I use a regex to capture the timestamp as the header then I use a regex to capture twice newline then the next timestamp as endline. However it still treat the second and last entry as ONE log entry.

Here's the output:

----------------

{  
   "EventReceivedTime":"2015-01-27 01:06:35",
   "SourceModuleName":"log4net",
   "SourceModuleType":"im_file",
   "time":"2015-01-27 01:06:18,859",
   "thread":"7",
   "level":"ERROR",
   "logger":"Web.Cms.Content.Base.Taxonomy.TaxonomyDetectionProvider",
   "ndc":"(null)",
   "message":"Get taxonomy Type Failed for Tools"
}{  
   "EventReceivedTime":"2015-01-27 06:34:35",
   "SourceModuleName":"log4net",
   "SourceModuleType":"im_file",
   "time":"2015-01-27 06:34:31,051",
   "thread":"26",
   "level":"ERROR",
   "logger":"www.Status404",
   "ndc":"(null)",
   "message":"  ErrorId: 20150127_102b01c6-3208-48c5-8c8b-ae4f92cf2b20\r\n  UserAgent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.99 
Safari/537.36\r\n  HostAddress: 192.168.10.2\r\n  RequestUrl: /ErrorPages/404.aspx\r\n  MachineName: QA01\r\n  
Raw Url:/undefined/\r\n  Referrer: http://qa1.www.something.com/toolset.aspx\r\n\r\n2015-01-27 06:34:33,270 [26] DEBUG Web.Caching.Core.CacheManagerBase [(null)] - Custom CacheProvider:Web.Caching.Core.AppFabricCacheManager,Web.Caching.Core Disabled"
}

I used this to produce that output:

----------------

Exec        if $raw_event =~ /^(\d{4}\-\d{2}\-\d{2} \d{2}\:\d{2}\:\d{2},\d{3}) \[(\S+)\] (\S+) (\S+) \[(\S+)\] \- (.*)/s \
                { \
                    $time = $1; \
                    $thread = $2; \
                    $level = $3; \
                    $logger = $4; \
                    $ndc = $5; \
                    $message = $6; \
                    to_json(); \
                } \
                else \
                { \
                    drop(); \
                }

 

 

I've also tried to tweak it by using this to avoid the combining the last two entries as one. However I am not able to get the last entry anymore.

----------------

Exec        if $raw_event =~ /^(\d{4}\-\d{2}\-\d{2} \d{2}\:\d{2}\:\d{2},\d{3}) \[(\S+)\] (\S+) (\S+) \[(\S+)\] \- ([\s\S]*?)(\r?\n\r?\n|$)/ \
                { \
                    $time = $1; \
                    $thread = $2; \
                    $level = $3; \
                    $logger = $4; \
                    $ndc = $5; \
                    $message = $6; \
                    to_json(); \
                } \
                else \
                { \
                    drop(); \
                }​