Your detections, correlation rules, and search results are only as good as the underlying data structure. A raw log line is a string. A structured JSON event is a set of named fields you can filter, pivot, and alert on.

Convert your logs to JSON at the collection layer before they reach your SIEM. Doing it early cuts ingest cost, keeps your schema consistent across sources, and makes your detections less fragile.

This post is for SecOps engineers feeding logs from mixed sources — Windows, Linux, network gear, applications — into a SIEM or data lake. We’ll cover why edge conversion wins, and how to set it up with NXLog Agent.

What structured logging buys you

You already know the difference between grepping a text file and querying a field. Structured logging makes that difference systematic. Every event becomes a JSON object with keys — user.name, source.ip, event.action — instead of a line you parse on read.

That structure pays off where SecOps spends its time:

  • Search speed — Field-indexed queries return faster than full-text scans across raw lines.

  • Correlation — When a username lives in the same field across Windows, syslog, and app logs, you can join events across sources with a single query.

  • Detection stability — Rules that operate on fields don’t break when a vendor reorders a message or adds a token. Rules that match regex against raw text do.

The catch: most log sources don’t emit clean JSON on their own, so you have to convert them. The question is where.

Why conversion belongs at the collection layer

It’s tempting to ship raw logs and let the SIEM sort them out at ingest. That approach costs you in four ways.

Ingest cost

When your SIEM bills by ingest volume, sending pre-structured JSON cuts reprocessing and lets you drop fields you don’t need before they count against your license.

Consistency

Parse once, at the edge, and every destination gets the same schema. Parse at each SIEM, and you maintain a different ruleset for every tool you feed.

Resilience

Structured data survives the trip. Ingest-time regex and grok patterns are brittle — one format change upstream and parsing fails silently, often unnoticed until a detection goes quiet. You don’t miss it until you need it.

Portability

Once an event is formatted in structured JSON, you can fan it out to a SIEM, an archive, and a data lake at the same time, with no per-destination parsing.

Converting at the edge only works if the logic is disciplined: keep it somewhere controlled and versioned, close to the source where the format is known.

How NXLog Agent converts logs to JSON

NXLog Agent handles JSON conversion with the JSON extension:

  • to_json() takes the event fields, serializes them into a JSON object, and writes it to the $raw_event core field.

  • parse_json() does the reverse — it reads a JSON string and breaks it into structured fields you can process further.

The simplest case — read syslog messages from a file, parse them into fields, write JSON back out:

<Extension json>
    Module    xm_json
</Extension>

<Extension syslog>
    Module    xm_syslog
</Extension>

<Input system_logs>
    Module    im_file
    File      '/var/log/messages'
    Exec      parse_syslog(); (1)
</Input>

<Output json_file>
    Module    om_file
    File      '/var/log/messages.json'
    Exec      to_json(); (2)
</Output>

<Route syslog_to_json>
    Path      system_logs => json_file
</Route>
1 The parse_syslog() procedure, from the Syslog extension, turns each line into fields.
2 The to_json() procedure serializes those fields on the way out.

Windows is where this becomes more useful. The Event Log for Windows input module reads Windows events directly through the native API, so events arrive already structured — full details and XML intact. From there, you serialize to JSON and forward:

<Extension json>
    Module    xm_json
</Extension>

<Input windows_events>
    Module    im_msvistalog
</Input>

<Output siem>
    Module    om_tcp
    Host      siem.example.com:1514
    Exec      to_json();
</Output>

<Route windows_events_to_siem>
    Path      windows_events => siem
</Route>

Fields whose names begin with a dot or underscore are hidden by default; set IncludeHiddenFields to TRUE to have to_json() include them in the output. For inbound JSON that spans multiple lines or arrives as an array, use the InputType directive instead of calling parse_json() per line.

Mapping to a common schema

Converting to JSON solves the format problem. It doesn’t solve the naming problem. If Windows calls it AccountName, syslog calls it user, and your app calls it username, your correlation rules still have to account for all three.

A shared schema fixes that. NXLog Agent’s Rewrite extension renames and reshapes fields before converting to JSON, so you can map every source onto one model — for example, the Elastic Common Schema (ECS):

<Extension json>
    Module        xm_json
</Extension>

<Extension syslog>
    Module        xm_syslog
</Extension>

<Extension ecs>
    Module        xm_rewrite
    Rename        Hostname, host.name
    Rename        Message, message
    Rename        MessageSourceAddress, host.ip
</Extension>

<Input udp_listen>
    Module        im_udp
    ListenAddr    0.0.0.0:514
    <Exec>
        parse_syslog();
        ecs->process();
    </Exec>
</Input>

<Output siem>
    Module        om_tcp
    Host          siem.example.com:1514
    Exec          to_json();
</Output>

<Route udp_to_siem>
    Path          udp_listen => siem
</Route>

This is a trimmed example — a full ECS mapping covers many more fields, and our integration guides include complete templates. The mapping lives in one place, under version control, and is applied the same way to every source before the data leaves the host.

Details worth getting right

A few things separate JSON that works from JSON that causes silent problems downstream:

Timestamps

By default, the JSON extension outputs dates as YYYY-MM-DDThh:mm:ss.sTZ, which Elasticsearch reads as a date field out of the box. If a destination needs a different format, set the DateFormat directive on the module rather than fixing it after ingest.

Nested vs. flat

Some destinations want flat keys; others handle nested objects. Check what your SIEM expects and shape the output to match at the agent.

Field discipline

Pick field names once and keep them stable. Renaming a field your detections depend on is a breaking change, even when the data looks identical.

Know what you emit

NXLog Agent excludes the $raw_event core field from the JSON it builds, so the output carries your parsed fields while the original raw line stays out of it. Use the Rewrite extension’s Delete directive to drop fields you don’t want before they go out.

Where NXLog fits in your pipeline

NXLog Agent runs the conversion described above at the edge, on the host where the log originates. It runs on Windows, Linux, macOS, BSD, Solaris, and AIX, and reads Windows events natively, so you can collect and structure data with one agent per host instead of stitching together multiple forwarders.

NXLog Platform sits above the fleet. From its web interface, you manage agent configurations with templates, route structured output to multiple destinations with visual pipelines, and watch data move through in real time. The conversion logic you write once applies to every agent, and the same JSON can land in your SIEM, your archive, and your storage of choice at once.

If you’re moving from raw log forwarding to structured collection, that’s a shift worth making. Parse and convert close to the source, standardize on one schema, and let every downstream tool work from clean data. Our integration guides walk through the setup for specific SIEMs if you want a full configuration for your stack.

NXLog Platform is an on-premises solution for centralized log management with
versatile processing forming the backbone of security monitoring.

With our industry-leading expertise in log collection and agent management, we comprehensively
address your security log-related tasks, including collection, parsing, processing, enrichment, storage, management, and analytics.

Start free Contact us
Share
Related Posts