Step 1: Parse JSON Logs with a Fallback

Log streams can contain both structured JSON and plain text. This step configures a JSON parse attempt with a plain-text fallback.

This step teaches you how to parse a log line as JSON if possible, and if not, to treat it as a plain text message without failing.

The Goal

You will build a pipeline that can process both of these incoming messages:

// Input 1 (JSON String)
{"raw_log": "{\"level\":\"info\",\"message\":\"User logged in\"}"}

// Input 2 (Plain Text)
{"raw_log": "web server started"}

And turn them into a consistent format, identifying which was which.

The `.parse_json().catch()` Pattern

The key to this pattern is the .catch() method. When .parse_json() fails on a string that isn't valid JSON, .catch() allows you to provide a fallback value instead of letting the processor fail.

Implementation

Create the Parser Pipeline: Copy the following configuration into a file named robust-parser.yaml.

Complete pipeline

robust-parser.yaml
name: robust-json-parser
description: A pipeline that handles a mix of JSON and plain-text logs.

config:
  input:
    generate:
      count: 1
      interval: 1s
      mapping: |
        root = [
          {"raw_log": "{\"level\":\"info\",\"message\":\"User logged in\"}"},
          {"raw_log": "web server started"}
        ]

  pipeline:
    processors:
      - unarchive:
          format: json_array
      - mapping: |
          let parsed = this.raw_log.parse_json().catch(null)
          if $parsed.type() == "object" {
            root = $parsed.merge({"was_json": true})
          } else {
            root = {
              "message": this.raw_log,
              "was_json": false,
              "level": "unknown"
            }
          }

  output:
    stdout:
      codec: lines

Deploy and Observe: Watch the logs. You will see a stream of output messages. Some will be fully structured with level and message fields, and was_json: true. Others will have the raw string inside the message field, with level: "unknown" and was_json: false.

Verification

For the authored sample, JSON logs become structured records and plain-text logs follow the configured fallback shape. The example does not test other malformed inputs or failure modes.

The Goal​

The .parse_json().catch() Pattern​

Implementation​

Verification​

The Goal

The `.parse_json().catch()` Pattern

Implementation

Verification