Skip to main content

Step 1: Parse JSON Logs with a Fallback

In the real world, log streams are often messy and contain a mix of formats. A robust pipeline needs to handle both structured (JSON) and unstructured (plain text) logs gracefully.

This step teaches you how to parse a log line as JSON if possible, and if not, to treat it as a plain text message without failing.

The Goal​

You will build a pipeline that can process both of these incoming messages:

// Input 1 (JSON String)
{"raw_log": "{\"level\":\"info\",\"message\":\"User logged in\"}"}

// Input 2 (Plain Text)
{"raw_log": "web server started"}

And turn them into a consistent format, identifying which was which.

The .parse_json().catch() Pattern​

The key to this pattern is the .catch() method. When .parse_json() fails on a string that isn't valid JSON, .catch() allows you to provide a fallback value instead of letting the processor fail.

Implementation​

  1. Create the Parser Pipeline: Copy the following configuration into a file named robust-parser.yaml.

    robust-parser.yaml
    name: robust-json-parser
    description: A pipeline that handles a mix of JSON and plain-text logs.

    config:
      input:
        generate:
          interval: 1s
          mapping: |
            # This generates a mix of JSON strings and plain-text strings
            root.raw_log = if random_int() % 2 == 0 {
              '{"level": "info", "message": "User logged in"}'
            } else {
              "web server started"
            }

      pipeline:
        processors:
          - mapping: |
              # Try to parse the raw_log field as JSON.
              # If it fails, .catch() will return the original raw_log string.
              let parsed = this.raw_log.parse_json().catch(this.raw_log)

              # Now, check if the result is a structured object or still a string.
              if parsed.type() == "object" {
                root = parsed
                root.was_json = true
              } else {
                root = {
                  "message": parsed,
                  "was_json": false,
                  "level": "unknown"
                }
              }

      output:
        stdout:
          codec: lines

  2. Deploy and Observe: Watch the logs. You will see a stream of output messages. Some will be fully structured with level and message fields, and was_json: true. Others will have the raw string inside the message field, with level: "unknown" and was_json: false.

Verification​

This pattern successfully handles a mixed-format log stream, converting the JSON logs to a structured format and wrapping the plain-text logs in a consistent way. This prevents your pipeline from crashing on unexpected data and is the first step to building a truly robust log processing system.