Enrich with Metadata

Raw logs often lack context. To make them useful for debugging and auditing, we need to know where they were processed, which pipeline handled them, and when.

Goal

Add the following metadata to every log event:

• Processing Node: ID and region of the edge node
• Pipeline Info: Version and environment
• Audit: Ingestion timestamp and trace ID

Configuration

We use the mapping processor to inject environment variables and generated values.

1. Define Metadata

We'll add a metadata object to keep things organized.

Partial snippet

- mapping: |
    root = this
    root.metadata = {
      "ingest_timestamp": now(),
      "trace_id": uuid_v4(),
      "node_id": env("NODE_ID").or("unknown_node"),
      "region": env("AWS_REGION").or("unknown_region"),
      "environment": env("ENVIRONMENT").or("dev")
    }

Complete Step 3 Configuration

See the complete reference for the maintained input, named rate-limit resource, processors, and failure boundary.

Deployment & Verification

1. Set Env Vars:

   export NODE_ID="node-01"

2. Test:

   curl -X POST http://localhost:8080/logs/ingest \
     -H "X-API-Key: $LOG_API_KEY" \
     -d '{"level": "ERROR", "message": "DB fail"}'

   Result should include metadata with processing_node: "node-01" and urgent: true.

Next Steps

With enriched data, we can now make intelligent decisions about what to keep and how to prioritize it.

👉 Step 4: Filter & Score Logs