Skip to main content

Inspect log enrichment

Compare a synthetic log as metadata is added, fields are regrouped, and object output is configured.

How to use this explorer

Move between stages with the arrows or the numbered stage list. Each stage shows the input it receives on the left and the output it produces on the right, with the lines that changed marked. The configuration that makes the change sits below the comparison, and the final stage shows the complete pipeline.

The 5 stages in order

  1. Original Log Data
    A synthetic application log before processing fields are added or regrouped.
  2. Add Lineage Metadata
    Add configured node, pipeline, and processing-time fields to the synthetic record.
  3. Restructure to Event/Metadata
    Regroup the authored fields under event and processing-metadata objects.
  4. Configure Batching
    Configure a batch boundary of ten messages or ten seconds.
  5. Configured S3 Object
    Show the authored JSON Lines object key and record shape for the configured S3 output.

Interactive pipeline

LOG ENRICHMENT

Curated export walkthrough

Stage 1 of 5

Original Log Data

A synthetic application log before processing fields are added or regrouped.

0HighlightsAuthored emphasis only—not a computed diff.
Copy & download

Input

{"id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890","timestamp": "2024-01-15T10:30:00Z","level": "INFO","service": "demo-service","message": "Demo log message from edge","user_id": "user_123","request_id": "b2c3d4e5-f6a7-8901-bcde-f12345678901"}

Output

{"id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890","timestamp": "2024-01-15T10:30:00Z","level": "INFO","service": "demo-service","message": "Demo log message from edge","user_id": "user_123","request_id": "b2c3d4e5-f6a7-8901-bcde-f12345678901"}
Partial snippet01-original-log-data.yaml
input:
  generate:
    interval: 2s
    mapping: |
      root.id = uuid_v4()
      root.timestamp = now()
      root.level = "INFO"
      root.service = "demo-service"
      root.message = "Demo log message from edge"
      root.user_id = "user_123"
      root.request_id = uuid_v4()