Inspect the multi-destination pipeline

Compare the synthetic record as each configured processing concern is introduced.

How to use this explorer

Move between stages with the arrows or the numbered stage list. Each stage shows the input it receives on the left and the output it produces on the right, with the lines that changed marked. The configuration that makes the change sits below the comparison, and the final stage shows the complete pipeline.

The 6 stages in order

1. Raw HTTP Input
   A synthetic HTTP record before parsing, enrichment, selection, or field handling.
2. Parse & Validate
   Parse JSON, validate schema with required fields (msg, user). Invalid logs route to DLQ for debugging. Add pipeline metadata showing when/where processing occurred.
3. Enrich Metadata
   Add authored processing time, correlation, version, and node fields.
4. Filter & Score
   Apply the authored priority mapping and remove records below its configured threshold.
5. Redact PII
   Delete the configured source field and add an unsalted hash for the synthetic email value.
6. Fan-Out
   Configure Elasticsearch, Kafka, and S3 outputs with one priority check on the Elasticsearch branch.

Interactive pipeline

MULTI-DESTINATION LOG PIPELINE

Curated architecture walkthrough

Stage1. Raw HTTP Input2. Parse & Validate3. Enrich Metadata4. Filter & Score5. Redact PII6. Fan-Out

Stage 1 of 6

Raw HTTP Input

A synthetic HTTP record before parsing, enrichment, selection, or field handling.

1HighlightsAuthored emphasis only—not a computed diff.

Highlights only

Copy & download

Input

 {"msg":"User login","user":"[email protected]"}

Output

•Authored highlight: ❌ Raw, Unstructured: No timestamp No correlation ID PII exposed (email) No priority/severity

Partial snippet01-raw-http-input.yaml

input:
  http_server:
    address: 0.0.0.0:8080
    path: /logs