Skip to main content

Fan-Out Explorer

Use the stage controls to compare the synthetic input, authored output, and configuration at each step.

How to use this explorer

Move between stages with the arrows or the numbered stage list. Each stage shows the input it receives on the left and the output it produces on the right, with the lines that changed marked. The configuration that makes the change sits below the comparison, and the final stage shows the complete pipeline.

The 5 stages in order

  1. Single Destination
    Basic pipeline with single file output - the typical starting point before implementing fan-out pattern.
  2. Broker Fan-Out Foundation
    Introduction of broker output with fan_out pattern enabling concurrent delivery to multiple destinations.
  3. Kafka Real-Time Streaming
    Add a Kafka branch with authored batching settings that require environment-specific tuning.
  4. S3 Long-Term Archive
    Object-storage destination with authored batching and compression settings that require environment-specific review.
  5. Elasticsearch Search & Analytics
    Search and analytics destination with balanced batching for near real-time indexing and operational monitoring.

Interactive pipeline

FAN-OUT PATTERN

Multi-destination branch configuration

Stage 1 of 5

Single Destination

Basic pipeline with single file output - the typical starting point before implementing fan-out pattern.

0HighlightsAuthored emphasis only—not a computed diff.
Copy & download

Input

{"event_id": "sensor-001","sensor_id": "temp-sensor-42","timestamp": "2025-01-20T10:30:00Z","temperature": 23.5,"humidity": 65.2}

Output

→ file:path: /var/data/events.jsonl// Single destination - no redundancy
Complete pipeline01-single-destination.yaml
name: single-destination-pipeline
type: pipeline

config:
  input:
    http_server:
      address: 0.0.0.0:8080
      path: /events

  output:
    file:
      path: /var/data/events.jsonl
      codec: lines