Skip to main content

Complete Pipeline

Copy pipeline YAML
Review it for your environment before you deploy.

This pipeline combines all fan-out techniques from this tutorial:

  • Multi-destination output - Send every event to multiple destinations simultaneously
  • Broker pattern - Use fan_out to duplicate messages to each output
  • Destination-specific config - Configure batching, compression per destination

Full Configuration​

Complete pipeline
fan-out-pattern.yaml
name: fan-out-pattern
description: Multi-destination fan-out pipeline for sensor data
type: pipeline
namespace: default
labels:
  pattern: fan-out
  data-type: sensor-metrics

config:
  input:
    http_server:
      address: 0.0.0.0:8080
      path: /events

  output:
    broker:
      pattern: fan_out
      outputs:
        # Local file for debugging
        - file:
            path: /tmp/events.jsonl
            codec: lines

        # Kafka for real-time streaming
        - kafka:
            addresses: ['${KAFKA_BROKERS}']
            topic: sensor-events
            key: '${!this.sensor_id}'

        # S3 for long-term archival
        - aws_s3:
            bucket: '${S3_BUCKET}'
            region: '${AWS_REGION}'
            path: 'events/${!timestamp_unix()}/${!uuid_v4()}.json'
            batching:
              count: 10
              period: 10s

        # Elasticsearch for search and analytics
        - opensearch:
            action: index
            urls: ['${ES_ENDPOINT}']
            index: 'sensor-events-${!timestamp_unix()}'
            id: '${!this.event_id.or(uuid_v4())}'
            batching:
              count: 10
              period: 10s

Quick Test​

# Send an event (goes to all configured destinations)
curl -X POST http://localhost:8080/events \
  -H "Content-Type: application/json" \
  -d '{
    "sensor_id": "temp-001",
    "temperature": 72.5,
    "humidity": 45,
    "timestamp": "2024-01-15T10:30:00Z"
  }'

Deploy​

# Deploy to Expanso orchestrator
expanso-cli job deploy fan-out-pattern.yaml

# Or run locally with expanso-edge
expanso-edge run --config fan-out-pattern.yaml

Download​

Download fan-out-pattern.yaml

What's Next?​