Fan-Out Destinations

A single destination is rarely enough. You need real-time search for debugging (Elasticsearch), cheap long-term storage for compliance (S3), and perhaps a local backup. The Fan-Out pattern sends data to all of them in parallel.

Goal

• Elasticsearch: Hot storage for recent logs (searchable)
• S3: Cold storage for all logs (cheap, compressed)
• Local File: Backup in case of network issues

Configuration

We use the broker output pattern with fan_out.

1. The Broker Output

output:
  broker:
    pattern: fan_out
    outputs:
      # Destination 1: Elasticsearch
      - elasticsearch:
          urls: ["${ELASTICSEARCH_URL}"]
          index: "logs-${!timestamp_unix_date()}"
          
      # Destination 2: S3
      - aws_s3:
          bucket: "${S3_BUCKET}"
          path: "logs/${!timestamp_unix_date()}/${!uuid_v4()}.jsonl.gz"
          batching:
             count: 1000
             period: 30s
             processors: [ { compress: { algorithm: gzip } } ]

2. Handling Failures

In a fan-out, if one output fails, we don't want to block the others. The fan_out pattern handles this, but individual outputs can also have their own fallback (e.g., write to disk if S3 is down).

Complete Step 6 Configuration

production-pipeline-step-6.yaml

input:
  http_server:
    address: "0.0.0.0:8080"
    path: /logs/ingest
    rate_limit: "1000/1s"
    auth:
      type: header
      header: "X-API-Key"
      required_value: "${LOG_API_KEY}"

pipeline:
  processors:
    - mapping: 'root = this.parse_json().catch({"message": content()})'

output:
  broker:
    pattern: fan_out
    outputs:
      # 1. Console (Simulating real-time view)
      - stdout:
          codec: lines

      # 2. Local File (Simulating S3/Archive)
      - file:
          path: "./output_logs/${!timestamp_unix_date()}_logs.jsonl"
          codec: lines
          
      # 3. Metrics (Simulating observability)
      - http_client:
          url: "http://localhost:8080/metrics_endpoint" # Mock endpoint
          verb: POST
          # Use 'drop_on_error' to prevent pipeline blocking
          drop_on_error: true 

Deployment & Verification

1. Send Data:

   curl -X POST http://localhost:8080/logs/ingest \
     -H "X-API-Key: $LOG_API_KEY" \
     -d '{"message": "multicast test"}'

2. Verify Outputs:

   • Check the terminal (stdout)
   • Check the output_logs/ directory for the file.

Troubleshooting

Issue	Solution
One output blocks others	Use drop_on_error: true on unreliable outputs
S3 connection fails	Check credentials and bucket permissions
File output missing	Verify the output directory exists and is writable

Conclusion

You have built a complete, production-grade log processing pipeline! You started with a basic HTTP server, added security, validated data, enriched it with context, filtered noise, redacted PII, and routed it to multiple destinations.

👉 View Complete Pipeline