Fan-Out Destinations

This architecture configures Elasticsearch, S3, and a local file as parallel destinations. Each destination remains an external dependency with its own delivery and retention contract.

Goal

• Elasticsearch: Configured searchable destination
• S3: Configured object-storage destination
• Local File: Configured on-site destination

Configuration

We use the broker output pattern with fan_out.

1. The Broker Output

output:
  broker:
    pattern: fan_out
    outputs:
      # Destination 1: Elasticsearch
      - elasticsearch:
          urls: ['${ELASTICSEARCH_URL}']
          index: 'logs-${!timestamp_unix_date()}'

      # Destination 2: S3
      - aws_s3:
          bucket: '${S3_BUCKET}'
          path: 'logs/${!timestamp_unix_date()}/${!uuid_v4()}.jsonl.gz'
          batching:
            count: 1000
            period: 30s
            processors: [{ compress: { algorithm: gzip } }]

2. Handling Failures

In a fan-out, if one output fails, we don't want to block the others. The fan_out pattern handles this, but individual outputs can also have their own fallback (e.g., write to disk if S3 is down).

Complete Step 6 Configuration

production-pipeline-step-6.yaml

input:
  http_server:
    address: '0.0.0.0:8080'
    path: /logs/ingest
    rate_limit: '1000/1s'
    auth:
      type: header
      header: 'X-API-Key'
      required_value: '${LOG_API_KEY}'

pipeline:
  processors:
    - mapping: 'root = this.parse_json().catch({"message": content()})'

output:
  broker:
    pattern: fan_out
    outputs:
      # 1. Console (Simulating real-time view)
      - stdout:
          codec: lines

      # 2. Local File (Simulating S3/Archive)
      - file:
          path: './output_logs/${!timestamp_unix_date()}_logs.jsonl'
          codec: lines

      # 3. Metrics (Simulating observability)
      - http_client:
          url: 'http://localhost:8080/metrics_endpoint' # Mock endpoint
          verb: POST
          # Use 'drop_on_error' to prevent pipeline blocking
          drop_on_error: true

Deployment & Verification

1. Send Data:

   curl -X POST http://localhost:8080/logs/ingest \
     -H "X-API-Key: $LOG_API_KEY" \
     -d '{"message": "multicast test"}'

2. Verify Outputs:

   • Check the terminal (stdout)
   • Check the output_logs/ directory for the file.

Troubleshooting

Issue	Solution
One output blocks others	Use drop_on_error: true on unreliable outputs
S3 connection fails	Check credentials and bucket permissions
File output missing	Verify the output directory exists and is writable

Conclusion

You have assembled the complete tutorial pipeline. You started with a basic HTTP server, added the tutorial's security controls, validated data, enriched it with context, filtered noise, redacted PII, and routed it to multiple destinations.

👉 View Complete Pipeline