Fan-Out Destinations

This architecture configures Elasticsearch, S3, and a local file as parallel destinations. Each destination remains an external dependency with its own delivery and retention contract.

Goal

• Elasticsearch: Configured searchable destination
• S3: Configured object-storage destination
• Local File: Configured on-site destination

Configuration

We use the broker output pattern with fan_out.

1. The Broker Output

Partial snippet

output:
  broker:
    pattern: fan_out
    outputs:
      # Destination 1: Elasticsearch
      - opensearch:
          action: index
          urls: ['${ELASTICSEARCH_URL}']
          index: 'logs-${!timestamp_unix_date()}'
          id: '${!this.request_id.or(uuid_v4())}'

      # Destination 2: S3
      - aws_s3:
          bucket: '${S3_BUCKET}'
          path: 'logs/${!timestamp_unix_date()}/${!uuid_v4()}.jsonl.gz'
          batching:
            count: 1000
            period: 30s
            processors: [{ compress: { algorithm: gzip } }]

2. Handling Failures

In a fan-out, if one output fails, we don't want to block the others. The fan_out pattern handles this, but individual outputs can also have their own fallback (e.g., write to disk if S3 is down).

Complete Step 6 Configuration

See the complete reference for the maintained input, named rate-limit resource, processors, and failure boundary.

Deployment & Verification

1. Send Data:

   curl -X POST http://localhost:8080/logs/ingest \
     -H "X-API-Key: $LOG_API_KEY" \
     -d '{"message": "multicast test"}'

2. Verify Outputs:

   • Check the terminal (stdout)
   • Check the output_logs/ directory for the file.

Troubleshooting

Issue	Solution
One output blocks others	Use drop_on_error: true on unreliable outputs
S3 connection fails	Check credentials and bucket permissions
File output missing	Verify the output directory exists and is writable

Conclusion

You have assembled the complete tutorial pipeline. You started with a basic HTTP server, added the tutorial's security controls, validated data, enriched it with context, filtered noise, redacted PII, and routed it to multiple destinations.

👉 View Complete Pipeline