Skip to main content

Advanced Fan-Out Patterns

After reviewing the basic fan_out broker, use these variants as unverified adaptation sketches.

Pattern 1: Failover with the try Broker​

While fan_out sends to all outputs, the try pattern sends to them in order, stopping at the first success. This is perfect for creating a primary/fallback setup.

Use Case: You want to send data to a primary Elasticsearch cluster, but if that is down, you want to fall back to a secondary cluster, and if that is also down, write to a local file.

Partial snippet
Failover Routing with the 'try' Broker
output:
  broker:
    pattern: try # Tries each output in order until one succeeds
    outputs:
      # 1. Primary Destination
      - opensearch:
          action: index
          urls: ['${ES_PRIMARY_ENDPOINT}']
          index: 'primary-index'
          id: '${!this.event_id.or(uuid_v4())}'

      # 2. Secondary/Fallback Destination
      - opensearch:
          action: index
          urls: ['${ES_SECONDARY_ENDPOINT}']
          index: 'secondary-index'
          id: '${!this.event_id.or(uuid_v4())}'

      # 3. Last Resort: Local File
      - file:
          path: /tmp/es_fallback_buffer.jsonl
          codec: lines

Pattern 2: Conditional Routing within a Fan-Out​

Sometimes a message should reach several branches while a conditional branch accepts a subset. Nest a switch inside the broker to express that selection.

Example: Send the authored logs to Kafka and S3, and select ERROR records for the alerting-service branch.

Partial snippet
Conditional Fan-Out Routing
output:
  broker:
    pattern: fan_out
    outputs:
      # Destination 1: Kafka (all messages go here)
      - kafka:
          addresses: ['${KAFKA_BROKERS}']
          topic: 'all-logs'

      # Destination 2: S3 (all messages go here)
      - aws_s3:
          bucket: 'my-log-archive'
          path: 'logs/${!timestamp_unix_date()}/${!uuid_v4()}.jsonl.gz'

      # Destination 3: Alerting Service (errors selected here)
      - switch:
          cases:
            - check: this.level == "ERROR"
              output:
                http_client:
                  url: 'http://my-alerting-service/ingest'
                  verb: 'POST'
            # If the check fails, the 'switch' does nothing,
            # effectively dropping the message for this branch.

This powerful combination allows you to build complex, business-aware routing logic while maintaining the core efficiency of the fan-out pattern.