Skip to main content

Troubleshooting

Quick Diagnosis​

# Check container status
docker ps | grep fan-out

# Check recent logs
docker logs fan-out-pattern --tail 50 2>&1 | grep -i error

# Test basic fan-out
curl -X POST http://localhost:8080/events \
  -H "Content-Type: application/json" \
  -d '{"sensor_id": "test", "temperature": 72}'

Common Issues​

Messages reaching some destinations but not others​

Cause: One destination failing, blocking others (if not using fan_out)

# Check each destination individually
curl -s http://localhost:9200/_cluster/health  # Elasticsearch
aws s3 ls s3://$BUCKET/ --max-items 1          # S3

Fix: Use broker with fan_out pattern (not fan_out_sequential):

Partial snippet
Partial snippet: validated fan-out destinations
output:
  broker:
    pattern: fan_out # Parallel, not sequential
    outputs:
      - opensearch:
          action: index
          urls: ['http://localhost:9200']
          index: events
          id: '${!this.event_id.or(uuid_v4())}'
      - aws_s3:
          bucket: '${S3_BUCKET:validation-bucket}'
          path: events/${!uuid_v4()}.jsonl

One slow destination slowing everything​

Cause: Using sequential fan-out or shared batching

Fix: Add independent batching per destination:

Partial snippet
Partial snippet: destination-specific batching
outputs:
  - opensearch:
      action: index
      urls: ['http://localhost:9200']
      index: events
      id: '${!this.event_id.or(uuid_v4())}'
      batching: { count: 100, period: 5s }
  - aws_s3:
      bucket: '${S3_BUCKET:validation-bucket}'
      path: events/${!uuid_v4()}.jsonl
      batching: { count: 1000, period: 60s }

Duplicate messages in destinations​

Cause: Retries without idempotency

Fix: Add idempotent writes:

Partial snippet
Partial snippet: idempotent fan-out destinations
output:
  broker:
    pattern: fan_out
    outputs:
      - kafka:
          addresses: ['localhost:9092']
          topic: events
          idempotent_write: true
      - opensearch:
          action: index
          urls: ['http://localhost:9200']
          index: events
          id: '${!this.event_id}' # Dedupe by ID

High memory usage​

Cause: Large batches backing up for slow destination

docker stats fan-out-pattern --no-stream

Fix: Add byte size limits to batching:

Partial snippet
batching:
  count: 500
  period: 30s
  byte_size: 10485760

Still stuck?​

  1. Add debug logging: logger: {level: DEBUG}
  2. Check the Complete Pipeline for reference config
  3. Review Circuit Breakers for handling destination failures