Troubleshooting

Quick Diagnosis

# Check container status
docker ps | grep fan-out

# Check recent logs
docker logs fan-out-pattern --tail 50 2>&1 | grep -i error

# Test basic fan-out
curl -X POST http://localhost:8080/events \
  -H "Content-Type: application/json" \
  -d '{"sensor_id": "test", "temperature": 72}'

Common Issues

Messages only reaching some destinations

Cause: One destination failing, blocking others (if not using fan_out)

# Check each destination individually
curl -s http://localhost:9200/_cluster/health  # Elasticsearch
aws s3 ls s3://$BUCKET/ --max-items 1          # S3

Fix: Use broker with fan_out pattern (not fan_out_sequential):

output:
  broker:
    pattern: fan_out  # Parallel, not sequential
    outputs:
      - elasticsearch: {...}
      - aws_s3: {...}

---

One slow destination slowing everything

Cause: Using sequential fan-out or shared batching

Fix: Add independent batching per destination:

outputs:
  - elasticsearch:
      batching: {count: 100, period: 5s}
  - aws_s3:
      batching: {count: 1000, period: 60s}

---

Duplicate messages in destinations

Cause: Retries without idempotency

Fix: Add idempotent writes:

- kafka:
    idempotent_write: true
- elasticsearch:
    id: ${!json("event_id")}  # Dedupe by ID

---

High memory usage

Cause: Large batches backing up for slow destination

docker stats fan-out-pattern --no-stream

Fix: Add byte size limits to batching:

batching:
  count: 500
  period: 30s
  byte_size: 10MB

Still stuck?

1. Add debug logging: logger: {level: DEBUG}
2. Check the Complete Pipeline for reference config
3. Review Circuit Breakers for handling destination failures