Step 1: HTTP Client Circuit Breakers

External APIs can be unreliable. If an API call fails, it can bring your entire pipeline to a halt. In this step, you will protect a fragile pipeline by adding a circuit breaker using a try/catch block.

1. The Fragile Pipeline

First, take a look at the circuit-breakers-foundation.yaml file in examples/data-routing/. It takes incoming sensor data, calls an API to enrich it with metadata, and prints the result.

It works perfectly when the API is healthy, but has no protection if the API fails.

2. See It Fail

Deploy the foundation pipeline and send it a request when the mock API is set to fail.

# Start the mock API server (from the setup step)
# Ensure it's running and accessible at $MOCK_API_URL

# Now, send a request that you know will time out
curl -X POST http://localhost:8080/sensor-data \
  -H "Content-Type: application/json" \
  -d '{"sensor_id": "temp_001"}' \
  --connect-timeout 5 \
  -v

You will notice the curl command hangs and eventually times out, and the pipeline logs will show an error. The pipeline is blocked, unable to process other requests.

3. Add the Circuit Breaker

To fix this, you'll wrap the http processor in a try block and add a catch block to handle failures gracefully.

1. Copy the Foundation:

   cp examples/data-routing/circuit-breakers-foundation.yaml http-circuit-breaker.yaml

2. Modify the Pipeline: Open http-circuit-breaker.yaml and replace the entire pipeline section with the one below. The key changes are the try and catch blocks.

   Insert this into http-circuit-breaker.yaml

   
   pipeline:
   processors: # Wrap the fallible processors in a 'try' block. - try: # This is the same HTTP call from the foundation, # but now with retries and a timeout defined. - http:
   url: $\{MOCK_API_URL\}/metadata/$\{!this.sensor_id\}
   verb: GET
   timeout: 3s
   retries: 2
   retry_period: 1s
   headers:
   Authorization: Bearer $\{!env("API_TOKEN")\}
   
       # This mapping runs after a successful HTTP call.
       - mapping: |
           root = this
           let api_response = content().parse_json()
           root.metadata = api_response.metadata
           root.enriched = true
           root.api_status = "success"
   
     # The 'catch' block executes when a processor in the 'try' block fails.
     catch:
       - mapping: |
           root = this
           root.enriched = false
           root.api_status = "failed"
           root.fallback_reason = "api_circuit_breaker_open"

4. Deploy and Test Again

Deploy the new, resilient pipeline and run the same test that failed before.

# Send the same request that timed out before
curl -X POST http://localhost:8080/sensor-data \
  -H "Content-Type: application/json" \
  -d '{"sensor_id": "temp_001"}'

This time, the command will return immediately. If you check the output, you will see a JSON object with enriched: false and api_status: "failed". The pipeline didn't crash; it failed gracefully and produced a fallback message.

You have now implemented a basic circuit breaker.