Skip to main content

Step 1: HTTP Client Circuit Breakers

External APIs can be unreliable. If an API call fails, it can bring your entire pipeline to a halt. In this step, you will protect a fragile pipeline by adding a circuit breaker using a try/catch block.

1. The Fragile Pipeline​

First, take a look at the circuit-breakers-foundation.yaml file in examples/data-routing/. It takes incoming sensor data, calls an API to enrich it with metadata, and prints the result.

It works perfectly when the API is healthy, but has no protection if the API fails.

2. See It Fail​

Deploy the foundation pipeline and send it a request when the mock API is set to fail.

# Start the mock API server (from the setup step)
# Ensure it's running and accessible at $MOCK_API_URL

# Now, send a request that you know will time out
curl -X POST http://localhost:8080/sensor-data \
  -H "Content-Type: application/json" \
  -d '{"sensor_id": "temp_001"}' \
  --connect-timeout 5 \
  -v

You will notice the curl command hangs and eventually times out, and the pipeline logs will show an error. The pipeline is blocked, unable to process other requests.

3. Add the Circuit Breaker​

To fix this, you'll wrap the http processor in a try block and add a catch block to handle failures gracefully.

  1. Copy the Foundation:

    cp examples/data-routing/circuit-breakers-foundation.yaml http-circuit-breaker.yaml

  2. Modify the Pipeline: Open http-circuit-breaker.yaml and replace the entire pipeline section with the one below. The key changes are the try and catch blocks.

pipeline: processors:

Wrap the fallible processors in a 'try' block.

  • try:

    This is the same HTTP call from the foundation,

    but now with retries and a timeout defined.

    • http: url: ${MOCK_API_URL}/metadata/${!this.sensor_id} verb: GET timeout: 3s retries: 2 retry_period: 1s headers: Authorization: Bearer ${!env("API_TOKEN")}

    This mapping only runs if the HTTP call succeeds.

    • mapping: | root = this let api_response = content().parse_json() root.metadata = api_response.metadata root.enriched = true root.api_status = "success"

    The 'catch' block executes ONLY if any processor in the 'try' block fails.

    catch:

    • mapping: | root = this root.enriched = false root.api_status = "failed" root.fallback_reason = "api_circuit_breaker_open"

## 4. Deploy and Test Again

Deploy the new, resilient pipeline and run the same test that failed before.

```bash
# Send the same request that timed out before
curl -X POST http://localhost:8080/sensor-data \
-H "Content-Type: application/json" \
-d '{"sensor_id": "temp_001"}'

This time, the command will return immediately. If you check the output, you will see a JSON object with enriched: false and api_status: "failed". The pipeline didn't crash; it failed gracefully and produced a fallback message.

You have now implemented a basic circuit breaker.