Skip to main content

Setup Environment for Fan-Out Pattern

Before building the multi-destination pipeline, you'll set up cloud service credentials and edge node configuration.

Prerequisites​

This example requires the following services to be running:

Before you begin, please ensure these services are set up and running according to their respective guides. Additionally, ensure you have completed the Local Development Setup guide for general environment configuration.

Step 1: Configure Example-Specific Variables​

After setting up the core services, configure fan-out-specific variables:

# Edge Node Configuration
export NODE_ID="edge-node-01"
export NODE_LOCATION="factory-floor-west"

# Optional: Cloud service credentials (for multi-cloud fan-out)
export S3_BUCKET="your-sensor-data-archive"
export AWS_REGION="us-west-2"

# Optional: Elasticsearch Configuration
export ES_USERNAME="your-elasticsearch-username"
export ES_PASSWORD="your-elasticsearch-password"
export ES_ENDPOINT="https://es-1.your-cluster.com:9200"

# Verify configuration
echo "Node ID: $NODE_ID"
echo "Kafka: $KAFKA_BROKERS"
echo "S3 Bucket: ${S3_BUCKET:-not set}"

Security Note: Never commit credentials to version control. Use your orchestrator's secret management for production deployments.

Step 2: Verify Optional Cloud Services​

If using optional cloud services, verify connectivity:

# Test S3 connectivity (if configured)
if [ -n "$S3_BUCKET" ]; then
  aws s3 ls s3://$S3_BUCKET/ --region $AWS_REGION
fi

# Test Elasticsearch connectivity (if configured)
if [ -n "$ES_ENDPOINT" ]; then
  curl -X GET "$ES_ENDPOINT/_cluster/health" \
    -u "$ES_USERNAME:$ES_PASSWORD" \
    -H "Content-Type: application/json"
fi

Step 3: Create Example-Specific Topics​

Create Kafka topics specific to this example (core topics already exist from local dev setup):

Create example-specific Kafka topic
# Create sensor-events topic for fan-out example
kafka-topics.sh --bootstrap-server $KAFKA_BROKERS \
  --topic sensor-events \
  --create \
  --partitions 3 \
  --replication-factor 1 \
  --if-not-exists

# Verify topic creation
kafka-topics.sh --bootstrap-server $KAFKA_BROKERS \
  --topic sensor-events \
  --describe

Create Elasticsearch index template (if using Elasticsearch):

Create Elasticsearch index template
# Create index template for daily sensor event indices
curl -X PUT "$ES_ENDPOINT_1/_index_template/sensor-events" \
  -u "$ES_USERNAME:$ES_PASSWORD" \
  -H "Content-Type: application/json" \
  -d '{
    "index_patterns": ["sensor-events-*"],
    "template": {
      "settings": {
        "number_of_shards": 3,
        "number_of_replicas": 1,
        "index.codec": "best_compression"
      },
      "mappings": {
        "properties": {
          "event_id": {"type": "keyword"},
          "sensor_id": {"type": "keyword"},
          "timestamp": {"type": "date"},
          "processing_timestamp": {"type": "date"},
          "temperature": {"type": "float"},
          "humidity": {"type": "float"},
          "edge_node_id": {"type": "keyword"},
          "edge_location": {"type": "keyword"},
          "device_type": {"type": "keyword"},
          "location": {
            "properties": {
              "building": {"type": "keyword"},
              "floor": {"type": "integer"},
              "zone": {"type": "keyword"}
            }
          }
        }
      }
    }
  }'

# Verify template creation
curl -X GET "$ES_ENDPOINT_1/_index_template/sensor-events" \
  -u "$ES_USERNAME:$ES_PASSWORD"

Next Steps​

✅ Setup Complete! Your environment is now configured for multi-destination fan-out pattern.

Ready to add destinations? Continue with the step-by-step tutorial:

Next: Configure the broker foundation for fan-out routing