Local Kafka Setup

This guide provides the configuration to run a local Kafka instance using Docker for Expanso examples.

1. Docker Compose Configuration

Save the following content as kafka.yml:

kafka.yml

services:
  # Kafka - Message streaming platform
  kafka:
    image: bitnami/kafka:3.7.0-debian-11-r0
    hostname: kafka
    container_name: expanso-kafka
    ports:
      - '9092:9092'
    environment:
      KAFKA_CFG_NODE_ID: 0
      KAFKA_CFG_PROCESS_ROLES: 'broker,controller'
      KAFKA_CFG_CONTROLLER_QUORUM_VOTERS: '0@kafka:9093'
      KAFKA_CFG_LISTENERS: 'PLAINTEXT://:9092,CONTROLLER://:9093'
      KAFKA_CFG_ADVERTISED_LISTENERS: 'PLAINTEXT://localhost:9092'
      KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP: 'PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT'
      KAFKA_CFG_INTER_BROKER_LISTENER_NAME: 'PLAINTEXT'
      KAFKA_CFG_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_CFG_AUTO_CREATE_TOPICS_ENABLE: 'true'
      KAFKA_BROKER_ID: 0
      ALLOW_PLAINTEXT_LISTENER: 'yes' # Required by Bitnami
    volumes:
      - kafka-data:/bitnami/kafka
    healthcheck:
      test:
        [
          'CMD-SHELL',
          'kafka-topics.sh --bootstrap-server localhost:9092 --list',
        ]
      interval: 10s
      timeout: 5s
      retries: 5

  # Kafka UI - Web interface for Kafka management
  kafka-ui:
    image: provectuslabs/kafka-ui:latest
    container_name: expanso-kafka-ui
    ports:
      - '8081:8080'
    environment:
      KAFKA_CLUSTERS_0_NAME: local
      KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: kafka:9092
    depends_on:
      kafka:
        condition: service_healthy

volumes:
  kafka-data:

2. Start the Service

Run the following command in the same directory where you saved kafka.yml:

docker compose -f kafka.yml up -d

3. Verify Connectivity

To ensure Kafka is running correctly, you can produce and consume a test message.

First, set the required environment variable:

export KAFKA_BROKERS="localhost:9092"

Now, run the test:

# Produce a test message
echo '{"test": "message"}' | kafka-console-producer.sh \
  --broker-list $KAFKA_BROKERS \
  --topic test-connectivity

# Consume the message
kafka-console-consumer.sh \
  --bootstrap-server $KAFKA_BROKERS \
  --topic test-connectivity \
  --from-beginning \
  --max-messages 1

You should see your test message {"test": "message"} in the output. You can also access the Kafka UI at http://localhost:8081.

4. Manage the Service

# View logs
docker compose -f kafka.yml logs -f

# Stop the service
docker compose -f kafka.yml down

# Stop and remove the data volume
docker compose -f kafka.yml down -v