docker compose example with Kraft and external consumer

[Vano Beridze]

Hello,

I've got Kafka (with Kraft) and Debezium running inside docker using docker compose.

I see that events are published via Debezium to the desired topic and I can consume them with Kafka console consumer from inside the docker machine.

But when I try to consume the same messages from my quarkus application which is running in dev mode on my host, it connects to the broker but messages do not arrive.

After some research I suspect it's related to setting proper env vars for Kafka to define different listeners but I was not able to succeed.

Can someone provide docker-compose with just 2 services: Kafka (with Kraft) and Debezium (Kafka Connect) which solves the problem?

Kind regards,

Vano

[jiri.p...@gmail.com]

Hi,

if you don't see messages arriving then IMHO the problem is that you are not connected on the app side. Could you please share log both from the broker and app?

J.

[Vano Beridze]

I finally made it working. posting the docker-compose.yml which worked for me.

version: '3'
 
services:
  kafka:
    image: quay.io/debezium/kafka:latest
    ports:
      - 29092:29092
    environment:
      - CLUSTER_ID=LRx92c9yQ+ws786HYosuBn
      - BROKER_ID=1
      - KAFKA_CONTROLLER_QUORUM_VOTERS=1@kafka:9093
      - KAFKA_LISTENERS=INTERNAL://:9092,EXTERNAL://:29092,CONTROLLER://:9093
      - KAFKA_ADVERTISED_LISTENERS=INTERNAL://kafka:9092,EXTERNAL://localhost:29092
      - KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT,CONTROLLER:PLAINTEXT
      - KAFKA_INTER_BROKER_LISTENER_NAME=INTERNAL
      - KAFKA_CONTROLLER_LISTENER_NAMES=CONTROLLER
  connect:
    image: quay.io/debezium/connect:latest
    ports:
      - 28083:8083
    environment:
      - BOOTSTRAP_SERVERS=kafka:9092
      - GROUP_ID=1
      - CONFIG_STORAGE_TOPIC=my_connect_configs
      - OFFSET_STORAGE_TOPIC=my_connect_offsets
      - STATUS_STORAGE_TOPIC=my_connect_statuses