Cannot produce/consume messages to/from Kafka when using Docker containers

[Nima Esmaili Mokaram]

Hi,

I am trying to use the Confluent solution in docker to run Zookeeper and Kafka on my local windows machine (I am running docker beta version 1.13.0-rc6-beta36 (9696) on Windows 10 Enterprise Anniversary Edition version 1607 build 14393.576) and ultimately on dedicated servers.

Here is the docker-compose.yml file

---
version: '2'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper
    ports:
      - "2181"
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000


  broker1:
    image: confluentinc/cp-kafka
    depends_on:
      - zookeeper
    ports:
      - '9092'
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
      KAFKA_ADVERTISED_LISTENERS: 'PLAINTEXT://broker1:9092'

after docker-compose up, everything seems to be up and running just fine.

and here are the properties that the kafka server was started with (this is what the Kafka image generates when it runs the launch script. I did not modify it)

and I am able to create a topic from the host machine (without going into any container)

However when I go to produce and consume messages using the kafka/zookeeper that I have running, I get the following error:

ERROR Error when sending message to topic foo with key: null, value: 16 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback) org.apache.kafka.common.errors.TimeoutException: Expiring 1 record(s) for foo-0 due to 2303 ms has passed since batch creation plus linger time

and nothing happens to my running consumer.

Now, if I go into the container and do the same thing (except with the local port number) everything works beautifully.

what am I missing/doing wrong that I am not able to produce and consume messages from outside of the container?

Thanks

Nima

[Nima Esmaili Mokaram]

The solution to my problem was the KAFKA_ADVERTISED_LISTENERS.

When I set the KAFKA_ADVERTISED_LISTENERS to be the IP address of the host that's running docker.

To get that, I ran ipconfig /all and tried different IPv4 addresses until one worked. The one that worked was the adapter for my Hyper-V Virtual Ethernet.