Need help setting up a simple Docker Compose (v2 or v3) for Docker, ZooKeeper, and Connect. All work but the Connect service.

290 views
Skip to first unread message

Behrang Saeedzadeh

unread,
Dec 13, 2021, 6:32:12 PM12/13/21
to debezium

I can create a compose file that contains both the zookeeper and connect services and it works fine:

version: "2"

services:
  zookeeper:
    image: bitnami/zookeeper:3.5.7
    container_name: zookeeper
    ports:
      - "2181:2181"
      - "2888:2888"
      - "3888:3888"
    environment:
      - ALLOW_ANONYMOUS_LOGIN=yes

  kafka:
    image: bitnami/kafka:2.5.0
    container_name: kafka
    depends_on:
      - zookeeper
    ports:
      - "9092:9092"
    environment:
      - KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181
      - ALLOW_PLAINTEXT_LISTENER=yes
      - KAFKA_CFG_LISTENERS=PLAINTEXT://:9092
      - KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://localhost:9092
    links:
      - zookeeper

From my host machine I can create a topic and publish and consume records from it:

$ bin/kafka-topics.sh --create --partitions 1 \
   --replication-factor 1 \
   --topic quickstart-events \
   --bootstrap-server localhost:9092
Created topic quickstart-events.

$ bin/kafka-console-producer.sh \
   --topic quickstart-events \
   --bootstrap-server localhost:9092
>1
>2
>3
>^C

$ ./bin/kafka-console-consumer.sh \
   -topic quickstart-events \
   --from-beginning --bootstrap-server localhost:9092
1
2
3
^CProcessed a total of 3 messages

However for some reason when I add a connect service to my Compose, it doesn't work and shows a warning message in a loop. Here's the updated compose file (new additions in bold/italic):

version: "2"

services:
  zookeeper:
    image: bitnami/zookeeper:3.5.7
    container_name: zookeeper
    ports:
      - "2181:2181"
      - "2888:2888"
      - "3888:3888"
    environment:
      - ALLOW_ANONYMOUS_LOGIN=yes

  kafka:
    image: bitnami/kafka:2.5.0
    container_name: kafka
    depends_on:
      - zookeeper
    ports:
      - "9092:9092"
    environment:
      - KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181
      - ALLOW_PLAINTEXT_LISTENER=yes
      - KAFKA_CFG_LISTENERS=PLAINTEXT://:9092
      - KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://localhost:9092
    links:
      - zookeeper

  connect:
    image: debezium/connect:1.7.1.Final
    container_name: connect
    depends_on:
      - zookeeper
      - kafka
    ports:
      - "8093:8083"
    environment:
      - BOOTSTRAP_SERVERS=kafka:9092
      - CONFIG_STORAGE_TOPIC=kafka_connect_configs
      - OFFSET_STORAGE_TOPIC=kafka_connect_offsets
    links:
      - zookeeper
      - kafka

And here are some relevant logs from the connect container:

connect | 2021-12-13 23:09:19,644 INFO || AdminClientConfig values:
connect | bootstrap.servers = [kafka:9092]
connect | client.dns.lookup = use_all_dns_ips
connect | client.id =
connect | connections.max.idle.ms = 300000
connect | default.api.timeout.ms = 60000
connect | metadata.max.age.ms = 300000
connect | metric.reporters = []
connect | metrics.num.samples = 2
connect | metrics.recording.level = INFO
connect | metrics.sample.window.ms = 30000
connect | receive.buffer.bytes = 65536
connect | reconnect.backoff.max.ms = 1000
connect | reconnect.backoff.ms = 50
connect | request.timeout.ms = 30000
connect | retries = 2147483647
connect | retry.backoff.ms = 100
connect | sasl.client.callback.handler.class = null
connect | sasl.jaas.config = null
connect | sasl.kerberos.kinit.cmd = /usr/bin/kinit
connect | sasl.kerberos.min.time.before.relogin = 60000
connect | sasl.kerberos.service.name = null
connect | sasl.kerberos.ticket.renew.jitter = 0.05
connect | sasl.kerberos.ticket.renew.window.factor = 0.8
connect | sasl.login.callback.handler.class = null
connect | sasl.login.class = null
connect | sasl.login.refresh.buffer.seconds = 300
connect | sasl.login.refresh.min.period.seconds = 60
connect | sasl.login.refresh.window.factor = 0.8
connect | sasl.login.refresh.window.jitter = 0.05
connect | sasl.mechanism = GSSAPI
connect | security.protocol = PLAINTEXT
connect | security.providers = null
connect | send.buffer.bytes = 131072
connect | socket.connection.setup.tim... = 30000
connect | socket.connection.setup.tim... = 10000
connect | ssl.cipher.suites = null
connect | ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
connect | ssl.endpoint.identification.algorithm = https
connect | ssl.engine.factory.class = null
connect | ssl.key.password = null
connect | ssl.keymanager.algorithm = SunX509
connect | ssl.keystore.certificate.chain = null
connect | ssl.keystore.key = null
connect | ssl.keystore.location = null
connect | ssl.keystore.password = null
connect | ssl.keystore.type = JKS
connect | ssl.protocol = TLSv1.3
connect | ssl.provider = null
connect | ssl.secure.random.implementation = null
connect | ssl.trustmanager.algorithm = PKIX
connect | ssl.truststore.certificates = null
connect | ssl.truststore.location = null
connect | ssl.truststore.password = null
connect | ssl.truststore.type = JKS
connect | [org.apache.kafka.clients.admin.AdminClientConfig]

...
connect | 2021-12-13 23:09:19,682 WARN || The configuration 'config.storage.topic' was supplied but isn't a known config. [org.apache.kafka.clients.admin.AdminClientConfig]
connect | 2021-12-13 23:09:19,682 WARN || The configuration 'rest.advertised.host.name' was supplied but isn't a known config. [org.apache.kafka.clients.admin.AdminClientConfig]
connect | 2021-12-13 23:09:19,682 WARN || The configuration 'status.storage.topic' was supplied but isn't a known config. [org.apache.kafka.clients.admin.AdminClientConfig]
connect | 2021-12-13 23:09:19,682 WARN || The configuration 'group.id' was supplied but isn't a known config. [org.apache.kafka.clients.admin.AdminClientConfig]
connect | 2021-12-13 23:09:19,682 WARN || The configuration 'rest.host.name' was supplied but isn't a known config. [org.apache.kafka.clients.admin.AdminClientConfig]
connect | 2021-12-13 23:09:19,682 WARN || The configuration 'rest.advertised.port' was supplied but isn't a known config. [org.apache.kafka.clients.admin.AdminClientConfig]
connect | 2021-12-13 23:09:19,682 WARN || The configuration 'task.shutdown.graceful.time...' was supplied but isn't a known config. [org.apache.kafka.clients.admin.AdminClientConfig]
connect | 2021-12-13 23:09:19,682 WARN || The configuration 'plugin.path' was supplied but isn't a known config. [org.apache.kafka.clients.admin.AdminClientConfig]
connect | 2021-12-13 23:09:19,682 WARN || The configuration 'offset.flush.timeout.ms' was supplied but isn't a known config. [org.apache.kafka.clients.admin.AdminClientConfig]
connect | 2021-12-13 23:09:19,682 WARN || The configuration 'config.storage.replication.factor' was supplied but isn't a known config. [org.apache.kafka.clients.admin.AdminClientConfig]
connect | 2021-12-13 23:09:19,682 WARN || The configuration 'offset.flush.interval.ms' was supplied but isn't a known config. [org.apache.kafka.clients.admin.AdminClientConfig]
connect | 2021-12-13 23:09:19,682 WARN || The configuration 'rest.port' was supplied but isn't a known config. [org.apache.kafka.clients.admin.AdminClientConfig]
connect | 2021-12-13 23:09:19,682 WARN || The configuration 'key.converter.schemas.enable' was supplied but isn't a known config. [org.apache.kafka.clients.admin.AdminClientConfig]
connect | 2021-12-13 23:09:19,682 WARN || The configuration 'internal.key.converter' was supplied but isn't a known config. [org.apache.kafka.clients.admin.AdminClientConfig]
connect | 2021-12-13 23:09:19,682 WARN || The configuration 'status.storage.replication.factor' was supplied but isn't a known config. [org.apache.kafka.clients.admin.AdminClientConfig]
connect | 2021-12-13 23:09:19,683 WARN || The configuration 'value.converter.schemas.enable' was supplied but isn't a known config. [org.apache.kafka.clients.admin.AdminClientConfig]
connect | 2021-12-13 23:09:19,683 WARN || The configuration 'internal.value.converter' was supplied but isn't a known config. [org.apache.kafka.clients.admin.AdminClientConfig]
connect | 2021-12-13 23:09:19,683 WARN || The configuration 'offset.storage.replication.factor' was supplied but isn't a known config. [org.apache.kafka.clients.admin.AdminClientConfig]
connect | 2021-12-13 23:09:19,683 WARN || The configuration 'offset.storage.topic' was supplied but isn't a known config. [org.apache.kafka.clients.admin.AdminClientConfig]
connect | 2021-12-13 23:09:19,683 WARN || The configuration 'value.converter' was supplied but isn't a known config. [org.apache.kafka.clients.admin.AdminClientConfig]
connect | 2021-12-13 23:09:19,683 WARN || The configuration 'key.converter' was supplied but isn't a known config. [org.apache.kafka.clients.admin.AdminClientConfig]
...

connect | 2021-12-13 23:09:19,683 INFO || Kafka version: 2.8.1 [org.apache.kafka.common.utils.AppInfoParser]
connect | 2021-12-13 23:09:19,683 INFO || Kafka commitId: 839b886f9b732b15 [org.apache.kafka.common.utils.AppInfoParser]
connect | 2021-12-13 23:09:19,683 INFO || Kafka startTimeMs: 1639436959683 [org.apache.kafka.common.utils.AppInfoParser]
connect | 2021-12-13 23:09:19,871 WARN || [AdminClient clientId=adminclient-1] Connection to node 1001 (localhost/127.0.0.1:9092) could not be established. Broker may not be available. [org.apache.kafka.clients.NetworkClient]
connect | 2021-12-13 23:09:19,973 WARN || [AdminClient clientId=adminclient-1] Connection to node 1001 (localhost/127.0.0.1:9092) could not be established. Broker may not be available. [org.apache.kafka.clients.NetworkClient]
connect | 2021-12-13 23:09:20,074 WARN || [AdminClient clientId=adminclient-1] Connection to node 1001 (localhost/127.0.0.1:9092) could not be established. Broker may not be available. [org.apache.kafka.clients.NetworkClient]
connect | 2021-12-13 23:09:20,375 WARN || [AdminClient clientId=adminclient-1] Connection to node 1001 (localhost/127.0.0.1:9092) could not be established. Broker may not be available. [org.apache.kafka.clients.NetworkClient]
connect | 2021-12-13 23:09:20,777 WARN || [AdminClient clientId=adminclient-1] Connection to node 1001 (localhost/127.0.0.1:9092) could not be established. Broker may not be available. [org.apache.kafka.clients.NetworkClient]
connect | 2021-12-13 23:09:21,679 WARN || [AdminClient clientId=adminclient-1] Connection to node 1001 (localhost/127.0.0.1:9092) could not be established. Broker may not be available. [org.apache.kafka.clients.NetworkClient]
connect | 2021-12-13 23:09:22,582 WARN || [AdminClient clientId=adminclient-1] Connection to node 1001 (localhost/127.0.0.1:9092) could not be established. Broker may not be available. [org.apache.kafka.clients.NetworkClient]
connect | 2021-12-13 23:09:23,686 WARN || [AdminClient clientId=adminclient-1] Connection to node 1001 (localhost/127.0.0.1:9092) could not be established. Broker may not be available. [org.apache.kafka.clients.NetworkClient]
connect | 2021-12-13 23:09:24,589 WARN || [AdminClient clientId=adminclient-1] Connection to node 1001 (localhost/127.0.0.1:9092) could not be established. Broker may not be available. [org.apache.kafka.clients.NetworkClient]

Any ideas what am I doing wrong and how can I fix these 

connect | 2021-12-13 23:09:24,589 WARN || [AdminClient clientId=adminclient-1] Connection to node 1001 (localhost/127.0.0.1:9092) could not be established. Broker may not be available. [org.apache.kafka.clients.NetworkClient]

errors?

Thanks in advance.

Reply all
Reply to author
Forward
0 new messages