Error when starting schema registry in docker container

1,967 views
Skip to first unread message

Eli Jordan

unread,
Apr 17, 2016, 10:19:51 AM4/17/16
to Confluent Platform
Hi All

I'm trying to get the docker container for the confluent schema-registry running, but not exactly as the examples in the repo show. I want to connect to a kafka cluster that I already have running on a VM running mesos. When I start the schema registry natively everything works just fine, however, when I use the latest docker images I get the following error.

metric.reporters = []

kafkastore.connection.url = 192.168.3.15:2181

avro.compatibility.level = backward

debug = false

shutdown.graceful.ms = 1000

response.mediatype.preferred = [application/vnd.schemaregistry.v1+json, application/vnd.schemaregistry+json, application/json]

kafkastore.commit.interval.ms = -1

response.mediatype.default = application/vnd.schemaregistry.v1+json

kafkastore.topic = _schemas

metrics.jmx.prefix = kafka.schema.registry

access.control.allow.origin = 

port = 8081

request.logger.name = io.confluent.rest-utils.requests

metrics.sample.window.ms = 30000

kafkastore.zk.session.timeout.ms = 30000

master.eligibility = true

kafkastore.topic.replication.factor = 3

kafkastore.timeout.ms = 500

host.name = 192.168.204.135

schema.registry.zk.namespace = schema_registry

kafkastore.init.timeout.ms = 60000

metrics.num.samples = 2

 (io.confluent.kafka.schemaregistry.rest.SchemaRegistryConfig:135)

[2016-04-17 14:06:14,499] INFO Initialized the consumer offset to -1 (io.confluent.kafka.schemaregistry.storage.KafkaStoreReaderThread:86)

[2016-04-17 14:06:24,743] ERROR Server died unexpectedly:  (io.confluent.kafka.schemaregistry.rest.SchemaRegistryMain:51)

org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 6000

at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:1120)

at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:147)

at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:122)

at kafka.utils.ZkUtils$.createZkClientAndConnection(ZkUtils.scala:89)

at kafka.utils.ZkUtils$.apply(ZkUtils.scala:71)

at kafka.consumer.ZookeeperConsumerConnector.connectZk(ZookeeperConsumerConnector.scala:181)

at kafka.consumer.ZookeeperConsumerConnector.<init>(ZookeeperConsumerConnector.scala:129)

at kafka.javaapi.consumer.ZookeeperConsumerConnector.<init>(ZookeeperConsumerConnector.scala:66)

at kafka.javaapi.consumer.ZookeeperConsumerConnector.<init>(ZookeeperConsumerConnector.scala:69)

at io.confluent.kafka.schemaregistry.storage.KafkaStoreReaderThread.<init>(KafkaStoreReaderThread.java:93)

at io.confluent.kafka.schemaregistry.storage.KafkaStore.<init>(KafkaStore.java:109)

at io.confluent.kafka.schemaregistry.storage.KafkaSchemaRegistry.<init>(KafkaSchemaRegistry.java:136)

at io.confluent.kafka.schemaregistry.rest.SchemaRegistryRestApplication.setupResources(SchemaRegistryRestApplication.java:53)

at io.confluent.kafka.schemaregistry.rest.SchemaRegistryRestApplication.setupResources(SchemaRegistryRestApplication.java:37)

at io.confluent.rest.Application.createServer(Application.java:109)

at io.confluent.kafka.schemaregistry.rest.SchemaRegistryMain.main(SchemaRegistryMain.java:43)


I was able to get the docker image that is currently on docker hub to work, by using host networking e.g. docker run --net="host" .. 
However, this image is running confluent platform 1.0. The latest Dockerfiles in the repo use confluent platform 2.0.1, so I would really like to get those working in my system. But I'm stumped as to why it works outside a docker container, but not inside with the same configuration.

Any help would be greatly appreciated.

Eli Jordan

unread,
Apr 17, 2016, 10:22:31 AM4/17/16
to Confluent Platform
Note: The docker command I used is 

docker run --net="host" -P -e SCHEMA_REGISTRY_KAFKASTORE_CONNECTION_URL=192.168.3.15:2181 -e SCHEMA_REGISTRY_HOST_NAME=192.168.204.135 confluent/schema-registry:2.0.1

Liquan Pei

unread,
Apr 17, 2016, 1:57:55 PM4/17/16
to confluent...@googlegroups.com
Hi Eli,

Just to confirm, are you using the repo https://github.com/confluentinc/docker-images and try to make it work with Kafka cluster VMs running on Mesos? A dumb suggestion would be increasing the Zookeeper connecting timeout. 

Thanks,
Liquan 

On Sun, Apr 17, 2016 at 7:22 AM, Eli Jordan <elias.k...@gmail.com> wrote:
Note: The docker command I used is 

docker run --net="host" -P -e SCHEMA_REGISTRY_KAFKASTORE_CONNECTION_URL=192.168.3.15:2181 -e SCHEMA_REGISTRY_HOST_NAME=192.168.204.135 confluent/schema-registry:2.0.1

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platf...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.
 To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/47b03235-b736-4479-9033-f98d05c9b447%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.



--
Liquan Pei | Software Engineer | Confluent | +1 413.230.6855
Download Apache Kafka and Confluent Platform: www.confluent.io/download

gerard...@dizzit.com

unread,
Apr 18, 2016, 6:21:25 AM4/18/16
to Confluent Platform
Can you ping the zookeeper from within the container, ti should be, because you use --net="host" but you might want to check out if this is the problem? Another possible issue, is port 8081 free to be used, or is it used by a sr running outside a container? 

On Sunday, April 17, 2016 at 7:57:55 PM UTC+2, Liquan Pei wrote:
Hi Eli,

Just to confirm, are you using the repo https://github.com/confluentinc/docker-images and try to make it work with Kafka cluster VMs running on Mesos? A dumb suggestion would be increasing the Zookeeper connecting timeout. 

Thanks,
Liquan 
On Sun, Apr 17, 2016 at 7:22 AM, Eli Jordan <elias.k...@gmail.com> wrote:
Note: The docker command I used is 

docker run --net="host" -P -e SCHEMA_REGISTRY_KAFKASTORE_CONNECTION_URL=192.168.3.15:2181 -e SCHEMA_REGISTRY_HOST_NAME=192.168.204.135 confluent/schema-registry:2.0.1

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.

Eli Jordan

unread,
Apr 18, 2016, 7:32:16 AM4/18/16
to Confluent Platform

@Liquan Pei yes thats correct. Yes I did try increasing the timeout, but it shouldn't matter since the mesos cluster is running in 2 VMs on my laptop.

@Gerard Yes, I can ping the zookeeper IP from within the container. I just ensured I wasn't running the registry externally, but I think there would be a failure from jetty when trying to bind to the port if that was the problem.

Has anyone else connected the schema registry docker container to a zookeeper/kafka setup that is not running in docker?

Eugene Dvorkin

unread,
Apr 18, 2016, 3:38:28 PM4/18/16
to Confluent Platform
I did not use this particular schema-registry container, I just build my own and had not problem. I think the problem is in resolving zookeeper address inside your container. 
Inside the docekr file you see where configuration are: schema-registry.properties.
It has connection to zookeeper:2181. When docker container starts, it should substitute with your environment variable. I don't think this happen. 
Here what you can do to solve the issue:
Start the container with bash because you can't start it with kafka:
docker run  -d --net="host" -P -e SCHEMA_REGISTRY_KAFKASTORE_CONNECTION_URL=192.168.3.15:2181 -e SCHEMA_REGISTRY_HOST_NAME=192.168.204.135 confluent/schema-registry:2.0.1  bash
login to container:
docker exec -it container_id bash
cd into schema-registry.properties file and check what you have as a connection string to zookeeper
I hope this work and give you some insights.

Eli Jordan

unread,
Apr 20, 2016, 8:49:24 AM4/20/16
to Confluent Platform
Thanks Eugene, but the command you provided doesn't quite work, because the docker image has an ENTRYPOINT defined, so running 'bash' has no effect. Additionally, if I use --entrypoint bash it doesn't look in the environment to update the properties file. Having said that, the schema registry does seem to be picking up the value from the environment variable (via the properties file), because it is printed in the logs (snippet from my first post)

metric.reporters = []

kafkastore.connection.url = 192.168.3.15:2181

avro.compatibility.level = backward

debug = false

shutdown.graceful.ms = 1000


I tried building my own docker image, but hit the same problem. The docker file and script is below. 


The thing that realy confuses me is that when I use the image on docker hub https://hub.docker.com/r/confluent/schema-registry/ it works! But this is an older version of registry


FROM ubuntu:trusty


RUN mkdir -p /opt/confluent

WORKDIR /tmp


RUN apt-get update && \

    apt-get upgrade -y && \

    apt-get install -y wget unzip openjdk-7-jre-headless


RUN wget http://packages.confluent.io/archive/2.0/confluent-2.0.1-2.11.7.zip && \

    unzip confluent-2.0.1-2.11.7.zip && \

    mv confluent-2.0.1/* /opt/confluent


COPY schema-registry.sh /opt/confluent/


EXPOSE 8081

ENTRYPOINT ["/opt/confluent/schema-registry.sh"]


#!/bin/bash


CONFIG_FILE="/opt/confluent/schema-registry.properties"


for var in $(env | grep '^SCHEMA_REGISTRY_' | sort); do

  key=$(echo $var | sed -r 's/SCHEMA_REGISTRY_(.*)=.*/\1/g' | tr A-Z a-z | tr _ .)

  value=$(echo $var | sed -r 's/.*=(.*)/\1/g')

  echo "${key}=${value}" >> ${CONFIG_FILE}

done


cat ${CONFIG_FILE}


/opt/confluent/bin/schema-registry-start ${CONFIG_FILE}

jer...@confluent.io

unread,
Apr 20, 2016, 7:09:27 PM4/20/16
to Confluent Platform
Hi Eli,

We just pushed new docker images today and we also  added a docker compose configuration. This might make things a little easier for you. 


What is your docker configuration for kafka? Which image are you using? 

J

Eli Jordan

unread,
Apr 21, 2016, 12:32:42 AM4/21/16
to Confluent Platform
Hi

So I just tried the version that is currently on docker hub, and I am hitting the same issue. When I was using the previous version from docker hub (version 1 of the confluent platform) this would work.

I am not running zookeeper in a docker container, I have a mesos/marathon/zookeeper installation all running natively on 2 VMs, with the goal being that the schema registry docker container will be deployed on marathon.

Jeremy Custenborder

unread,
Apr 21, 2016, 12:57:23 AM4/21/16
to Confluent Platform
Can you telnet to port 2181 on the ip address you are using for zookeeper? The message you are seeing is the standard timeout message meaning it cannot connect to zookeeper. Are you seeing any connections showing up in the zookeeper logs?

--
You received this message because you are subscribed to a topic in the Google Groups "Confluent Platform" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/confluent-platform/C3FmdsbJtLY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to confluent-platf...@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/5f54ace1-b84d-440a-bed3-aed687add26e%40googlegroups.com.

David Tucker

unread,
Apr 21, 2016, 1:00:13 AM4/21/16
to confluent...@googlegroups.com
Or simply test that zookeeper is ok with
    echo "ruok" | nc <zk_host> 2181

The response should be
   imok

(with no CR, so it might look a bit funny on your terminal)

-- David


--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platf...@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/CAHGXxfY-ePNyu_QSOJ%3DQ_O-YiZz7azXfbgaoe9aVfVTFwjmYhw%40mail.gmail.com.

Eli Jordan

unread,
Apr 21, 2016, 5:17:13 AM4/21/16
to Confluent Platform
Thanks for the pointers!

I was able 'nc' the zookeeper server from inside the container, and it replied with 'imok' so it seems at a low level the connection is working.

I tried starting the confluent server and got the logs below.

The 192.168.3.1 IP address seems fishy, because that is not the IP address of the docker container or the VM docker is running on or the zookeeper server, and I can't ping it from anywhere. Where does this IP come from? 
Although, I just ran the schema-registry natively on my laptop (which works fine), and saw the same IP??

I also checked the version of zookeeper I'm running it is 3.4.6, does confluent require a newer version?

Apr 17 09:28:38 mesos-master zkServer.sh[14618]: 2016-04-17 09:28:38,971 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection from /192.168.3.1:59271
Apr 17 09:28:38 mesos-master zkServer.sh[14618]: 2016-04-17 09:28:38,975 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@868] - Client attempting to establish new session at /192.168.3.1:59271
Apr 17 09:28:38 mesos-master zkServer.sh[14618]: 2016-04-17 09:28:38,976 [myid:] - INFO  [SyncThread:0:ZooKeeperServer@617] - Established session 0x1542305c0ba0e45 with negotiated timeout 30000 for client /192.168.3.1:59271
Apr 17 09:28:49 mesos-master zkServer.sh[14618]: 2016-04-17 09:28:49,282 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection from /192.168.3.1:59273
Apr 17 09:28:49 mesos-master zkServer.sh[14618]: 2016-04-17 09:28:49,285 [myid:] - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@357] - caught end of stream exception
Apr 17 09:28:49 mesos-master zkServer.sh[14618]: EndOfStreamException: Unable to read additional data from client sessionid 0x0, likely client has closed socket
Apr 17 09:28:49 mesos-master zkServer.sh[14618]: at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
Apr 17 09:28:49 mesos-master zkServer.sh[14618]: at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
Apr 17 09:28:49 mesos-master zkServer.sh[14618]: at java.lang.Thread.run(Thread.java:745)
Apr 17 09:28:49 mesos-master zkServer.sh[14618]: 2016-04-17 09:28:49,285 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed socket connection for client /192.168.3.1:59273 (no session established for client)
Apr 17 09:28:49 mesos-master zkServer.sh[14618]: 2016-04-17 09:28:49,730 [myid:] - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@357] - caught end of stream exception
Apr 17 09:28:49 mesos-master zkServer.sh[14618]: EndOfStreamException: Unable to read additional data from client sessionid 0x1542305c0ba0e45, likely client has closed socket
Apr 17 09:28:49 mesos-master zkServer.sh[14618]: at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
Apr 17 09:28:49 mesos-master zkServer.sh[14618]: at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
Apr 17 09:28:49 mesos-master zkServer.sh[14618]: at java.lang.Thread.run(Thread.java:745)
Apr 17 09:28:49 mesos-master zkServer.sh[14618]: 2016-04-17 09:28:49,730 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed socket connection for client /192.168.3.1:59271 which had sessionid 0x1542305c0ba0e45
Apr 17 09:29:20 mesos-master zkServer.sh[14618]: 2016-04-17 09:29:20,001 [myid:] - INFO  [SessionTracker:ZooKeeperServer@347] - Expiring session 0x1542305c0ba0e45, timeout of 30000ms exceeded
Apr 17 09:29:20 mesos-master zkServer.sh[14618]: 2016-04-17 09:29:20,001 [myid:] - INFO  [ProcessThread(sid:0 cport:-1)::PrepRequestProcessor@494] - Processed session termination for sessionid: 0x1542305c0ba0e45
To unsubscribe from this group and all its topics, send an email to confluent-platform+unsub...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.

Jeremy Custenborder

unread,
Apr 22, 2016, 12:49:05 PM4/22/16
to Confluent Platform
I wonder if the 192.168.3.1 could be a second NIC on one of the virtual machines. "/192.168.3.1:59271" is just calling .toString() on the endpoint of the socket that connected. This means that 192.168.3.1 did connect to zookeeper?  3.4.6 is the version that Kafka 0.9x of Confluent Platform 2.0.1 ships with. Is Kafka able to start under this configuration? Would it be possible to try a zookeeper container specific to Kafka?

To unsubscribe from this group and all its topics, send an email to confluent-platf...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platf...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.

--
You received this message because you are subscribed to a topic in the Google Groups "Confluent Platform" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/confluent-platform/C3FmdsbJtLY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to confluent-platf...@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/9e73acd8-90aa-49f0-a19f-fce26f0dfab0%40googlegroups.com.

Eli Jordan

unread,
Apr 27, 2016, 7:58:13 AM4/27/16
to Confluent Platform
Ok, so I just tried this on a linux machine and everything worked fine! It seems there is something broken / not configured correctly for my local boot2docker setup.

Thanks for all the help.
To unsubscribe from this group and all its topics, send an email to confluent-platform+unsub...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.

--
You received this message because you are subscribed to a topic in the Google Groups "Confluent Platform" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/confluent-platform/C3FmdsbJtLY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to confluent-platform+unsub...@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages