Existing Kafka, Zookeeper install with Docker Schema Registry

[Mark Davis]

We have an environment that already uses Kafka and Zookeeper (this would be a pretty common situation I think for people in companies tied to a standard distro with ambari/other).

I am trying to encourage people to move towards confluent, starting with the schema registry - if possible as a container since most of our ancilliary apps run in containers.

This works with our current kafka version:

docker run --net="host" -e SCHEMA_REGISTRY_KAFKASTORE_CONNECTION_URL=zk1.mycompany.com:2181 confluent/schema-registry

Couple of questions:

1. Is the confluent/schema-registry image an official supported image?
     - don't see tags on it for different versions, etc so it looks kind of experimental - i.e. any way to get 2.0.0 image
     - another chap cgwong seems to have a more recently updated registry - seems liek a fork of your one,  but not keen on using unsolicited containers.

2. Is there a kafka/zookeeper/schema-registry/rest-proxy compatibility matrix anywhere?  Which version of Kafka / ZK do we need to be running to be able to run e.g. 2.0.0 schema registry (albeit the container seems to be a 1.0 ver at the moment)
    - currently we run 0.8.2.2 - does this mean that I need to wait for 0.9.0 to go to 2.0 or..?

Any tips appreciated!

Cheers!

Mark.

[Ewen Cheslack-Postava]

On Tue, Feb 2, 2016 at 4:31 AM, Mark Davis <mark....@gmail.com> wrote:

We have an environment that already uses Kafka and Zookeeper (this would be a pretty common situation I think for people in companies tied to a standard distro with ambari/other).

I am trying to encourage people to move towards confluent, starting with the schema registry - if possible as a container since most of our ancilliary apps run in containers.

This works with our current kafka version:

docker run --net="host" -e SCHEMA_REGISTRY_KAFKASTORE_CONNECTION_URL=zk1.mycompany.com:2181 confluent/schema-registry

Couple of questions:

1. Is the confluent/schema-registry image an official supported image?
     - don't see tags on it for different versions, etc so it looks kind of experimental - i.e. any way to get 2.0.0 image
     - another chap cgwong seems to have a more recently updated registry - seems liek a fork of your one,  but not keen on using unsolicited containers.

These aren't supported at the moment, which is why we haven't published any 2.0 images. We're maintaining them on a best-effort basis right now.

That said, we're keeping track of when people are using them to understand demand -- knowing you're using them is useful info!

 

2. Is there a kafka/zookeeper/schema-registry/rest-proxy compatibility matrix anywhere?  Which version of Kafka / ZK do we need to be running to be able to run e.g. 2.0.0 schema registry (albeit the container seems to be a 1.0 ver at the moment)
    - currently we run 0.8.2.2 - does this mean that I need to wait for 0.9.0 to go to 2.0 or..?

These get a bit complicated because of the dependencies between them, so we provide the following docs to try to help: http://docs.confluent.io/2.0.0/upgrade.html In general, schema registry and REST proxy include *Kafka clients*, which require that the Kafka brokers be updated first. Since REST proxy depends on Schema Registry if you're using Avro messages, you'll need to upgrade Schema Registry after Kafka brokers and before REST proxy instances. I'll defer to the linked upgrade guide, but please ask if anything in the guide is unclear or incomplete and we'll try to clear it up.

-Ewen

 

Any tips appreciated!

Cheers!

Mark.

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platf...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/f21096a9-7eee-46ad-8ad6-711e5a27d441%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Thanks,
Ewen

[Mark Davis]

Thanks Ewen, that's useful.  So basically from that we can't look at 2.x until we upgrade our kafka to .9x, good to know!  

Looking forward to officially supported containers at some stage!  

On Friday, February 5, 2016 at 8:28:51 AM UTC, Ewen Cheslack-Postava wrote:

On Tue, Feb 2, 2016 at 4:31 AM, Mark Davis <> wrote:

We have an environment that already uses Kafka and Zookeeper (this would be a pretty common situation I think for people in companies tied to a standard distro with ambari/other).

I am trying to encourage people to move towards confluent, starting with the schema registry - if possible as a container since most of our ancilliary apps run in containers.

This works with our current kafka version:

docker run --net="host" -e SCHEMA_REGISTRY_KAFKASTORE_CONNECTION_URL=zk1.mycompany.com:2181 confluent/schema-registry

Couple of questions:

1. Is the confluent/schema-registry image an official supported image?
     - don't see tags on it for different versions, etc so it looks kind of experimental - i.e. any way to get 2.0.0 image
     - another chap cgwong seems to have a more recently updated registry - seems liek a fork of your one,  but not keen on using unsolicited containers.

These aren't supported at the moment, which is why we haven't published any 2.0 images. We're maintaining them on a best-effort basis right now.

That said, we're keeping track of when people are using them to understand demand -- knowing you're using them is useful info!

 

2. Is there a kafka/zookeeper/schema-registry/rest-proxy compatibility matrix anywhere?  Which version of Kafka / ZK do we need to be running to be able to run e.g. 2.0.0 schema registry (albeit the container seems to be a 1.0 ver at the moment)
    - currently we run 0.8.2.2 - does this mean that I need to wait for 0.9.0 to go to 2.0 or..?

These get a bit complicated because of the dependencies between them, so we provide the following docs to try to help: http://docs.confluent.io/2.0.0/upgrade.html In general, schema registry and REST proxy include *Kafka clients*, which require that the Kafka brokers be updated first. Since REST proxy depends on Schema Registry if you're using Avro messages, you'll need to upgrade Schema Registry after Kafka brokers and before REST proxy instances. I'll defer to the linked upgrade guide, but please ask if anything in the guide is unclear or incomplete and we'll try to clear it up.

-Ewen

 

Any tips appreciated!

Cheers!

Mark.

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.

To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/f21096a9-7eee-46ad-8ad6-711e5a27d441%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Thanks,
Ewen

[Hisham Mardam-Bey]

I've been using this:

https://hub.docker.com/r/socialorra/confluent-platform/

It's for Confluent 2.0.0 / Java 8.

On Mon, Feb 8, 2016 at 5:15 AM, Mark Davis <mark....@gmail.com> wrote:

Thanks Ewen, that's useful.  So basically from that we can't look at 2.x until we upgrade our kafka to .9x, good to know!  

Looking forward to officially supported containers at some stage!  

On Friday, February 5, 2016 at 8:28:51 AM UTC, Ewen Cheslack-Postava wrote:

On Tue, Feb 2, 2016 at 4:31 AM, Mark Davis <> wrote:

We have an environment that already uses Kafka and Zookeeper (this would be a pretty common situation I think for people in companies tied to a standard distro with ambari/other).

I am trying to encourage people to move towards confluent, starting with the schema registry - if possible as a container since most of our ancilliary apps run in containers.

This works with our current kafka version:

docker run --net="host" -e SCHEMA_REGISTRY_KAFKASTORE_CONNECTION_URL=zk1.mycompany.com:2181 confluent/schema-registry

Couple of questions:

1. Is the confluent/schema-registry image an official supported image?
     - don't see tags on it for different versions, etc so it looks kind of experimental - i.e. any way to get 2.0.0 image
     - another chap cgwong seems to have a more recently updated registry - seems liek a fork of your one,  but not keen on using unsolicited containers.

These aren't supported at the moment, which is why we haven't published any 2.0 images. We're maintaining them on a best-effort basis right now.

That said, we're keeping track of when people are using them to understand demand -- knowing you're using them is useful info!

 

2. Is there a kafka/zookeeper/schema-registry/rest-proxy compatibility matrix anywhere?  Which version of Kafka / ZK do we need to be running to be able to run e.g. 2.0.0 schema registry (albeit the container seems to be a 1.0 ver at the moment)
    - currently we run 0.8.2.2 - does this mean that I need to wait for 0.9.0 to go to 2.0 or..?

These get a bit complicated because of the dependencies between them, so we provide the following docs to try to help: http://docs.confluent.io/2.0.0/upgrade.html In general, schema registry and REST proxy include *Kafka clients*, which require that the Kafka brokers be updated first. Since REST proxy depends on Schema Registry if you're using Avro messages, you'll need to upgrade Schema Registry after Kafka brokers and before REST proxy instances. I'll defer to the linked upgrade guide, but please ask if anything in the guide is unclear or incomplete and we'll try to clear it up.

-Ewen

 

Any tips appreciated!

Cheers!

Mark.

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.

To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platf...@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/f21096a9-7eee-46ad-8ad6-711e5a27d441%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Thanks,
Ewen

--

You received this message because you are subscribed to the Google Groups "Confluent Platform" group.

To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platf...@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/0d4b6a6f-779e-41cf-bff4-d8ce45682515%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Hisham Mardam-Bey
http://hisham.cc/

[Hisham Mardam-Bey]

Jumped the gun on that last message potentially (the container runs Kafka and ZK itself); you might still find it useful though.

[Mark Davis]

Thanks Hisham - I'm really looking for a schema registry than can complement an existing environment - I think I might have to bake my own as the existing one falls over every so often for me.

To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/f21096a9-7eee-46ad-8ad6-711e5a27d441%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Thanks,
Ewen

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.

To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/0d4b6a6f-779e-41cf-bff4-d8ce45682515%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Hisham Mardam-Bey
http://hisham.cc/

[Ben Davison]

Hi Mark,

Here's how we changed the confluent-platform dockerfile image to be Kafka 0.9

FROM debian:8.2

ENV SCALA_VERSION="2.11.7"
ENV JAVA_VERSION="7"
ENV CONFLUENT_MAJOR_VERSION="2.0"

RUN apt-get update && \
    apt-get upgrade -y && \
    apt-get install -y curl openjdk-${JAVA_VERSION}-jre-headless && \
    curl -SL http://packages.confluent.io/deb/${CONFLUENT_MAJOR_VERSION}/archive.key | apt-key add - && \
    echo "deb [arch=amd64] http://packages.confluent.io/deb/${CONFLUENT_MAJOR_VERSION} stable main" >> /etc/apt/sources.list && \
    apt-get update && \
    apt-get install -y confluent-platform-${SCALA_VERSION} && \
    apt-get install -y libjtds-java

If you build the image as another other name then confluent-platform, you will need to update all the other docker files to inherit from your new image.

Also: Hisham, that's awesome, will be really useful for us to use in unittesting and dev work.

Thanks,

Ben

To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platf...@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/f21096a9-7eee-46ad-8ad6-711e5a27d441%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Thanks,
Ewen

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.

To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platf...@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/0d4b6a6f-779e-41cf-bff4-d8ce45682515%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Hisham Mardam-Bey
http://hisham.cc/

--

Hisham Mardam-Bey
http://hisham.cc/

--

You received this message because you are subscribed to the Google Groups "Confluent Platform" group.

To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platf...@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/4f54b99c-3af3-4889-bc2f-c73135b80356%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

This email, including attachments, is private and confidential. If you have received this email in error please notify the sender and delete it from your system. Emails are not secure and may contain viruses. No liability can be accepted for viruses that might be transferred by this email or any attachment. Any unauthorised copying of this message or unauthorised distribution and publication of the information contained herein are prohibited.

7digital Limited. Registered office: 69 Wilson Street, London EC2A 2BB.
Registered in England and Wales. Registered No. 04843573.

[Mark Davis]

Thanks Ben - that's very helpful!

To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/f21096a9-7eee-46ad-8ad6-711e5a27d441%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Thanks,
Ewen

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.

To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/0d4b6a6f-779e-41cf-bff4-d8ce45682515%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Hisham Mardam-Bey
http://hisham.cc/

--

Hisham Mardam-Bey
http://hisham.cc/

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.

To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/4f54b99c-3af3-4889-bc2f-c73135b80356%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[John Conlon]

Hi Ben,

Is there any reason why you kept Java 7 and not used Java 8?

thanks

John

To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/f21096a9-7eee-46ad-8ad6-711e5a27d441%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Thanks,
Ewen

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.

To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/0d4b6a6f-779e-41cf-bff4-d8ce45682515%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Hisham Mardam-Bey
http://hisham.cc/

--

Hisham Mardam-Bey
http://hisham.cc/

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.

To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/4f54b99c-3af3-4889-bc2f-c73135b80356%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Ben Davison]

Hi John,

No reason, it will probably work with Java 8.

Regards,

Ben

To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platf...@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/f21096a9-7eee-46ad-8ad6-711e5a27d441%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Thanks,
Ewen

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.

To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platf...@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/0d4b6a6f-779e-41cf-bff4-d8ce45682515%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Hisham Mardam-Bey
http://hisham.cc/

--

Hisham Mardam-Bey
http://hisham.cc/

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.

To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platf...@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/4f54b99c-3af3-4889-bc2f-c73135b80356%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

This email, including attachments, is private and confidential. If you have received this email in error please notify the sender and delete it from your system. Emails are not secure and may contain viruses. No liability can be accepted for viruses that might be transferred by this email or any attachment. Any unauthorised copying of this message or unauthorised distribution and publication of the information contained herein are prohibited.

7digital Limited. Registered office: 69 Wilson Street, London EC2A 2BB.
Registered in England and Wales. Registered No. 04843573.

--

You received this message because you are subscribed to the Google Groups "Confluent Platform" group.

To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platf...@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/64f1c14b-675d-43fd-8e48-a59aa64e5e4c%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Ben Davison]

I just tried it with dockerhub image java:8 and it worked fine.

[Andy Chambers]

FYI, I use those docker-images currently for development. I like that you can set KAFKA_ADVERTISED_HOST_NAME

which is required for those of us on OSX/docker-machine

After a few weeks of starting up the confluent services manually, deleting files from /tmp/kafka, /tmp/zookeeper

as part of my workflow, I now very much enjoy the workflow below. I tried a few different "distributions" of

the constituent parts (e.g. there's a spotify one floating around, and another one that I can't remember the name

of) but the confluent images seemed to be the most reliable.

docker-compose up -d

run-test-job

docker-compose stop && docker-compose rm

Here's the contents of my docker-compose in-case it helps anyone else round here.

zookeeper:

  image: confluent/zookeeper

  ports:

    - "0.0.0.0:2181:2181"

kafka:

  environment:

    - KAFKA_ADVERTISED_HOST_NAME=192.168.99.100

    - KAFKA_AUTO_CREATE_TOPICS_ENABLE=true

  image: confluent/kafka

  links:

    - zookeeper:zookeeper

  ports:

    - "0.0.0.0:9092:9092"

schema-registry:

  image: confluent/schema-registry

  environment:

    - SCHEMA_REGISTRY_KAFKA_STORE_CONNECTION_URL=zookeeper:2181

    - SCHEMA_REGISTRY_DEBUG=true

    - SCHEMA_REGISTRY_KAFKASTORE_TOPIC_REPLICATION_FACTOR=1

  links:

    - kafka:kafka

    - zookeeper:zookeeper

  ports:

    - "0.0.0.0:8081:8081"

On Friday, 5 February 2016 00:28:51 UTC-8, Ewen Cheslack-Postava wrote:

On Tue, Feb 2, 2016 at 4:31 AM, Mark Davis <mark....@gmail.com> wrote:

We have an environment that already uses Kafka and Zookeeper (this would be a pretty common situation I think for people in companies tied to a standard distro with ambari/other).

I am trying to encourage people to move towards confluent, starting with the schema registry - if possible as a container since most of our ancilliary apps run in containers.

This works with our current kafka version:

docker run --net="host" -e SCHEMA_REGISTRY_KAFKASTORE_CONNECTION_URL=zk1.mycompany.com:2181 confluent/schema-registry

Couple of questions:

1. Is the confluent/schema-registry image an official supported image?
     - don't see tags on it for different versions, etc so it looks kind of experimental - i.e. any way to get 2.0.0 image
     - another chap cgwong seems to have a more recently updated registry - seems liek a fork of your one,  but not keen on using unsolicited containers.

These aren't supported at the moment, which is why we haven't published any 2.0 images. We're maintaining them on a best-effort basis right now.

That said, we're keeping track of when people are using them to understand demand -- knowing you're using them is useful info!

 

2. Is there a kafka/zookeeper/schema-registry/rest-proxy compatibility matrix anywhere?  Which version of Kafka / ZK do we need to be running to be able to run e.g. 2.0.0 schema registry (albeit the container seems to be a 1.0 ver at the moment)
    - currently we run 0.8.2.2 - does this mean that I need to wait for 0.9.0 to go to 2.0 or..?

These get a bit complicated because of the dependencies between them, so we provide the following docs to try to help: http://docs.confluent.io/2.0.0/upgrade.html In general, schema registry and REST proxy include *Kafka clients*, which require that the Kafka brokers be updated first. Since REST proxy depends on Schema Registry if you're using Avro messages, you'll need to upgrade Schema Registry after Kafka brokers and before REST proxy instances. I'll defer to the linked upgrade guide, but please ask if anything in the guide is unclear or incomplete and we'll try to clear it up.

-Ewen

 

Any tips appreciated!

Cheers!

Mark.

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.

To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/f21096a9-7eee-46ad-8ad6-711e5a27d441%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Thanks,
Ewen