Docker and vert.x Event Bus

[Rudolf Traunmüller]

Hello, 

I'm struggling with a (hopefully) simple deployment scenario: 

I've two fat jars and want to communicate using the vert.x eventbus. 

The following is the exact infrastructure setup I'm using: 

Host running ubuntu 14.04 with docker, IP: 172.20.28.1

• Container A, IP 172.17.0.6
• Container B, IP 172.17.0.12

I've experimented with many settings, such as configuring cluster.xml of hazelcast and setting the clustered public host and port settings of the vert.x event bus but didn't succeed although both containers run on the same host. I've tried approaches mentioned in the blog posts below, but again without success: 

• https://groups.google.com/forum/#!searchin/vertx/docker/vertx/MvKcz_aTaWM/QM6CfllX9HQJ
• https://groups.google.com/forum/#!searchin/vertx/docker/vertx/TBqbsPcIEds/d5dbb6nBBAoJ

For the example described in this post, I deactivated all additional settings and 

• expose or publish no ports with docker
• configure no additional settings for clustered public host and port
• and use the default-cluster.xml without modification delivered with vertx-hazelcast 3.2.0

As can be seen in the logs (containera.logs, containerb.logs) Hazelcast finds the members of the cluster: 

Members [2] {

Member [172.17.0.6]:5701

Member [172.17.0.12]:5701 this

}

Can anybody give me a hint what I have to configure so that not only hazelcast finds it members but also the eventbus communicates correctly? 

If hazelcast can find all members, why is the eventbus struggling? Do I have to expose or publish special ports for the eventbus (in addition to e.g. 5701 for hazelcast) in order to make it work? 

Thanks a lot!

Best,

Rudolf 

[Tim Fox]

Hi Rudolf,

Please step through the trouble shooting clustering guide.

We should really have a FAQ for this as it comes up often:

https://groups.google.com/forum/#!searchin/vertx/troubleshooting$20clustering

--
You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.
Visit this group at https://groups.google.com/group/vertx.
To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/f5a64f20-333f-4c53-bfed-82a56c3b46dd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Kegan Holtzhausen]

Whats in your Cluster.xml 

/Kegan

[Kegan Holtzhausen]

I remember when I played with this I had to add the following docker opts.

DOCKER_OPTS="-H tcp://127.0.0.1:5555 --bip=10.0.0.1/16"

And also had to add -cluster -cluster-host to my java opts specifying appropriate IP's 

Its a bit out of date but see https://github.com/unixunion/vertx-dockermod/tree/master

Though its designed for EC2, I used to test it on Docker. 

/K

On Thursday, February 11, 2016 at 11:46:53 AM UTC+1, Rudolf Traunmüller wrote:

[Rudolf Traunmüller]

I've attached the default-cluster.xml which is included with vertx-hazelcast 3.2.1. 

Best,

Rudolf

[Kegan Holtzhausen]

I think you need to configure that. I see TCPIP is disabled. Also you need to expose the ports in docker.

/K

> On 11 Feb 2016, at 13:23, Rudolf Traunmüller <celum...@gmail.com> wrote:
>
> <default-cluster.xml>

[David Bush]

It's super easy if you use Docker Compose which comes with Docker. Just create a docker-compose.yml file like this:

vertx-cluster:
  hostname: cluster
  image: vertx/vertx3-exec
  command: bare
vertx-inst1:
  build: ./inst1
  hostname: inst1
  links:
    - vertx-cluster:cluster
vertx-inst2:
  build: ./inst2
  hostname: inst2
  links:
    - vertx-cluster:cluster

vertx-cluster is a stand-alone ha instance.

vertx-inst1 is your first far-jar container. The 'build:' command is pointing to the directory where its Dockerfile  is located.

vertx-inst2 is your other one.

Don't change your cluster.xml file at all. The out-of-box one works just fine as all of the instances you mention in the docker-compose file can reach each other through multicast. Just make sure your fat jars are started with --cluster. 

When you look at the logs with 'docker-compose logs' you will see that your two instances are joining the cluster.

[Rudolf Traunmüller]

Hi David, 

thanks for the hint with Docker Compose. I tried it out and receive the same error when accessing the event bus. I'm using the docker compose script below (the images are already on the system so I simply reference them and expose the http ports, otherwise it's the same that you provided). Attached you'll find the log of the cluster. 

vertx-cluster:

  hostname: cluster

  image: vertx/vertx3-exec

  command: bare

vertx-inst1:

  image: sample/image1

  hostname: inst1

  ports:

   - "7777:7777"

  links:

    - vertx-cluster:cluster

vertx-inst2:

  image: sample/image1

  hostname: inst2

  ports:

   - "8888:8888"

  links:

    - vertx-cluster:cluster

The exception from the eventbus

vertx-inst1_1   | /ping

vertx-inst1_1   | /ping

vertx-inst1_1   | (TIMEOUT,-1) Timed out waiting for a reply

vertx-inst1_1   | (TIMEOUT,-1) Timed out waiting for a reply

vertx-inst1_1   |       at io.vertx.core.eventbus.impl.HandlerRegistration.sendAsyncResultFailure(HandlerRegistration.java:117)

vertx-inst1_1   |       at io.vertx.core.eventbus.impl.HandlerRegistration.lambda$new$147(HandlerRegistration.java:64)

vertx-inst1_1   |       at io.vertx.core.impl.VertxImpl$InternalTimerHandler.handle(VertxImpl.java:738)

vertx-inst1_1   |       at io.vertx.core.impl.VertxImpl$InternalTimerHandler.handle(VertxImpl.java:709)

vertx-inst1_1   |       at io.vertx.core.impl.ContextImpl.lambda$wrapTask$18(ContextImpl.java:335)

vertx-inst1_1   |       at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:358)

vertx-inst1_1   |       at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357)

vertx-inst1_1   |       at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112)

vertx-inst1_1   |       at java.lang.Thread.run(Thread.java:745)

The sample is working fine in the development environment and if I start the jars without docker. Do you have any other ideas or a working sample with a fat jar in a docker container which uses the vert.x eventbus? 

Thanks again!

Best,

Rudi

[David Bush]

I see from the log that they've both joined the cluster. So at least that part is working. I'll post a complete working example on GitHub later today and post the link here. Maybe that will help.

[David Bush]

Here's an example of using Docker Compose to cluster two Groovy verticals. There's no need for a customized cluster.xml as Docker will allow multicast networking between the containers mentioned in the compose file.

https://github.com/diabolicallabs/vertx-examples/tree/master/docker-examples/docker-compose-multicast

[Clement Escoffier]

Hi,

It looks cool. Could we use your example in vertx-examples/docker-examples (https://github.com/vert-x3/vertx-examples/tree/master/docker-examples) ? 

Clement

On 14 févr. 2016, at 05:05, David Bush <dbu...@gmail.com> wrote:

Here's an example of using Docker Compose to cluster two Groovy verticals. There's no need for a customized cluster.xml as Docker will allow multicast networking between the containers mentioned in the compose file.

https://github.com/diabolicallabs/vertx-examples/tree/master/docker-examples/docker-compose-multicast

--
You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.
Visit this group at https://groups.google.com/group/vertx.

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/a012e459-43fb-4282-bf93-83ef7e59f589%40googlegroups.com.

[Chris Jones]

Hi Rudolf,

I see this question come up a lot, and what a lot of people miss in the documentation (myself included) is that Event Bus does not use the cluster manager to send event bus messages. I.e. in your example with Hazelcast as the cluster manager, you have the Hazelcast cluster up and communicating properly (so your Cluster Manager is fine); however, the Event bus is failing to communicate with your other docker instances due to one or more of the following:

1. It is attempting to use an incorrect IP address to the other node (i.e. the IP of the private interface on the Docker instance, not the publicly mapped one)
   
2. It is attempting to communicate on a port Docker is not configured to forward (the event bus picks a dynamic port if you don't specify one)

What you need to do is: 

1. Tell Vertx the IP address that the other nodes should use to talk to each instance ( using the -cluster-host [command line]
   , setClusterPublicHost [VertXOptions] or "vertx.cluster.public.host" [System Property] options)
   
2. Tell Vertx explicitly the Port to use for event bus communication and ensure Docker is forwarding traffic for those ports ( using the "vertx.cluster.public.port" [System Property], setClusterPublicPort [VertXOptions] or -cluster-port [command line] options). In the past, I have used 15701 because it is easy to remember (just a '1' in fromt of the Hazelcast ports).

The Event bus only uses the Cluster Manager to manage the IP/Port information of the other Vertx Instances and the registration of the Consumers/Producers. The communications are done independently of the cluster manager, which is why you can have the cluster manager configured properly and communicating, but still have no Event bus communications.

You may not need to do both the steps above if both your containers are running on the same host, but you definitely will once you start running them on separate hosts. 

Note: The above is based on experience with Docker from about 6-8 months ago before it supported Multi-host networking and before Swarm was released. They probably fix a lot of issues that made working across hosts a complete PITA. I have not used Docker-networking or Swarm since they were released in 1.9, but if you are not using Docker-networking or Swarm, the above should help you.

[Farzad Pezeshkpour]

FWIW - perhaps worth checking out weave for creating multicast capable network (no hard wiring) to host a vert.x cluster.

We tried it successfully in a recent spike.

What was neat:
. Creates multicast capable network on environments that don't support multicast
. DNS
. Useful UI and network visualisation

http://weave.works

PS I don't work for weave; this is a personal post; it doesn't imply the views of my employer

[David Bush]

Farzad, This is what I'm doing as well. It also works across hosts. It's a great relief as the tcp configuration is a total pain. There are some other alternate drivers that support multicast but Weave seems to be the most popular as of now.

Clement: I'll make an example of this as well for the example project.

[Rudolf Traunmüller]

Hi David,

thanks for the example. It works perfectly for me. I now tried to create a corresponding example with a Java based verticle in a fat jar which is deployed via Docker (the example is based on the vertx-docker-java-fatjar, see. https://github.com/vert-x3/vertx-examples/tree/master/docker-examples/vertx-docker-java-fatjar). The eventbus is working as well. 

There the fat jar is created using the maven-shade-plugin and invokes the io.vertx.core.Launcher which launches the verticle. 

Where I'm still stuck is when I build a fat jar using spring boot and want to launch the verticle not with the io.vertx.core.Launcher but from the java app using the java based Vertx.clusteredVertx API. Otherwise it's exactly the same setup. I've an example here: https://github.com/rudolftraunmueller-celum/vertx-boot. 

If you want to give it a try run mvn package, put the jar file into inst1 and inst2 and then it's exactly the same setup you described in your example (docker-compose up etc.). 

Thanks again for your help! 

Best,

Rudi

[David Bush]

Did you fix it? I gave your project a try and it works for me.

BTW, you don't need to mention --cluster in the CMD in your Dockerfiles. That is a command understood by the Launcher. Since you're not using the Launcher, it doesn't do anything.

[Rudolf Traunmüller]

Sorry, the setup is maybe a bit confusing. Both instances consume and send messages on the same eventbus address and therefore always receive at least one event (their own), even if the eventbus is not properly working between both instances. 

I use 

String name = ManagementFactory.getRuntimeMXBean().getName();

to identify the instances, which results in <processid>@<hostname> identifiers for the log messages.

If I start the example using docker-compose up I'll receive the following logoutput ...

vertx-inst2_1   | Members [3] {
vertx-inst2_1   |       Member [172.17.0.6]:5701
vertx-inst2_1   |       Member [172.17.0.7]:5701
vertx-inst2_1   |       Member [172.17.0.8]:5701 this
vertx-inst2_1   | }
...
vertx-inst1_1   | Reply: ping from 5@inst1 Mon Feb 29 06:35:24 UTC 2016
vertx-inst2_1   | Reply: ping from 5@inst2 Mon Feb 29 06:35:25 UTC 2016
vertx-inst1_1   | Reply: ping from 5@inst1 Mon Feb 29 06:35:26 UTC 2016
vertx-inst2_1   | Reply: ping from 5@inst2 Mon Feb 29 06:35:27 UTC 2016

... so instance 1 is always receiving its own message (@inst1). The same is true for instance 2. 

As soon as I start the two instances without docker 

java -jar inst1/hello-verticle-fatjar-3.2.1.jar

and

java -jar inst2/hello-verticle-fatjar-3.2.1.jar

the log output changes:

For inst1 (processid 22666) to: 

Reply: ping from 22666@ubuntu1404.sky Mon Feb 29 07:39:21 CET 2016
Reply: ping from 22751@ubuntu1404.sky Mon Feb 29 07:39:22 CET 2016
Reply: ping from 22666@ubuntu1404.sky Mon Feb 29 07:39:23 CET 2016
Reply: ping from 22751@ubuntu1404.sky Mon Feb 29 07:39:24 CET 2016

For inst2 (processid 22751) to: 

Reply: ping from 22751@ubuntu1404.sky Mon Feb 29 07:39:20 CET 2016
Reply: ping from 22666@ubuntu1404.sky Mon Feb 29 07:39:21 CET 2016
Reply: ping from 22751@ubuntu1404.sky Mon Feb 29 07:39:22 CET 2016
Reply: ping from 22666@ubuntu1404.sky Mon Feb 29 07:39:23 CET 2016

So now, without docker, both processes always receive messages from themselves as well as from the other process. 

I use the eventBus().send command so at most one consumer receives the message. To rule out any problems with this constraint I changed the example to eventBus().publish and now the following behaviour can be observed: 

If I start the example using docker-compose up I'll receive the following log output ...

vertx-inst1_1   | ping from 5@inst1
vertx-inst2_1   | ping from 5@inst2
vertx-inst1_1   | ping from 5@inst1
vertx-inst2_1   | ping from 5@inst2

... so again instance 1 is always receiving its own message (@inst1). The same is again true for instance 2. 

Without docker: 

For inst1 (processid 25476): 

ping from 25476@ubuntu1404.sky Mon Feb 29 08:33:02 CET 2016
ping from 25523@ubuntu1404.sky Mon Feb 29 08:33:02 CET 2016
ping from 25476@ubuntu1404.sky Mon Feb 29 08:33:03 CET 2016
ping from 25523@ubuntu1404.sky Mon Feb 29 08:33:03 CET 2016

For inst2 (processid 25523): 

ping from 25523@ubuntu1404.sky Mon Feb 29 08:33:05 CET 2016
ping from 25476@ubuntu1404.sky Mon Feb 29 08:33:06 CET 2016
ping from 25523@ubuntu1404.sky Mon Feb 29 08:33:06 CET 2016
ping from 25476@ubuntu1404.sky Mon Feb 29 08:33:07 CET 2016

In comparison to the previous example with eventBus().send both processes now receive all events. 

Both examples show again that something is missing in the docker example. 

I've committed the example to github so maybe you could update and rerun the example, then it should be clearer whether the docker version is working for you.

Thanks!

Best,

Rudolf

[David Bush]

Oh. Sorry, I missed that.

The problem is that the Vert.x Message Bus is binding to localhost and not the interface that Docker provides. The Hazelcast logs show that it is doing the right thing. However, the Message Bus binds using its own mechanism. It seems that's a constant source of confusion.

Anyway, you can solve the problem by specifying the cluster host to be the address of the Docker interface for the container. If you are using the Launcher, you can specify it on the command line. Since you are not, it will have to be done in code.

I think this will work for you as it should resolve to the Docker network interface:

App.java
String local = InetAddress.getLocalHost().getHostAddress();
VertxOptions options = new VertxOptions();
options.setClusterHost(local);

Vertx.clusteredVertx(options, vertxAsyncResult -> vertxAsyncResult.result().deployVerticle(new HelloVerticle()));

docker-compose.yml

vertx-inst1:
 build: ./inst1
 hostname: inst1
vertx-inst2:
 build: ./inst2
 hostname: inst2

[David Bush]

Actually this is even better but we have to change your HelloVerticle to prove that it works since the host name is the same.

App.java

String dockerIp = InetAddress.getByName("vertx").getHostAddress();

VertxOptions options = new VertxOptions();

options.setClusterHost(dockerIp);

Vertx.clusteredVertx(options, vertxAsyncResult -> vertxAsyncResult.result().deployVerticle(new HelloVerticle()));

docker-compose.yml

vertx-inst1:
 build: ./inst1
 hostname: vertx
vertx-inst2:
 build: ./inst2
 hostname: vertx

HelloVerticle.java

@Override
public void start() throws Exception {
 String name = UUID.randomUUID().toString();
 vertx.setPeriodic(1000, id -> {
 vertx.eventBus().publish("ping", "ping from " + name + " " + new Date());
 });
 vertx.eventBus().consumer("ping", message -> {
 System.out.println(message.body());
 });
}

[Rudolf Traunmüller]

That's it! 

I applied your changes and now both instances receive events from each other. Awesome :-) 

That's the log output: 

vertx-inst1_1 | ping from 66b4818a-bf7d-4257-9024-d49bf8cb9327 Tue Mar 01 06:53:21 UTC 2016
vertx-inst1_1 | ping from 6069ce7b-828c-4837-b796-ab04b93887e0 Tue Mar 01 06:53:21 UTC 2016
vertx-inst2_1 | ping from 6069ce7b-828c-4837-b796-ab04b93887e0 Tue Mar 01 06:53:21 UTC 2016
vertx-inst2_1 | ping from 66b4818a-bf7d-4257-9024-d49bf8cb9327 Tue Mar 01 06:53:22 UTC 2016

I've committed the change to github. 

Thanks a lot for everybody's help! 

Best,

Rudolf

[Harmeet Singh]

Awesome it worked, but I don't understand me running two containers on same machine, then also I need this to work. But at last it worked. Thank you very mush.

[teivah]

Hello,

Very interesting topic, it helped me very much especially your reply Chris.

One question though. I did understand that the Event Bus was not using the cluster manager to send messages. But is it mandatory nonetheless to have defined a cluster manager to send messages on a distributed Event Bus ?

I had exactly the same problem than Rudolf so I tried to use the option setClusterHost and did not work. I have in addition configured a cluster manager using Zookeeper (keeping the setClusterHost option) and then I was able to have my Docker instances communicating across the Event Bus.

Do you confirm?

Thanks

[Erwin de Gier]

Is there a full example of the Docker - Hazelcast - Zookeeper solution available?

[Jochen Mader]

https://github.com/vert-x3/vertx-examples/tree/master/docker-examples

2016-11-27 9:02 GMT+01:00 Erwin de Gier <er...@edegier.nl>:

Is there a full example of the Docker - Hazelcast - Zookeeper solution available?

--
You received this message because you are subscribed to the Google Groups "vert.x" group.

To unsubscribe from this group and stop receiving emails from it, send an email to vertx+unsubscribe@googlegroups.com.

Visit this group at https://groups.google.com/group/vertx.

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/26981443-869d-458c-8dfd-3961c90d5c68%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Jochen Mader | Lead IT Consultant

codecentric AG | Elsenheimerstr. 55a | 80687 München | Deutschland
tel: +49 89 215486633 | fax: +49 89 215486699 | mobil: +49 152 51862390
www.codecentric.de | blog.codecentric.de | www.meettheexperts.de | www.more4fi.de

Sitz der Gesellschaft: Düsseldorf | HRB 63043 | Amtsgericht Düsseldorf
Vorstand: Michael Hochgürtel . Rainer Vehns
Aufsichtsrat: Patric Fedlmeier (Vorsitzender) . Klaus Jäger . Jürgen Schütz