Howto: running a multi-module Vert.x application distributed over a multi-node docker environment

sANTo L

unread,

Sep 23, 2014, 7:46:07 AM9/23/14

to ve...@googlegroups.com

Those are the steps I took to get my multi-module vert.x app running distributed over a multi-node docker environment

- Description:

My app consists of two modules: a main module which is responsible for all the backend stuff and a web module which serves static files and forwards some requests (e.g. authentication) to the main module over the eventbus.

The intention is to get the main module running inside a docker container on one host and the web module inside another container on another host, both communicating with each other over a clustered eventbus.

Both modules are maven projects developed in Netbeans.

- Before we start:

Note 1: This topic is about how I set up my application in docker, not how you should write such an application

Note 2: my app will be located in /opt/app and vert.x will be loace in /opt/vertx

Note 3: obviously you should adjust the settings below to match you own environment

Disclaimer: I am by no means a docker expert (i only started using it about a week ago) so there might be better ways to do this.

Credits: Thanks to Nick Scavelli for providing a lot of useful information

- Here we go:

[1] Create a script named "launch.sh" in the root of your main module repository which will be used to launch your vert.x app inside the container:

#!/bin/sh


# get private ip address of container
local_ip=$(ifconfig eth0 | awk '/inet addr:/ {print $2}'|sed 's/addr://')
# set local cluster port (port used in the container to communicate with the eventbus)
local_cluster_port=4000

# launch the module
cd /opt/app
vertx runmod com.example~main~1.0-SNAPSHOT -cp . -cluster -cluster-host $local_ip -cluster-port $local_cluster_port

[2] do the same for the web module, but replace "com.example~main~1.0-SNAPSHOT" with the name of the web module, e.g. com.example~web~1.0-SNAPSHOT

[3] Create a file named "Dockerfile" in the root of your main module. This will be used to create our docker image off of the module code:

FROM ubuntu:14.04

MAINTAINER Santo   s_a_n_t_o_._p_r_i_v_e_(at)_g_m_a_i_l_._c_o_m

###
### Install Oracle java
###

# Enable PPA's
RUN sudo apt-get -y install software-properties-common

# Add PPA for Oracle Java (v8)
RUN sudo add-apt-repository ppa:webupd8team/java

# Update package index
RUN sudo apt-get update

# Automatically accept license for installations of java
RUN sudo echo oracle-java8-installer shared/accepted-oracle-license-v1-1 select true | sudo /usr/bin/debconf-set-selections

# Install Oracle Java (v8)
RUN sudo apt-get -y install oracle-java8-installer

# Optional: Check java version
RUN java -version

###
### Install Vert.x
###

# Download Vert.x package (currently v2.1.2)
RUN sudo wget http://dl.bintray.com/vertx/downloads/vert.x-2.1.2.tar.gz -P /tmp

# Extract Vert.x package
RUN cd /opt
RUN sudo tar -zxvf /tmp/vert.x-2*.tar.gz -C /opt

# Create symlink to this vert.x version
RUN cd /opt
RUN sudo ln -s /opt/vert.x-2.1.2 /opt/vertx

# Add vert.x to system classpath
ENV PATH $PATH:/opt/vert.x-2.1.2/bin

# Increase log level of hazelcast, so we can see some useful information
RUN sudo sed -i "s/\(com\.hazelcast\.level=\).*\$/\1INFO/" /opt/vertx/conf/logging.properties

# Optional: Check vert.x version
RUN vertx version

###
### Install our vert.x module
###

# Add the application
ADD target/mods/com.example~main~1.0-SNAPSHOT/. /opt/app

# Add the application launcher
ADD launch.sh /opt/app/
RUN chmod +x /opt/app/launch.sh

# Expose application ports (only required for the web module)
EXPOSE 8888

# Run the application
CMD ["/opt/app/launch.sh"]

[4] do the same for the web module, but replace "com.example~main~1.0-SNAPSHOT" with the name of the web module, e.g. com.example~web~1.0-SNAPSHOT

[5] create docker image for the main module:

cd <path-to-main-module-repository>
sudo docker build -t main-module --rm .

[6] do the same for the web module:

cd <path-to-web-module-repository>
sudo docker build -t web-module --rm .

[7] run a docker container for our main module and start a shell:

sudo docker run -t -i --rm --name main -h main -P -p 5701:5701 -p 4001:4000 web-module /bin/bash

### we are now inside our container, let's update some final settings for this container

[8] add the following lines to the vertx script (/opt/vertx/bin/vertx) to get the eventbus working:

## The public host is the public ip address of the host where the container is running
## The public port is the "host port" that maps to the container port as specified via the docker run command (-p <hostport>:<containerport>, e.g. -p 4001:4000)
VERTX_OPTS="-Dvertx.cluster.public.host=192.168.1.205 -Dvertx.cluster.public.port=4001"

[9] change the cluster.xml file to get hazelcast working (/opt/vertx/conf/cluster.xml)

[9.1] update the <properties> section like this:

    <properties>
        <property name="hazelcast.mancenter.enabled">false</property>
        <property name="hazelcast.memcache.enabled">false</property>
        <property name="hazelcast.rest.enabled">false</property>
        <property name="hazelcast.wait.seconds.before.join">0</property>
        <property name="hazelcast.logging.type">jdk</property>
 <!-- CHANGE: add next 3 lines, first line is the PRIVATE ip address of the container -->
        <property name="hazelcast.local.localAddress">172.17.0.3</property>
        <property name="hazelcast.socket.server.bind.any">false</property>
        <property name="hazelcast.socket.client.bind">false</property>
    </properties>

[9.2] update the <network> section like this:

 <!-- CHANGE: This is the public ip address of the host: -->
        <public-address>192.168.1.205</public-address>
 <!-- CHANGE: This port should be mapped on the docker level, e.g. -p 5701:5701: -->
        <port auto-increment="false" port-count="100">5701</port>
        <outbound-ports>
            <!--
            Allowed port range when connecting to other nodes.
            0 or * means use system provided port.
            -->
            <ports>0</ports>
        </outbound-ports>
        <join>
    <!-- CHANGE: disable multicast -->
            <multicast enabled="false">
                <multicast-group>224.2.2.3</multicast-group>
                <multicast-port>54327</multicast-port>
            </multicast>
            <!-- CHANGE: enable tcp-ip and specify the public ip addresses of all the hosts that should communicate -->
            <tcp-ip enabled="true">
                <interface>192.168.1.49</interface>
                <interface>192.168.1.205</interface>
            </tcp-ip>
[...snip...]

[10] launch the main module inside the container:

/opt/app/launch.sh

### Go back to the host machine and perform those final steps to start the web container on another host:

[11] transfer the web module to the other host:

sudo docker save -o ./web-module.tar web-module
scp web-module.tar administrator@192.168.1.49:/home/administrator/tmp

[12] ssh into the other host and import the image

ssh administrator@192.168.1.49
cd /home/administrator/tmp
sudo docker -i web-module.tar

[13] run a docker container for our web module and start a shell:

sudo docker run -t -i --rm --name web -h web -P -p 5701:5701 -p 4002:4000 web-module /bin/bash

### we are now inside our container, let's update some final settings for this container

[14] add the following lines to the vertx script (/opt/vertx/bin/vertx) to get the eventbus working:

## The public host is the public ip address of the host where the container is running
## The public port is the "host port" that maps to the container port as specified via the docker run command (-p <hostport>:<containerport>, e.g. -p 4002:4000)
VERTX_OPTS="-Dvertx.cluster.public.host=192.168.1.49 -Dvertx.cluster.public.port=4002"

[15] change the cluster.xml file to get hazelcast working (/opt/vertx/conf/cluster.xml)

[15.1] update the <properties> section like this:

    <properties>
        <property name="hazelcast.mancenter.enabled">false</property>
        <property name="hazelcast.memcache.enabled">false</property>
        <property name="hazelcast.rest.enabled">false</property>
        <property name="hazelcast.wait.seconds.before.join">0</property>
        <property name="hazelcast.logging.type">jdk</property>
 <!-- CHANGE: add next 3 lines, first line is the PRIVATE ip address of the container -->
        <property name="hazelcast.local.localAddress">172.17.0.23</property>
        <property name="hazelcast.socket.server.bind.any">false</property>
        <property name="hazelcast.socket.client.bind">false</property>
    </properties>

[15.2] update the <network> section like this:

 <!-- CHANGE: This is the public ip address of the host: -->
        <public-address>192.168.1.49</public-address>
 <!-- CHANGE: This port should be mapped on the docker level, e.g. -p 5701:5701: -->
        <port auto-increment="false" port-count="100">5701</port>
        <outbound-ports>
            <!--
            Allowed port range when connecting to other nodes.
            0 or * means use system provided port.
            -->
            <ports>0</ports>
        </outbound-ports>
        <join>
    <!-- CHANGE: disable multicast -->
            <multicast enabled="false">
                <multicast-group>224.2.2.3</multicast-group>
                <multicast-port>54327</multicast-port>
            </multicast>
            <!-- CHANGE: enable tcp-ip and specify the public ip addresses of all the hosts that should communicate -->
            <tcp-ip enabled="true">
                <interface>192.168.1.205</interface>
                <interface>192.168.1.49</interface>
            </tcp-ip>
[...snip...]

[16] launch the web module inside the container:

/opt/app/launch.sh

[17] That's it. You should now see that the two containers cluster together. Also any messages sent over the eventbus from the web module to the main module should work.

Hopefully this can be of any help for those who are struggling with getting a vert.x app running inside a multi-node docker environment.

sANTo

Nikolai Raitsev

unread,

Sep 24, 2014, 7:48:00 AM9/24/14

to ve...@googlegroups.com

Hi Santo,

many thanks for sharing it!

Nabeel Ali Memon

unread,

Apr 20, 2017, 11:12:10 AM4/20/17

to vert.x

Many thanks for sharing this config.

I notice you're specifying tcp-ip interfaces in the config. What about dynamic host addresses and what about auto-scale group hosts? I'm talking about scenario in which you have multiple docker containers spread across multiple auto-scaling hosts. Any suggestions?

sANTo L

unread,

Apr 20, 2017, 4:13:29 PM4/20/17

to vert.x

Hi Nabeel,

I'm currently not using any docker containers anymore because I had too much trouble keeping them up and running in my environment.

This had nothing to do with the vert.x clustering, but with the orchestration tools I used which were apparently too buggy.

Anyway, I indeed used tcp-ip interfaces here because multicast was not an option in this environment.

To handle dynamic host addresses I created another shell script that retrieved all known (dynamic) hostnames for the whole environment from a metadata service.

The script then injected those hostnames as members in the tcp-ip section.

Because it's sufficient to have at least one other reachable member defined, this worked like a charm.

When a new host was added (manually or via auto-scaling), that host was able to retrieve all other hostnames from the metadata service, update the cluster.xml file with this information and join the other nodes.

So the key here is to have a metadata service that can provide a list of hosts in the cluster and then have a script for each host that uses this information to update the cluster.xml file

Hope this helps.

regards,

Santo

Nabeel Ali Memon

unread,

Apr 21, 2017, 8:45:59 AM4/21/17

to vert.x

Hi Santo,

I was thinking about a similar registry which could give other member's info that are connected in the cluster.

To keep the tests simple, i'm trying to connect simply two nodes on my AWS VPC (under same security group where all traffic is open) and even they don't seem to work. The remove verticle doesn't respond. In the logs, I see the following membership info:

```

Members [4] {

Member [172.30.1.21]:5701

Member [172.30.1.21]:5702 this

Member [172.30.1.188]:5701

Member [172.30.1.188]:5702

}

```

I'm not sure if the hosts are connecting correctly for vertx cluster to work. Any insights on that?

Following is my hazelcast.xml file fragment:

```

<property name="hazelcast.mancenter.enabled">false</property>

<property name="hazelcast.memcache.enabled">false</property>

<property name="hazelcast.rest.enabled">false</property>

<property name="hazelcast.shutdownhook.enabled">false</property>

<property name="hazelcast.socket.bind.any">false</property>

</properties>

...

<outbound-ports>

</outbound-ports>

...

<tcp-ip enabled="true">

</tcp-ip>

</join>

...

```

Nabeel Ali Memon

unread,

Apr 21, 2017, 10:42:26 AM4/21/17

to vert.x

Ok I change the configuration on both hosts to the following:

<tcp-ip enabled="true">

</tcp-ip>

I have verified that a hazelcast distributed map is replicate on both hosts ( I can set/get the map values on any host). Only when I send a message to a verticle, it seems like the eventbus has no information on the deployed verticle on the other host. Have you seen such a condition?

Nabeel Ali Memon

unread,

Apr 21, 2017, 4:59:29 PM4/21/17

to vert.x

I got it working. All I was missing was to call `setClusterHost(host_ip)` on clusterManager initializer.

BTW, I did this with using Apache Ignite rather than Hazelcast for clustering. Would love to have reviews/experiences if anyone has with Ignite.

sANTo L

unread,

Apr 21, 2017, 6:49:06 PM4/21/17

to vert.x

Hi Nabeel,

Apologies for the late response, but i see you got it working now.

Indeed, when cluster can be established but eventbus communication is not working then it's very likely you didn't specify the cluster host and/or cluster port.
This is by far one of the most common mistakes made when setting up a Vert.x cluster.

Unfortunately I have no experience with Apache Ignite so I'm afraid I can't help you with that.