Distributed OrientDB via Multi-host Docker

Esen Sagynov

unread,

Jan 5, 2015, 1:19:42 AM1/5/15

to

Using ODB 2.0-SNAPSHOT (Jan 5, 2015).

My nodes in Docker containers on separate physical machines cannot join each other. The establish a connection however eventually they refuse joining because some conditions aren't met.

The following are logs on Host 2:

2015-01-05 06:07:15:691 INFO  [172.17.0.3]:2434 [orientdb] [3.3] Accepting socket connection from /123.123.123.124:55647 [SocketAcceptor]
2015-01-05 06:07:15:692 INFO  [172.17.0.3]:2434 [orientdb] [3.3] Established socket connection between /172.17.0.3:2434 and /123.123.123.124:55647 [TcpIpConnectionManager]
2015-01-05 06:07:15:694 WARNING [172.17.0.3]:2434 [orientdb] [3.3] Wrong bind request from Address[172.17.0.12]:2434! This node is not requested endpoint: Address[123.123.123.124]:2434 [TcpIpConnectionManager]

Host 1 (123.123.123.123) running ODB in a container 172.17.0.3:2434.

Host 2 (123.123.123.124) running ODB in a container 172.17.0.12:2434.

Host 3 (123.123.123.125) running ODB in a container 172.17.0.10:2434.

Because docker on each host manages its own bridge each ODB container doesn't see it directly. This is why in config/hazelcast.xml I specify the IP of the host machines.

                        <tcp-ip enabled="true">
                                <member> 123.123.123.123:2434</member>
                                <member> 123.123.123.124:2434</member>
                                <member> 123.123.123.125:2434</member>
                        </tcp-ip>

All ODB containers expose all 3 ports (2424, 2480, 2434). I can telnet and ping. However, because each container requests a dynamic docker IP address, each node cannot join the same cluster because host IP doesn't not match the actual Docker IP inside the container as shown in the above error log.

Any solutions for this?

Edit 1:

Just in case, 172.17.0.2 is accessible only on the same host. However, my ODB nodes are running on separate physical machines each of which has its own 172.17.0.2 free or occupied. On each host Docker creates the same networking bridge.

What I don't understand is that when ODB on host B receives a connection from host A on port 2434, why does it reject the connection request just because the host A tried to connect to host B on IP 123.123.123.123 (via the host network) while ODB in host B is running in an isolated/containerized network under IP 172.17.0.2?

Attaching ODB container directly to the host network still generate the same error:

2015-01-05 07:06:03:912 INFO  [172.17.42.1]:2434 [orientdb] [3.3] Accepting socket connection from /123.123.123.124:44369 [SocketAcceptor]
2015-01-05 07:06:03:913 INFO  [172.17.42.1]:2434 [orientdb] [3.3] Established socket connection between /123.123.123.123:2434 and /123.123.123.124:44369 [TcpIpConnectionManager]
2015-01-05 07:06:03:914 WARNING [172.17.42.1]:2434 [orientdb] [3.3] Wrong bind request from Address[172.17.42.1]:2434! This node is not requested endpoint: Address[123.123.123.123]:2434 [TcpIpConnectionManager]
2015-01-05 07:06:03:915 INFO  [172.17.42.1]:2434 [orientdb] [3.3] Connection [/123.123.123.124:44369] lost. Reason: Socket explicitly closed [TcpIpConnection]

Luca Garulli

unread,

Jan 5, 2015, 7:27:31 AM1/5/15

to orient-database

Hi Esen,

Seems OrientDB servers can't see each other. I suggest you to try if connections are allowed between hosts with CURL.

Lvc@

On 5 January 2015 at 07:19, Esen Sagynov <kadi...@gmail.com> wrote:

Using ODB 2.0-SNAPSHOT (Jan 5, 2015).

My nodes in Docker containers on separate physical machines cannot join to each other. The establish a connection however eventually they refuse joining because some conditions aren't met.

The following are logs on Host 2:

2015-01-05 06:07:15:691 INFO [172.17.0.3]:2434 [orientdb] [3.3] Accepting socket connection from /123.123.123.124:55647 [SocketAcceptor] 2015-01-05 06:07:15:692 INFO [172.17.0.3]:2434 [orientdb] [3.3] Established socket connection between /172.17.0.3:2434 and /123.123.123.124:55647 [TcpIpConnectionManager] 2015-01-05 06:07:15:694 WARNING [172.17.0.3]:2434 [orientdb] [3.3] Wrong bind request from Address[172.17.0.12]:2434! This node is not requested endpoint: Address[123.123.123.124]:2434 [TcpIpConnectionManager]

Host 1 (123.123.123.123) running ODB in a container 172.17.0.3:2434.
Host 2 (123.123.123.124) running ODB in a container 172.17.0.12:2434.
Host 3 (123.123.123.125) running ODB in a container 172.17.0.10:2434.

Because docker on each host manages its own bridge each ODB container doesn't see it directly. This is why in config/hazelcast.xml I specify the IP of the host machines.

<tcp-ip enabled="true">
<member> 123.123.123.123:2434</member>
<member> 123.123.123.124:2434</member>
<member> 123.123.123.125:2434</member>
</tcp-ip>

All ODB containers expose all 3 ports (2424, 2480, 2434). I can telnet and ping. However, because each container requests a dynamic docker IP address, each node cannot join the same cluster because host IP doesn't not match the actual Docker IP inside the container as shown in the above error log.

Any solutions for this?

--

---
You received this message because you are subscribed to the Google Groups "OrientDB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to orient-databa...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Esen Sagynov

unread,

Jan 6, 2015, 2:19:42 AM1/6/15

to orient-...@googlegroups.com

I can successfully telnet to each server from any other server because incoming requests are allowed. In fact, each server successfully receives a connection request, however, there seems to be some kind of a validation. When the host A receives a connection request from host B, host A checks if the requested IP address is host A's IP address. Now here this validation fails according to the above error message.

I wonder if it is necessary to do such validations in ODB?

Tobie Morgan Hitchcock

unread,

Mar 4, 2015, 8:41:02 AM3/4/15

to orient-...@googlegroups.com

Hi Esen,

I've just been through the same problem, but have a solution.

Basically, Hazelcast attaches itself to the private ip address of the docker container, and notifies other nodes of this ip address (meaning that it does not ever match the members). There are two solutions...

1. Use docker run --net=host to run the docker container (works ok, but the container is therefore not running in it's separate network).

2. Pass the private/public IP address of the machine to the docker container so that Hazelcast knows which IP address to bind to.

Add <properties><property name="hazelcast.local.localAddress">IPV4ADDRESSHERE</property></properties> into hazelcast.xml
Add <network><public-address>IPV4ADDRESSHERE</public-address></network>

I've setup a docker container to do exactly this, which can be used to setup a distributed OrientDB running on CoreOS using Fleet (fleetctl). This will work on Vagrant or Amazon EC2/VPC.

The repository is available here: https://github.com/abcum/docker-orientdb.git

Tobie

Luca Garulli

unread,

Mar 4, 2015, 1:24:24 PM3/4/15

to orient-database

Hi Tobie,

How your project compares to https://github.com/orientechnologies/orientdb-docker ?

Lvc@

Tobie Morgan Hitchcock

unread,

Mar 4, 2015, 2:32:36 PM3/4/15

to orient-...@googlegroups.com

Hi Luca,

Docker differences

1. Using `centos:7` not `debian`

2. Using `java-1.7.0-openjdk-headless` instead of `java-1.7.0-openjdk`

3. The OrientDB install tar file (https://abcum-deploy.s3.amazonaws.com/orient/orientdb-community-2.0.4.tar.gz) is an exact copy of the one available at http://www.orientechnologies.com/download/ except that it has had the demo database removed to make it smaller.

4. The OrientDB configs are included in the docker image in the /conf/ folder, and do not need to be included on a shared volume.

5. The OrientDB config and startup script has been simplified slightly so that it runs in the foreground on CoreOS in a distributed setup.

6. Running the docker image will launch a 'ready to go' distributed OrientDB instance using either TCP or AWS node discovery.

7. The OrientDB instance is launched in the foreground (not as a daemon) so that it can be controlled by Fleet on CoreOS

8. Using Hazelcast variables (http://blog.hazelcast.com/2013/08/14/xml-variables/) for putting the docker environment variables into the hazelcast.xml file.

If running OrientDB on a CoreOS cluster in Amazon EC2/VPC you should be able to launch any number of OrientDB instances which connect with each other using the Fleetctl service file, which is included in the repository.

I think those are the main differences off the top of my head!

Tobie

Simon Gemmell

unread,

Mar 5, 2015, 5:42:19 AM3/5/15

to

Here is my set up: orientdb running in a docker container on an AWS server in Frankfurt, and a similar container on AWS in Sydney. IPSEC tunneling between them. My docker file looks like this:

FROM dockerfile/java:oracle-java8

MAINTAINER Simon Gemmell

# Update the default application repository sources list

RUN apt-get update

# Install supervisord

RUN apt-get -y install supervisor

RUN mkdir -p /var/log/supervisor

# Install orientdb

ENV ORIENT_VERSION orientdb-community-2.0.3

ENV ORIENT_TAR ${ORIENT_VERSION}.tar.gz

ENV ORIENT_URL http://www.orientechnologies.com/download.php?file=${ORIENT_TAR}

ENV ROOT /opt

RUN cd ${ROOT} \

&& curl -o ${ORIENT_TAR} ${ORIENT_URL} \

&& tar -xzf ${ORIENT_TAR} \

&& ln -s ${ROOT}/${ORIENT_VERSION} ${ROOT}/orientdb \

&& ln -s ${ROOT}/orientdb/bin/*.sh /usr/local/bin/ \

&& mkdir /usr/local/log

# cleanup

RUN apt-get -y -qq --force-yes clean \

&& rm -rf /opt/downloads/linux /var/lib/apt/lists/* /tmp/* /var/tmp/*

# use supervisord to start orientdb

ADD supervisord.conf /etc/supervisor/conf.d/supervisord.conf

#EXPOSE 2424

EXPOSE 2480

EXPOSE 5701

# Set the user to run OrientDB daemon

USER root

# Default command when starting the container

CMD ["/usr/bin/supervisord", "-c", "/etc/supervisor/conf.d/supervisord.conf"]

I don't expose 2424 as it is "linked" to another container.

Then I run it like this:

#!/bin/bash

sudo docker run --name orientdb -d -v `pwd`/orientdb/config:/opt/orientdb/config -v /mnt/EBS/databases:/opt/orientdb/databases -v /mnt/EBS/backup:/opt/orientdb/backup -v `pwd`/orientdb/log:/var/log -p 2424:2424 -p 2480:2480 -p 5701:5701 tkbt/orientdb

cd orientdb

./updateip.sh

cd ..

updateip.sh does this:

#!/bin/bash

IP=`sudo docker inspect --format '{{ .NetworkSettings.IPAddress }}' orientdb`

sed -i 's/<property name="hazelcast.local.localAddress">.*/<property name="hazelcast.local.localAddress">'$IP'<\/property>/' config/hazelcast.xml

My supervisord:

[supervisord]

nodaemon=true

[program:orientdb]

directory=/opt/orientdb/bin/

command=server.sh -Ddistributed=true -Xmx1024m

autostart=true

autorestart=true

stderr_logfile=/var/log/orientdb.err.log

stdout_logfile=/var/log/orientdb.out.log

; startsecs 5 delays starting orient by 5 secs, which will allow any external scripts to update IPs etc

startsecs = 5

stopwaitsecs = 120

; make sure we send the signals to any process started by the top scripts

stopasgroup=true

My hazelcastconfig:

....

<property name="hazelcast.mancenter.enabled">false</property>

<property name="hazelcast.memcache.enabled">false</property>

<property name="hazelcast.rest.enabled">false</property>

<property name="hazelcast.socket.server.bind.any">false</property>

<property name="hazelcast.socket.client.bind">false</property>

</properties>

<public-address>10.0.1.96</public-address>

<outbound-ports>

<!--

Allowed port range when connecting to other nodes.

0 or * means use system provided port.

-->

</outbound-ports>

<join>

<multicast-group>224.2.2.3</multicast-group>

<multicast-port>54327</multicast-port>

</multicast>

<tcp-ip enabled="true">

</tcp-ip>

</join>

</network>

<executor-service>

<pool-size>16</pool-size>

</executor-service>

...

This all works, but orientdb in distributed mode is horribly broken (https://github.com/orientechnologies/orientdb/issues/3690) so we haven't been able to run it for any version of 2.0 in distributed mode and have it work.

Esen Sagynov

unread,

Mar 6, 2015, 6:00:41 AM3/6/15

to orient-...@googlegroups.com

Awesome! I'm happy to see so many options now. Back when I had this problem, couldn't find much of a solution. Glad that users will now have multiple options where to start from when using ODB with Docker.

@Tobie thank you for your image. Haven't tried yet, but looks solid.

--

---
You received this message because you are subscribed to a topic in the Google Groups "OrientDB" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/orient-database/ehfKcInWZXs/unsubscribe.
To unsubscribe from this group and all its topics, send an email to orient-databa...@googlegroups.com.

Message has been deleted

Simon Gemmell

unread,

Mar 6, 2015, 3:13:00 PM3/6/15

to orient-...@googlegroups.com

@tobie thanks for the heads up about hazelcast variables, didn't know you could do that. And for sharing your setup

Stefan Bergh

unread,

Jul 6, 2017, 1:56:03 PM7/6/17

to OrientDB

Hi,

I know this thread is long dead, but I am trying to do this exact same thing. I was able to access the repo https://github.com/abcum/docker-orientdb.git about a week ago, but it is now 404-ing. Is that code gone for good, or is there any way I could grab it from somewhere?

Thanks in advance,

Stefan Bergh

Luca Garulli

unread,

Jul 6, 2017, 4:38:38 PM7/6/17

to OrientDB

Hi Stefan,

If you want to use OrientDB in a clustered configuration with Docker, you could leverage on Kubernates.

For Kubernates we have a working configuration. Rename the attached file as hazelcast.xml and put in the config directory overwriting the default one. Then you need this extra jar:

<dependency>
      <groupId>com.hazelcast</groupId>
      <artifactId>hazelcast-kubernetes</artifactId>
      <version>${hazelcast-kubernetes-version}</version>
    </dependency>