Running a Docker Hazelcast cluster

[Sutanu Dalui]

Hi,

I am trying to run a Hazelcast cluster using Docker containers (same Docker machine and NOT using Docker compose). However having problem in joining.

I am using a custom Dockerfile, extending the base Hazelcast image (for Hazelcast v3.8.2). Have verified that the container runs fine standalone. However I could not make it join. I am using TCP joiner. After going through various posts online, I could finally make the containers join by setting,

<public-address>${broadcast.ip}</public-address>

<property name="hazelcast.local.localAddress">${broadcast.ip}</property>

<interfaces enabled="true">
            <interface>${broadcast.ip}</interface>
</interfaces>

the above properties with the Host IP (which is different from Docker internal IP),

and start the Docker container with

--net=host

switch.

However, now the client is not able to connect anymore. As I understand this is because the ports are now not visible due to he usage of --net=host.

So now I am stuck as in -

• If I do not provide --net=host, Hazelcast does not join

• If I do provide it, then client does not connect.

Can you please help. Let me know if you need any more information.

Thanks,
Sutanu

[baris....@gmail.com]

Hi Sutanu, 

How many host machines do you have VM or physical? As I understand, all containers are on the same machine. is it correct?

Could you please share full config xml and docker run commands?

thanks 

[Sutanu Dalui]

Hi,

I have a single Linux host machine hosting a VM on private cloud. It is running a single Docker machine. So yes, all containers are on the same machine.

Docker run:

sudo docker run -d -p --net=host 5701:5701 --name hazelcast1 \
-e "BROADCAST_ADDRESS=<host_ip>" \
-e "CLUSTER_MEMBERS=<host_ip>" \
-e "LISTEN_PORT=5701" \
-e "CASSANDRA_HOST=<host_ip>" \
-e "CASSANDRA_CLUSTER=dm_wfm" \
-e "MIN_HEAP=256M" \
-e "MAX_HEAP=256M" \
cass_hazelcast:4.1

<?xml version="1.0" encoding="UTF-8"?>
<hazelcast xsi:schemaLocation="http://www.hazelcast.com/schema/config hazelcast-config-3.8.xsd"
           xmlns="http://www.hazelcast.com/schema/config"
           xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
   
    <group>
        <name>${group.name}</name>
        <password>${group.pass}</password>
    </group>
    <management-center enabled="${mancenter.enabled}">${mancenter.url}</management-center>
    <properties>

        <property name="hazelcast.local.localAddress">${broadcast.ip}</property>

    </properties>
    <network>
        <public-address>${broadcast.ip}</public-address>
        <port auto-increment="true" port-count="100">${port}</port>
        <outbound-ports>
            <!--
            Allowed port range when connecting to other nodes.
            0 or * means use system provided port.
            -->
            <ports>0</ports>
        </outbound-ports>
        <join>
            <multicast enabled="false">
                <multicast-group>224.2.2.3</multicast-group>
                <multicast-port>54327</multicast-port>
            </multicast>
            <tcp-ip enabled="true">
                __CLUSTER_MEMBERS__
            </tcp-ip>
            <aws enabled="false">
                <access-key>my-access-key</access-key>
                <secret-key>my-secret-key</secret-key>
                <!--optional, default is us-east-1 -->
                <region>us-west-1</region>
                <!--optional, default is ec2.amazonaws.com. If set, region shouldn't be set as it will override this property -->
                <host-header>ec2.amazonaws.com</host-header>
                <!-- optional, only instances belonging to this group will be discovered, default will try all running instances -->
                <security-group-name>hazelcast-sg</security-group-name>
                <tag-key>type</tag-key>
                <tag-value>hz-nodes</tag-value>
            </aws>
            <discovery-strategies>
            </discovery-strategies>
        </join>

        <interfaces enabled="true">
            <interface>${broadcast.ip}</interface>
        </interfaces>

        <ssl enabled="false"/>
        <socket-interceptor enabled="false"/>
        <symmetric-encryption enabled="false">
            <!--
               encryption algorithm such as
               DES/ECB/PKCS5Padding,
               PBEWithMD5AndDES,
               AES/CBC/PKCS5Padding,
               Blowfish,
               DESede
            -->
            <algorithm>PBEWithMD5AndDES</algorithm>
            <!-- salt value to use when generating the secret key -->
            <salt>thesalt</salt>
            <!-- pass phrase to use when generating the secret key -->
            <password>thepass</password>
            <!-- iteration count to use when generating the secret key -->
            <iteration-count>19</iteration-count>
        </symmetric-encryption>
    </network>
    <partition-group enabled="false"/>
    <executor-service name="default">
        <pool-size>16</pool-size>
        <!--Queue capacity. 0 means Integer.MAX_VALUE.-->
        <queue-capacity>0</queue-capacity>
    </executor-service>
    <queue name="default">
        <!--
            Maximum size of the queue. When a JVM's local queue size reaches the maximum,
            all put/offer operations will get blocked until the queue size
            of the JVM goes down below the maximum.
            Any integer between 0 and Integer.MAX_VALUE. 0 means
            Integer.MAX_VALUE. Default is 0.
        -->
        <max-size>0</max-size>
        <!--
            Number of backups. If 1 is set as the backup-count for example,
            then all entries of the map will be copied to another JVM for
            fail-safety. 0 means no backup.
        -->
        <backup-count>1</backup-count>

        <!--
            Number of async backups. 0 means no backup.
        -->
        <async-backup-count>0</async-backup-count>

        <empty-queue-ttl>-1</empty-queue-ttl>
    </queue>
   
    <map name="default">
        <!--
           Data type that will be used for storing recordMap.
           Possible values:
           BINARY (default): keys and values will be stored as binary data
           OBJECT : values will be stored in their object forms
           NATIVE : values will be stored in non-heap region of JVM
        -->
        <in-memory-format>BINARY</in-memory-format>

        <!--
            Number of backups. If 1 is set as the backup-count for example,
            then all entries of the map will be copied to another JVM for
            fail-safety. 0 means no backup.
        -->
        <backup-count>1</backup-count>
        <!--
            Number of async backups. 0 means no backup.
        -->
        <async-backup-count>0</async-backup-count>
        <!--
            Maximum number of seconds for each entry to stay in the map. Entries that are
            older than <time-to-live-seconds> and not updated for <time-to-live-seconds>
            will get automatically evicted from the map.
            Any integer between 0 and Integer.MAX_VALUE. 0 means infinite. Default is 0.
        -->
        <time-to-live-seconds>0</time-to-live-seconds>
        <!--
            Maximum number of seconds for each entry to stay idle in the map. Entries that are
            idle(not touched) for more than <max-idle-seconds> will get
            automatically evicted from the map. Entry is touched if get, put or containsKey is called.
            Any integer between 0 and Integer.MAX_VALUE. 0 means infinite. Default is 0.
        -->
        <max-idle-seconds>0</max-idle-seconds>
        <!--
            Valid values are:
            NONE (no eviction),
            LRU (Least Recently Used),
            LFU (Least Frequently Used).
            NONE is the default.
        -->
        <eviction-policy>NONE</eviction-policy>
        <!--
            Maximum size of the map. When max size is reached,
            map is evicted based on the policy defined.
            Any integer between 0 and Integer.MAX_VALUE. 0 means
            Integer.MAX_VALUE. Default is 0.
        -->
        <max-size policy="PER_NODE">0</max-size>
        <!--
            `eviction-percentage` property is deprecated and will be ignored when it is set.

            As of version 3.7, eviction mechanism changed.
            It uses a probabilistic algorithm based on sampling. Please see documentation for further details
        -->
        <eviction-percentage>25</eviction-percentage>
        <!--
            `min-eviction-check-millis` property is deprecated  and will be ignored when it is set.

            As of version 3.7, eviction mechanism changed.
            It uses a probabilistic algorithm based on sampling. Please see documentation for further details
        -->
        <min-eviction-check-millis>100</min-eviction-check-millis>
        <!--
            While recovering from split-brain (network partitioning),
            map entries in the small cluster will merge into the bigger cluster
            based on the policy set here. When an entry merge into the
            cluster, there might an existing entry with the same key already.
            Values of these entries might be different for that same key.
            Which value should be set for the key? Conflict is resolved by
            the policy set here. Default policy is PutIfAbsentMapMergePolicy

            There are built-in merge policies such as
            com.hazelcast.map.merge.PassThroughMergePolicy; entry will be overwritten if merging entry exists for the key.
            com.hazelcast.map.merge.PutIfAbsentMapMergePolicy ; entry will be added if the merging entry doesn't exist in the cluster.
            com.hazelcast.map.merge.HigherHitsMapMergePolicy ; entry with the higher hits wins.
            com.hazelcast.map.merge.LatestUpdateMapMergePolicy ; entry with the latest update wins.
        -->
        <merge-policy>com.hazelcast.map.merge.PutIfAbsentMapMergePolicy</merge-policy>

        <!--
           Control caching of de-serialized values. Caching makes query evaluation faster, but it cost memory.
           Possible Values:
                        NEVER: Never cache deserialized object
                        INDEX-ONLY: Caches values only when they are inserted into an index.
                        ALWAYS: Always cache deserialized values.
        -->
        <cache-deserialized-values>INDEX-ONLY</cache-deserialized-values>

    </map>

    <multimap name="default">
        <backup-count>1</backup-count>
        <value-collection-type>SET</value-collection-type>
    </multimap>

    <list name="default">
        <backup-count>1</backup-count>
    </list>

    <set name="default">
        <backup-count>1</backup-count>
    </set>

    <jobtracker name="default">
        <max-thread-size>0</max-thread-size>
        <!-- Queue size 0 means number of partitions * 2 -->
        <queue-size>0</queue-size>
        <retry-count>0</retry-count>
        <chunk-size>1000</chunk-size>
        <communicate-stats>true</communicate-stats>
        <topology-changed-strategy>CANCEL_RUNNING_OPERATION</topology-changed-strategy>
    </jobtracker>

    <semaphore name="default">
        <initial-permits>0</initial-permits>
        <backup-count>1</backup-count>
        <async-backup-count>0</async-backup-count>
    </semaphore>

    <reliable-topic name="default">
        <read-batch-size>10</read-batch-size>
        <topic-overload-policy>BLOCK</topic-overload-policy>
        <statistics-enabled>true</statistics-enabled>
    </reliable-topic>

    <ringbuffer name="default">
        <capacity>10000</capacity>
        <backup-count>1</backup-count>
        <async-backup-count>0</async-backup-count>
        <time-to-live-seconds>30</time-to-live-seconds>
        <in-memory-format>BINARY</in-memory-format>
    </ringbuffer>

    <serialization>
        <portable-version>0</portable-version>
    </serialization>

    <services enable-defaults="true"/>

    <lite-member enabled="false"/>

</hazelcast>

Thanks for your help!

[baris....@gmail.com]

Hi,

Your docker run command is not correct. However, we have an compose example, I believe that it will be easy for you to deploy on a single machine with this example.

https://github.com/hazelcast/hazelcast-code-samples/tree/master/hazelcast-integration/docker-compose

Just replace your custom image and client app hazelcast.yml.

If you need further help please do not hesitate.

warm regards

[Sutanu Dalui]

Hi Baris,

Could you please elaborate when you say command is not correct?

Also replacing the custom image is not an option for us since we have a Cassandra map store backed server implementation. 

Thanks,

Sutanu

--
You received this message because you are subscribed to a topic in the Google Groups "Hazelcast" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/hazelcast/Lob0f_UIPCM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to hazelcast+unsubscribe@googlegroups.com.
To post to this group, send email to haze...@googlegroups.com.
Visit this group at https://groups.google.com/group/hazelcast.

To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/bfedca06-404c-4af6-8d4a-5cdd0d3bbb12%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[baris....@gmail.com]

Hi,

In the run command you try to bind to host and port simulatenously. 

It gives: docker: Invalid containerPort: --net=host. error

You said you are already using custom image for hazelcast, therefore, I recommended you to put this image instead of image: hazelcast/hazelcast on sample deployment file and create your own.

I am repeating example again, since, I suggest that every service that you have, should be deployable in docker compose or similar deployment methods, in order to leverage container deployments. And not to deal with routing and connection problems with workarounds.

However, If you have mutlicast supporting network, running hazelcast(s) directly on host will solve your discovery problem as well. 

Regarding solution:

I also think that `--net=host` flag with host binding 

<interfaces enabled="true">
            <interface>${broadcast.ip}</interface>
</interfaces>

should be enough to solve the problem, I think that client IP and cluster members IPs also network interfaces do not match, is it also possible for you to share client and cluster member logs? 

thanks and regards

Sutanu

To unsubscribe from this group and all its topics, send an email to hazelcast+...@googlegroups.com.

To post to this group, send email to haze...@googlegroups.com.
Visit this group at https://groups.google.com/group/hazelcast.