Hazelcast member discovery through Eureka does not when running in Docker swarm

693 views
Skip to first unread message

ama...@gmail.com

unread,
Apr 26, 2019, 9:30:52 AM4/26/19
to Hazelcast
I have embedded Hazelcast members in my Spring Boot application. I am using Eureka for discovery mechanism. When I run multiple instances of my service on a single machine, the members are able to discover each other successfully. But when I run the services in docker swarm, they never detect each other and run as independent instances. However, I can see both the instances registered under Eureka (on Eureka registered apps link). Below is how I configure the members:

        Config config = new Config();
        config.getProperties().setProperty("hazelcast.discovery.enabled", Boolean.toString(true));
        JoinConfig joinConfig = config.getNetworkConfig().getJoin();
        joinConfig.getMulticastConfig().setEnabled(false);
        joinConfig.getEurekaConfig().setEnabled(true).setProperty("namespace", "hazelcast")
                .setProperty("self-registration", "true").setProperty("use-classpath-eureka-client-props", "false")
                .setProperty("shouldUseDns", "false").setProperty("name", "hazelcast_cache")
                .setProperty("serviceUrl.default", eurekaURL)
                .setProperty("skip-eureka-registration-verification", "true");
        return Hazelcast.newHazelcastInstance(config);

What am I missing here?

Thank you.

Rafal Leszko

unread,
Apr 26, 2019, 9:46:16 AM4/26/19
to haze...@googlegroups.com
Hi,

Could you check your "public ip" for Hazelcast instances? That may be the issue. Please have a look here: https://github.com/hazelcast/hazelcast-docker#hazelcast-hello-world

Otherwise, we've registered your GH Issue: https://github.com/hazelcast/hazelcast-eureka/issues/29 . So, we'll have a look at this.

Cheers,
Rafał

--
You received this message because you are subscribed to the Google Groups "Hazelcast" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hazelcast+...@googlegroups.com.
To post to this group, send email to haze...@googlegroups.com.
Visit this group at https://groups.google.com/group/hazelcast.
To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/a10238e3-0118-4f7a-b4f2-4a178616940e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


--
Rafał Leszko
Software Engineer
@RafalLeszko

Guglielmo Nigri

unread,
May 3, 2019, 11:15:49 AM5/3/19
to Hazelcast
Hi! Did the suggestion about public ip work for you in the end?

Cheers,
Guglielmo


On Friday, April 26, 2019 at 3:46:16 PM UTC+2, Rafal Leszko wrote:
Hi,

Could you check your "public ip" for Hazelcast instances? That may be the issue. Please have a look here: https://github.com/hazelcast/hazelcast-docker#hazelcast-hello-world

Otherwise, we've registered your GH Issue: https://github.com/hazelcast/hazelcast-eureka/issues/29 . So, we'll have a look at this.

Cheers,
Rafał

On Fri, Apr 26, 2019 at 3:30 PM <ama...@gmail.com> wrote:
I have embedded Hazelcast members in my Spring Boot application. I am using Eureka for discovery mechanism. When I run multiple instances of my service on a single machine, the members are able to discover each other successfully. But when I run the services in docker swarm, they never detect each other and run as independent instances. However, I can see both the instances registered under Eureka (on Eureka registered apps link). Below is how I configure the members:

        Config config = new Config();
        config.getProperties().setProperty("hazelcast.discovery.enabled", Boolean.toString(true));
        JoinConfig joinConfig = config.getNetworkConfig().getJoin();
        joinConfig.getMulticastConfig().setEnabled(false);
        joinConfig.getEurekaConfig().setEnabled(true).setProperty("namespace", "hazelcast")
                .setProperty("self-registration", "true").setProperty("use-classpath-eureka-client-props", "false")
                .setProperty("shouldUseDns", "false").setProperty("name", "hazelcast_cache")
                .setProperty("serviceUrl.default", eurekaURL)
                .setProperty("skip-eureka-registration-verification", "true");
        return Hazelcast.newHazelcastInstance(config);

What am I missing here?

Thank you.

--
You received this message because you are subscribed to the Google Groups "Hazelcast" group.
To unsubscribe from this group and stop receiving emails from it, send an email to haze...@googlegroups.com.

ama...@gmail.com

unread,
May 6, 2019, 4:15:17 AM5/6/19
to Hazelcast
Hello,
By exposing the Hazelcast port from container the issue was solved.
Thank you.

Guglielmo Nigri

unread,
May 6, 2019, 8:12:40 AM5/6/19
to haze...@googlegroups.com
Glad to hear that!
Cheers,
Guglielmo

You received this message because you are subscribed to a topic in the Google Groups "Hazelcast" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/hazelcast/qj7Tp_GqTiY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to hazelcast+...@googlegroups.com.

To post to this group, send email to haze...@googlegroups.com.
Visit this group at https://groups.google.com/group/hazelcast.

For more options, visit https://groups.google.com/d/optout.


--

Guglielmo Nigri
Software Engineer

mark.c...@gmail.com

unread,
Jun 3, 2019, 5:25:37 PM6/3/19
to Hazelcast
I'm having a similar issue. I've logged onto one of the docker containers and I can curl the cluster rest endpoint for each individual container that should be in the cluster and I get a response, the problem is though that each one is returning only a single member rather than showing them as being clustered together.

/Mark
To unsubscribe from this group and all its topics, send an email to haze...@googlegroups.com.

To post to this group, send email to haze...@googlegroups.com.
Visit this group at https://groups.google.com/group/hazelcast.

Rafal Leszko

unread,
Jun 4, 2019, 2:47:08 AM6/4/19
to haze...@googlegroups.com
Hi Mark,

It means that the members didn't form the cluster, so the discovery didn't work. Could you first check the public IPs of your members as mentioned in this thread. Also check if your Hazelcast port (5701) is exposed (and published) from the container. If nothing helps. please create a GH Issue here: https://github.com/hazelcast/hazelcast-eureka (please include all the information and the steps to reproduce).

Cheers,
Rafał

To unsubscribe from this group and stop receiving emails from it, send an email to hazelcast+...@googlegroups.com.

To post to this group, send email to haze...@googlegroups.com.
Visit this group at https://groups.google.com/group/hazelcast.
To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/15802262-47a2-4f1f-9eb6-cd90d09628e2%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

mark.c...@gmail.com

unread,
Jun 4, 2019, 6:38:42 AM6/4/19
to Hazelcast
Thanks for the reply Rafal,

all the ports and ips are public to each other, i.e. I can log onto any of the docker containers and hit the Rest Endpoints so we know they are available to each other, they just aren't discovering each other.

I think my issue may be the fact that my cluster is over a number of docker containers that aren't using the same spring application name. Does hazelcast-eureka handle such a set up?

/Mark


On Tuesday, 4 June 2019 07:47:08 UTC+1, Rafal Leszko wrote:
Hi Mark,

It means that the members didn't form the cluster, so the discovery didn't work. Could you first check the public IPs of your members as mentioned in this thread. Also check if your Hazelcast port (5701) is exposed (and published) from the container. If nothing helps. please create a GH Issue here: https://github.com/hazelcast/hazelcast-eureka (please include all the information and the steps to reproduce).

Cheers,
Rafał

Rafal Leszko

unread,
Jun 4, 2019, 6:59:36 AM6/4/19
to haze...@googlegroups.com
Hi Mark,

You need to have the same name set in eureka-client.properties (field: hazelcast.name). This is probably the issue you encounter.
Read more at:

Cheers,
Rafał

To unsubscribe from this group and stop receiving emails from it, send an email to hazelcast+...@googlegroups.com.

To post to this group, send email to haze...@googlegroups.com.
Visit this group at https://groups.google.com/group/hazelcast.
To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/005876e9-f1fd-4685-8658-50deedbcae07%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

mark.c...@gmail.com

unread,
Jun 4, 2019, 7:21:23 AM6/4/19
to Hazelcast
Is there a way of setting those eureka-client.properties values via Java config instead? We current do set it in java config:

hazelcastConfig.getGroupConfig().setName("name of config")


We're setting these other values and I'm wondering if there is some conflict with these settings and our use case of the members being in different spring applications:

final InterfacesConfig interfaces = new InterfacesConfig();
interfaces.setEnabled(true);
interfaces.addInterface("10.10.0.*");
hazelcastConfig.getNetworkConfig().setInterfaces(interfaces);
hazelcastConfig.getNetworkConfig().getJoin().getMulticastConfig().setEnabled(false);
hazelcastConfig.getNetworkConfig()
               .getJoin()
               .getEurekaConfig()
               .setEnabled(true)
               .setProperty("self-registration", "true")
               .setProperty("use-classpath-eureka-client-props", "true")
               .setProperty("shouldUseDns", "false")
               .setProperty("namespace", "hazelcast")
               .setProperty("use-metadata-for-host-and-port", "true");

hazelcastConfig.getNetworkConfig().getJoin().getMulticastConfig().setEnabled(false);




On Tuesday, 4 June 2019 11:59:36 UTC+1, Rafal Leszko wrote:
Hi Mark,

You need to have the same name set in eureka-client.properties (field: hazelcast.name). This is probably the issue you encounter.
Read more at:

Cheers,
Rafał

Rafal Leszko

unread,
Jun 4, 2019, 8:03:27 AM6/4/19
to haze...@googlegroups.com
Hi Mark,


Hope it helps,
Rafał

To unsubscribe from this group and stop receiving emails from it, send an email to hazelcast+...@googlegroups.com.

To post to this group, send email to haze...@googlegroups.com.
Visit this group at https://groups.google.com/group/hazelcast.
To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/04c2f544-e5b6-429f-b802-219b23894468%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

mark.c...@gmail.com

unread,
Jun 4, 2019, 12:35:21 PM6/4/19
to Hazelcast
Hi Rafal,

sorry to pester, but I've been trying to get this to work with the hazelcast-sample code and I can to a certain extent. For the hazelcast-separate-client code base I've changed the config bean slightly:

@Bean
   public Config hazelcastConfig() {
       Config config = new Config();
       config.getNetworkConfig().setPort(hazelcastPort);
       config.getNetworkConfig().getJoin().getMulticastConfig().setEnabled(false);
       config.getNetworkConfig().getJoin().getEurekaConfig()
             .setEnabled(true)
             .setProperty("self-registration", "true")
             .setProperty("use-classpath-eureka-client-props", "false")
             .setProperty("shouldUseDns", "false")
             .setProperty("name", "blag")
             .setProperty("serviceUrl.default", "http://localhost:8761/eureka/")
             .setProperty("namespace", "hazelcast");
       return config;
   }

and running them using different spring app names works as you'd expect (i.e. we've two separate spring boot apps registered in Eureka + the hazelcast cluster blag):

$ java -jar hazelcast-separate-client/target/hazelcast-separate-client-0.1-SNAPSHOT.jar --server.port=8080 --hazelcast.port=5701 --spring.application.name=app1

$ java -jar hazelcast-separate-client/target/hazelcast-separate-client-0.1-SNAPSHOT.jar --server.port=8081 --hazelcast.port=5703 --spring.application.name=app2


When I try the same with the hazelcast-metadata sample, I still have an issue where they are not connecting together as a cluster. There my code for the config looks like:

@Bean
public Config hazelcastConfig(EurekaClient eurekaClient) {
EurekaOneDiscoveryStrategyFactory.setEurekaClient(eurekaClient);
Config config = new Config();
config.getNetworkConfig().setPort(hazelcastPort);
config.getNetworkConfig().getJoin().getMulticastConfig().setEnabled(false);
config.getNetworkConfig().getJoin().getEurekaConfig()
.setEnabled(true)
.setProperty("self-registration", "true")
.setProperty("use-classpath-eureka-client-props", "false")
.setProperty("namespace", "hazelcast")
.setProperty("name", "blag")
.setProperty("use-metadata-for-host-and-port", "true");
return config;
}



On Tuesday, 4 June 2019 13:03:27 UTC+1, Rafal Leszko wrote:
Hi Mark,


Hope it helps,
Rafał

Rafal Leszko

unread,
Jun 5, 2019, 9:07:54 AM6/5/19
to haze...@googlegroups.com
Hi Mark,

Your configuration for the second example (Metadata) does not include serviceUrl.default and other parameters like defined here: https://github.com/hazelcast/hazelcast-code-samples/blob/master/hazelcast-integration/eureka/springboot-embedded/hazelcast-separate-client/src/main/resources/eureka-client.properties.

Cheers,
Rafał

To unsubscribe from this group and stop receiving emails from it, send an email to hazelcast+...@googlegroups.com.

To post to this group, send email to haze...@googlegroups.com.
Visit this group at https://groups.google.com/group/hazelcast.
To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/28ccc1fe-befd-44ed-80d8-32b5931f1bd3%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

mark.c...@gmail.com

unread,
Jun 6, 2019, 6:46:14 AM6/6/19
to Hazelcast
Hi Rafal,

thanks for pointing that out. I've not set all those additional properties and yet I'm still not getting the cluster being created with the metadata example code:

    @Bean
   public Config hazelcastConfig(EurekaClient eurekaClient) {
       EurekaOneDiscoveryStrategyFactory.setEurekaClient(eurekaClient);
       Config config = new Config();
       config.getNetworkConfig().setPort(hazelcastPort);
       config.getNetworkConfig().getJoin().getMulticastConfig().setEnabled(false);
       config.getNetworkConfig().getJoin().getEurekaConfig()
             .setEnabled(true)
             .setProperty("self-registration", "true")
             .setProperty("use-classpath-eureka-client-props", "false")
             .setProperty("namespace", "hazelcast")
             .setProperty("name", "blag")
             .setProperty("shouldUseDns", "false")              
             .setProperty("serviceUrl.default", "http://localhost:8761/eureka/")              
             .setProperty("use-metadata-for-host-and-port", "true");
       return config;
   }




Should I raise an issue on Github for this or is there something I'm obviously missing here?

/Mark


On Wednesday, 5 June 2019 14:07:54 UTC+1, Rafal Leszko wrote:
Hi Mark,

Your configuration for the second example (Metadata) does not include serviceUrl.default and other parameters like defined here: https://github.com/hazelcast/hazelcast-code-samples/blob/master/hazelcast-integration/eureka/springboot-embedded/hazelcast-separate-client/src/main/resources/eureka-client.properties.

Cheers,
Rafał

Rafal Leszko

unread,
Jun 6, 2019, 9:25:22 AM6/6/19
to haze...@googlegroups.com
Hi Mark,


If you set all these values and it still doesn't work, then please raise a GH Issue to https://github.com/hazelcast/hazelcast-eureka with the steps to reproduce.

Cheers,
Rafał

To unsubscribe from this group and stop receiving emails from it, send an email to hazelcast+...@googlegroups.com.

To post to this group, send email to haze...@googlegroups.com.
Visit this group at https://groups.google.com/group/hazelcast.
To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/570f343b-7792-4866-a07c-e1922a97baa6%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.
Reply all
Reply to author
Forward
0 new messages