Vert.x server-client application with Hazelcast clients - configuring cluster-host

[przemyslaw...@connectpoint.pl]

Hey guys,

I am a member of a team working on Java project with client-server architecture. Some time ago we switched to vert.x on server-side and used vertx-tcp-eventbus-bridge to connect it with clients (we had our own's Java Bridge Client implementation as there is no official one yet). But shortly after, we switched to clustered vert.x with Hazelcast. As our client-server approach isn't exactly what Hazelcast cluster stands for, we decided to use Hazelcast clients as they seemed to be what we needed.

The problem we encountered and I wanted to ask you for help is configuration of cluster-host option in client application. It seems that even if Hazelcast connects successfully this value is not "passed on" from Hazelcast connection but is required to be filled explicitly. I would like to use the same host as the Hazelcast is using and not need to provide it explicitly.

Currently, it looks like that:

HazelcastClusterManager clusterManager = new HazelcastClusterManager(hazelcast);

VertxOptions opts = new VertxOptions()
      .setClusterManager(clusterManager)
      .setClusterHost(ConfigurationManager.EB_CLUSTER_HOST)
      .setClusterPort(ConfigurationManager.EB_CLUSTER_PORT);

return Vertx.rxClusteredVertx(opts);

Best,

Przemek

[Tim Fox]

On Thursday, 27 April 2017 09:29:41 UTC+1, przemyslaw...@connectpoint.pl wrote:

Hey guys,

I am a member of a team working on Java project with client-server architecture. Some time ago we switched to vert.x on server-side and used vertx-tcp-eventbus-bridge to connect it with clients (we had our own's Java Bridge Client implementation as there is no official one yet). But shortly after, we switched to clustered vert.x with Hazelcast. As our client-server approach isn't exactly what Hazelcast cluster stands for, we decided to use Hazelcast clients as they seemed to be what we needed.

I'm not sure I understand what you mean here. Could you explain in some more detail?

 

The problem we encountered and I wanted to ask you for help is configuration of cluster-host option in client application.

Which client application are you referring to?

[przemyslaw...@connectpoint.pl]

On Thursday, 27 April 2017 11:02:01 UTC+2, Tim Fox wrote:

On Thursday, 27 April 2017 09:29:41 UTC+1, przemyslaw...@connectpoint.pl wrote:

Hey guys,

I am a member of a team working on Java project with client-server architecture. Some time ago we switched to vert.x on server-side and used vertx-tcp-eventbus-bridge to connect it with clients (we had our own's Java Bridge Client implementation as there is no official one yet). But shortly after, we switched to clustered vert.x with Hazelcast. As our client-server approach isn't exactly what Hazelcast cluster stands for, we decided to use Hazelcast clients as they seemed to be what we needed.

I'm not sure I understand what you mean here. Could you explain in some more detail?

We have a single server side application (running on a separate VM) with multiple client applications connecting to it. For communication we use clustered vert.x with Hazelcast. The cluster currently consists of only one member (server side application). Client side applications connects as cluster clients to this specific cluster member. All the communication between server and clients is via Event Bus.

Server side application (Java) uses Hazelcast cluster configured with TCP/IP (we don't want to use multicast). Currently our cluster is a single member / single server model with client apps connecting to this particular node. The IP/host/port of the server is permanent so there is no problem with specifying it as below:

ClasspathXmlConfig hazelcastConfig = new ClasspathXmlConfig(HAZELCAST_CONFIG_URL);
HazelcastClusterManager clusterManager = new HazelcastClusterManager(hazelcastConfig);

VertxOptions opts = new VertxOptions()

      .setClusterManager(clusterManager)
      .setClusterHost(ConfigurationManager.EB_CLUSTER_HOST)
      .setClusterPort(ConfigurationManager.EB_CLUSTER_PORT);

Vertx.rxClusteredVertx(opts)
      .doOnError(throwable -> log.fatal("Failed to create clustered vert.x!", throwable))
      .doOnSuccess(this::deploy).subscribe();

 

 

The problem we encountered and I wanted to ask you for help is configuration of cluster-host option in client application.

Which client application are you referring to?

 

I was referring to "our client side application". The problem is that "our client side application" can be run on any machine meaning the host / IP is unknown until runtime.I want to emphasize that we don't have issues with configuring Hazelcast or Event Bus to make it work. The case is that it seems that Event Bus Options ignore the host name which is being used for Hazelcast cluster connection and require it to be specified explicitly. I wanted to know if there is a "nice way" for VertxOptions to inherit this information from Hazelcast config.

 

It seems that even if Hazelcast connects successfully this value is not "passed on" from Hazelcast connection but is required to be filled explicitly. I would like to use the same host as the Hazelcast is using and not need to provide it explicitly.

Currently, it looks like that:

HazelcastClusterManager clusterManager = new HazelcastClusterManager(hazelcast);

VertxOptions opts = new VertxOptions()
      .setClusterManager(clusterManager)
      .setClusterHost(ConfigurationManager.EB_CLUSTER_HOST)
      .setClusterPort(ConfigurationManager.EB_CLUSTER_PORT);

return Vertx.rxClusteredVertx(opts);

Best,

Przemek

I hope this time I managed to clarify the context better!

[Tim Fox]

On Thursday, 27 April 2017 10:35:25 UTC+1, przemyslaw...@connectpoint.pl wrote:

On Thursday, 27 April 2017 11:02:01 UTC+2, Tim Fox wrote:

On Thursday, 27 April 2017 09:29:41 UTC+1, przemyslaw...@connectpoint.pl wrote:

Hey guys,

I am a member of a team working on Java project with client-server architecture. Some time ago we switched to vert.x on server-side and used vertx-tcp-eventbus-bridge to connect it with clients (we had our own's Java Bridge Client implementation as there is no official one yet). But shortly after, we switched to clustered vert.x with Hazelcast. As our client-server approach isn't exactly what Hazelcast cluster stands for, we decided to use Hazelcast clients as they seemed to be what we needed.

I'm not sure I understand what you mean here. Could you explain in some more detail?

We have a single server side application (running on a separate VM) with multiple client applications connecting to it. For communication we use clustered vert.x with Hazelcast. The cluster currently consists of only one member (server side application). Client side applications connects as cluster clients

Not sure what you mean by "cluster clients" here. Do you mean that your clients use hazelcast directly (not Vert.x) ?

 

to this specific cluster member. All the communication between server and clients is via Event Bus.

if your clients are just Hazelcast clients, how can they use the Vert.x event bus?

Maybe I am being slow, but I can't get my head around your setup here.

[przemyslaw...@connectpoint.pl]

On Thursday, 27 April 2017 12:26:42 UTC+2, Tim Fox wrote:

On Thursday, 27 April 2017 10:35:25 UTC+1, przemyslaw...@connectpoint.pl wrote:

On Thursday, 27 April 2017 11:02:01 UTC+2, Tim Fox wrote:

On Thursday, 27 April 2017 09:29:41 UTC+1, przemyslaw...@connectpoint.pl wrote:

Hey guys,

I am a member of a team working on Java project with client-server architecture. Some time ago we switched to vert.x on server-side and used vertx-tcp-eventbus-bridge to connect it with clients (we had our own's Java Bridge Client implementation as there is no official one yet). But shortly after, we switched to clustered vert.x with Hazelcast. As our client-server approach isn't exactly what Hazelcast cluster stands for, we decided to use Hazelcast clients as they seemed to be what we needed.

I'm not sure I understand what you mean here. Could you explain in some more detail?

We have a single server side application (running on a separate VM) with multiple client applications connecting to it. For communication we use clustered vert.x with Hazelcast. The cluster currently consists of only one member (server side application). Client side applications connects as cluster clients

Not sure what you mean by "cluster clients" here. Do you mean that your clients use hazelcast directly (not Vert.x) ?

Client side application is a regular clustered vert.x node, it doesn't use Hazelcast directly. I just wanted to emphasize that these are not "regular" Cluster Members but Cluster Clients (http://docs.hazelcast.org/docs/3.8/manual/html-single/index.html#hazelcast-java-client). Everything is done according to the docs (http://vertx.io/docs/vertx-hazelcast/java/#_using_an_existing_hazelcast_cluster) and the exact code is:

ClientConfig hazelcastConf = new XmlClientConfigBuilder(HAZELCAST_CLIENT_CONFIG_URL).build();
hazelcast = HazelcastClient.newHazelcastClient(hazelcastConf);

HazelcastClusterManager clusterManager = new HazelcastClusterManager(hazelcast);

VertxOptions opts = new VertxOptions()
      .setClusterManager(clusterManager)

      .setClusterHost(ConfigurationManager.EB_CLUSTER_HOST) // I want the cluster host to be the same value as the one
                                                            // used by Hazelcast, not to set it explicitly here
      .setClusterPort(ConfigurationManager.EB_CLUSTER_PORT);

return Vertx.rxClusteredVertx(opts);

[Tim Fox]

Cluster host and port are the values used by Vert.x, not Hazelcast.

If you don't configure them, then Vert.x will use the usable interface it can see, this might not be the one you want if you have more than one interface, so you need to configure it in that case.

Vert.x doesn't have any understanding of Hazelcast config so can't parse that in order to extract values (and that wouldn't be the right thing to do in many cases anyway).

Why don't you just pass in your desired client host/port as a system property or environment variable and use that to programmatically configure both Hazelcast and Vert.x?

[przemyslaw...@connectpoint.pl]

The default hostname to use when clustering is "localhost" (http://vertx.io/docs/apidocs/io/vertx/core/VertxOptions.html#DEFAULT_CLUSTER_HOST), so it seems Vert.x doesn't put much effort into discovering network interfaces. In case of Hazelcast Client config I don't have to specify the interface nor hostname to be used, only the cluster member address which is very convenient.

<network>
   <cluster-members>
      <address>server-host:port</address>
   </cluster-members>
</network>

Hazelcast figures out the proper network interface (and hostname) during runtime. 

In this case I will try to extract the hostname from Hazelcast and pass it to VertxOptions.

I don't want to set it as a system property or environment variable as I don't know what hostname to use at launch time of the client side application. It would be great if setting hostname to "0.0.0.0" would work. But as I understand other Vert.x nodes use this value to reply to sender, so 0.0.0.0 obviously won't work.

[Tim Fox]

On Thursday, 27 April 2017 12:35:19 UTC+1, przemyslaw...@connectpoint.pl wrote:

The default hostname to use when clustering is "localhost" (http://vertx.io/docs/apidocs/io/vertx/core/VertxOptions.html#DEFAULT_CLUSTER_HOST), so it seems Vert.x doesn't put much effort into discovering network interfaces.

When running with Starter Vert.x uses this code to select an interface:

https://github.com/eclipse/vert.x/blob/master/src/main/java/io/vertx/core/impl/launcher/commands/BareCommand.java#L327

which basically selects the first non local, non multicast interface.

 

In case of Hazelcast Client config I don't have to specify the interface nor hostname to be used, only the cluster member address which is very convenient.

<network>
   <cluster-members>
      <address>server-host:port</address>
   </cluster-members>
</network>

Hazelcast figures out the proper network interface (and hostname) during runtime. 

But there is no such thing as a "proper network interface". It's common for machines to be configured with multiple interfaces and in that case there is no way for either Hazelcast or Vert.x to know which one you want to use, or maybe you want to use different ones for hazelcast and Vert.x :)

Point is, you can't get away from configuring it in the general case as neither Vert.x nor Hazelcast can do magic.

[Tim Fox]

On Thursday, 27 April 2017 13:14:04 UTC+1, Tim Fox wrote:

On Thursday, 27 April 2017 12:35:19 UTC+1, przemyslaw.kuczynski@connectpoint.pl wrote:

The default hostname to use when clustering is "localhost" (http://vertx.io/docs/apidocs/io/vertx/core/VertxOptions.html#DEFAULT_CLUSTER_HOST), so it seems Vert.x doesn't put much effort into discovering network interfaces.

When running with Starter Vert.x uses this code to select an interface:

https://github.com/eclipse/vert.x/blob/master/src/main/java/io/vertx/core/impl/launcher/commands/BareCommand.java#L327

which basically selects the first non local, non multicast interface.

 

In case of Hazelcast Client config I don't have to specify the interface nor hostname to be used, only the cluster member address which is very convenient.

<network>
   <cluster-members>
      <address>server-host:port</address>
   </cluster-members>
</network>

Hazelcast figures out the proper network interface (and hostname) during runtime. 

But there is no such thing as a "proper network interface". It's common for machines to be configured with multiple interfaces and in that case there is no way for either Hazelcast or Vert.x to know which one you want to use, or maybe you want to use different ones for hazelcast and Vert.x :)

Point is, you can't get away from configuring it in the general case as neither Vert.x nor Hazelcast can do magic.

 

In this case I will try to extract the hostname from Hazelcast and pass it to VertxOptions.

That might work in many cases, but in the case the client has more than one interface, Hazelcast might pick the wrong one and your client won't work. (Things like VPNs also create virtual interfaces).

 

I don't want to set it as a system property or environment variable as I don't know what hostname to use at launch time of the client side application. It would be great if setting hostname to "0.0.0.0" would work.

When creating a server socket 0.0.0.0 just means "listen on all available local addresses", it's not an address that can be used as the target of a connection (Not a Vert.x specific thing)

[Tim Fox]

For your use case, it might make sense just to run a non clustered Vert.x on the server side and use an event bus bridge to connection to clients - that way you don't need Hazelcast at all, and your clients won't need to be configured with a cluster host as they won't be accepting incoming connections.