Obtaining isolation in hazelcast communication...

[ErikEngerd]

Hi,


We are dealing with a scenare where multiple parties are developing
software where hazelcast is used for a number of components.

What we would like to achieve is to be able for these different
components to support the use different hazelcast versions by these
components. That would solve the problem of upgrade where an
individual party could update his own software to use a more recent
version of hazelcast without impacting any of the other components
that use hazelcast. Of course, if all of these components use the same
hazelcast configuration, then problems can arise if different
hazelcast versions try to communicate with each other. To solve this,
one piece of the puzzle is to be able to use different hazelcast
configurations for these components. This I believe is now possible
since hazelcast 1.7.1.

A next question is then how to achieve this isolation through
configuration. In particular there is the port setting in hazelcast
configuration with the autoincrement flag. My question now is how far
apart the port ranges should be to achieve this isolation.

In particular, I have used <port autoincrement='true">5701</port> and
I run one process with this configuration. Next, I start up the same
application again but using different values of the port. What happens
now is that I see ports 5699, 5700, and 5701 leading to group
communication and all other ports (> 5701 or < 5699) lead to isolation
(no group is formed).

What in general are the guidelines for configuration to make sure that
I can achieve this isolation?
Are there also some API examples of how to programmatically set the
hazelcast configuration?

Cheers
Erik

[Atle Prange]

Hi Erik ...
(answer below)

On Wed, 2009-10-21 at 10:50 -0700, ErikEngerd wrote:
> Hi,
>
>
> We are dealing with a scenare where multiple parties are developing
> software where hazelcast is used for a number of components.
>
> What we would like to achieve is to be able for these different
> components to support the use different hazelcast versions by these
> components. That would solve the problem of upgrade where an
> individual party could update his own software to use a more recent
> version of hazelcast without impacting any of the other components
> that use hazelcast. Of course, if all of these components use the same
> hazelcast configuration, then problems can arise if different
> hazelcast versions try to communicate with each other. To solve this,
> one piece of the puzzle is to be able to use different hazelcast
> configurations for these components. This I believe is now possible
> since hazelcast 1.7.1.
>
> A next question is then how to achieve this isolation through
> configuration. In particular there is the port setting in hazelcast
> configuration with the autoincrement flag. My question now is how far
> apart the port ranges should be to achieve this isolation.
>

I think you can use the "group" feature in the configuration:

http://code.google.com/docreader/#p=hazelcast&s=hazelcast&t=ConfigGroup

> In particular, I have used <port autoincrement='true">5701</port> and
> I run one process with this configuration. Next, I start up the same
> application again but using different values of the port. What happens
> now is that I see ports 5699, 5700, and 5701 leading to group
> communication and all other ports (> 5701 or < 5699) lead to isolation
> (no group is formed).
>
> What in general are the guidelines for configuration to make sure that
> I can achieve this isolation?
> Are there also some API examples of how to programmatically set the
> hazelcast configuration?

You can do this:

Config config = new
XmlConfigBuilder("/path/to/config/file.xml").build();

config.setGroupName( "groupName" );

HazelcastInstance instance = Hazelcast.newHazelcastInstance( config );

-atle

>
> Cheers
> Erik
>
>
>
> >

[Talip Ozturk]

>> In particular, I have used <port autoincrement='true">5701</port> and
>> I run one process with this configuration. Next, I start up the same
>> application again but using different values of the port. What happens
>> now is that I see ports 5699, 5700, and 5701 leading to group
>> communication and all other ports (> 5701 or < 5699) lead to isolation
>> (no group is formed).

> Config config = new

> XmlConfigBuilder("/path/to/config/file.xml").build();
>
> config.setGroupName( "groupName" );
>
> HazelcastInstance instance = Hazelcast.newHazelcastInstance( config );

Atle is right. Just to make it clear:

Config config1= new XmlConfigBuilder().build();
config1.setGroupName("app1");
HazelcastInstance instanceApp1 = Hazelcast.newHazelcastInstance( config1);

Config config2= new XmlConfigBuilder().build();
config2.setGroupName("app2");
HazelcastInstance instanceApp2 = Hazelcast.newHazelcastInstance( config2);

Now instanceApp1 and instanceApp2 are not going to be members of the
same cluster because they have different group names.

I like to emphasis the fact that cluster isolation cannot achieved by
changing ports. You don't even have to change the ports at all,
changing the groupName should be enough.

Hope it is clear.

Regards,
-talip

[ErikEngerd]

>
> Atle is right. Just to make it clear:
>
> Config config1= new XmlConfigBuilder().build();
> config1.setGroupName("app1");
> HazelcastInstance instanceApp1 = Hazelcast.newHazelcastInstance( config1);
>
> Config config2= new XmlConfigBuilder().build();
> config2.setGroupName("app2");
> HazelcastInstance instanceApp2 = Hazelcast.newHazelcastInstance( config2);
>
> Now instanceApp1 and instanceApp2 are not going to be members of the
> same cluster because they have different group names.

That is simple enough.

>
> I like to emphasis the fact that cluster isolation cannot achieved by
> changing ports. You don't even have to change the ports at all,
> changing the groupName should be enough.
>
> Hope it is clear.

I understand that changing the group name will provide isolation.
However, I am still a bit puzzled how that could work when different
applications (i.e. different groups) use different hazelcast versions.
Isn't it possible that a new version of hazelcast could implement a
different method to setup a group creating incompatibilities and in
the end resulting in problems.

Or would the behavior be like this:
* a new version of hazelcast is released (version X) that breaks
network compatibility with a previous version Y of hazelcast
* an application A uses version X and an application B uses version Y
* application A tries to form a group with other members of group A
but in doing so also communicates to hazelcast instances of
application B. These members of application B refuse to join the group
because of an error in setting up the communication (incompatibility)

As a result, group formation for application A succeeds. Application B
does not join the group of application A simply because of
communication incompatibility. If B would have used the same hazelcast
version as application A, then B would not have joined either but in
that case because of the configuration of the group names.

Or is there perhaps another solution at hazelcast level where all
processes first communicate a version number of their network protocol
level (could be identical to hazelcast version as the simplest case)
together with a nice error message mentioning failure to join because
of incompatibility.

I am just concerned about these versioning issues because at our
company we have different products/organisations that work as
independenlty as possible on their own products, yet are likely to be
colocated on the same Java EE application server.


Cheers
Erik

>
> Regards,
> -talip

[ErikEngerd]

I am trying out the solution with the groupnames with hazelcast
version 1.7.1 and whatever I do, I keep on getting a group of two
members even if I run one member with group 'x' and another one with
group 'y'.

Is there something I am doing wrong? I have checked the latest source
code from the trunk and there it looks like the group name (and
password) are being used in the join request.

The configuration and code are below.

hazelcast.xml
==========

<hazelcast>
<!--
<group>
<name>dev</name>
<password>dev-pass</password>
</group>
-->
<network>
<port auto-increment="true">5698</port>
<join>
<multicast enabled="false">
<multicast-group>224.2.2.3</multicast-
group>
<multicast-port>54327</multicast-port>
</multicast>
<tcp-ip enabled="false">
<interface>127.0.0.1</interface>
</tcp-ip>
</join>
<interfaces enabled="true">
<interface>127.0.0.1</interface>
</interfaces>
</network>
<executor-service>
<core-pool-size>16</core-pool-size>
<max-pool-size>64</max-pool-size>
<keep-alive-seconds>60</keep-alive-seconds>
</executor-service>
<queue name="default">
<!--
Maximum size of the queue. When a JVM's local queue size
reaches the maximum,
all put/offer operations will get blocked until the queue
size
of the JVM goes down below the maximum.
Any integer between 0 and Integer.MAX_VALUE. 0 means
Integer.MAX_VALUE. Default is 0.
-->
<max-size-per-jvm>10000</max-size-per-jvm>
<!--
Maximum number of seconds for each item to stay in the
queue. Items that are
not consumed in <time-to-live-seconds> will automatically
get evicted from the queue.
Any integer between 0 and Integer.MAX_VALUE. 0 means
infinite. Default is 0.
-->
<time-to-live-seconds>0</time-to-live-seconds>
</queue>
<map name="default">
<!--
Number of backups. If 1 is set as the backup-count for
example,
then all entries of the map will be copied to another JVM
for
fail-safety. Valid numbers are 0 (no backup), 1, 2, 3.
-->
<backup-count>1</backup-count>
<!--
Valid values are:
NONE (no eviction),
LRU (Least Recently Used),
LFU (Least Frequiently Used).
NONE is the default.
-->
<eviction-policy>NONE</eviction-policy>
<!--
Maximum size of the map. When max size is reached,
map is evicted based on the policy defined.
Any integer between 0 and Integer.MAX_VALUE. 0 means
Integer.MAX_VALUE. Default is 0.
-->
<max-size>0</max-size>
<!--
When max. size is reached, specified percentage of
the map will be evicted. Any integer between 0 and 100.
If 25 is set for example, 25% of the entries will
get evicted.
-->
<eviction-percentage>25</eviction-percentage>
</map>
</hazelcast>



Sample.java
==========


import java.io.IOException;
import java.util.Collection;

import com.hazelcast.config.Config;
import com.hazelcast.config.XmlConfigBuilder;
import com.hazelcast.core.Hazelcast;
import com.hazelcast.core.HazelcastInstance;
import com.hazelcast.core.ITopic;
import com.hazelcast.core.Instance;
import com.hazelcast.core.InstanceEvent;
import com.hazelcast.core.InstanceListener;
import com.hazelcast.core.MessageListener;

public class Sample implements InstanceListener,
MessageListener<String> {

private ITopic<String> topic;

private HazelcastInstance hazelcast;

public Sample(String aConfig, String aGroup) throws IOException {
Config config = new XmlConfigBuilder(aConfig).build();
System.out.println("Group name from config is '" +
config.getGroupName() + "'");

System.out.println("Group name '" + aGroup + "'");
config.setGroupName(aGroup);
System.out.println("Group name from config is '" +
config.getGroupName() + "'");

hazelcast = Hazelcast.newHazelcastInstance(config);


topic = hazelcast.getTopic("topic");
topic.addMessageListener(this);
hazelcast.addInstanceListener(this);


}

public void run() throws Exception {
int count = 0;
while (true) {
Thread.sleep(1000);
System.out.print(".");
topic.publish("" + count++);
}
}

public HazelcastInstance getHazelcast() {
return hazelcast;
}

public static void main(String[] args) throws Exception {
if ( args.length != 2 ) {
System.err.println("Usage: sample <hazelcast.xml> <groupname>");
System.exit(1);
}
String config = args[0];
String group = args[1];
Sample sample = new Sample(config, group);

Collection<Instance> instances = sample.getHazelcast().getInstances
();
for (Instance instance : instances) {
System.out.println(instance.getInstanceType() + ","
+ instance.getId());
}

sample.run();

}

public void instanceCreated(InstanceEvent event) {
Instance instance = event.getInstance();
System.out.println("Created " + instance.getInstanceType() + ","
+ instance.getId());
}

public void instanceDestroyed(InstanceEvent event) {
Instance instance = event.getInstance();
System.out.println("Destroyed " + instance.getInstanceType() + ","
+ instance.getId());
}

@Override
public void onMessage(String aMsg) {
System.out.println("Message received: " + aMsg);
}
}

[ErikEngerd]

I did some further browsing in the 1.7.1 code and switched on
logging.

It looks like ClusterManager.handleJoinRequest() does not inspect the
group name and password
and always accepts new members.

[Talip Ozturk]

> That is simple enough.

I love hearing this.

> I understand that changing the group name will provide isolation.
> However, I am still a bit puzzled how that could work when different
> applications (i.e. different groups) use different hazelcast versions.
> Isn't it possible that a new version of hazelcast could implement a
> different method to setup a group creating incompatibilities and in
> the end resulting in problems.
>
> Or would the behavior be like this:
> * a new version of hazelcast is released (version X) that breaks
> network compatibility with a previous version Y of hazelcast
> * an application A uses version X and an application B uses version Y
> * application A tries to form a group with other members of group A
> but in doing so also communicates to hazelcast instances of
> application B. These members of application B refuse to join the group
> because of an error in setting up the communication (incompatibility)

I see your concern. Yes you are right, all these can happen. We can
eliminate communication protocol conflict. It is not hard to handle, I
already have the solution in mind. Your protocol version idea is also
good and will work. Next snapshot will have it. If protocols are
different then a warning will be logged and nothing else will get
affected, every application will keep working as they were. You still
have to make sure that all application A deployments do have the same
version of Hazelcast though. If not, we can log a warning.

-talip

[Talip Ozturk]

You are right again. MulticastService do check the group name and
password but as you said ClusterManager.handleJoinRequest doesn't. It
is bug!

Thanks,
-talip

[ErikEngerd]

> You still
> have to make sure that all application A deployments do have the same
> version of Hazelcast though. If not, we can log a warning.

That part is the responsibility of the application project so that is
realisable. For applications that consist of one ear which is deployed
on a cluster, this version consistenty is arranged automatically.

On Oct 23, 4:01 pm, Talip Ozturk <ta...@hazelcast.com> wrote:
> You are right again. MulticastService do check the group name and
> password but as you said ClusterManager.handleJoinRequest doesn't. It
> is bug!

Thanks for confirming this.
Can you inform me when the issue is solved? Is there a bug number and
website I can look at to monitor this?

>
> Thanks,
> -talip

[Fuad Malikov]

Hi Erik,
 

Can you inform me when the issue is solved? Is there a bug number and
website I can look at to monitor this?

Issue for related bug:
http://code.google.com/p/hazelcast/issues/detail?id=151

fuad

[ErikEngerd]

I have a suggestion about additional configuration flexibility using a
simple mechanism.

What we have now in our applications is singleton usage of Hazelcast
(i.e. Hazelcast.getTopic(...)) which is fine. Now, if we want to start
isolating application using the current configuration APIs then this
has some consequences in Java EE.

The first is lifecyclemanagement. I have to perform the required
configuration API calls when the application starts. This may sound
trivial but in fact is not in Java EE. In particular, the only
container that provides this lifecycle management is the web
container, but when a war is embedde within an EAR, it might be that
the EJBs in an ear are available and handling traffic (i.e. using
hazelcast) before the war is used.

The second is issue is non-singleton access of Hazelcast. In
particular, I can no longer use the Hazelcast singleton but have to
provide my own. That own singleton could of course also provide the
configuration API logic, thereby solving the lifecycle management
problem above. In other words, the first call on the singleton would
initialize hazelcast properly.

As good as that may sound, all applications using hazelcast would have
to provide their own singleton.

There is however a simple way to provide this custom configuration,
yet still use the Hazelcast singleton.
What I would like to propose is an extension to the Hazelcast
singleton mechanism as follows:
1. at initialization of the default instance, Hazelcast checks if a
property file with a defined name (e.g. com.hazelcast.properties)
exists on the classpath.
2. if it does, it reads the property
com.hazelcast.ConfigProviderClass from that property file.
3. If the property exists and the class can be constructed using a
default constructor and is an instance of a new interface
HazelcastConfigProvider, its create() method is called to create a
Config object which is then used for initializing Hazelcast.

(NOTE: the above mechanism could be directly based on
java.util.ServiceLoader which is then used to locate the first
instance
of HazelcastConfigProvider in its path. That would be then based on
the standard Java SPI mechanism).


Advantages of this method would be that:
1. applications can still use the standard singleton mechanism: i.e.
no changes to existing code
2. any custom initialization logic can be added and can be packaged as
a separate library in the application (non-intrusive).
3. the solution is not limited to XML configuration
4. life cycle management problems are avoided

Impacts to the code would be as follows:
1. A new interface HazelcastConfigProvider

public interface HazelcastConfigProvider {
Config create();
}

2. Modification to FactoryImpl.newFactory(Config config).

Replace

if (config == null) {
config = new XmlConfigBuilder().build();
}

By

ServiceLoader<HazelcastConfigProvider> configProviderLoader =
ServiceLoader.load(HazecastConfigProvider.class);
if ( configProviderLoader.hasNext() ) {
// take the first config provider.
HazelcastConfigProvider provider =
configProviderLoader.next();
config = provider.create();
}
else {
config = new XmlConfigBuilder().build();
}


Cheers
Erik

[ErikEngerd]

On Oct 26, 4:34 pm, ErikEngerd <erik.brak...@gmail.com> wrote:
>
> By
>
> ServiceLoader<HazelcastConfigProvider> configProviderLoader =
> ServiceLoader.load(HazecastConfigProvider.class);
> if ( configProviderLoader.hasNext() ) {
> // take the first config provider.
> HazelcastConfigProvider provider =
> configProviderLoader.next();
> config = provider.create();
> }
> else {
> config = new XmlConfigBuilder().build();
> }
>

Of course, the above part must also be surrounded by if (config ==
null) { ....}

[Talip Ozturk]

I liked the idea. I started thinking about its implementation. Two
things makes me move slower:

1. ServiceLoader is Java6+. we want to be Java5+. should we have
ServiceLoader-like implementation for java5?
2. ServiceLoader is still classpath based solution and problematic in
OSGi environments. The thing is that HazelcastConfigProvider
implementation is a user class and hazelcast osgi bundle will have
hard time getting it from user's osgi bundles.


-talip

[ErikEngerd]

On Oct 27, 4:10 pm, Talip Ozturk <ta...@hazelcast.com> wrote:
> I liked the idea. I started thinking about its implementation. Two
> things makes me move slower:
>
> 1. ServiceLoader is Java6+. we want to be Java5+. should we have
> ServiceLoader-like implementation for java5?

It is not required to use ServiceLoader. Instead a basic machenism
based on properties could also be used.

> 2. ServiceLoader is still classpath based solution and problematic in
> OSGi environments. The thing is that HazelcastConfigProvider
> implementation is a user class and hazelcast osgi bundle will have
> hard time getting it from user's osgi bundles.

I have been reading about this a bit and indeed, the OSGI
specification only defines the bundle class loader, but does not
define what happens to the context classloader when a bundle boundary
is crossed. See for instance,

http://impalablog.blogspot.com/2008/10/using-threads-context-class-loader-in.html

As is described in the above link, solutions for this problem exist,
but are non-standard.

One area for inspiration is JPA which has to solve a similar problem
for finding persistence providers when used in OSGI. There a
persistence provider would be exposed as a service so that the
framework can find it. See for instance
http://www.slideshare.net/shaunmsmith/osgi-persistence-with-eclipselink

Perhaps a similar solution would be possible for hazelcast whereby a
bundle could expose its own config service.

There is also a possible other solution based on an OSGI
ServiceFactory. The ServiceFactory gets the Bundle object of the
caller when a service is requested. The bundle object then in turn can
be queried using the findEntries() method of the Bundle. The OSGI spec
(OSGI 4.1, core spec, section 4.3.15) says that the findEntries()
methd is suitable for finding setup or configuration information from
another bundle. In other words, this appears to be the mechanism that
is intended for this purpose. The good thing about a service factory
is that it would allow different applications to use independent
hazelcast instances, thus reducing the chance of side effects between
applications.

Is there some more information available on the current hazelcast osgi
bundle and the way it is intended to work? That way, I can help think
about these deployment issues. In particular, is the singleton style
access to hazelcast still intended to work in OSGI? Also, I am curious
how the current implementation for the OSGI bundle would keep the API
usage for getting a hazelcast instance the same as without OSGI. One
solution would be to abandon the singleton mechanism alltogether and
use non-singleton access or an application-specific singleton.

Cheers
Erik

>
> -talip

[ErikEngerd]

Note the the ServiceFactory approach would not be as flexible as the
ServiceLoader approach, but it would solve the problem. Configuration
information of a bundle could consist of the groupname for instance
(we have to assume that classloading of a class from the Caller by the
Callee is not possible).

[Talip Ozturk]

This topic is moved to an issue for development purposes. Additional
comments should go here:
http://code.google.com/p/hazelcast/issues/detail?id=161

-talip