partition grouping CUSTOM mechanism

[Edmund Cheng]

hi,

I have a question also regarding partition grouping in regards to the pri/backup replica distribution.

Suppose I have this setup.

Site A:

machine 1 with 2 Hz JVMs

machine 2 with 2 Hz JVMs

Site B:

machine 3 with 2 Hz JVMs

machine 4 with 2 Hz JVMs

Total of 8 across.

I can use declarative config:

  <partition-group enabled="true" group-type="CUSTOM">

<member-group>

  <interface>machine1</interface>

  <interface>machine2</interface>

</member-group>

<member-group>

  <interface>machine3</interface>

  <interface>machine4</interface>

</member-group>

</partition-group>

Now by using this CUSTOM grouping for sites, I can safely distribute the pri/backup replicas across sites.

When 1 site goes down, I will still have 2 machine with 4 Hz JVMs with all data intact.

However, at this time the pri/backup replicas distribution are screwed between these 2 machines, meaning the distribution is random.

I presume this is the expected behaviour since the grouping is like 1 layered.

How do I further split this correctly?

I see that there is ZONE AWARE, which seems to suit what I need, but it seems like it is only supported for cloudbased with their discovery plugins.

[Alparslan Avcı]

Hi Edmund,

Hazelcast always distributes the partitions evenly among the cluster members even if you set a custom partition grouping. Partition grouping means that the backup replicas of the partitions won't be kept in the same partition group that the primary replica lives.

Can you please elaborate more what you are trying to achieve?

Regards,

Alparslan Avci

[Edmund Cheng]

Hi Alparslan,

Let me elaborate,

With my above mentioned setup, I put a CUSTOM grouping as:

  <partition-group enabled="true" group-type="CUSTOM">

<member-group>

  <interface>machine1</interface>

  <interface>machine2</interface>

</member-group>

<member-group>

  <interface>machine3</interface>

  <interface>machine4</interface>

</member-group>

</partition-group>

So each machine has 2 Hz JVMs.

Now I started all JVMs, and looked at the entries count and the backup entries count.

1)

I can see that the pri entry count matches the backup entry count in the other group (red box) and vice versa (black box)

I assume this means machine1,machine2 group's backup replicas sit in the other group of machine3,machine4 by looking at this.

Unless this indication is not correct?

2) I then killed off machine3 and machine4 JVMs, and I can see that the total entry count is still the same, which is correct.

But it seems like now, machine2's backup replica is distributed to machine1 and to itself.

So right now, if i kill off another 2 JVMs on either machine, I would lose some pri entries.

Is this understanding correct?

So I just want to understand to this stage, can I make machine2's backup replicas be in machine1 and vice versa?

[Alparslan Avcı]

Hi again Edmund,

Yes, your understanding is correct. Hazelcast's out-of-the-box partition group implementations are only one layered. You cannot specify two different partition grouping strategy at the same time. But you can use SPI to implement your own partition grouping strategy. Please see here: http://docs.hazelcast.org/docs/latest/manual/html-single/index.html#grouping-types

Regards,

Alparslan

[Edmund Cheng]

Thanks Alparslan,

I read about it, then this requires some development code change.

Are these 2 options also viable?

1) Is ZONE AWARE only applies to cloud based implementation? For normal on-premises it does not support?

2) WAN replication, is this worth to explore to help in such a situation?

[Alparslan Avcı]

Hi again Edmund,

The answers are inline:

1) Is ZONE AWARE only applies to cloud based implementation? For normal on-premises it does not support?

No, you can use ZONE_AWARE on-premise environments but you need to provide at least one member attributes below when starting the members: hazelcast.partition.group.zone, hazelcast.partition.group.rack, or hazelcast.partition.group.host. If none of them are provided, it won't work. Please see here how to provide member attributes: http://docs.hazelcast.org/docs/latest/manual/html-single/index.html#defining-member-attributes

2) WAN replication, is this worth to explore to help in such a situation?

WAN Replication helps you replicate a cluster to another DR cluster (active or passive) in another data center mostly placed in a different geographical location. It is helpful when the network latencies are high between the sites. Actually, if the network latency is high between your sites; it is not recommended to build a single cluster placing the instances on both sites. Low network latency among cluster members is critical for Hazelcast clusters. 

Regards,

Alparslan Avci

[Edmund Cheng]

Thanks Alparslan,

1) I think what i meant to ask is if ZONE AWARE can only be used with AWS, jcloud and Azure discovery plugins, if so this will require code changes,

if not, you mean I can use Zone aware for normal tcp-ip discoveries as long as I defined member attributes?

2) Yes i understand your explanation regarding the WAN.

[Edmund Cheng]

Hi Alparslan,

For 1, I tried the config myself as this:

e.g machine 1:

<partition-group enabled="true" group-type="ZONE_AWARE" />

  <member-attributes>

      <attribute name="hazelcast.partition.group.zone">ZoneA</attribute>

      <attribute name="hazelcast.partition.group.host">machine1</attribute>

    </member-attributes>

e.g machine 4

<partition-group enabled="true" group-type="ZONE_AWARE" />

  <member-attributes>

      <attribute name="hazelcast.partition.group.zone">ZoneB</attribute>

      <attribute name="hazelcast.partition.group.host">machine4</attribute>

    </member-attributes>

This config seems to be working to what I need.

- when all 8 nodes are up, the partitions are grouped into Zones

- after i killed off one of the zones, the 2 remaining machines in the same zone does their distribution of pr/backup across 2 machines correctly. (meaning all backup replicas of 1 machine went to the other)

. 

I presume its the nature of Hazelcast looking in this order, zone, rack, host and determines the partitioning strategy based on this top down order whenever possible?

[Edmund Cheng]

Hi, seems like I was wrong, the distribution didnt really work. (though sometimes it does, but I guess that was lucky?)

Perhaps I misunderstood this ZONE AWARE, it still is 1 layered. Even though if I define something like this, it doesnt matter what I give after the zone attribute.

Only 1 will be taken into effect? is my understanding correct?

<partition-group enabled="true" group-type="ZONE_AWARE" />

  <member-attributes>

      <attribute name="hazelcast.partition.group.zone">SiteA</attribute>

      <attribute name="hazelcast.partition.group.rack">rack2</attribute>

      <attribute name="hazelcast.partition.group.host">machine2</attribute>

    </member-attributes>

[Alparslan Avcı]

Hi Edmund,

Yes, it is only one layered as other partition grouping strategies. Hazelcast looks for zone, rack, and host information in the given order and create partition groups with available information. 

Your understanding is correct.

Regards,

Alparslan Avci

[Edmund Cheng]

Thank you Alparslan,

Your feedback is very much appreciated.