Switching between cluster platforms (Hazelcast <-> Atomix ONOS built-in) ?

[Andreas Gilbert]

Hi!

Actually i use Hazelcast to create a 2-node cluster.

For now, only the LeadershipService is used from distributed primitives.

I used/modified the HazelcastLeadershipService from ONOS 1.x for my needs.

A component HazelcastLeaderShipApp is using the HazelcastLeadershipService producing an InstanceEvent (PRIMARY, STANDBY).

The component StoreManager starts the HazelcastInstance in it's activate method.

Nest step is to switch to the ONOS built-in cluster platform - BUT the possibility to use the 2-node Hazelcast cluster should be remain.

So component StoreManager/HazelcastLeaderShipApp is disabled and a new OnosLeaderShipApp component uses the built-in LeadershipService producing same InstanceEvent.

Is it possible to switch between Hazelcast<->Onos cluster via e.g. Properties or similar ?

Would be enough that this switching occurs on startup (not required during runtime).

BR

Andreas

[Andreas Gilbert]

I think it could be done using Properties:

The StoreManager evaluates the properties regarding cluster type (HazelCast, Onos) and starts HazelCast instance or forms an ONOS cluster (via network service).

BR

Andreas

[Jordan Halterman]

It certainly should be feasible to do this. However, ONOS has a lot of custom Atomix primitives, and the same would probably be required to entirely implement Hazelcast. In other words, it would require significant work to implement the primitive APIs using Hazelcast.

--
You received this message because you are subscribed to the Google Groups "ONOS Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to onos-dev+unsubscribe@onosproject.org.
To post to this group, send email to onos...@onosproject.org.
Visit this group at https://groups.google.com/a/onosproject.org/group/onos-dev/.
To view this discussion on the web visit https://groups.google.com/a/onosproject.org/d/msgid/onos-dev/8595593e-426e-469c-a678-7715bcf9d0aa%40onosproject.org.

--

Jordan Halterman

Member of Technical Staff, ONF

[Andreas Gilbert]

Hi!

Yes that's right - but actually only the LeaderShip is needed - so manageable work :-)

br, Samir

[Jordan Halterman]

Just curious, what is the rationale for doing this? Is this just an experiment? The only reasonable rationale I could think of for doing this is to improve availability. Hazelcast can keep nodes available under a wider array of failure scenarios than Atomix can. But if you don’t entirely replace Atomix with Hazelcast then you’re really just getting a less reliable leadership/mastership election service (which can e.g. allow multiple masters in the same term in a network partition) with no greater availability since Raft partitions would still be running in the cluster and still lead to a partial loss of availability in certain cases that Hazelcast could otherwise handle. In other words, there are probably arguments for using Hazelcast or Atomix, but not great arguments for using Hazelcast for leadership elections if Atomix/Raft is still running inside the cluster.

--

You received this message because you are subscribed to the Google Groups "ONOS Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to onos-dev+u...@onosproject.org.

To post to this group, send email to onos...@onosproject.org.
Visit this group at https://groups.google.com/a/onosproject.org/group/onos-dev/.

To view this discussion on the web visit https://groups.google.com/a/onosproject.org/d/msgid/onos-dev/d8795116-1b20-4a48-9c3b-812a62e6f8ad%40onosproject.org.

[Andreas Gilbert]

We have a customer configuration with only 2 nodes - for ONOS using Atomix (Raft) a cluster requires at minimum 3 nodes.

That's the reason to use Hazelcast for master election - other primitives are not used in this configuration.

On the Atmoix homepage there is an interesting statement:

Clusters typically consist of 3 or 5 active nodes.

Smaller clusters can be used but may lose write availability if a node fails.

So may it would be possible to use ONOS LeaderShip with Atomix in a 2-node cluster ?

[Jordan Halterman]

Although the other primitives may not be in use in the application, they’re still used throughout the ONOS internal stores, so the cluster is still prone to the same types of failures resulting in lost availability. If one node goes down in a two node cluster, the controller will lose mastership for all devices and be unable to control the network. To be able to tolerate n-1 failures, the consensus algorithm must be replaced altogether. Thus, it’s not really clear to me what you gain by changing the protocol underlying only one of the primitives. All of the consistent primitives would need to be rewritten with support for Hazelcast and the Raft partitions removed to affect the availability of the ONOS cluster as a whole.

Atomix 2.1 will be in the next ONOS release. A core component of Atomix 2.1 is the ability to switch between consensus and primary-backup protocols for replicating primitives. It’s also capable of operating without the use of a consensus algorithm altogether (basically the same as Hazelcast), though like Hazelcast and any other system it’s prone to network partitions causing major problems without the use of consensus. Waiting for the Atomix Upgrade is likely the only practical way to increase the availability of an ONOS cluster.

On Apr 4, 2018, at 5:35 AM, Andreas Gilbert <onos....@gmail.com> wrote:

We have a customer configuration with only two nodes - for Atomix (Raft) a cluster requires at minimum 3 nodes.

That's the reason to use Hazelcast for master election - other primitives are not used in this configuration.

--

You received this message because you are subscribed to the Google Groups "ONOS Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to onos-dev+u...@onosproject.org.
To post to this group, send email to onos...@onosproject.org.
Visit this group at https://groups.google.com/a/onosproject.org/group/onos-dev/.

To view this discussion on the web visit https://groups.google.com/a/onosproject.org/d/msgid/onos-dev/d9e1878b-e819-43ac-8ddf-12f264bbc096%40onosproject.org.

[Andreas Gilbert]

Simply spoken: in our 2-node configuration each node operates as standalone node - i would say, this is not a traditional cluster, its a failover system (primary/standby)

Currently we use Hazelcast in our Application to elect a leader (custom leadership service) for several task (which should be handled only be one node at a time) - no other cluster features are used.

Is the switching configuration between consensus and primary-backup protocols available in ONOS (eg setting), or will ONOS use a fixed protocol ?

So what i read with Atomix 2.1 it could be possible to eliminate Hazelcast for our needs ...

It will be in the ONOS 1.13.x release ?

[Jordan Halterman]

Gotcha. That makes more sense.

Yes, there are a couple of changes being made with respect to Atomix 2.1:

• In-memory primary-backup primitives will be supported, and eventually strong consistency algorithms (CRDTs) can be used for cluster membership, primary elections, epochs (distributed counters), etc. This allows for an ONOS cluster that doesn’t use consensus at all, but it will still be highly discouraged

• Instead, the goal is to also separate the consensus layer from ONOS core. So, ONOS will be able to connect to an external Atomix cluster of consensus nodes, allowing the ONOS controller to again practically scale to much smaller and larger numbers of nodes without losing current consistency guarantees

• In practice, both types of clusters will be supported since the peer-to-peer protocol may simplify development workflows.

But to be clear, the upgrade to Atomix 2.1 is not a trivial task. Even within the context of the current usage of Atomix, we need to rewrite StoragePartitionServer and StoragePartitionClient to use the generic protocol interfaces, and update all the Atomix primitives and state machines to use the generic primitive interfaces as well. These changes are necessary to make them useable with either protocol (Raft or primary-backup).

But ultimately, Atomix 2.1 should replace the vast majority of the primitive code in ONOS. The only things that will have to be kept are the wrapper interfaces/classes for backwards compatibility. Eventually, Atomix will provide for all primitives, messaging, events, cluster management, upgrade management, etc so most of the distributed systems code no longer has to be maintained within ONOS.

The timeline for this refactoring is still about a month away. It will likely start next week, but will take a few more weeks to get to a stable release of Atomix that can safely support the primitive changes.

On Apr 5, 2018, at 12:26 AM, Andreas Gilbert <onos....@gmail.com> wrote:

Simply spoken: in our 2-node configuration each node operates as standalone node - i would say, this is not a traditional cluster, its a failover system (primary/standby)

Currently we use Hazelcast in our Application to elect a leader for several task (which should be handled only be one node at a time) - no other cluster features are used.

So what ii read with Atomix 2.1 it could be possible to eliminate Hazelcast for our needs ...

--
You received this message because you are subscribed to the Google Groups "ONOS Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to onos-dev+u...@onosproject.org.
To post to this group, send email to onos...@onosproject.org.
Visit this group at https://groups.google.com/a/onosproject.org/group/onos-dev/.

To view this discussion on the web visit https://groups.google.com/a/onosproject.org/d/msgid/onos-dev/7f3dcb1b-1c44-4514-b59b-590edfd5ce16%40onosproject.org.

[Andreas Gilbert]

Thanks for explanation!

That sounds good, so i'm looking forward to trying out Atomix 2.1 in ONOS.

In the meantime i will use either Hazelcast or ONOS builtin (atomix) cluster implementations (leadership primitive) in my application, switched by e.g. property.