distributing work among cluster members

[j...@panix.com]

I'm looking for ideas on how to manage a workload of several hundred tasks that I'd like to balance among hazelcast cluster members.  

My first thought was to use consistent hashing (see: https://arxiv.org/pdf/1406.2294.pdf ), and to do so I would need know how many

members my cluster has, and what my ordinal cluster member number is.   Hazelcast.getCluster().getMembers() can provide me both.

I can then loop through my job IDs and if my cluster ordinal

number matches the bucket it is assigned to (+1), then I should perform work on that job.  

Job IDs are strings, a list of which may vary over time.

What other ways are there to do this?  

[Guido Medina]

You can decide by using some hash function, I developed for a FX trading system to distribute jobs among several Akka nodes, assume the following:

• Your IDs have a hashcode implementation (String and other Java wrapper classes have this)
• You have a way to know each node member on the cluster; in other words, they are ordered using some criteria,
• maybe there is a way to get the cluster information and get the order on which they joined; you need this for consistency so that the same hashcode always goes to the same node consistently.

With this assumptions in place all you have to do on the node is:

public static Member getMemberFor(Object id, List<Member> members) {

  final int hashcode = id.hashcode();

  if (hashcode >= 0) {

    return members.size() + (members.get(hashcode % members.size()) - 1;

  } else {

    return hashcode % members.size();

  }

}

This is what I used, it is a dumb algorithm provided that your IDs are mostly integer and come from an AtomicLong or something,

here is a similar approach for Akka which I happen to contribute to https://github.com/akka/akka/pull/17743/files

I'm not adding much to what you already posted, I'm mostly confirming that your approach is what it is generally used for this, it is simple and efficient.

Guido.

[Guido Medina]

Sorry, the first function had the if inverted:

public static Member getMemberFor(Object id, List<Member> members) {

  final int hashcode = id.hashcode();

  if (hashcode >= 0) {

    return hashcode % members.size();

  } else {

[Guido Medina]

Darn it, I coded this on notepad, here is again the corrected function :D

public static Member getMemberFor(Object id, List<Member> members) {

 final int hashcode = id.hashcode();

 if (hashcode >= 0) {

   return members.get(hashcode % members.size());

 } else {

   return members.get(members.size() + (hashcode % members.size()) - 1);

 }

}

[Randy May]

Hi,

Hazelcasts’s IExecutorService can do this for you. I think  anExecutor.submitToKeyOwner(myTask, job_id) should do the trick. See https://docs.hazelcast.org/docs/3.12.2/manual/html-single/index.html#executing-code-in-the-cluster

-- 
You received this message because you are subscribed to the Google Groups "Hazelcast" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hazelcast+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/627ba5e0-dcf7-40b0-9b24-f0038560d8bb%40googlegroups.com.