Clustered Vert.x Hazelcast maps clustering or not

[bytor99999]

So we have a new part of our application that we are sending messages via RabbitMQ instead of Vert.x. Why, because if we embedded Vertx then Hazelcast will add it as a node to its cluster, and the data that we store in Hazelcast would then be replicated on this other node, that has no need for that data and instead would just cost us more money on AWS for a larger instance just to store that data.

So what I was wondering, if anyone knew of a way that we can still have Vert.x register its node with Hazelcast for clustering, without Hazelcast bringing over all the App Data we store in it for our app.

p.s. We have tried creating two separate Hazelcast clusters. One for Vert.x and another for our data, but it didn't work in that it was complex to set up and keep separate.

Thanks

Mark

[qsys]

I don't know if I get the question right, but you seem to use Hazelcast as a sort of local (?) cache or something, besides vertx clustering? Anyway, I have no issues whatsoever to have 2 (or more) Hazelcast instances in one application (or verticle). Using still vertx 2.x, I have verticles with 3 Hazelcast instances:
1/ vertx clustering
2/ shared cache between several modules
3/ hibernate cache per module (for some modules).
For some verticles, a spring container is started, for others, there's no spring, nor hibernate, and for one 'vertx eb connector', it's embedded in a rather big spring (running in a tomcat) application. It all seems to work pretty fine without any issues to keep all the Hazelcast instances separate.

So, what're your issues separating two (or more) Hazelcast clusters?

qsys


Op vrijdag 31 juli 2015 23:07:02 UTC+2 schreef bytor99999:

[bytor99999]

I was more hoping for some property to set that turns off Hazelcast except for Vert.x clustering, not the shared cache data.

Thanks

Mark

[stream liu]

I'm not sure, but you can store your data to local verticle instead of share to all verticle. http://vertx.io/docs/vertx-core/java/#_using_shared_data_with_vert_x

BTW, you can try vertx-zookeeper for clustering. 

https://github.com/vert-x3/vertx-zookeeper

--
You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.
Visit this group at http://groups.google.com/group/vertx.
To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/5c72f2e7-7577-4993-b7af-f2e15959cb0f%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[qsys]

In that case, something like this might probably work:
http://docs.hazelcast.org/docs/latest/manual/html-single/hazelcast-documentation.html#map-configuration

and set backup-count to 0:

 <backup-count>0</backup-count>

You can do this for 'all' maps, lists, sets, multimaps, ..., or per 'region'. For example, in the application I mentioned earlier, several regions are defined in the same hazelcast instance, depending on the objects to be cached. And there one 'microcache' region as well. They all have different setting. Hazelcast is really customizable, but keep in mind that Hazelcast doesn't want you to interfere what is stored where... So having a cluster means "it doesn't matter where something is stored", so it's probably beyond your control.

I suppose if you set backup-count to 0, no backups will be made. However, the data might probably be stored on other hazelcast nodes, as far as I understand. Anyway, you can choose the number of backups, and this might be the important part, considering you don't want any (or many?) copies of the data. By the way, in the default settings, backups are limited (I thought 1), so your data is not copied to all hazelcast nodes (i.e. verticles).

Oh yeah, what's the problem with using a separate Hazelcast instance per verticle? (You might use Hazelcast, or Zookeeper, or any other cluster manager in the future. I'd keep cluster manager and the 'local data manager' separate, to keep things simple. Combining those two, well, doesn't feel very clean/SoC to me.)

qsys




Op zaterdag 1 augustus 2015 02:12:24 UTC+2 schreef bytor99999:

[bytor99999]

Thanks. Yes, probably having more than one Hazelcast cluster would be the ultimate solution. Unfortunately, to get that working right now would take a while. Or implementing using Zookeeper, same thing. Many reasons why in our current large codebase. 

So I was hoping for a quick simple solution like a property. backup-count won't work either because we do need a backup, and even with, doesn't mean that that one node wouldn't have any data. I just need that node to have no data but be clustered with Vert.x only. 

But thanks for all the help and suggestions.

Mark

[jordan.h...@gmail.com]

To point you in another potential direction, I think a potential solution could be partition groups:

http://docs.hazelcast.org/docs/latest/manual/html/partitiongroupconfig.html

It seems that you can configure Hazelcast to replicate to specific nodes or groups of nodes via this route. I'm assuming that partition groups relate to replication and not access, so nodes outside a partition group could access data inside that group. I'm not entirely sure that's true though. No doubt, it will at least be a PITA, but probably no more so than managing two clusters. Perhaps best to use a cluster manager that uses an external cluster if you need to encapsulate the replication.

If be interested to know if this suggestion works (I have no idea)

--

You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.
Visit this group at http://groups.google.com/group/vertx.

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/07adc389-d28f-404f-ae63-4b39f7fb5c4d%40googlegroups.com.

[bytor99999]

Thanks Jordan. I think for now, we will stick with communication to this "server" via RabbitMQ instead of the eventBus. That way we aren't starting Vert.x on that server app and wouldn't join the cluster.

Mark

[Chris Jones]

Hi Mark,

A potential way to accomplish this is using the Hazelcast client. From the documentation, it appears that the purpose of the client is to provide exactly what you are after - a client on the Hazlecast cluster which can do pretty much everything a full cluster member can do, but does not participate in the cluster replication/communication/storage.

http://docs.hazelcast.org/docs/latest/manual/html/javaclient.html

http://docs.hazelcast.org/docs/latest/manual/html/javaclientoverview.html

I did a quick proof-of-concept to try this and it was pretty easy to get working - I changed the Web server Verticle of my application so that it was a client of the cluster only and did not participate in the cluster itself.

To get it to work (again, this was a PoC, I'm not using the code like this):

1. Created a new class which I would use to replace: io.vertx.spi.cluster.hazelcast.HazelcastClusterManager
2. Copy and pasted the entire body of io.vertx.spi.cluster.hazelcast.HazelcastClusterManager into the new class and fixed up the standard bits like the constructors (I only needed to override one method of that class, but everything in that class is private)
3. Change the 'join' method to join the cluster as a client instead of a cluster member.

I've set up a Gist which has the code I tested with: https://gist.github.com/cdjones32/8f4fd6247d6f98ff24a3

And started VertX according to: http://vertx.io/docs/vertx-hazelcast/#_using_this_cluster_manager

The end result was as expected, with the Verticle starting up and being able to send and receive events on the Event bus and use the VertX shared data API.

As I said, I just did this as a Proof of Concept today, so there may be other considerations from a VertX perspective, but from a Hazelcast perspective it definitely looks feasible.

Creating your own Cluster Manager definitely seems to be an option, but if there are no additional issues, I don't see why it couldn't make sense for the existing HazelcastClusterManager to offer the ability to connect as a client. 

Regards

Chris

[bytor99999]

Thanks Chris, will definitely try this. Funny is that we already have a custom HazelcastClusterManager class. The purpose of this class was so that if Spring had already created a Hazelcast instance and beans for all our Maps, that Vert.x should use it instead of creating a brand new one and therefore all the places that we Autowire them in will all use the same cluster. 

Before, Spring would create one instance, and Vert.x a different one and they would share the data.

Creating HazelcastClusterManagers in Vert.x is really cool and easy. 

Mark

[bytor99999]

Could you have also instead extended

 io.vertx.spi.cluster.hazelcast.HazelcastClusterManager 

and just override the join method? Or where there other methods you had to implement to get it to work?

Thanks again.

Mark

[Chris Jones]

Hi Mark,

Yes, it is only the join method that needs to be changed; however, all the fields in the HazelcastClusterManager class are private and don't have setter methods. That means you can't set anything that needs to be set in the join method unless you do something dodgy like using reflection (which I did initially, but it made the code in the join method too complex for a demonstration).

If the  io.vertx.spi.cluster.hazelcast.HazelcastClusterManager class had protected setters for things like the Hazelcast instance, the config etc, then yes, it would be just a matter of extending that class and overriding the join method.

Chris

[bytor99999]

Ah, yeah I see, for my other implementation I added those properties to my subclass. 

class CustomHazelcastClusterManager extends HazelcastClusterManager {

  @Autowired
  private Vertx vertx

  @Autowired
  private HazelcastInstance hazelcast
  private String nodeID
  private String membershipListenerId

  private volatile boolean active

  private Config conf
  /**
   * Constructor - gets config from classpath
   */
  public CustomHazelcastClusterManager() {
    // We have our own shutdown hook and need to ensure ours runs before Hazelcast is shutdown
    super()
  }

  /**
   * Constructor - config supplied
   * @param conf
   */
  public CustomHazelcastClusterManager(Config conf) {
    super(conf)
  }

  @Override
  public synchronized void join(Handler<AsyncResult<Void>> resultHandler) {
    log.error("**************** Calling our custom implementation")
    vertx.executeBlocking({ fut ->
      if (!active) {
        active = true
        if (conf == null) {
          conf = loadConfigFromClasspath()
          if (conf == null) {
            log.warn("Cannot find cluster configuration on classpath and none specified programmatically. Using default hazelcast configuration")
          }
        }
        log.error("%%%%%%%%%Hazelcast set ${hazelcast}")

        if (hazelcast == null) {
          hazelcast = Hazelcast.getOrCreateHazelcastInstance(conf)
        }
        log.error("%%%%%%%%%Hazelcast set ${hazelcast}")
        nodeID = hazelcast.getCluster().getLocalMember().getUuid()
        membershipListenerId = hazelcast.getCluster().addMembershipListener(this)
        fut.complete()
      }
    }, resultHandler)
  }
}

Thanks

Mark

[Chris Jones]

You must be getting some help there from your Dependency Injection framework. Whithout DI, 'vertx' would be null and if I use any of the super classes functions (like getNodeID()) then I get nulls back from them as well (instead of the values you are setting in the join method).

So it you did this:

customManager = new CustomHazelcastClusterManager();

// ... join happens in here...

System.out.println( customManager.getNodeID() );  // I would expect that to print null, not the node ID you stored in the join method - because the superclass doesn't have access to your copy of nodeID.

That would be ideal if that would work, but I didn't think Java would allow that... I would also expect the 'hazelcast' field in the superclass to be null, which would stop most of it from working. If I do everything as you showed, I just get NullPointer exceptions from the HazelcastClusterManager class whenever I try to access anything like shared maps etc. This is because the 'hazelcast' field on the superclass is never actually set (which is what I expected to happen).

Anyway, if it works for you, that's awesome... I was just expecting that you would need all those fields to be either changed to 'protected' or have getters/setters on the HazelcastClusterManager class for the extend to work. 

[bytor99999]

It is more of a, "I think it works" I haven't had time to go back to see if for sure it is being used. I don't get errors, but I might not be using it at the moment. All I know now is the issue I was trying to fix before, being Spring starting up a HZ instance that is different from the one Vert.x starts up. And that now is one and only one HZ instance that is the same one.

I will have to check again sometime. It is just trying to find that time. ;)

Mark