High availability and deploying/undeploying verticles programmatically

[Zachary Pry]

Hello,

I just recently started working with vert.x. After some POCs with NodeJS and Vert.x - Vert.x was selected (not just for performance reasons).
The more I work with it - the more I like it, I believe you are creating great product!

Right now I am investigating high availability (version 2.0.3/2.1M3).

From documentation I see this topic:
http://vertx.io/core_manual_java.html#deploying-and-undeploying-verticles-programmatically
In there is described how programmatically (via Container) deploy and undeploy verticles/modules etc.

In here: http://vertx.io/manual.html#high-availability-with-vertx
I see description how to configure High Availability and all examples given are only via command line.

My questions are:
1. I see that I can set how much instances todeploy via Container class, but, by default, are they run in HA "mode" ?
2. Can I set up high availability option (explicitly) via Container class ?
3. Can I set HA groups programmatically ?

and on similar topic:
4. In HA mode "failed" instances are restarted, but how can I restart (programmatically) instance if I see it is not stable (context.exit() might not work as it shuts down all verticles deployed with it) ?
(http://vertx.io/api/java/org/vertx/java/platform/Container.html#undeployModule%28java.lang.String%29 might be a solution, but I don't see where to get deploymentId for individual verticles).

General idea is to create code which allows to control verticles status and restart/stop those which are not stable, or just restart/stop them for maintenance purposes (e.g. like heroku does - restart every 24h).

I did a quick search and reviewed 50 (or so) topics but couldn't find this topic, if it was already asked - please redirect me to it.

Thank you in advance

[Jordan Halterman (kuujo)]

I may be able to answer a few of these questions,

When you run a module in HA mode via the command line, you're essentially starting a new Vert.x instance with HA. As the documentation specifies, if that Vert.x instance fails then the module will be restarted in another node in the cluster. That also means that any modules/verticles deployed programmatically (from within that top level module) will be re-deployed as well, simply due to the fact that they're programmatically deployed. I think the problem with automatically redeploying modules/verticles that were programmatically deployed is that the very fact that they were programmatically deployed may indicate that some programming logic is involved in their deployment, and when the verticle that deployed them fails, if that verticle is restarted then it will automatically restart those modules/verticles it deployed as well, so Vert.x will simply redeploy that top level module and it's up to that module to redeploy any modules/verticles that may have been deployed under it. Essentially, HA applies to the Vert.x instance, so if that instance fails then the top level module deployed in that instance will be redeployed in another instance. Does that make sense?

As for setting HA groups programmatically, someone else will have to answer that one :-)

And as for the deployment ID, if you want separate deployment IDs for separate module/verticle instances, you can just deploy instances separately (one at a time) and thus the resulting deployment ID (from deployModule or deployVerticle) is unique to that instance.

Jordan Halterman

[Zachary Pry]

Hello Jordan,

Thank you for your response.

Yep, it makes sense to me. Such organization seems logical (if I understood it correctly) and I will create logic to accommodate those requirements.
In such case there is no actual need to set and configure HA programmatically.

But one question remains then - how, within programmatically deployed instances, restart/stop specified ones. For example if (via Container) I've started: feed, mail and contacts verticles but I want to restart mail verticle only (e.g. due to malfunction: memory leak issues, unusually long responses, or simply scheduled restart)... continuation of that - below...

And as for the deployment ID, if you want separate deployment IDs for separate module/verticle instances, you can just deploy instances separately (one at a time) and thus the resulting deployment ID (from deployModule or deployVerticle) is unique to that instance.

 

I looked at http://vertx.io/api/java/org/vertx/java/platform/Container.html and deployModule and deployVerticle are returning void; I still don't see where I can get deploymentId, am I looking in the wrong place ?

[Jordan Halterman (kuujo)]

Module/verticle deployment is asynchronous, so the deployment ID is not available to be returned when the method call is made, but instead it is provided after the module or verticle is deployed using the async result handler.

So, for instance:

// Deploy a verticle
container.deployVerticle("SomeVerticle.java", new Handler<AsyncResult<String>>() {
  public void handle(AsyncResult<String> result) {
    // If the verticle was successfully deployed
    if (result.succeeded()) {
      // Get the deployment ID
      String deploymentID = result.result();

      // Undeploy the verticle using the deployment ID
      container.undeployVerticle(deploymentID);
    }
  }
});

That's assuming you're using Java, but in the other language implementations the deployment ID is provided in the same manner, whether it's a closure in Javascript and PHP or a function in Python.

Jordan Halterman

[Jordan Halterman (kuujo)]

By the way, I've actually worked on some projects that manage deployments as well. You can see some example code in Via for instance:

https://github.com/kuujo/via/blob/master/src/main/java/net/kuujo/via/ViaNode.java#L616

This is the section where modules and verticles are deployed by Via. When a module or verticle is deployed, I check whether the deployment was successful, and if it was I store the deployment info including the Vert.x deployment ID in a map. Then, when the module or verticle needs to be undeployed, I reference that deployment info here:

https://github.com/kuujo/via/blob/master/src/main/java/net/kuujo/via/ViaNode.java#L817

Hope that helps

Jordan Halterman

On Wednesday, February 5, 2014 12:38:32 PM UTC-8, Zachary Pry wrote:

[Jordan Halterman (kuujo)]

By the way, in relation to your original post, as for as running modules in HA mode, Via provides similar features as far as failover. Via allows you to deploy modules/verticles on different nodes over the event bus. If a Via node dies then it's deployments are reassigned. This differs from the Vert.x command line option in that it spreads deployments across a cluster and redeploys failed instances separately. It was actually designed for deploying Vertigo (http://github.com/kuujo/vertigo) networks across a Vert.x cluster.

Maybe Via can give you some ideas for alternatives in general. The development version (0.3.0) is a fault tolerant implementation that's not stable and may be pretty complicated, but 0.1.0 is pretty simple and a good example of deployment management.

Jordan Halterman

[Zachary Pry]

It was a great help Jordan, thank you very much.

I will definitely investigate project you mentioned...

[santo...@gmail.com]

I also implemented basic auto-redeployment of modules in my application, but based on file change detection.

More specifically, I wrote a verticle that "listens" for file changes (add/modify/delete) in specified directories, based on the java 7 nio package.
When such a file change is detected, it is send to a specific address on the eventbus.
The main module that deploys all other modules in my applicaiton registers a handler on that address and reacts to those events by deploying, redeploying or undeploying the related modules.
For this to work, I register all deployment ID's in a map, as Jordan already suggested.

There was also another discussion in this group about the same topic, which might be of help as well:
https://groups.google.com/forum/#!searchin/vertx/json$20from$20file/vertx/z0nD4eYCa2s/hZLJ6Ggh7BoJ

Santo


Op donderdag 6 februari 2014 10:32:20 UTC+1 schreef Zachary Pry:

[Tim Fox]

On 06/02/14 19:40, santo...@gmail.com wrote:

I also implemented basic auto-redeployment of modules in my application, but based on file change detection.

More specifically, I wrote a verticle that "listens" for file changes (add/modify/delete) in specified directories, based on the java 7 nio package.
When such a file change is detected, it is send to a specific address on the eventbus.
The main module that deploys all other modules in my applicaiton registers a handler on that address and reacts to those events by deploying, redeploying or undeploying the related modules.

I'm curious as to why this is necessary and is it sounds very much like what Vert.x does out of the box with the auto-redeploy functionality....

--
You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[santo...@gmail.com]

The main reason is that vert.x automatically redeploys the whole application, as I use a parent module that starts all other modules.
With my own implementation I can keep my application running while just redeploying the changed module.

Also I can easily undeploy a module on one system and redeploy it on another without having to stop the whole application.
Not sure this can be done with the out of the box auto-redeploy functionality of vertx?

Santo

Op vrijdag 7 februari 2014 09:17:42 UTC+1 schreef Tim Fox:

[Tim Fox]

Ah, but in the general case, you can't assume that an application will continue to run correctly after the restart of a single module as opposed to the entire application. For instance, a particular module might do something in it's startup that the app won't like if it's done twice, or perhaps the deployer has kept note of the deployment id of a module - if you redeploy it, that deploy id will change and the one the deployer has will no longer work. That's why the only easy/safe way to do this is to redeploy the entire app :)

[santo...@gmail.com]

I understand your concern, but I only do this for my own custom modules and I know I shouldn't do anything fancy in there that can cause issues with the rest of the application ;-)
Regarding the deployment id's, I keep them in a map and remove/update them when un/re-deploying a module.

Santo


Op vrijdag 7 februari 2014 10:51:14 UTC+1 schreef Tim Fox:

[Tim Fox]

On 07/02/14 10:00, santo...@gmail.com wrote:

I understand your concern, but I only do this for my own custom modules and I know I shouldn't do anything fancy in there that can cause issues with the rest of the application ;-)
Regarding the deployment id's, I keep them in a map and remove/update them when un/re-deploying a module.

That sounds like a reasonable approach :)

[dennyd...@sina.cn]

Hi Jordan,

I can not find the page about "Via", could you please tell me the new page?

Thank you.