Re: Can we dynamically deploy verticles with vertx 3?

[Jez P]

I don't know about dependency jars (I've never tried dynamically adding the classes within a jar to the current classloader) but certainly dynamically deploying verticles is possible, the (commercial) project I'm currently doing does exactly this. 

However, with the fact that the vert.x 3 classloader setup is now very straightforward, if you can add the classes within a jar to a normal running Java application, there's no reason why you shouldn't be able to do the same in vert.x 3. 

I guess what I'm saying is there are two separate orthogonal parts to your question, the second I know the answer to but the first i don't. You might need to exert some control over naming (both of the event bus address they register at and the classnames themselves).

I think OSGI might have some dynamic capabilities - maybe that's worth a look to see if that solves the dynamic loading part of the problem?

On Friday, March 6, 2015 at 12:28:52 AM UTC, javadevmtl wrote:

I'm working on some sort of platform where we would like to upload and deploy verticles and have them register on the event bus so we can send to them as some type "service"

Is it possible in vertx 3 to deploy a verticle and any dependency jars dynamically?

[Julien Viet]

Vert.x 3 provides a dynamic service resolver that loads from the classpath using a descriptor.

service:com.mycompany:cleverdbservice

in addition it can also deploy from Maven repositories and download the jar to deploy (using the maven service factory) using the GAV as service value.

this way you have something very dynamic.

-- 
Julien Viet
www.julienviet.com

--
You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Jez P]

That looks awesome, Julien

[javadevmtl]

So the nomment you call service:... it looks for it in maven repo? And add it to the class path or you have to add it to class path?

[Julien Viet]

it will not be added to the classpath of the JVM (otherwise that would be kind of magic)

 the archive will be fetch from an maven repository (using Aether) and used to create a new classloader to load the service.

look here : https://github.com/vert-x3/vertx-maven-service-factory

the code is fairly straightforward to understand : https://github.com/vert-x3/vertx-maven-service-factory/blob/master/core/src/main/java/io/vertx/maven/MavenVerticleFactory.java

-- 
Julien Viet
www.julienviet.com

On 6 Mar 2015 at 14:41:58, javadevmtl (java.d...@gmail.com) wrote:

So the nomment you call service:... it looks for it in maven repo? And add it to the class path or you have to add it to class path?

[javadevmtl]

Are services akin to modules now?

I assume it will download the service from the repo and all it's dependencies which can be other vertx services or 3rd party jars?

[javadevmtl]

Btw when we are speaking of dynamic we are saying that we already have a vertx application running and we want to deploy new service on top of it with out restarting the application.

[Julien Viet]

that should work.

you can use this service locator in the vertx.deployVerticle programmatic operation.

-- 
Julien Viet
www.julienviet.com

[javadevmtl]

As an example....

vertx.deployVerticle("service:io.vertx:vertx-jdbc-service:3.0.0-milestone2");

This downloads the dependencies in my local maven repo but then the following exception gets thrown...

io.vertx.core.VertxException: Cannot find file io.vertx.vertx-jdbc-service.3.0.0-milestone2.json on classpath

at io.vertx.service.ServiceVerticleFactory.resolve(ServiceVerticleFactory.java:53)

at io.vertx.maven.MavenVerticleFactory.resolve(MavenVerticleFactory.java:157)

at io.vertx.core.impl.DeploymentManager.doDeployVerticle(DeploymentManager.java:124)

at io.vertx.core.impl.DeploymentManager.deployVerticle(DeploymentManager.java:96)

at io.vertx.core.impl.VertxImpl.deployVerticle(VertxImpl.java:526)

at io.vertx.core.impl.VertxImpl.deployVerticle(VertxImpl.java:513)

at com.wp.Main.main(Main.java:14)

[javadevmtl]

Julien it's abit confusing. I read the docs and from your comment....

"Vert.x 3 provides a dynamic service resolver that loads from the classpath using a descriptor."

That means that the .json has to be preloaded with the application.

I guess an analogy of what I'm trying to create would be what stored procedure are to an sql server.

So the SQL Server is the container and stored procedure is the service.

I can create stored procedure on the sql server and run them without having to restart the sql server.

Then I can connect to the SQL server using my JDBC driver and call any particular stored procedure.

I can also delete any stored procedure any time I like.

We are attempting to build this type of functionality using the new isolation group and setExtraClassPath() functions of vertx3.

Right now we can upload verticles to a folder using HTTP POST and we can load them using the extraClassPath and they become available to the event bus.

Just wondering if we should build all this or try to use vertx 3 service mechanism.

[Julien Viet]

The maven verticle factory loads the classes (and its dependencies) via Aether and then creates an URLClassLoader to create the service from it.

so it does not have to be in the classpath but rather in the maven repository (you can configure this maven repository URL) and then you deploy a particular version using maven coordinates : groupId, artifactId, version.

so a scenario might that while your application is running, a service is deployed in a maven repository and then you ask your application to deploy the verticle that correspond to this maven coordinate.

-- 
Julien Viet
www.julienviet.com

[javadevmtl]

OK but when programmatically calling deployVerticle () using maven service factory an exception is thrown saying it can't find the .json

[Julien Viet]

it needs a json descriptor indeed.

-- 
Julien Viet
www.julienviet.com

On 6 Mar 2015 at 23:55:37, javadevmtl (java.d...@gmail.com) wrote:

OK but when programmatically calling deployVerticle () using maven service factory an exception is thrown saying it can't find the .json

[javadevmtl]

So how is that dynamic anymore?

[Julien Viet]

maybe we are not discussing the same thing :-)

what means “dynamic” for you ? can you give a example / use case explaining it ?

-- 
Julien Viet
www.julienviet.com

[javadevmtl]

Ok so I have an existing vertx application running and I want to add new functionality to it with out stopping it.

Just like dropping a WAR file into a web container. It's possible to with verticles and a bit of classloader tricks, I was just hopping to use existing service infrastructure of vertx3. If not custom soultion can be built.

[Julien Viet]

you can install several versions of the same service in the maven repository and then use a different verticle name when you deploy the verticle at runtime.

-- 
Julien Viet
www.julienviet.com

[javadevmtl]

That;'s fine. but you still require the json descriptor so that means I need to stop my application provide the json so it can be put in classpath and then restart the application. Or am i missing something?

[Tim Fox]

On 09/03/15 15:34, javadevmtl wrote:

That;'s fine. but you still require the json descriptor so that means I need to stop my application provide the json so it can be put in classpath and then restart the application. Or am i missing something?

Yes. The json descriptor is owned by the service, not the service user.

In Vert.x 2.0 terms you can think of it like the mod.json file in the module. It's something provided by the module author.

[Tim Fox]

On 09/03/15 15:37, Tim Fox wrote:

On 09/03/15 15:34, javadevmtl wrote:

That;'s fine. but you still require the json descriptor so that means I need to stop my application provide the json so it can be put in classpath and then restart the application. Or am i missing something?

Yes. The json descriptor is owned by the service, not the service user.

This might help https://github.com/vert-x3/vertx-service-factory/blob/master/README.md

[javadevmtl]

Hi Tim thank i read that! But what I'm trying to say is as a service user I need to stop myself add the json descriptor to my classpath restart myself and then ask to load the service.

[Tim Fox]

On 09/03/15 15:44, javadevmtl wrote:

Tim thanks I read all that. But I guess we cannot get away without stopping, putting the json in classpath and restarting.

Yes, you can.

You could either use the maven-service-factory as Julien mentioned.

Or you could deploy your own stuff dynamically using isolation groups.

Basically Vert.x 3 is very flexible in this area.

[Tim Fox]

On 09/03/15 15:47, Tim Fox wrote:

On 09/03/15 15:44, javadevmtl wrote:

Tim thanks I read all that. But I guess we cannot get away without stopping, putting the json in classpath and restarting.

Yes, you can.

You could either use the maven-service-factory as Julien mentioned.

Or you could deploy your own stuff dynamically using isolation groups.

https://vertx.ci.cloudbees.com/view/vert.x-3/job/vert.x3-website/ws/target/site/docs/vertx-core/java/index.html#_verticle_isolation_groups

[javadevmtl]

oh ok then maybe this is where the confusion stems from....

I tried to deploy the JDBC service from maven repository and an exception got thrown...

public static void main(String[] args) {

Vertx vertx = Vertx.vertx();

DeploymentOptions options = new DeploymentOptions().setInstances(1);

vertx.deployVerticle("service:io.vertx:vertx-jdbc-service:3.0.0-milestone2",options);

}

SEVERE: Cannot find file io.vertx.vertx-jdbc-service.3.0.0-milestone2.json on classpath

io.vertx.core.VertxException: Cannot find file io.vertx.vertx-jdbc-service.3.0.0-milestone2.json on classpath

[Tim Fox]

On 09/03/15 15:53, Tim Fox wrote:

On 09/03/15 15:47, Tim Fox wrote:

On 09/03/15 15:44, javadevmtl wrote:

Tim thanks I read all that. But I guess we cannot get away without stopping, putting the json in classpath and restarting.

Yes, you can.

You could either use the maven-service-factory as Julien mentioned.

Or you could deploy your own stuff dynamically using isolation groups.

https://vertx.ci.cloudbees.com/view/vert.x-3/job/vert.x3-website/ws/target/site/docs/vertx-core/java/index.html#_verticle_isolation_groups

The Maven service factory stuff internally uses isolation groups to do its thing.

But if you didn't want to use Maven, say you just wanted to load stuff dynamically by dropping something in a folder, then you can write something that scans a folder for changes and when it sees something it does a vertx.deployVerticle() specifying an isolation group where you can add the extra classpath for whatever it is you are loading.

[Tim Fox]

I'd guess you haven't added the maven service factory to your classpath, but without seeing your project I'm just guessing.

[javadevmtl]

I'm pretty sure I have... I'll check a bit later...

[Tim Fox]

If you can push something to GitHub I can take a look, but probably not
for a while as I will be travelling shortly :)

On 09/03/15 16:25, javadevmtl wrote:

[javadevmtl]

Can't do github but it really is dead simple...

public class Main {

public static void main(String[] args) {

Vertx vertx = Vertx.vertx();

DeploymentOptions options = new DeploymentOptions().setInstances(1);

vertx.deployVerticle("service:io.vertx:vertx-jdbc-service:3.0.0-milestone2",options);

}

}

And the build.gradle is...

apply plugin: 'java'

apply plugin: 'eclipse'

sourceCompatibility = 1.8

version = 'XYZ'

jar {

    manifest {

        attributes 'Implementation-Title': '...', 'Implementation-Version': version

    }

}

repositories {

    mavenCentral()

maven {

        url = 'http://oss.sonatype.org/content/repositories/snapshots/'

}

}

dependencies {

compile group: 'io.vertx', name: 'vertx-maven-modules', version: '3.0.0-SNAPSHOT'

compile group: 'io.vertx', name: 'vertx-core', version: '3.0.0-SNAPSHOT'

}

[dgo...@squaredfinancial.com]

At least when using vertx-maven-service-factory as a dep, I do see that it does seem to be adding the service's jar retrieved from maven to the extra classpath and so forth.

Is this just naming convention discrepancies in vertx-jdbc-service in particular ? its embedded service descriptor file in the jar seems to be named io.vertx.vertx-jdbc.json instead of the io.vertx.vertx-jdbc-service.json that the service factory is looking for, so ...I guess it can't find it?

[Tim Fox]

Yes, that looks like it. I've now made the service name consistent with
the way the other services are named.

>
>
>
>
>

[dgo...@squaredfinancial.com]

On Friday, 13 March 2015 11:13:47 UTC, Tim Fox wrote:

Yes, that looks like it. I've now made the service name consistent with
the way the other services are named.

just minor type/thinko: I don't think there's any logic to coalesce the maven groupId tail and artifactId head when constructing the service identifier (and IMO there probably shouldn't be, seems a bit "clever"), so I think it has to be (admittedly redundant looking but it's just a degenerate case):

io.vertx.vertx-jdbc-servce.json

but at time of writing you've still got io.vertx.jdbc-service.json

so I'm still seeing:

Mar 13, 2015 5:16:19 PM io.vertx.core.impl.DeploymentManager
SEVERE: Cannot find file io.vertx.vertx-jdbc-service.json on classpath

[dgo...@squaredfinancial.com]

Turns out quite a few of the 1st party vertx service descriptors are currently named that way, in contrast e.g. com.englishtown.vertx.vertx-solr-service.json gets it "right" - i.e. what the maven service factory currently expects.  I note e.g. the felix project's maven plugin for osgi bundles actually applies a sort of groupId/artifactId tail/head coalescing for naming in some places which sounds similar to the mental rule people have presumably been using when naming. Beh, still seems dangerously "clever" to me...

But a more tricky issue arises:  at least one of the source trees looks like they end up (possibly quite reasonably) having several service descriptors (e.g. vertx-mysql-postresql-service) inside artifacts, thus an assumption of a 1:1 maven GAV coord correspondence may have been broken some time ago. Unless copies are uploaded multiple times under all the different coords...
 
I dunno, the intra-jar service descriptor, while arguably DRY and enough (and already working ...if the naming is right),  may thus prove a  problematic approach, one could consider also maintaining detached service.descriptor.json copies outside jars i.e. upload service descriptions themselves to maven repos under "vertxservice" (or whatever) classifier - and have the maven service factory discover based on those. (mentioned something like that a fair while back on vertx-dev)   i.e.  "another level of indirection solves everything (except too many levels of indirection)".

[dgo...@squaredfinancial.com]

FWIW, stuck the test code I was using for issue here: https://github.com/dgoldensquared/scratch-vertx3-mvnsvc-test

(so, just to confirm: dynamic service deployment from remote maven repos does work when the names match properly)

[Tim Fox]

On 13/03/15 19:49, dgo...@squaredfinancial.com wrote:

Turns out quite a few of the 1st party vertx service descriptors are currently named that way, in contrast e.g. com.englishtown.vertx.vertx-solr-service.json gets it "right" - i.e. what the maven service factory currently expects.

Right, we should fix this them. I think what happened is they were all correct at one point, then we changed the naming conventions for the maven artifacts but forgot to change the json servicee descriptor name :(

I note e.g. the felix project's maven plugin for osgi bundles actually applies a sort of groupId/artifactId tail/head coalescing for naming in some places which sounds similar to the mental rule people have presumably been using when naming. Beh, still seems dangerously "clever" to me...

But a more tricky issue arises:  at least one of the source trees looks like they end up (possibly quite reasonably) having several service descriptors (e.g. vertx-mysql-postresql-service) inside artifacts, thus an assumption of a 1:1 maven GAV coord correspondence may have been broken some time ago.

A couple of possible solutions to this:

1. For those projects that contain more than one service, they could be refactored to create more than one artifact.

2. Allow the ability to override the service name when deploying

E.g. normally one would do:

vertx.deployVerticle("service:com.mycompany:myartifact:1.0");

and this would install the artifact then look for a file com.mycompany.myartifact.json on the CP.

However if one did:

vertx.deployVerticle("service:com.mycompany:myartifact:1.0::serviceA");

this would install the artifact then look for a file com.mycompany.serviceA.json on the CP.

Unless copies are uploaded multiple times under all the different coords...
 
I dunno, the intra-jar service descriptor, while arguably DRY and enough (and already working ...if the naming is right),  may thus prove a  problematic approach, one could consider also maintaining detached service.descriptor.json copies outside jars i.e. upload service descriptions themselves to maven repos under "vertxservice" (or whatever) classifier - and have the maven service factory discover based on those. (mentioned something like that a fair while back on vertx-dev)   i.e.  "another level of indirection solves everything (except too many levels of indirection)".

On Friday, 13 March 2015 17:50:30 UTC, dgo...@squaredfinancial.com wrote:

On Friday, 13 March 2015 11:13:47 UTC, Tim Fox wrote:

Yes, that looks like it. I've now made the service name consistent with
the way the other services are named.

just minor type/thinko: I don't think there's any logic to coalesce the maven groupId tail and artifactId head when constructing the service identifier (and IMO there probably shouldn't be, seems a bit "clever"), so I think it has to be (admittedly redundant looking but it's just a degenerate case):

io.vertx.vertx-jdbc-servce.json

but at time of writing you've still got io.vertx.jdbc-service.json

so I'm still seeing:

Mar 13, 2015 5:16:19 PM io.vertx.core.impl.DeploymentManager
SEVERE: Cannot find file io.vertx.vertx-jdbc-service.json on classpath

[David De La Harpe Golden (Squared Financial)]

Well, the first can be made work without code changes, but the second, allowing additional final qualifier in addition to the maven-based coords like that, seems quite useful, and also a little bit like the versatile "entry points" in a python distribution or the way an osgi bundle can provide multiple services.

[Marco Ellwanger]

Also how would one deploy a Ruby verticle inside a jar file with the maven-service-factory in vertx-x3? It currently fails after locating and downloading the jar from the repo with ClassNotFoundException on the value specified for the main class in the descriptor as it is likely expecting a Java class to be found on the classpath. Dynamic, language-agnostic module deployment (via runmod or not) was one of the killer features of vertx 2, it would be great to get that support back soon into vertx-x3 (this is likely dependent on the outstanding work on codegen I assume).

cheers

--Marco

[Tim Fox]

On 26/03/15 20:41, Marco Ellwanger wrote:

Also how would one deploy a Ruby verticle inside a jar file with the maven-service-factory in vertx-x3?

Just put your Ruby verticles inside the jar.... but we don't support Ruby in Vert.x 3 (yet)

--

[Marco Ellwanger]

Thanks, Tim. One follow-up question I didn't want to start a separate thread for: Programatically any verticle can deploy other verticles, I was looking to restrict this feature by a gatekeeper, but it seems the only way to achieve that would be to swap out VertxImpl or DeploymentManager with a modified version that guards the entry points for deploying verticles. It would be nice to be able to plug-in classes at Vert.x start-up to add behavior to critical entry points/system functions (with a default no-op implementation). Is anything like this planned?

cheers

--Marco