Collide Plugins

[James Nelson]

I am working on a few collide plugins at the moment, and the closest one to being review-worthy is a gwt compiler / super dev mode plugin.

I have a prototype up and running which simply deploys a verticle, GwtPlugin, to which I send compile request dtos, and I have the compiler forward all output from TreeLogger to vertx eventbus.

So, I can now officially use collide to recompile collide ;-}

But, there are lots of rough edges, and a few design decisions that I would like to address before submitting my branch for review.

a) - Classpath.  For the sake of rapid prototyping, I just added my plugin's dependencies to server runtime, and let it live inside the collide process.

This is less than favorable for a number of reasons: 

• Plugins like compilers might need a lot of memory, and could destabilize the server.
  
• Adding piles of jars for any sort of plugin to runtime classpath will be messy, and could introduce nasty conflicts (especially if plugin uses libs of different version from collide)
• Plugin build process gets tangled into main server build process, which is just plain bad
• I'm looking at either running the plugins as a separate process and using stdin/out to send signals, or using a URLClassLoader to launch plugin tasks in a thread with it's own effective classpath.

b) - Extensibility.  Requiring plugins to be integrated as verticles in the main server runtime would make it harder to drop in a plugin to use without potentially breaking server functionality / startup.

To help other contributors build plugins, we might be wise to setup a basic framework for registering plugins.

• Should plugins be forced to live in a separate process to keep server runtime clean?
• Should plugins have a client-side interface to easily integrate with the run button (and possibly supply a regex to match files which can be run by the plugin)
• Should plugins have a server-side interface / shared dto superinterface for registering with vertx?  Currently I just use an interface with .getAddressBase(), for building eventbus routing table.

c) - Modularity.  We should have a common package structure for organizing plugins.  Perhaps com.google.collide.plugin.your_name_here.client / .server / .shared?

d) - Isolation.  A buggy plugin should never crash the server.  (Adding a reconnect socket button might also be nice for development, but that's another matter entirely)
e) - Reusability.  Dtos and shared code should be accessible between plugins, while keeping the core project free from any dependencies on specific client/server code.

All in all, from my work so far, I would have to say I am most in favor of running plugins isolated from the main server, and simply proxying requests around.

If my compiler plugin overloads memory on my VPS, I want to be able to kill it and reload it without taking down the server.

This could mean something messy like Runtime.exec(), but at least then we could build a single Plugins verticle which handles requests to launch and control any plugin process.

Then, each plugin merely has to implement a sane stdin/stdout protocol, which can exist entirely behind a DtoRouter class of some kind, 

and would encourage developers in any language to implement native plugins.

I could go on, but I think it would be best to get some group opinions on how we should design this feature.

[Scott Blum]

I think I'd have to see what you did to get more context, but in general I would say err on the side of underdesign.  I would want to see several different plugins before I'd have a clear idea of what the commonalities are and what a good plugin model would be.

The very simplest integration would be to deploy a sibling verticle, and serve up its own UI at a different URL.  Vert.x has some design philosophy already in place regarding verticle isolation, classpaths, and communication, and it also has ways to scale up to multiple processes that share the same global event bus, if you really want to move the compile out of process rather than just out of classloader.  I would read up on this a bit.

[James Nelson]

Awesome, thanks for the heads up.

I'll move towards having the plugin integrate it's own dependencies to it's verticle, and just concern myself with the implementation.
I really didn't like the idea of using URLClassLoader or a seperate process anyway. =}

Now, this brings me to my next topic of inquiry...

Generating the bootstrap.js and .collide run scripts.

I'm sure this is possible in ant, but it would be far easier and more extensible to build and export a plugin in maven.

With ant, I'm pretty much forced to put all the jars I need for every module in the project classpath,

but with maven they can be developed and deployed as standalone modules,

and plugins can easily be written to give programmatic access to the xml module configuration

{which would be perfect for scraping config files and generated code into approriate bootstraps}. 

So, I guess the question here is, would adding dependency on maven be too much technical debt?

Even if the project sticks with ant, I will likely just symlink /client, /server, /dto, /clientlibs and /plugins into maven modules.

The allure of having a central repository and extensible, scoped, modular components is simply too much for me to resist. =}

If nothing else, I can just treat collide itself as a single, core module that be built with maven's ant command,

then build and distribute the plugins as pure maven, and maybe export ant build scripts to reduce barrier to entry...

At least then I will never have to worry about leaking dependencies into collide core. ;-}

Collide is already extremely modular, which is great.  

So, if a mavenholic like me wanted to drop it in,

do you think the community would be for or against it?

[James Nelson]

Although, to go back on my previous statement a little...

I think the gwt super dev mode will have to use URLClassLoader no matter what,

since it will be responsible for compiling any configuration of gwt modules you would like to create.

This means using the IDE to select jars and source, so no way to know ahead of time what to send to vertx.

My plan is to build a highly modular set of standalone gwt components which expose their controller as either javascript api or html5 webworker.

The master page will just be a thin layer which hooks up and controls jsni components, each of which can be recompiled and hotswapped without a page reload.

When compiling for production, any modules which do not run in an iframe widget or web worker will just become .runAsync() split points, guaranteed to be wholly isolated.

Best of all, I'm going to rebind GWT.create(java.lang.Thread.class) to generate two artifacts: 

One is a synthetic, single-script-linker js glob that can run the thread independently in a web worker (or iframe, or just in the top window if ie doesn't like either)

The other is a controller class returned by the generator which will interface with the emulated thread in the root host page.

Whatever methods are annotated @Export on your thread become the communication protocol to your web worker,

and the actual Thread methods, sleep(), yield(), etc. will be emulated as best as possible.

I'm currently using AsyncCallback to get all data, which may seem obtuse, but actually allows multiple return types by convention,

so calling worker.calculate(listOfStuff<Dto>, listOfCallbacks<Async<Dto>>, Async<Void> onDone) would stash the callbacks, serialize the rest and send it to the worker in batches.

The web worker environment performs your actual heavyweight calculations, ui thread stays happy, and you can send back responses whenever they are ready. 

This will allow me to properly emulate (and achieve) concurrency in the browser,

so when I write PlayN games, I will be able to run heavy A* without glitching out my redraw,

and actually use ThreadLocal in shared code when that is the behavior I want in the server.

So, for me, collide is the perfect environment to rapidly iterate;

why reload or recompile everything when I can get runtime concurrency, hotswapping, module isolation and tight code splitting?

One-click of a test button will recompile whatever little bit of code you need, run in fresh window as soon as it is ready, immediately run/debug the result.

After all that rant, I'll get to my point:

Maven has the strict classpath management I need to enforce isolation, control dependencies and keep out code bloat,

but there's no reason to require a dependency management toolkit when the current dependencies are java 7 & ant 1.8.4.

So, I am just going to symlink my fork and just make the poms defer to ant tasks until I need to write some Mojos.

In order to fulfil my life-long obsession to write a program which codes, compiles, recompiles and hotswaps itself,

I just need gwt super dev mode, an ant runner, and a maven runner plugged in.

All three of these plugins require essentially the same data:
list of source / jars, property map, and a string command.

The rest is just "perform long and boring task in background, pipe log to user",

Ideally, all of these plugins would just be run targets we could infer from a filename regex,

and the run configuration / dialog / controls can just be an IsEditor<RunConfig>

The whole plugin would look something like:

interface PluginRunner{

String getMain();//target

JsoArray[String] getClasspath();//libs

String getAddressBase();//socket channel, so subclasses play nice

//maybe project id in here or something 

}

class GwtPluginClient implements PluginRunner, IsEditor<GwtConfig>{

Editor<GwtConfig> getEditor();//widget to edit plugin configuration

String getFilenameRegex();//to integrate with run button on selected files

}

class GwtPluginServer extends BusModBase implements PluginRunner{

//compile stuff.  send messages.  be cool.

}

Anyway,

that's my plan.

[Scott Blum]

Hi James,

I might suggest coming at this from the other end.  Instead of worrying about the technical / organizational details, I think people would rather see a sweet demo.  If I were you, I'd just make a clone (http://code.google.com/p/collide/source/clones), and push the minimal thing that runs, however hacky it might be, with instructions on how to run it locally.  A hacked bootstrap.js / collide would be totally fine.  Once people actually see it running, then we can have a more intelligent discussion about organizational details.

I will say tho, none of the original committers know Maven well, so there's probably a bit of an uphill battle there.  There's really not much (at least, on the local system) that you can't do with Ant, it's more a matter of style.

Cheers,

Scott

--
You received this message because you are subscribed to the Google Groups "collide-discuss" group.
To post to this group, send email to collide...@googlegroups.com.
To unsubscribe from this group, send email to collide-discu...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msg/collide-discuss/-/ivdZpzSAaq8J.

For more options, visit https://groups.google.com/groups/opt_out.
 
 

[James Nelson]

Well, it took some hideous code to get the minimal version to run, but I just hacked it in, and it works.

It turns out that vertx already uses URLClassLoader to do it's business,

and what made it difficult was that it loaded libraries which gwt-dev also loads, and causes the classloader to bomb.

So, I wound up using a parentless URLClassLoader to get a thread without vertx dependencies in it,

and then I had to do some fairly ugly reflection to get messages in and out of the event bus...

...but it works!

If I can get everything in and out of the thread to funnel through json, it should be trivial to setup some reflection utils to encapsulate the ugliness...

I'm going to clean up the gui a little, and throw in a menu to select the .gwt.xml file & classpath, and toss it up on one of my linux servers.

Too bad I didn't get preview on a google computer instance; I'll likely host images there once it's public. ;-}