Cast roadmap and plans for the next release

[Tomaž Muraus]

First preview version has just been release and now it is time to talk about the roadmap and plans for the next release.

For the next release (0.2.0) we will mostly focus on fixing the bugs, adding some new features and working on improving a single server experience.

Here is a short list of tasks on which will work for the 0.2.0 release (some of them are "infrastructure" related):

• improving the current build system and "dist" target
• improving the install script
• adding more integration tests for the Cast client
• improving our continuous integration and build system to produce nightly tarballs and adding more integration tests so we can be sure that the install scripts works properly on all the supported platforms
• improving a single server experience which will probably mean modifying the client to make some actions more accessible and easy to use
• refactoring the REST API and making all the actions which are currently accessible through the API also accessible directly in a clean way through the code
• a plugin system. This task is a pretty big one and directly related to the task above so I will post a new thread with more details in the upcoming days.
• a plugin for integrating Cast with git repositories and github (more about this in the upcoming post)

After we finish the 0.2.0 release we will focus on cluster capabilities and multi-node configuration, but more about this when we finish the next release.

[Paul Querna]

2011/4/26 Tomaž Muraus <ka...@k5-storitve.net>:

Other tasks that might be a little more detailed:
- Switch to Express for the HTTP server layer. (When we started it
wasn't clear express would win, it has, it wil give us long term wins)

- Use node-swiz for Serialization & Validation in the API layers
(working to get it open sourced right now)

- Solidify the CA / multi-node CA situation / configurations.

[Tomaž Muraus]

Here are a few more tasks targeted for the 0.2.0 release.

Core:

• Refactor code which still references the old sys module to use the new util one
• Refactor code which uses sys.pump to use the new pipe function
• Port the tests to Whiskey 0.3.0 (Wiskey 0.3.0 should be released this week)

Website, Documentation:

• Fix the heading tags (we should use the h1 tag for the main heading on every page) and make the website more SEO friendly
• Update the "Get started" page to include the link to the Installation page and replace the Redis example with the example which shows how to create a NodeJS application instance and how to upgrade it
• Add "Get it" button to the homepage which links to the Downloads page
• Add "social" widget to the homepage which shows latest tweets mentioning Cast

[Russell]

I've been thinking about the API refactoring (and discussing it with Tomaž in #cast-project), and what I'd like to do is:

1. Move all non-http logic out of lib/services/http/ into lib/control/ or somewhere.

2. The new code in lib/control/ should return (or pass back via a callback) "Job" objects on all calls that modify agent state (ie, anything not currently behind a GET). Jobs should go into a queue corresponding to the resource they operate on. In this case our top level resources are bundles, instances, and certificate signing requests. Logically services belong to instances, so service operations should go into a queue based on the instance they belong to (Tomaž and I were discussing modifying the public API to reflect this, but for now we want to limit the scope of our changes).

Job objects should be EventEmitters that emit:

'queue' - Emitted when a job validated, given a globally unique 'id' and assigned to a queue.

'error' - Emitted if the job fails due to an error.

'message' - I'd like to be able to emit 'message' events so we can provide users feedback on what is actually happening on the agent.

'success' - Emitted when the job succeeds.

I've gone back and forth on whether jobs should be returned directly (implying no validation of the request, since most validation would require async calls) or passed back via a callback. Node's APIs all seem to pass back event emitters directly, even if they will be useless due to an error (see net.createConnection), but on the other hand it would be nice to, for example, 404 requests to instances that don't exist.

The solution would be to, for the HTTP API, not respond to the request until the Job emits a 'queue' event. At that point the Job has an 'id', so we return either the id or the entire serialized job (id included).

3. At this point someone is free to branch off and work on the plugin system which will basically give user plugin code access to the 'control' API.

4. The HTTP layer should be reworked to use node-swiz. If we want to switch to Express, this would probably be where we should do it.

5. A new HTTP API (and corresponding 'control' API) should be added for accessing jobs. Given a job's id, callers should be able to retrieve the current state of a job, "tail" it for "messages" (ie, "GET /1.1/jobs/102/messages?next=0" should return a list of 1 or more messages as soon as at least 1 is available), or wait for it to complete ("GET /1.1/jobs/102/result").

Obviously the Jobs stuff would be quite a bit of work, but I think would give us a lot of good things. We can stream messages back to the user (its nice to have UI feedback for long running jobs), our API becomes "non-blocking" (personally I don't care about this, but I know Tomaž and Paul expressed an interest), but the "tail job" and "wait for job to complete" calls eliminate the need for polling. Plus it solves a lot of locking problems.

Anyway, curious to know what others think of this, if anyone has suggestions/improvements, etc.

-Russell