Next step for Colony documentation

[Tiago Silva]

One of the hardest things in writing documentation is without a shadow of a doubt, figuring out what the end-user frustrations and sticking points will be. Since we are working with Colony every single day, we often assume end-users know things that they actually don't. 

Therefore, I'd like to know what you currently find is your greatest pain in either understanding what Colony is, what Colony is for, how can you get started, and how you can you develop your project with it.

This will help me understand what is the 20% of missing documentation that will make 80% of the difference, and help give a quick boost to the docs.

[Bruno Costa]

Hey y'all,

My experience with the framework is still limited, so this is my very first small contribution.

Following the quick starting guide, the first time I tried to run Colony, I already had a service running on port 8080 (Colony Web Server's default port). A quick grep helped me find the configuration file used for that and I changed it. I would suggest a easier way to do that (e.g. '$ colony -p 8090') or at least a info in the guide with the location of the configuration file.

Do you think that could be helpful for other first time users?

--
Bruno Costa | Co-Founder

hardquarters, freshly cooked code served with style.

[Luís Martinho]

Grep is always a good way to get out of a sticky situation :)

Colony configuration is not really well documented at this point. Probably we could be more detailed, especially for other first time users as you said.

Regarding the CLI arguments you can't really do "colony -p XXXX" simply because the port argument is not something Colony would take, but rather the Http Service Main Plugin would.

This is because you could launch a Colony instance without any HTTP plugin. You could also launch a Colony instance with several plugins that bind to a TCP port.

What would work would be something like "colony -pt.hive.colony.plugins.main.service.http:port XXXX" or something.

What do you think?

Anyway the configuration is going to have to be reworked in the near future, to make it more flexible and powerful.

This may include other CLI arguments but is especially focused on the configuration file structure.

Ideas will be very welcome. :)

--
Luís Martinho
lmar...@hive.pt
+351 93 405 72 40

[João Magalhães]

Just to add some context.

Configuration is one of the most important (and oldest) parts of colony. There are some major flaws in it that make it not flexible enough to provide the required support for the distribution part of colony.

In the current/next month it's going to be completely redesigned to make it more flexible (eg: better support for multiple configuration file formats) and more portable (easy to "install" configuration in a colony instance).

We're are more than open to receive any suggestion or complain regarding configuration as we hope to make right (forever) this time :).

I hope this helps everyone understand the reason for the configuration infra-structure redesign.

[Markus Gattol]

- what are the flaws (short enumeration)?
- what formats are you thinking of?
- I find what Mongrel2 does interesting; use sqlite but write/read
to it from any language binding to sqlite there is.

[Markus Gattol]

> Therefore, I'd like to know what you currently find is your greatest pain in
> either understanding what Colony is, what Colony is for, how can you get
> started, and how you can you develop your project with it.

Imo how events are used and work should be documented e.g. from the
point where a plugin sends an event, the trail through the codebase
(all bits and pieces touched; modules, classes, functions, decorators,
etc.) it takes (into the plugin manager and back out to any plugin
that is a subscriber). I think understanding how this works (what
classes/functions/decorators are involved and how) is a must for
anyone who wants to hack colony itself and probably also anybody who
writes a plugin. Some text, a few code snippets, all spiced up with a
nice infographic ...

[Markus Gattol]

Got another one: How does one plug his favorite WSGI and web server
into Colony. 9 out of 10 folks will want to use proven httpd for the
very same reasons no one runs e.g. Django's built-in httpd in
production.

Maybe to extend on that bit, a howto using
- Colony
- http://projects.unbit.it/uwsgi/wiki/Mongrel2
- Mongrel2
- optional: ZeroMQ to e.g talk to MongoDB (but that's probably worth
a tutorial on it's own)

since that's what I would consider a truly scalable solution stack
these days, add ZeroMQ and you got no limits on scaling out any tier
(logicl/data/presentation tier).

[João Magalhães]

It's quite difficult to explain all the technical changes that we want to make to configuration without an extensive knowledge of the inner working of the framework.

The overall picture is that the configuration files are currently attached to the resource manager plugin (pt.hive.colony.plugins.resources.resource_manager), this is extremely negative for a scenario where configuration must be distributed across a grid of colony instances. We want to extract it to a more neutral place, the plugin meta directory and then interpreted at runtime by the resource manager. This will require the creation of the CCX (colony configuration) as the configuration equivalentt of the CBX (colony bundle) and CPX (colony plugin) files.

The other major problem is that the resource manager plugin currently supports format extensibility on the kind of configuration that handles (JSON, YAML, XML, etc.), the problem is that the adaptation of the inner structures of the configuration files is not exposed in the target language in a simple uniform way so that the resource manager can apply templates to any of the fields. This one is quite complicated, I hope I've explained in a simple way.

Keep in mind that these changes are going to occur slowly during the next one/two months.

[João Magalhães]

Completly agree, we need to provide WSGI and FCGI integration with colony, and then some how tos on integrating it with known web servers (nginx, cherokee, mongrel, etc).

We had (a long time ago) integration with apache through mod_python and we intend to provide integration with 3rd party web server in the near future, but for know that must be considered a feature request and must be thrown for at least a 2-3 month time frame.

As for the ZeroMQ it would be a new feature to be added to colony and must be discussed as such. As explained by Luis before there are many things in ZeroMQ that match features of colony distributed.

[Markus Gattol]

Sure, good things take time but the important bit is that you can
start a process by starting to discuss matters. Now, I've looked quite
hard but couldn't find a roadmap page for colony (there was a posting
here on the list, but that's all). Maybe we should have some wiki page
where we could put things like

- ZeroMQ
- WSGI
- etc.

even when it's only put there for initial discussion of the matter.
Also, folks would be able to see which feature has priority over
another etc. In general I find the two most important things people
are looking for first with any FLOSS project are

- a FAQs page
- then the roadmap,
- followed by a quickstart tutorial.

So the only thing that's still lacking with Colony is a wiki roadmap
page :)

[João Magalhães]

You're right we should have a roadmap / timeline page where we explain the next steps for the development of the Colony Framework.

We have been discussing the roadmap (internally) for quite some time now, we need to figure it out in more detail and than post it there.

We'll try to get that up and running as fast as possible.