Consider splitting off volofile capability from dependency capability

[James Burke]

Damon Oehlman commented here:
https://github.com/volojs/volo/issues/25#issuecomment-8413344

asking about splitting out the volofile/project automation stuff from
the dependency resolution stuff. This has come up before, and so this
is a thread to talk through what that might look like. This is a bit
rambling, just trying to list out the different things affecting the
decision.

I liked having an easy way to do synchronous-style scripting in node
that had some helpers for some common tasks. I liked using promises
too, since it gave a linear flow to the code. I also wanted to include
these helpers once, in a globally installed package, so that it was
easy to use in a volofile-type of file, and not require each use to
need to npm install a bunch of files in a local node_modules to get
that capability, since coming from my web background, that seemed
silly and bloated to do so.

However, I think that runs a bit counter to how packages in node/npm
are used. Not saying what they do now is good (people pushing a
package's test files to npm is just lunacy), but there is a culture
there. And one of the reasons for trying volo's approach, to something
like grunt[1] was to try to go more with the grain of node/npm instead
of having to do a bunch of adapter modules to use the node stuff
directly.

For reusable volo commands, like volo-ghdeploy[2] it was nice to be
able to reuse the globally installed helpers too. I am wondering
though if it is better to just encourage folks to create node packages
with a 'bin', see uglifyjs as an example, and then get the reusability
that way. I think it is still useful to have a way to call a locally
installed node_modules/.bin over the global script though, so I like
the 'n.' support in volofiles[3] for that.

The 'v.' support for just doing strings for a sequential list of
async-completed strings is also nice. Maybe it just means encouraging
shelljs for folks instead. Getting windows support without needing
cygwin is a big driver for this automation functionality.

The other reason for including the automation piece with a globally
installed package, and inside volo in particular, it is nice to have
the onAdd: and onCreate: hooks as part of `volo add` and `volo create`
to allow the newly fetched dependency/project template to do some work
after getting fetched. Maybe this can be accomplished by a
volo.scripts area in the package.json that just runs plain node
scripts though.

That is it for now on the brain dump. Feedback welcome.

James

[1] https://github.com/volojs/volo/wiki/volo-and-grunt
[2] https://github.com/volojs/volo-ghdeploy
[3]https://github.com/volojs/volo/wiki/Creating-a-volofile
[4] https://github.com/arturadib/shelljs

[Damon Oehlman]

Hey James,

Thanks for the brain dump - it certainly helps provide some background on the architecture of volo.  I'm definitely one of those who has become quite accustomed to the "Node way" of doing things, and can see the advantages of most of the patterns encouraged in the node.js ecosystem.  Like you said, there is some bloat, but in general I find it's not too bad.  With regards to promises vs callbacks, I'm happy to go either way, but mostly use the node callback pattern most in my own code.  

I guess from my side, I'm interested in volo primarily due to the thought process and implementation you have put in behind the "volo add" functionality. This in my opinion is the part of volo that should be talked about, promoted and used to influence change in the way people both create and consume JS libraries (and other web assets).  Other package management solutions are doing a very poor job of this (IMO), either because they haven't implemented a good strategy for dealing with versioning (bower I think falls into this category) or require a separate dedicated package repository (or use NPM) for resources that can be consumed via either github or the web already.

From having a look through the volo codebase, in my mind I can see two strong candidates for separately managed node modules:

• Firstly, I'd love to see what I will call the "fetch" functionality pushed into a separate package (perhaps volo-fetch).  While I know you prefer a promises abstraction, I would definitely encourage this project to embrace more of a callback style pattern and utilize existing modules such as async, request, etc for it's implementation.   The volo-fetch package in this form, would easily trump bower and I think we could hit Addy and Paul up to consider it for inclusion into tools like Yeoman over bower.  The reason I suggest using the callback style here is to make it more comfortable for the existing node community to embrace.
  
  Truth be told, I was thinking of replicating the volo fetch approach as part of the buildJS suite as "slurp": https://github.com/buildjs/slurp, but decided it made much more sense to get involved with volo instead...
  
  
• Secondly, as per my original comment in the issue, I'd separate out the automation functionality into it's own package.  It's useful in it's own right (with a little crossover perhaps with tools like Jake - https://github.com/mde/jake).  I think your suggestion of using shelljs is a great idea.  

I'm happy to jump on Skype anytime if you are keen to have more of a chat about any of this, but also respect that you may already have a roadmap in your own mind about how volo should evolve.  I'm definitely keen to contribute regardless.

Thanks again for your hard work on Volo :)

Cheers,

Damon.

[James Burke]

On Tue, Sep 11, 2012 at 4:47 PM, Damon Oehlman <damon....@gmail.com> wrote:
> Thanks for the brain dump - it certainly helps provide some background on
> the architecture of volo. I'm definitely one of those who has become quite
> accustomed to the "Node way" of doing things, and can see the advantages of
> most of the patterns encouraged in the node.js ecosystem. Like you said,
> there is some bloat, but in general I find it's not too bad. With regards
> to promises vs callbacks, I'm happy to go either way, but mostly use the
> node callback pattern most in my own code.

I don't mind exposing the public interface in a way that it could be
used with Node's error first callback style, but I do not plan on
using that style internally. It should be enough to modify the top
level main function to do the node style:

https://github.com/volojs/volo/blob/master/volo.js#L20

but then also pass back a promise if the consumer wanted to use it in
a promise style.

> I guess from my side, I'm interested in volo primarily due to the thought
> process and implementation you have put in behind the "volo add"
> functionality. This in my opinion is the part of volo that should be talked
> about, promoted and used to influence change in the way people both create
> and consume JS libraries (and other web assets). Other package management
> solutions are doing a very poor job of this (IMO), either because they
> haven't implemented a good strategy for dealing with versioning (bower I
> think falls into this category) or require a separate dedicated package
> repository (or use NPM) for resources that can be consumed via either github
> or the web already.

It is possible to use just the volo in another package, see this as an example:

https://github.com/volojs/ghvolo

Hopefully that can be a starting point if you want to just consume
volo as part of another package.

So what about this path:

* Update the main volo.js file to match what style you think works for
using the volo stuff as a component. Also, if the style of reuse via
ghvolo seems awkward in some ways, lets fix those things.
* Work out a volo.scripts approach that gives the onAdd and onCreate
capability but just by running plain node scripts. so maybe
volo.scripts.add and volo.scripts.create.
* Try converting something like
https://github.com/volojs/volo-ghdeploy to be just a plain node
package with a script file. See what the iceberg effect is for its
dependencies.
* Separate out the volofile work as a separate package that volo can consume.

If that stuff worked out, then I could see still shipping the volofile
capability with volo for a few more releases, just to ease backward
compatibility/transitions, then consider moving away from it if the
"focused scripts" approach and guidelines worked out.

As far as naming, I'm tending to want to keep the volo add/create
stuff as volo, and then the volofile stuff as a separate module. Maybe
just call it volofile. If that is too confusing on names, maybe vauto.
Anyway, we can sort that out as we go. The first three items should be
addressed first anyway, and if we keep the name as 'volo' we can do
that without needing to get into naming.

How does that sound? If there is anything in particular you want to
take on, give a holler.

James