what are we building - strawman

[Tom Yandell]

I'm not yet sure what we are thinking of building. How about the following?

* A public repository of software packages, containing resources to be used in web browsers (including JavaScript modules in AMD format).

* A network accessible API for downloading and uploading packages and associated metadata.

* A client library that allows JavaScript programs to interact with this and other repositories (search/browse, register/authenticate, upload, etc). This would be used by each of the following:

** One or more client applications (e.g. command-line), allowing people to search/browse, register/authenticate, upload, download and manage local installations.

** A website that allows people to search/browse, register/authenticate, upload, etc.

Thoughts?

Tom

[James Burke]

I think most of that is what npm does today. So ideally we could
piggy-back on that, but it may not be possible if more direct AMD
support is needed. Maybe npm can be used as a basis if the npm
maintainer(s) do not want to add the extra AMD considerations:

* Specify if the file is in AMD format. Depending on the answer, have
a way for the client tool to be able to convert to either
Node/CommonJS or AMD format.
* Allow package.json contents to be a comment in a file.
* Allow a single JS file to be registered/used, and do not require
folder structure, separate package.json file, but use that comment as
the package.json contents.

I would also look at what Kris Zyp did with cpm
(https://github.com/kriszyp/cpm) to see what other differences it has
with npm, as they are probably important for AMD modules. Part of that
is probably stamping some sort of config for the web app so that it
can find modules with unusual settings.

So a common approach to modifying an AMD loader's config as part of
the tool might be necessary, but I would really like to have it so
that modules downloaded via the tool are just placed relative to a
baseUrl and that package.json's "main" is not used, so that there
would be no configuration. I'm sure there is a bit of nuance there
though.

In summary: I think an AMD-friendly npm, perhaps just a slightly
tweaked cpm is a nice start. Eventually I would like to see other
tools build on that, to help with initial app scaffolding, but that
can be a second tier goal.

James

[Michael Mathews]

I think there are two related aspects being discussed here, and both seem important. If I can use CPAN as an analogy, there is (was?) the cpan shell, a command line interface for installing modules, and then there was the CPAN repository, the place where users could search for and distribute modules.

I'm not actually sure how you can have one without the other. I suppose you could require that modules be self-hosted, or git-hub hosted (but why limit it to git?), and then the user could specify a complete URL to define where the module should be installed from. This sounds like a simple way around the problem but it means giving up a great deal of control over how the repository is managed. I'd worry that it could lead to a reputation for broken/poor quality modules. What do others think?

Also, James, you mentioned package.json in comments. This is intriguing! Firstly because I wasn't sure if we had decided that the meta format would be the same or an extension/subset of package.json. Would it be confusing to use that established format but then alter it slightly? Or would it be a great benefit to reuse such a familiar standard? And I like the idea of meta data in structured JS comments :)

Michael Mathews
mic...@gmail.com

[James Burke]

On Tue, Oct 11, 2011 at 3:50 PM, Michael Mathews <mic...@gmail.com> wrote:
> I think there are two related aspects being discussed here, and both seem important. If I can use CPAN as an analogy, there is (was?) the cpan shell, a command line interface for installing modules, and then there was the CPAN repository, the place where users could search for and distribute modules.

Agreed, although I think cpm and npm allow just installing from a zip
file. cpm has knowledge of git, and I think that is useful too. In
other words, there should be a distributed option: one that does not
require a registry, it is just that a registry can help make some
things easier.

IIRC, that is how npm works, the registry just tells the client where
to find the bundled code, not necessarily host the bundled code,
although it may allow for that as a bootstrap/helper.

> I'm not actually sure how you can have one without the other. I suppose you could require that modules be self-hosted, or git-hub hosted (but why limit it to git?), and then the user could specify a complete URL to define where the module should be installed from. This sounds like a simple way around the problem but it means giving up a great deal of control over how the repository is managed. I'd worry that it could lead to a reputation for broken/poor quality modules. What do others think?
>
> Also, James, you mentioned package.json in comments. This is intriguing! Firstly because I wasn't sure if we had decided that the meta format would be the same or an extension/subset of package.json. Would it be confusing to use that established format but then alter it slightly? Or would it be a great benefit to reuse such a familiar standard? And I like the idea of meta data in structured JS comments :)

package.json style already has traction, I'm not sure what the other
choice would be. I say use it until we find a problem with it -- it
will allow for easier bootstrapping.

I do not expect to use all of the package.json entries that npm uses,
and we may introduce new properties. I think that is fine. There is
some flexibility in the package.json properties. But I think there are
some properties already in use in a package.json that can just be
picked up and used as-is.

James

[Tom Yandell]

On 12 October 2011 00:43, James Burke <jrb...@gmail.com> wrote:

On Tue, Oct 11, 2011 at 3:50 PM, Michael Mathews <mic...@gmail.com> wrote:
> I think there are two related aspects being discussed here, and both seem important. If I can use CPAN as an analogy, there is (was?) the cpan shell, a command line interface for installing modules, and then there was the CPAN repository, the place where users could search for and distribute modules.

Agreed, although I think cpm and npm allow just installing from a zip
file.

Yeah, I like this too, although I mainly install from the repository. I also like that with cpan/npm/apt/rpm/etc. you can add different repositories (public and private).

 

cpm has knowledge of git, and I think that is useful too. In
other words, there should be a distributed option: one that does not
require a registry, it is just that a registry can help make some
things easier.

I'm not sure if I consider the ability to interact with a version control system to be a desirable feature or not. Maybe that's because I expect to perform processing to create a release artefact from the source form (more on that below). Github already exposes urls to get tarballs/zip of tags. How does CPM interact with git?

 

IIRC, that is how npm works, the registry just tells the client where
to find the bundled code, not necessarily host the bundled code,
although it may allow for that as a bootstrap/helper.

I think npm hosts the package. They do a little normalisation of the package contents as well - that's why the folder inside http://registry.npmjs.org/spectrum/-/spectrum-0.4.1.tgz is called "package" (for example). This can make for a more consistent experience, although I guess an (alternative) NPM repository could link to author hosted packages.

> I'm not actually sure how you can have one without the other. I suppose you could require that modules be self-hosted, or git-hub hosted (but why limit it to git?), and then the user could specify a complete URL to define where the module should be installed from. This sounds like a simple way around the problem but it means giving up a great deal of control over how the repository is managed. I'd worry that it could lead to a reputation for broken/poor quality modules. What do others think?

I think that NPM/CPAN/RPM/APT/etc. strike a good balance - being able to easily install packages from a central repository that are generally as available as the repo, being able to add other repositories and being able to install a package from a tarball (either on the local filesystem or located at a URL).

 

>
> Also, James, you mentioned package.json in comments. This is intriguing! Firstly because I wasn't sure if we had decided that the meta format would be the same or an extension/subset of package.json. Would it be confusing to use that established format but then alter it slightly? Or would it be a great benefit to reuse such a familiar standard? And I like the idea of meta data in structured JS comments :)

package.json style already has traction, I'm not sure what the other
choice would be. I say use it until we find a problem with it -- it
will allow for easier bootstrapping.

I do not expect to use all of the package.json entries that npm uses,
and we may introduce new properties. I think that is fine. There is
some flexibility in the package.json properties. But I think there are
some properties already in use in a package.json that can just be
picked up and used as-is.

 Yeah, I'd like to have a single metadata file for packages that contain resources that are useful in server and browser environments. After discussing, I think you can split the kind of JavaScript modules into:

* JavaScript modules that don't need to drastically change depending on the environment that they run in. In this case you should be able to apply a mechanical transformation to the format for a given environment (e.g. CommonJS -> AMD for loading in a browser, AMD -> CommonJS for loading in a server-side environment, etc). Examples of modules in this class include underscore, js-signals, etc.

* JavaScript modules that depend on environment specific functionality. Examples of this are template libraries (that have different mechanisms for loading templates in different environments - e.g. XHR, node's fs module, rhino's Java apis), dom libraries (using native dom in the browser, libxml2 bindings in node, Java dom apis in rhino), etc.

I think the complicated thing about the latter case if you are authoring modules in CommonJS format and intend to convert to AMD (as I am), the dependencies are essentially static. I can't make it so that the spectrum library is mechanically translated to AMD and remove the dependency on node's fs module (or replace it with something that uses XHR/script tag insertion/etc). I'd like to be able to provide both CommonJS modules that can be used in server-side environments (potentially using synchronous dynamic require to support different server-side environments), and AMD versions of the same modules that work with the browser. This would allow clients to not have to care whether the "spectrum" they depend on is the browser or server version.

I'd like to get two things out of this:

* Establish a convention for where resources (including AMD modules) that should be exposed to, and used by, browsers go in packages.

* Make it easy for users to consume AMD modules, whatever server-side technology they use. I think it's valuable to take npm modules and make them so that all requirejs users can use them in browsers with minimal effort, but I also think it would be good for those same people to be able to use packages containing only modules authored in AMD. This isn't a goal of NPM and I'm not sure if it's a goal of CPM. If not then I'd be up for building something that makes the widest range of AMD and CommonJS modules available to the widest range of developers.

Thoughts?

Tom

James

[James Burke]

On Wed, Oct 12, 2011 at 2:44 PM, Tom Yandell <t...@yandell.me.uk> wrote:
>  Yeah, I'd like to have a single metadata file for packages that contain
> resources that are useful in server and browser environments. After
> discussing, I think you can split the kind of JavaScript modules into:

[snip]

> * JavaScript modules that depend on environment specific functionality.
> Examples of this are template libraries (that have different mechanisms for
> loading templates in different environments - e.g. XHR, node's fs module,
> rhino's Java apis), dom libraries (using native dom in the browser, libxml2
> bindings in node, Java dom apis in rhino), etc.
> I think the complicated thing about the latter case if you are authoring
> modules in CommonJS format and intend to convert to AMD (as I am), the
> dependencies are essentially static. I can't make it so that the spectrum
> library is mechanically translated to AMD and remove the dependency on
> node's fs module (or replace it with something that uses XHR/script tag
> insertion/etc). I'd like to be able to provide both CommonJS modules that
> can be used in server-side environments (potentially using synchronous
> dynamic require to support different server-side environments), and AMD
> versions of the same modules that work with the browser. This would allow
> clients to not have to care whether the "spectrum" they depend on is the
> browser or server version.

Right, I wonder if one of the search criteria for a module is also
"environment". I think some version of the commonjs package.json had
an "engine" thing, maybe reuse that. But maybe the query or registry
key should contain module ID + target environment.

> I'd like to get two things out of this:
> * Establish a convention for where resources (including AMD modules) that
> should be exposed to, and used by, browsers go in packages.
> * Make it easy for users to consume AMD modules, whatever server-side
> technology they use. I think it's valuable to take npm modules and make them
> so that all requirejs users can use them in browsers with minimal effort,
> but I also think it would be good for those same people to be able to use
> packages containing only modules authored in AMD. This isn't a goal of NPM
> and I'm not sure if it's a goal of CPM. If not then I'd be up for building
> something that makes the widest range of AMD and CommonJS modules available
> to the widest range of developers.

I think we need to get more familiarity with CPM, I just have not had
time to do so, and expect I will not have time in the near future, but
it is aware of AMD modules. I'm relying more on you all to sort some
of it out.

But I do think it may be as "simple" as:

* take what npm does and how it does its registry, and add a
package.json property that indicates module format, maybe format:
'amd' or moduleFormat: 'amd' (check CPM, it may have something for
this already). Stand up a new registry if the one that CPM uses by
default is not going to work, or work with that registry to work out
the kinks.
* allow the project's package.json to indicate its format, and if
different from the dependency's format, run a conversion.
* Ideally choose modules that work in an "environment" setting in the
project's package.json.
* Look at, and probably tweak, how CPM does the AMD project config
when adding new modules.

It is probably worth reaching out to Kris Zyp at this point, and get
his take on where CPM is at and where it is going. If you like, I can
ask him to join the list, otherwise, feel free to reach out to him
directly.

James

[Michael Mathews]

Michael Mathews
mic...@gmail.com

On it.

>
> But I do think it may be as "simple" as:
>
> * take what npm does and how it does its registry, and add a
> package.json property that indicates module format, maybe format:
> 'amd' or moduleFormat: 'amd' (check CPM, it may have something for
> this already). Stand up a new registry if the one that CPM uses by
> default is not going to work, or work with that registry to work out
> the kinks.
> * allow the project's package.json to indicate its format, and if
> different from the dependency's format, run a conversion.
> * Ideally choose modules that work in an "environment" setting in the
> project's package.json.
> * Look at, and probably tweak, how CPM does the AMD project config
> when adding new modules.

I'm in the process of setting up a backlog of requirements, that way we can maintain a list like this more formally.

>
> It is probably worth reaching out to Kris Zyp at this point, and get
> his take on where CPM is at and where it is going. If you like, I can
> ask him to join the list, otherwise, feel free to reach out to him
> directly.

I have already invited Kris to join, about 2 weeks ago, but I'd appreciate if James, you would nudge him. I agree that getting his feedback would be very valuable

[Kris Zyp]

> * Establish a convention for where resources (including AMD modules) that should be exposed to, and used by, browsers go in packages.

Since there is much less configuration and manipulation involved in keeping modules at the root level in a package and now that NPM is effectively encouraging modules at the root level as well, this has become the de facto standard. I am not sure if there is as much convergence on other categories, although what I have mostly seen is something like:
root
- my-module.js
+ css
+ test
+ docs

* Specify if the file is in AMD format. Depending on the answer, have
a way for the client tool to be able to convert to either
Node/CommonJS or AMD format.
* Allow package.json contents to be a comment in a file.
* Allow a single JS file to be registered/used, and do not require
folder structure, separate package.json file, but use that comment as
the package.json contents.

I think these sound like great ideas for package repositories.

I'm not sure if I consider the ability to interact with a version control system to be a desirable feature or not. Maybe that's because I expect to perform processing to create a release artefact from the source form (more on that below). Github already exposes urls to get tarballs/zip of tags. How does CPM interact with git?

There are two components to what we have with the Dojo foundation package repository (http://packages.dojofoundation.org). One of them is CPM, which is the command line tool, and this doesn't have any special git integration. The other part is CPR, which is the package repository. This does have git integration. It exposes the versions of a package based on the tags listed in git. This means that to release a new version, you only have create a tag in git, there are no discrepancies between a git tag and a repo version to worry about.

Both of these are based on the package registry specification that Isaac Schlueter (NPM author) wrote: http://wiki.commonjs.org/wiki/Packages/Registry. In fact, the Dojo foundation package repo can even create links in the NPM registry to the DF one so that you can install DF packages directly with NPM without even having to point to a different repo.

Anyway, that being said, I would be perfectly happy if someone else wants to create or expand on the tools available for AMD packages. Let me know if there is anything I can do to help and/or collaborate.