Questions about the new packaging system

[Eric Dobbertin]

Just watched the devshop talk on the packaging system. Questions and comments:

• It wasn't explicitly stated, but since it uses `api.use`, I'm assuming on_test dependencies will be naturally distinct from package dependencies? Currently everything gets installed, even if only needed for running tests, so I'm just making sure that annoyance is resolved.
• It looks like the `username:` piece is simply part of the defined name. Is there any checking against your Meteor developer account when you publish, or can anyone publish a package beginning with my username?
• Some packages are maintained by multiple people. Can the name of, for example, a GitHub organization be used for the username?
• When you publish the package, where do the files go? ("Uploading source..." to where?) Are they stored somewhere public or stored on some Meteor group server and only accessible through the command line and public API?
• I'm assuming you can add a local package dependency (while developing and testing the package)?
• I'm on board with version-specific dependencies, but maybe it would be useful to provide a way to optionally specify minimum and maximum version, too? Your algorithm for resolving dependency conflicts could be smarter then, attempting to find a version that is known to work with all packages that depend on it.
• Is the N.N.N format required for `version`?
• There should be a way to mark a package retired. Still can be added as necessary, but doesn't show in search/atmosphere. I would also argue that it should be possible for someone other than the owner to mark a package as retired. If it's retired by someone without publish rights, then send the owner a few warning e-mails and wait a month, allowing them to cancel the retirement before it happens. And allow undo retirement at any time. There are already some abandoned packages on atmosphere that should ideally be retired.

[ek...@meteor.com]

Hi Eric! Thanks for the awesome and reasonable questions, responses inline. :)

• It wasn't explicitly stated, but since it uses `api.use`, I'm assuming on_test dependencies will be naturally distinct from package dependencies? Currently everything gets installed, even if only needed for running tests, so I'm just making sure that annoyance is resolved.

Yup! We are still working out the details of how the new api would work exactly, but yeah, test dependencies will no longer be installed when not running tests.

• It looks like the `username:` piece is simply part of the defined name. Is there any checking against your Meteor developer account when you publish, or can anyone publish a package beginning with my username?

Yes, of course. We are going to check at publish time to enforce this.

• Some packages are maintained by multiple people. Can the name of, for example, a GitHub organization be used for the username?

We will allow multiple maintainers on a package. Support for organizations in meteor developer accounts is coming soon.

• When you publish the package, where do the files go? ("Uploading source..." to where?) Are they stored somewhere public or stored on some Meteor group server and only accessible through the command line and public API?

The files will go to the meteor package server (packages.meteor.com, but it is not up yet). The package server will have a public API to access package data. We are going to work with atmosphere to integrate their gorgeous UI with it as well.

• I'm assuming you can add a local package dependency (while developing and testing the package)?

Yes! We will continue to have local packages and you can continue to depend on them.

• I'm on board with version-specific dependencies, but maybe it would be useful to provide a way to optionally specify minimum and maximum version, too? Your algorithm for resolving dependency conflicts could be smarter then, attempting to find a version that is known to work with all packages that depend on it.

So, the way that we currently think that dependencies should work, is that when you specify a dependency (jqu...@1.10.0), you are really just specifying the lower bound. The upper bound is going to be determined by semver/earliestCompatibleVersion flag specified by the publisher of, in this case, jquery. So, 2.0.0 probably would not be compatibe, 1.11.0 would be. We are not currently going to support more explicit setting of version dependency bounds, but we could in the future. (If you wanted to lock your version to 1.10.0, you could request jquery@=1.10.0, in which case, no other version of jquery would ever be considered OK.

• Is the N.N.N format required for `version`?

Yeah -- we are planning to use semver versions, so the format has to be a valid semver: N.N.N. (More info on semver, for reference: http://semver.org/)

• There should be a way to mark a package retired. Still can be added as necessary, but doesn't show in search/atmosphere. I would also argue that it should be possible for someone other than the owner to mark a package as retired. If it's retired by someone without publish rights, then send the owner a few warning e-mails and wait a month, allowing them to cancel the retirement before it happens. And allow undo retirement at any time. There are already some abandoned packages on atmosphere that should ideally be retire

We will probably have a way for a maintainer to deprecate a version, but mostly we trust the community to make it clear which packages are no longer in use (for example, right now, atmosphere provides an awesome UI showing how often a package has been updated). If it becomes needed, it is certainly a thing that we could implement later.

Thanks!

-Ekate

[Eric Dobbertin]

Thanks, Ekate. I like all of those answers. :) I had one other thought today, which is that losing the dependency on git is nice, but also could add some complexity for those who are using git, which most packages will. For example, I probably don’t want to publish when there are uncommitted modifications, and I might still like to tag each version with git. It becomes several related tasks that all have to be done for every release, so detecting git, checking for modifications, auto-tag-and-push, etc. would be useful features. I suspect your answer will be “Someone from the community can make a wrapper that deals with git and then calls meteor publish”, and I think that’s absolutely true, which is why I’m publicly suggesting it so that “someone from the community” can get to work. :)

--
You received this message because you are subscribed to a topic in the Google Groups "meteor-core" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/meteor-core/MNzKXHlCGE0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to meteor-core...@googlegroups.com.
To post to this group, send email to meteo...@googlegroups.com.
Visit this group at http://groups.google.com/group/meteor-core.
For more options, visit https://groups.google.com/d/optout.

[Andrew Mao]

I've had some longstanding problems with the packaging system, filed as outstanding Meteor issues, which I hope will be addressed in 0.9.0:

• Provide an easier way to include files loaded from NPM packages: https://github.com/meteor/meteor/issues/1259
• Allow packages to be loaded with arguments, settings, or roles. Generally, provide a way to specify options to packages. https://github.com/meteor/meteor/issues/1292
• Related to the above, allow packages to load different sets of files depending on arguments/settings that it is called with. It's become far too cumbersome to provide different functionality under an umbrella package and people have started dividing things up into different packages (can you count how many cfs-* packages there are on Atmosphere?)

Thanks!

[Morten Henriksen]

Ekate one question in addition:

Is it possible to have default weak packages and allow the user to remove these later?

(like the insecure + auto publish)

Eric sounds like you are referring to mbr or a similar tool?

[Eric Dobbertin]

Yes, Morten, either mrtbulkrelease or maybe meteorite could be rewritten as a slim wrapper that helps with the git aspects. Rather than wrapper commands, though, I think it would be nicer if the packaging system eventually supported plugins or hooks. A couple pre-publish or post-publish hooks is all that would really be needed. Then we can all use the same commands.

+1 for default weak packages

[Eric Dobbertin]

And on the subject of umbrella packages, it would be nice to be able to somehow easily search for the available add-on packages. Some way to say with metadata: “I am an add-on/plugin for X package” and then get a list of all those with something like `meteor search --related X package`. (Even if Andrew’s suggestion for loading different sets of files with settings were implemented, there would still be cases where one developer makes a package that is an add-on for another developer’s package.)

And one more question: Are all files uploaded, or only those listed with `add_files`? If not all, then what about a file like LICENSE.md that should stay with the code? Do we need to call `add_files` for the license file? On the other hand, if all are uploaded, the ability to list files to exclude could be nice. Like .gitignore, but for package publish/upload. There are files like `.editorconfig` and `.jshintrc` and `.travis.yml` that don’t really need to be uploaded and stored for any packages.

[ek...@meteor.com]

Sorry for the delay -- I have been out of town for a few days! Yeah, we are probably not going to do any sort of git integration this round, but excited to see what solution the community comes up with.

>>And one more question: Are all files uploaded, or only those listed with `add_files`? 

At the moment, we are only uploading files that are explicitly listed as such with add_files. We are currently discussing some changes to the control file format for the package, that may make it easier.

>>Is it possible to have default weak packages and allow the user to remove these later?

>> Allow packages to be loaded with arguments, settings, or roles. Generally, provide a way to specify options to packages; allow packages to be loaded with different sets of files depending on options.

That's not something that we are explicitly planning to do right now, but we are discussing various ways to make a multitude of packages easier to manage. Hopefully, then, some of the underlying concerns can be answered, and if that doesn't help, we will look another look at this afterwards! 

Thanks!

-Ekate

On Thursday, April 10, 2014 9:16:04 AM UTC-4, Eric Dobbertin wrote:

And on the subject of umbrella packages, it would be nice to be able to somehow easily search for the available add-on packages. Some way to say with metadata: “I am an add-on/plugin for X package” and then get a list of all those with something like `meteor search --related X package`. (Even if Andrew’s suggestion for loading different sets of files with settings were implemented, there would still be cases where one developer makes a package that is an add-on for another developer’s package.)

[J Bruni]

Meteor would gain a huge boost of flexibility by using some approaches which the Mimosa build tool already uses.

For non-Meteor projects, I use Mimosa to build my deployable ("production-ready") app from my development files. Mimosa has a very straightforward modules system. Each module is easily configurable.

I think the Meteor packages could follow a similar approach, bringing a lot of flexibility and benefits for the whole community.

In fact, while Meteor is absorbing/merging/integrating the best from what appears anywhere (Handlebars became Spacebars... Facebook React influenced Blaze... Rails-like design at Iron Router... MongoDB inspiration for DDP, and so on), it should seriously consider what Mimosa is doing regarding the build process. Consider making extremely easy to interfere in - and thus make very flexible - the build process. It would give wings for many of us front-end developers!

One of the things which Meteor packages could do, similar to Mimosa modules, would be to influence the build process. Meteor could provide hooks at specific steps of the build workflow, just as Mimosa does.

I can elaborate more on this. By now, I will point to the blog post explaining "how to write a Mimosa module", which gives a tutorial-like approach to the raw information also available at the docs in the Mimosa website.

Andrew asked for "a way to specify options to packages" - which is an idea I totally agree and support. And I think this "way" could be similar to what Mimosa already implemented to "specify options to modules"...

In fact, to be honest, I feel I would love to see Mimosa baked into Meteor. Both are amazing tools - cool, forward-thinking, ahead of their time, and so on. Put famo.us in the basket... wow.

[Slava Kim]

Looking at Mimosa's feature's list it looks like most of the things are already implemented in the Meteor's build tool. They were there a year ago as of 0.5.0 (engine release):

- JS Transpilers - check! coffeescript built-in, typescript, wisp, es6-harmony, clojurescript available at atmosphere. Even with sourcemaps, stacktraces and debugging support
- CSS Pre-processors - check! Less and stylus builtin, compass and SASS available on atmosphere. Less has sourcemaps support!
- HTML Micro-templaters - what is this? Spacebars are built-in, jade is available on the atmosphere
- Server integration - erhh, looks irrelevant to Meteor
- Project Creation - `meteor create` creates the project :) Chris Mather has built `em` tool for scaffolding
- File Watching - meteor has it
- Bower integration - well, there is a package that integrates Bower, but it looks like an irrelevant feature to Meteor's packaging system

9. Live Reload - hard code push is even better, it preserves your Session state, CSS reload can be faster though, there is a room for improvements
10. Minification - have it built-in, could give more control over minification but this is coming
11. AMD/RequireJS Support - looks irrelevant
12. Concatenation/Optimization - have it
13. JSHinting - don't have it
14. CSSLinting - have it

 JIT notifications - what is this?

A lot of features actually overlap :)

[Gadi Cohen]

Awesome reply :)  I'll add that there are still cases where bower/requirejs support is useful:

1. Avoiding "smart" packages which just include an old version of a complete library

2. Selectively bundling only parts of the library used by your app.

Both these needs are addressed by raix's famono package, which supports famo.us too (which the previous posted asked about).

[Morten Henriksen]

I’ll just add to this - there is already a package that adds jsHint… It just need support for multiple source handlers or pr 1207. https://github.com/raix/Meteor-jshint (its about 10 month old now…)

For now you can add the .jshintrc to your project folder adding Meteor specific jshint - https://github.com/raix/Meteor-jshintrc

yep, famono adds bower/famo.us/lazyloading capabilities etc. but more important more fine-grained library dependencies.

Venlig hilsen

Morten N. O. Nørgaard Henriksen

Grøn idé ApS - 30 13 12 41