Pub - package manager

[Bob Nystrom]

We've mentioned on the list a few times that we're starting to work on a package manager for Dart. Since a package manager is as much a cultural artifact as a technical one, I'd like to get community feedback on the plan as soon as possible.

Here's what we're thinking for the first phase: https://docs.google.com/document/d/13y7yCwq9GtPChXtd6t0YMcUtMJLZq2IVwq546Dve_No/edit

You can comment here or in the doc itself. Let me know what you think. :)

Cheers!

- bob

[Lars Tackmann]

Hi Bob

I like your decision to go with the something simple and managed like NPM. However I am very doubtful  about the entire pub server thing. It seams overly complex for me, one of the good things Maven brought to Java was a standardized layout (convention over configuration). I think you should look into getting out of the entire Dartium package problem by means of enforcing some sane standard. 

That being said i definitely understand that people might want to make strage package layouts, me I for instance symlink log4dart into some of my apps as I can then work with one log4dart code base directly from many apps and thus ensure I am always running the latest.

--

Lars Tackmann

[Bob Nystrom]

On Tue, Apr 3, 2012 at 12:43 PM, Lars Tackmann <ltac...@gmail.com> wrote:

I like your decision to go with the something simple and managed like NPM. However I am very doubtful  about the entire pub server thing. It seams overly complex for me, one of the good things Maven brought to Java was a standardized layout (convention over configuration).

Those aren't mutually exclusive. We are definitely a fan of convention over configuration. However, I don't think we want to bake that convention directly into the browser. It will likely need to evolve over time (like Ruby has gone from vanilla RubyGems to bundler) and baking our package management details into the browser and language spec makes that very hard.

Instead, what I'd like is to have just a couple of language primitives, and then build pub (and any future superior package managers) in terms of those. There still will be a convention (and definitely not reams of XML), but it will be one specified by the package manager you're using.

The pub server stuff is just one mechanism (and very possibly not a great one) for enabling that. If it ends up not working, we'll ditch it. What I think is important is to starting building something that works, so that we can try to build packages and see how they feel.

- bob

[Lars Tackmann]

On Wed, Apr 4, 2012 at 2:16 AM, Bob Nystrom <rnys...@google.com> wrote:

The pub server stuff is just one mechanism (and very possibly not a great one) for enabling that. If it ends up not working, we'll ditch it. What I think is important is to starting building something that works, so that we can try to build packages and see how they feel.

- bob

Experimenting sounds like the right path forward with this thing. I have a some of open source projects to try it on and I'll happily change the way I package them a couple of times ;)

--
Yours sincerely

Lars Tackmann

[Bob Nystrom]

Thank you! There will likely be some churn, but this is one of those software engineering areas where real-world empirical evidence is helpful for making sure we're doing the right thing.

- bob

[godlove...@gmail.com]

[Chris Buckett]

Something I've just remembered. Will pub also make use of the
#resource keyword?
So if I use #resource("someImage.png"); or #resource("myCss.css"); in
a library file, pub would also know to pull these resources into the
package?

(I don't think that #resource is used anywhere at the moment- please
correct me if I'm wrong. It used to let the Dart Editor show non-dart
files in the Library view, but that's now redundant as we view the
file system folders directly).

Cheers,
Chris.

On Apr 3, 5:08 pm, Bob Nystrom <rnyst...@google.com> wrote:
> We've mentioned on the list a few times that we're starting to work on a
> package manager for Dart. Since a package manager is as much a cultural
> artifact as a technical one, I'd like to get community feedback on the plan
> as soon as possible.
>

> Here's what we're thinking for the first phase:https://docs.google.com/document/d/13y7yCwq9GtPChXtd6t0YMcUtMJLZq2IVw...

[Nathan Weizenbaum]

The current plan doesn't have Pub parsing Dart files at all, so #resource probably won't be supported. Any static resources in your package directory would be preserved, though, I think.

[Bob Nystrom]

On Fri, Apr 6, 2012 at 11:04 AM, Nathan Weizenbaum <nw...@google.com> wrote:

The current plan doesn't have Pub parsing Dart files at all, so #resource probably won't be supported. Any static resources in your package directory would be preserved, though, I think.

What Nathan said. I don't think any mention of non-code resources belongs in your source code. Everything in the directory for your package will implicitly be part of your package. If we find a need for it, we may also support listing resources in your package's pubspec, but definitely not in your Dart code itself.

My hope is that pub will let us get rid of #resource entirely from the language.

- bob

[Chris Buckett]

>The current plan doesn't have Pub parsing Dart files at all, so #resource
> probably won't be supported.

How will you deal with #source if pub isn't going to be parsing Dart
files? Especially for #import's that are url based?

> If we find a need for it, we may also
> support listing resources in your package's pubspec,

I think that some reference to non-dart resources will probably be
needed. For example, if I make available a client side library which
relies on some images and css files, then there needs to be some
mechanism to tie these into the library.
The equivalent to Java would be to think about how many non .class
files end up inside a jar (properties, translations, config, Licence
etc...)

>. Everything in the directory for your package will
> implicitly be part of your package.

How would you identify "everything in the directory" if the library
was accessed by a url? (or do libraries have to be local first?)

Cheers,
Chris.

[Ladislav Thon]

How would you identify "everything in the directory" if the library
was accessed by a url?  (or do libraries have to be local first?)

I don't want to speak for Bob, but importing by URL is a terrible idea, and especially for Dart, which has fast startup as one of its primary design goals. You should only be able to import those packages that are somehow "installed", which means accessible from the local filesystem. (There's a problem with running in the browser, because you can't "install" packages there. I'm not exactly sure how to solve it -- the Pub people have some ideas, but I personally don't like running a special server for serving packages. On the other hand, I don't think that JavaScript people solved this problem either.)

LT

[Seth Ladd]

There are two forces at work, and the team is trying to find the right balance.

On one hand, web development is awesome because you can simply use URI's to pull in various resources for your project. It's very quick and easy to start, simply by pointing to a URI.

On the other hand, as you point out, complex apps with many many dependencies won't use the URI scheme because it doesn't make sense to load 100 individual files over HTTP just to run an app. Not only is speed a concern, but too many 3rd party servers increases the chance of downtime for your app.

I also agree, I don't want to run a separate server app just to develop. Chances are, I'm already running a web server anyway (even for development, most modern web apps don't work from file://), so I feel like I should be able to take advantage of this fact. Running a web server during development is the norm these days, look at Sinatra, App Engine, Rails, etc. Let's try for a single server during development.

It would be interesting if the Dart runtime (and thus dev web server) itself acted like a caching proxy. I might continue to use URI's in my #import, but my tools like dart2js, Dart VM, Dart Dev Server, and Dart Editor, would run a heavily optimized cache proxy. This might be the best of both worlds.  Don't have the cache proxy? No problem, URI resolution happens like normal. Uber-simple scripts continue to work. Have the cache proxy built into the tools? Awesome, everything is resolved from local disk and is super fast.

[Christian Grobmeier]

Hi all,

I just read through the document - thanks for compiling it! I really
love the name "pub".

There are some questions in my mind...

First, I think there needs to be a clear separation between the use
cases. I have seen the following:

1) deploying to and serving from a remote machine
2) structure of your project on the local machine
3) using dependencies as a browser app
4) using dependencies as a server app

At the moment I feel we mix all that use cases and the result is a
little bit confusing (maybe its just me :-))

I saw already a structure suggestion for a local project. It's fine so
far, but this structure is not really what I want to see on my
dependencies server.
If I write a server app which takes the dependency from such a
structure it might results in multiple http requests. Including 20x
#source might result in 20 http requests. Therefore I would prefer a
clean and easy structure on the remote server and a single file for
each dependency, like for example Javas jar files.

Having a single file per dependency also helps pub to check signatures
and md5 files. If I make pub install (or whatever the equivalent will
be) pub should check md5 files per default. It would be great if the
md5 file could be on another, third server.

Another question which come to my mind: can browser apps depend on
packages from a remote server? Not sure if I like that. If they can,
does a user of my app need to download all dependencies? If yes, then
how do these deps get cached? Browser cache only?

Server apps might depend to remote servers more easily, as they have
the chance to store deps on a local place. It would be good to be able
to define that place. On Unix based machines I would expect .pub in my
home directory or something like that.

As for the "pub serve" command - why do we need that? imho every
webserver should do the trick. If I want to have a pub-repository on
my mass hosters webspace it should not be a problem for me: upload,
go.
Therefore I would love to see commands like:

pub list <channel> // list the available dependencies for the channel
pub install <oackage definition> // install a dependency to my local
repos, checks md5 files, if available
pub delete <oackage definition> // deletes from my local repos
pub package // packages the project i am currently in: creates a
deployable version including all dependencies for my webapp
pub deploy <channel> // creates a deployable for my remote repository
and deploys it (until i add --dry-run )

pulll update makes sense to me also. Not sure if "pub install" is
obsolete with that

I think, once the remote repository layout is there, things will
lighten up for me. Also the pub-file definition will bring more light

Speaking of the directory layout:

src/
myapp.dart
myapp.packages/
interwebs.dart
ui/
widgets.dart
base/
utils.dart

Why do we need /src/myapp.dart/ and /src/myapp.packages?
Can't it be:
/src/dart
/src/packages
?
I think it is pretty clear that we develop a specific app... or is it
usus to work on different apps in a single project?

Sorry if I missed something which should be clear already. I was off
for a while, and this is what the pub-document raised for
questions/confusion in me.

Cheers
Christian

--
http://www.grobmeier.de
https://www.timeandbill.de

[Chris Buckett]

On Apr 7, 5:27 pm, Ladislav Thon <ladi...@gmail.com> wrote:
> > How would you identify "everything in the directory" if the library
> > was accessed by a url?  (or do libraries have to be local first?)
>
> I don't want to speak for Bob, but importing by URL is a terrible idea, and
> especially for Dart, which has fast startup as one of its primary design

My thought re using URL's would be that within an internal company's
build structure, I and my fellow developers might reference shared
libraries on an internal library server -
eg: #import("http://dartserver.local/acmeLib/acmeLib.dart");

These would be pulled to our local machines by pub, and likewise, when
the build server builds the entire app to JavaScript, the libs would
be incorporated into the output .js file (which would ultimately be
deployed to the customer).

Likewise, when I publish a library, I (or more preferably the build
system) would send (a "versioned" copy of) it to my local library
server for other developers.

(This is similar to maven/nexus, which allows deploying of packages to
a central, internal http server within the organization).

Cheers,
Chris.

[Bob Nystrom]

On Sat, Apr 7, 2012 at 7:24 AM, Chris Buckett <chrisb...@gmail.com> wrote:

>The current plan doesn't have Pub parsing Dart files at all, so #resource
> probably won't be supported.

How will you deal with #source if pub isn't going to be parsing Dart
files?  Especially for #import's that are url based?

Nathan is overly optimistic here. :) Pub probably will do some Dart file parsing, at least for certain operations. Certainly the "pub serve" command in the doc will parse (and rewrite) Dart code. Fortunately, this is pretty easy to do since we already have a couple of Dart parsers written in Dart.

 

> If we find a need for it, we may also
> support listing resources in your package's pubspec,

I think that some reference to non-dart resources will probably be
needed.  For example, if I make available a client side library which
relies on some images and css files, then there needs to be some
mechanism to tie these into the library.
The equivalent to Java would be to think about how many non .class
files end up inside a jar (properties, translations, config, Licence
etc...)

Right, resources are definitely important. There's two cases (that I can think of) where we need to worry about resources.

1. When you install a package. Here, I think the "everything in the package directory comes with the package" model works fine. So when you install a package, you'll get its entire package directory.

2. When you deploy an app that uses a package. Here, you may also need to deploy some of those resources (CSS and templares are the obvious ones). We haven't spent a lot of time thinking about this yet, but we probably will have some way to indicate these resources, either in some "build step" description, or in the pubspec. In either case, though, I don't think your Dart source code is the best place to list those resources.

- bob

[Bob Nystrom]

On Sat, Apr 7, 2012 at 10:09 AM, Christian Grobmeier <grob...@gmail.com> wrote:

Hi all,

I just read through the document - thanks for compiling it! I really
love the name "pub".

Thanks. I've been sitting on that name for almost a year now. :)

 

There are some questions in my mind...

First, I think there needs to be a clear separation between the use
cases.

Agreed, the doc isn't as clear (or complete) as it could be. Writing docs is time consuming!

 

I have seen the following:

1) deploying to and serving from a remote machine

This is what "pub deploy" is for.

 

2) structure of your project on the local machine

Right. This is what conventions and "pub install" and "pub update" will worry about.

 

3) using dependencies as a browser app

The idea is that "pub serve" will handle this (modulo the valid concerns that have been raised about not wanting to have to run a server for this).

 

4) using dependencies as a server app

I believe the plan is to implement "package:" and link library support directory into the standalone VM for this.

 

At the moment I feel we mix all that use cases and the result is a
little bit confusing (maybe its just me :-))

I saw already a structure suggestion for a local project. It's fine so
far, but this structure is not really what I want to see on my
dependencies server.

What would you like to see here? Concrete examples are really helpful.

 

If I write a server app which takes the dependency from such a
structure it might results in multiple http requests. Including 20x
#source might result in 20 http requests. Therefore I would prefer a
clean and easy structure on the remote server and a single file for
each dependency, like for example Javas jar files.

Agreed completely. The doc doesn't go into this, but we've discussed having "pub deploy" also essentially minify and concatenate your Dart code the same way JS compilers do for JS. We hope SPDY will help, but we are definitely sensitive to wanting to minimize roundtrips to pull down your Dart app. That's a big reason why we feel there will be a deploy step at all.

Having a single file per dependency also helps pub to check signatures
and md5 files. If I make pub install (or whatever the equivalent will
be) pub should check md5 files per default. It would be great if the
md5 file could be on another, third server.

Another question which come to my mind: can browser apps depend on
packages from a remote server?

In the current system, sure, you can #import from whatever URL you like. In practice, I personally think this is crazy talk and (outside of possibly a few blessed CDNs) almost no real developer would want to serve their app like this.

 

Not sure if I like that. If they can, does a user of my app need to download all dependencies?

Yup. #imports() are static which means they must all be successfully resolved and loaded before your app can start (at least right now).

 

If yes, then how do these deps get cached? Browser cache only?

I believe whatever normal caching behavior the browser has will kick in here, but I'm not an expert on this.

 

Server apps might depend to remote servers more easily, as they have
the chance to store deps on a local place. It would be good to be able
to define that place. On Unix based machines I would expect .pub in my
home directory or something like that.

As for the "pub serve" command - why do we need that? imho every
webserver should do the trick.

The problem I'm aware of is that you will often link to libraries that are stored outside of your application's root directory. Unless you start symlinking stuff in (which, alas, doesn't work on Windows) it isn't easy for a stock web server to handle that.

 

pub list <channel> // list the available dependencies for the channel

Is "channel" here a place you can install packages from?

 

pub install <oackage definition> // install a dependency to my local
repos, checks md5 files, if available

+1.

 

pub delete <oackage definition> // deletes from my local repos

I would use "uninstall", but, yup, this sounds good.

 

pub package // packages the project i am currently in: creates a
deployable version including all dependencies for my webapp

Once we have some kind of package hosting, we'll want some commands like this, but we aren't worrying about this in phase one.

 

pub deploy <channel> // creates a deployable for my remote repository
and deploys it (until i add --dry-run )

Right now, at least, deploy doesn't actually push your "compiled" app to anything remote because that's a bit outside of pub's job description. There are two different tasks here. Deploying an application is for getting a complete Dart program and all of its dependencies and making them ready for production use. Publishing is for getting a reusable package and putting it up somewhere where other Dart developers can pull it down and install it.

Speaking of the directory layout:

src/
         myapp.dart
         myapp.packages/
           interwebs.dart
           ui/
           widgets.dart
           base/
       utils.dart

Why do we need /src/myapp.dart/

myapp.dart is just a file: the entrypoint Dart program for myapp.

 

and /src/myapp.packages?

My thinking here was that it allows multiple entrypoints in the same directory. Each entrypoint needs its own packages directory and we need to ensure they don't collide.

 

I think it is pretty clear that we develop a specific app... or is it
usus to work on different apps in a single project?

In a single directory, yes.

 

Sorry if I missed something which should be clear already.

Nope, it's probably just not clear in the doc. We've been discussing this internally a lot, but we haven't written much down (yet), so there's lots of gaps and unstated assumptions. I'm trying to improve that situation over time, but it's a lot of work. Thanks for pointing out the stuff that isn't clear; that helps me know where more detail is needed. :)

 

Cheers!

- bob

[John Messerly]

On Mon Apr 09 10:45:36 GMT-700 2012, Bob Nystrom <rnys...@google.com> wrote:

On Sat, Apr 7, 2012 at 10:09 AM, Christian Grobmeier <grob...@gmail.com> wrote:

Hi all,

I just read through the document - thanks for compiling it! I really
love the name "pub".

Thanks. I've been sitting on that name for almost a year now. :)

Hey, I thought I coined that one. Wishful thinking perhaps ;)

- John

[Nathan Weizenbaum]

Remember that there's a distinction between your #imports and the dependencies you declare to Pub. When using pub, you always #import "package:" URIs which resolve to local files. If you want to get a package (e.g. acmeLib) from your local package server, you would declare that in your dependency file. Pub would then download that package when you run "pub install".

[Ross Smith]

> Agreed completely. The doc doesn't go into this, but we've discussed having
> "pub deploy" also essentially minify and concatenate your Dart code the

> ?same way JS compilers do for JS. We hope SPDY will help, but we are

> definitely sensitive to wanting to minimize roundtrips to pull down your
> Dart app. That's a big reason why we feel there will be a deploy step at
> all.

This is probably not in the scope of pub (I like the name too btw!)
but it might be? Is there any plan to support code-splitting in
deployed code? Something like GWT's runasync()?

http://code.google.com/p/google-web-toolkit/wiki/CodeSplitting

-Ross

[Bob Nystrom]

I don't know if we have something I'd call a plan, but yes, this is definitely a topic we've discussed. You're right that it's mostly outside the scope of pub (though pub would need to be aware of it) and is more a language/isolate/import issue, but it's on our radar.

Cheers,

- bob

[dvoytenko]

It's kind of disappointing to see almost no successful experience re-
used in Pub from Maven. I know that the discussion is kind of split b/
w the document and this thread, so I apologize if I repeat anything...

(I believe Jason van Zyl (https://plus.google.com/
110566245181212600381), the principal creator of Maven, is usually
fairly happy to share his good/bad experience while creating Maven)

First, few numbers (http://search.maven.org/#stats):
- Maven Central Repository currently hosts over 300K artifacts. This
is almost ten times the number of Rubygems on rubygems.org. It's the
biggest repository, but not the only one.
- Maven Central Repository currently boasts over 80M downloads a weak!
I.e. every 2 month more artifacts are downloaded from Maven than from
rubygems.org since its inception.

Certainly, this success couldn't have been achieved if they didn't get
some things right. Not that Dart couldn't improve on many things too.

On top of this, Maven imposes a simple standard conventional
structure, provides very few core commands, is well integrated with
modern IDEs, works well with web environments, etc. In my experience
most of issues when working with Maven arise not from Maven itself,
but from some outdated issues with Java.

If I may comment on some points made in the document. I'm not saying
that Dart Pub should be done exactly this way, but it's certainly
worth considering.

>> #import('../../../some/other/lib.dart');
>> #import('http:/some.com/other/lib.dart');

The "Maven way" the import could look like:

#import('other/lib.dart') // import in the local package/project
#import('package-name:other/lib.dart') // an import resolved through
other package named "package-name"

The package itself is declared in the dependencies file as suggested,
including version constraints.

>> even imports between the files in packages work and are resolved relative to the top packages directory
>> So each package has a copy of every package it uses?
>> The idea here is that you flatten the entire dependency graph and put it in a directory in your entrypoint package

"Maven way" would either have an option to have external packages next
to "my" package (not inside) or in repository. The resolution is
simple - either the package is right next to the folder of "my"
package/project, or look in ~/.dartpub/package-name. That would really
be "flat".

>> pub update

"Maven way" doesn't need update command (or it can be optional). The
artifacts are resolved with version constraints from dependencies
file, downloaded from central repository and cached locally
automatically. The transitive dependencies are handled exactly the
same way. Nothing is copied into the project. Local cache is somewhere
in ~/.dartpub/

>> How does it know what packages you're using?
>> it may be able to infer them by the #imports in your code

No need to infer, #import can state package's name (namespace)
explicitly (#import('package-name:lib.dar')). The only thing that
dependencies file supplies are version constraints.

>> How does pub make the packages directory?
>> #library('link!/actual/real/path/to/utils.dart')
>> No pointless duplication.

"Maven way": no need for this. The dependencies are resolved against
and used directly from the local cache (or sibling folders). This way
there truly is no pointless duplication and no extra entities to
support.

>> How do link libraries work in Dartium?

The production mode should be very similar to development mode with
few allowances. The added benefit - less entities for Dart team to
create and less things for Dart users to learn. A simple Dart-based
HTTP server could completely handle all issues here:
a. There's already HTTP server available in Dart, so it can be
extended.
b. It can be integrated very naturally with Dart Editor (and other
IDEs)
c. It can implement the same package resolution as necessary via
global repository/local cache and expose packages this way via HTTP
interface with no security implications.
d. <script> can use the same form as import "package-name:abc/
lib.dar". Again, less to learn.
e. No "paths" outside of the project's directory except for those
strictly enforced by convention in adjacent folders and local cache.
f. If there's web server already, a simple plugin should do.
g. Finally, Dartium could support this package resolution natively.
Again, it's strict and well-defined, so some allowances could be made.

Nearly the same model can be used in production too.

>> pub deploy

There could still be something similar to this in the "Maven way"
approach. Maven today provides different packaging routines as well.


Thus, something like "Maven" approach could potentially create a more
standard package layout, provide simpler discovery and force fewer
conventions, "special files" and commands on developers. Why not
consider it?

--
Dimitry Voytenko

[Ladislav Thon]

In my experience
most of issues when working with Maven arise not from Maven itself,
but from some outdated issues with Java.

I personally think that Maven is overly complicated, and mostly without reasons. But I agree that the repository/packages part of Maven is pretty well done -- it's the "project automation" part that looks horrible to me. Luckily, Pub isn't trying to be "project automation" tool, it's only a package manager.

 

"Maven way" doesn't need update command (or it can be optional).

Sure it does! Maven resolves artifacts when building the project. There is nothing like this in Dart (it might be in the future, but it probably won't be connected to Pub in any way, or only very loosely), so Pub has to have a command like this.

>> #library('link!/actual/real/path/to/utils.dart')

"Maven way": no need for this. The dependencies are resolved against
and used directly from the local cache (or sibling folders).

I don't like it either. AFAIK, all package managers have a predefined location(s) where to load packages from ("local cache")... but the Dart team seems to think that there is an extra degree of flexibility needed. Maybe it is, maybe it is not, only time will tell.

LT

[dvoytenko]

> I personally think that Maven is overly complicated, and mostly without
> reasons. But I agree that the repository/packages part of Maven is pretty
> well done -- it's the "project automation" part that looks horrible to me.

I agree that some parts are complicated and Pub probably won't have
anything to do with those parts. But the core is strong and very
simple:
- Layout convention
- Repository
- Package resolution
- Namespace

A lot of complexities can be also attributed to Java. Dart should not
have those problems.

> Sure it does! Maven resolves artifacts when building the project. There is

You're right. It does exist. But, I guess, the presence of such a
command probably doesn't change things that much. I understand why
implicit behavior might be undesirable.

thanks,
Dimitry Voytenko

[Bob Nystrom]

On Tue, Apr 10, 2012 at 12:40 PM, dvoytenko <dvoy...@gmail.com> wrote:

It's kind of disappointing to see almost no successful experience re-
used in Pub from Maven.

Maven is on my list of technologies to understand better. Just because this doc doesn't make lessons from Maven clear, that doesn't mean we're choosing to actively ignore it. :)

 

I know that the discussion is kind of split b/w the document and this thread, so I apologize if I repeat anything...

(I believe Jason van Zyl (https://plus.google.com/
110566245181212600381), the principal creator of Maven, is usually
fairly happy to share his good/bad experience while creating Maven)

First, few numbers (http://search.maven.org/#stats):
- Maven Central Repository currently hosts over 300K artifacts. This
is almost ten times the number of Rubygems on rubygems.org. It's the
biggest repository, but not the only one.
- Maven Central Repository currently boasts over 80M downloads a weak!
I.e. every 2 month more artifacts are downloaded from Maven than from
rubygems.org since its inception.

Certainly, this success couldn't have been achieved if they didn't get
some things right. Not that Dart couldn't improve on many things too.

On top of this, Maven imposes a simple standard conventional
structure,

We have something similar in mind for pub, it just hasn't been documented yet. We'll get there. The Dart in general strongly prefers convention over configuration.

 

provides very few core commands,

I can guarantee pub will have few commands, at least at first. :)

 

is well integrated with modern IDEs,

I am coordinating this with the editor team.

If I may comment on some points made in the document.

Yes, please do!

 

>> #import('../../../some/other/lib.dart');
>> #import('http:/some.com/other/lib.dart');

The "Maven way" the import could look like:

#import('other/lib.dart') // import in the local package/project

This works now in that that will be resolved relative to the importing file's location. The pub document doesn't discuss this because this is already how Dart works now.

 

#import('package-name:other/lib.dart') // an import resolved through
other package named "package-name"

This is a URL where the scheme is "package-name". I don't think we want a scheme per-package because scheme resolution is handled directly by the Dart implementation.

 

>> even imports between the files in packages work and are resolved relative to the top packages directory
>> So each package has a copy of every package it uses?
>> The idea here is that you flatten the entire dependency graph and put it in a directory in your entrypoint package

"Maven way" would either have an option to have external packages next
to "my" package (not inside) or in repository. The resolution is
simple - either the package is right next to the folder of "my"
package/project, or look in ~/.dartpub/package-name. That would really
be "flat".

In practice, I expect users will have a single global package directory like ~/.dartpub. Pub will then pull packages from that to generate your application's packages directory using the link library stuff. This doc doesn't make it clear, but it's mostly focused on mechanism (i.e. "package:" and link libraries) and less so on usage. When you use pub, you'll be using "package:" but you generally won't have to think about link libraries or the application-specific packages directory. Pub will just do that for you.

>> pub update

"Maven way" doesn't need update command (or it can be optional). The
artifacts are resolved with version constraints from dependencies
file, downloaded from central repository and cached locally
automatically.

Pub works the same way, but our opinion is that we don't want that process to happen constantly and automatically. Upgrading to new versions of stuff is a relatively big deal and I think most developers want to know when they are choosing to do that.

Also, keep in mind that Dart is a scripting language, and doesn't have a required "build step", so there's no well-defined time where we could do this even if we did want to automate it.

 

The transitive dependencies are handled exactly the
same way. Nothing is copied into the project. Local cache is somewhere
in ~/.dartpub/

Right, this is the plan for pub too. You will have a cache. You won't really "copy" things into your project. Instead, pub will generate "link libraries" that point back to your cache.

 

>> How does it know what packages you're using?
>> it may be able to infer them by the #imports in your code

No need to infer, #import can state package's name (namespace)
explicitly (#import('package-name:lib.dar')).

Imports will specify libraries but those have a slightly different granularity than packages. A package may (and likely will) contain multiple libraries.

Namespacing, at least in terms of what names are bound in your Dart code when you import a library is a different area and is outside of pub's scope. 

>> How does pub make the packages directory?
>> #library('link!/actual/real/path/to/utils.dart')
>> No pointless duplication.

"Maven way": no need for this. The dependencies are resolved against
and used directly from the local cache (or sibling folders). This way
there truly is no pointless duplication and no extra entities to
support.

Right, but Maven does this at build time, right? Dart doesn't have a "build time". In either case, there should be no extra entities to support: the "packages" directory is just an artifact created and automatically managed by pub.

 

>> How do link libraries work in Dartium?

The production mode should be very similar to development mode with
few allowances. The added benefit - less entities for Dart team to
create and less things for Dart users to learn.

I agree that for sanity's sake it's good to keep your development and deployment experiences similar so you don't get surprises when your deployed app behaves differently.

However, we have some challenging constraints with Dart. It is a client-side language that runs in a browser. That makes all kinds of stuff hard. The workflow that makes the most sense for development is different from how you want outside users to use your app.

In particular, when deployed:

- You want to minimize server round-trips. That likely means concatenating code.

- You'll probably want to minify code.

- Since the browser is requesting your app, you can't rely on any complicated import resolution logic. Browsers don't understand search paths, or version constraints or any of the other sundry details of package management.

Meanwhile, when you develop:

- You don't want to do a slow "compile step" every time you make a change to your code. Client-side web developers expect to be able to just refresh to see their changes.

- You don't want to have to have lots of copies of the same package laying around.

To satisfy all of those constraints, we're loosening things so that deployed apps are a bit different than in-development apps.

 

A simple Dart-based
HTTP server could completely handle all issues here:
a. There's already HTTP server available in Dart, so it can be
extended.
b. It can be integrated very naturally with Dart Editor (and other
IDEs)
c. It can implement the same package resolution as necessary via
global repository/local cache and expose packages this way via HTTP
interface with no security implications.
d. <script> can use the same form as import "package-name:abc/
lib.dar". Again, less to learn.
e. No "paths" outside of the project's directory except for those
strictly enforced by convention in adjacent folders and local cache.
f. If there's web server already, a simple plugin should do.

Yes, this is what "pub serve" is.

 

g. Finally, Dartium could support this package resolution natively.
Again, it's strict and well-defined, so some allowances could be made.

Nearly the same model can be used in production too.

I think this model works well in development, but for deployed apps, I think users want as few moving parts as possible. We don't want to say "if you're using Dart, you must run this specific web server in production". Instead, "pub deploy" will bake in all of the import resolution logic into your Dart code (by rewriting the imports to vanilla URLs) so that you can run a deployed Dart app on any stock web server.

 

standard package layout,

Yes.

 

provide simpler discovery

Package hosting and discovery is planned, but farther out. This doc just covers the first phase.

- bob

[Sam McCall]

On Tue, Apr 10, 2012 at 10:08 PM, dvoytenko <dvoy...@gmail.com> wrote:
>
>> I personally think that Maven is overly complicated, and mostly without
>> reasons. But I agree that the repository/packages part of Maven is pretty
>> well done -- it's the "project automation" part that looks horrible to me.

I spent plenty of time doing things the maven way, and I agree, the
project automation is a disaster. Using it as a simple package manager
works okay, but it's orders of magnitude more complex than other
languages' package managers for no benefit.
It's more popular than e.g. gems only because Java is more popular
than Ruby; almost every Ruby developer consumes gems, the same is not
true for Java/Maven.

> I agree that some parts are complicated and Pub probably won't have
> anything to do with those parts. But the core is strong and very
> simple:
> - Layout convention

Maven's layout convention is pretty terrible: traversing
mypackage/src/main/java/com/mypackage gets really old, most of this is
Maven's fault.

> - Repository
This is mostly solid: packages served as versioned archives over HTTP
is good, with package metadata alongside. The proliferation of brittle
maven-metadata.xml files, and .md5, .sha1, .asc, .asc.md5, .asc.sha1
files is a mess. Custom artifacts aren't needed for package
management.

> - Package resolution
It's hard for me to tell whether maven's "I'm going to download the
internet now" tendencies were due to design or implementation, but the
user experience was pretty terrible.
If they've stopped encouraging people to pin dependencies to specific
versions, then the model should work...

> - Namespace
Package names are too complex in maven (org.apache.commons:commons-math:2.2).
Lots of simpler approaches work here: gems ('commonmath'), github
('apache/math'), cpan ('Math::Commons'), even java
('org.apache.commons.math').

> A lot of complexities can be also attributed to Java.

To me it feels the other way around - java is a fairly simple
language, albeit verbose.
Much of the complexity around Java is cultural, and Maven is a shining
example of Java culture gone wrong (e.g. a great example of how not to
use XML!)

Just my 2c, I'm sure lots of people feel more warmly towards Maven and
I'd be interested in more details. But I spent over a week learning
enough about Maven to get a (somewhat complex) set of packages
deployed, and for me that qualifies it as a failure.
(Or myself, but I prefer not to think about that...)

Cheers,
Sam

[dvoytenko]

Bob,

>> Upgrading to new versions of stuff is a relatively big deal and I think most developers want to know when they are choosing to do that.

Rereading this, I agree. This make sense. Explicit download/update
could have its pitfalls, so it'd be better to avoid.

>> Namespacing, at least in terms of what names are bound in your Dart code when you import a library is a different area and is outside of pub's scope.

>> Imports will specify *libraries* but those have a slightly different granularity than packages. A package may (and likely will) contain multiple libraries.

This might actually be a sticky point, or it might be just me misusing
terminology. I'm aware that Dart allows optionally to use namespaces
when importing. Here, by namespacing, I meant general convention and
expectation from package names and what's inside. I.e. where Dart
would expect to find the Pub "dependencies" file. My feeling was that
if this is not standardized right away the bad conventions in layout
could quickly multiply. I guess this is why Maven addresses artifacts
via group id and artifact id. I personally never really liked it, but
I'm not sure if there are other options. Confusion might come in if
someone starts deploying "dependencies" file irregularly, i.e. trying
to create sub-packages with nested dependencies file. That could
quickly render repository "less flat" and hard to navigate.

I guess the question here is that could you clarify some specifics
behind considered repository structure?

>> Right, but Maven does this at build time, right? Dart doesn't have a "build time". In either case, there should be no extra entities to support: the "packages" directory is just an artifact created and automatically managed by pub.

But the explicit "update" command solves this, right? The question
here is, though, is "packages" directory strictly necessary here?
Clearly, something like this might be needed, if one decided to
override an existing package for some reason (fork it?). But
otherwise, do these packages really solve the purpose. Or would
something like a (hidden) "expanded dependencies" file with transitive
dependencies do just as well?

As far as an option to override an existing package inside of
"packages" directory - it certainly sounds interesting. But, what if
someone had a project with a set of adjacent packages and wanted to
override an existing external package for all of them, not just one?
This could get messy quickly.

Another point I wanted to stress as well - version control. You
haven't yet addressed this issue I understand, but you will certainly
have to do it sooner or later. I guess the question becomes what ends
up being committed into a version control and what doesn't. With
version constraints (vs pinning), it's possible that two runs of
update command would "link" a project to different versions of
package. It'd certainly be dangerous if one person ran update and
committed a project to git and another developer pulled it up, ran
update again and get the different set of dependencies versions. How
do you see this working? If there's a "packages" directory, would it
be committed with all internals? This might also be easier to
accomplish with a single "expanded dependencies" file.

>> - You want to minimize server round-trips. That likely means concatenating code.
>> - You'll probably want to minify code.

Isn't this what "pub deploy" for? I mean, we all do this today all the
time in Java and other environments.

>> - Since the browser is requesting your app, you can't rely on any complicated import resolution logic. Browsers don't understand search paths, or version constraints or any of the other sundry details of package management.

I wasn't suggesting a more complicated resolution logic here. As far
as I can tell, a trip to ~/.dartpub/commonweb/mylib.dart would be just
as fast as to ./packages/commonweb/mylib.dar?

>> "pub deploy" will bake in all of the import resolution logic into your Dart code (by rewriting the imports to vanilla URLs) so that you can run a deployed Dart app on any stock web server.

This makes sense. So, in a way, when you create one big Dart package,
you sort of plan to "materialize" all packages inside of "packages"
directory? And then rewrite imports/script tags throughout? It'd be
great to hear more about this. I'm still mostly considering Dart in
the world where you can't get too far w/o some form of server-side
logic where you pretty quickly have to choose a specific web server...

[dvoytenko]

Well, thanks! This is super subjective. Maven is an optional system
and no one is required to use it - just download a library and put it
into your project. Yet, dozens of millions of download per week prove
different. All while, Oracle (and Sun before it) have rather cold
relationship with Maven. I specifically outlined _only_ package/
repository management features of Maven not to mix in "project
automation", not sure even how it got implied.

> works okay, but it's orders of magnitude more complex than other
> languages' package managers for no benefit.

This is also subjective. Maven's package management core is tiny and
extremely simple - combined size is probably under 2M of compiled
bytecode or less. The whole thing is managed by about 10 interfaces.

> Maven's layout convention is pretty terrible: traversing
> mypackage/src/main/java/com/mypackage gets really old, most of this is
> Maven's fault.

No, that's not Maven's fault. Other packaging systems for Java often
have a more complicated layout than this. Bunch of that stuff comes
from Java being Java. And what does this has to do with Dart? Why
would Dart want to borrow on a Java conventions? Plus, who ever
traverses over Java packages?

> maven-metadata.xml files, and .md5, .sha1, .asc, .asc.md5, .asc.sha1

Well, Dart doesn't have to have these artifacts. Though if an HTTP
download involved it's kind of hard to imagine not to have some form
of hash involved sooner or later. Not all of these are useful and
those that are could probably be arranged a bit differently, but then
again, no one is proposing using Maven itself for Dart.

> files is a mess. Custom artifacts aren't needed for package

Which custom artifacts? Sources? Well, Dart is a source-only language,
it doesn't need sources anyway.

> Package names are too complex in maven (org.apache.commons:commons-math:2.2).
> Lots of simpler approaches work here: gems ('commonmath'), github
> ('apache/math'), cpan ('Math::Commons'), even java
> ('org.apache.commons.math').

Well, Dart doesn't have to borrow on a complex 3-tier package
addressing. But addressing early own the namespace issues is
important. As long as the namespace simple, clear and unique - it's
fine - pick the simplest solution. Java suffered through this a lot.
How many libraries out there have package "org.xml.sax"?

> To me it feels the other way around - java is a fairly simple
> language, albeit verbose.
> Much of the complexity around Java is cultural, and Maven is a shining
> example of Java culture gone wrong (e.g. a great example of how not to
> use XML!)

Not substantiated. Why would anyone use Maven if it didn't make their
lives easier? Use Ant?

What does the XML have to do with anything? I wasn't proposing that
Dart should use XML...

> Just my 2c, I'm sure lots of people feel more warmly towards Maven and
> I'd be interested in more details. But I spent over a week learning
> enough about Maven to get a (somewhat complex) set of packages
> deployed, and for me that qualifies it as a failure.

Java comes with a pretty steep learning curve. But I'd maintain Maven
does NOT make it harder. You could checkout any one of massive open
source Java projects (from Apache or others) and be modifying and
recompiling it within minutes thanks to Maven. You could also find any
specific piece of code on Maven Central repository in a matter of
seconds and add it to your list of dependencies.

Then again, I wasn't trying to plug for Maven here. I was just trying
to highlight few good features that Dart could borrow on.

[Sam McCall]

I guess my point was that once you strip off the parts where 'dart doesn't have to do it that way' and 'that's Java's fault', there's nothing maven-specific left, just ideas common to all package managers.

On Apr 11, 2012 12:58 AM, "dvoytenko" <dvoy...@gmail.com> wrote:
>
> Well, thanks! This is super subjective. Maven is an optional system
> and no one is required to use it - just download a library and put it
> into your project. Yet, dozens of millions of download per week prove
> different. All while, Oracle (and Sun before it) have rather cold
> relationship with Maven. I specifically outlined _only_ package/
> repository management features of Maven not to mix in "project
> automation", not sure even how it got implied.
>
> > works okay, but it's orders of magnitude more complex than other
> > languages' package managers for no benefit.
>
> This is also subjective. Maven's package management core is tiny and
> extremely simple - combined size is probably under 2M of compiled
> bytecode or less. The whole thing is managed by about 10 interfaces.

2M byte code and 10 interfaces is what I'd call large and complex!

> > Maven's layout convention is pretty terrible: traversing
> > mypackage/src/main/java/com/mypackage gets really old, most of this is
> > Maven's fault.
>
> No, that's not Maven's fault. Other packaging systems for Java often
> have a more complicated layout than this.

The src/main/java is from Maven. At most, you need src there.

> And what does this has to do with Dart? Why
> would Dart want to borrow on a Java conventions?

If you mean maven conventions, that was my point :-)

> Plus, who ever
> traverses over Java packages?

Everyone using command-line tools to manipulate the filesystem.

> Which custom artifacts? Sources?
Maven allows you to have arbitrary named artifacts per POM, this is part of the repository format.

> > To me it feels the other way around - java is a fairly simple
> > language, albeit verbose.
> > Much of the complexity around Java is cultural, and Maven is a shining
> > example of Java culture gone wrong (e.g. a great example of how not to
> > use XML!)
>
> Not substantiated. Why would anyone use Maven if it didn't make their
> lives easier? Use Ant?

I'm not arguing that maven isn't the best solution within open-source Java-land. I think it's poor compared to package managers *in general* and am not sure what features it is a good example of (again, compared to offerings across languages).

Sorry for my strong reaction... My hope is that a package manager should feel like part of the language, require minimal effort to integrate with, and largely be invisible most of the time. These are aesthetic concerns and so often come with strong feelings and little data...

[Vadim Tsushko]

Bob,
I see pub in last dart-sdk builds. Does it mean we may try to use pub
already?
If so, would you please explain how to use it on some simple example?
I've read initial document but did not actually undartstand how apply
it to concrete sutuation.
Suppose I have such an directory structure on my windows:
c:\projects\my_project\
c:\projects\my_project\lib
c:\projects\my_project\tests
I'm using unittest and currently as unittest library is included in
dart-sdk my tests contains import lines such as:
#import('../../../dart/dart-sdk/lib/unittest/unittest.dart');

What steps I need to do for using new package import syntax in that
configuration?

[Bob Nystrom]

On Fri, May 25, 2012 at 2:37 AM, Vadim Tsushko <vadimt...@gmail.com> wrote:

Bob,
I see pub in last dart-sdk builds. Does it mean we may try to use pub
already?

Yes! However, it's very bare bones right now. We haven't tried to get people to use it yet just because I haven't spent the time writing any docs yet. I figured that time was better spent cranking on core functionality first.

If so, would you please explain how to use it on some simple example?

Conveniently, I just gave a demo on pub, so I can reproduce that here. So let's say you're making a web app that shows framed pictures of cats. (Because, why wouldn't you want that?) You set up something like:

~/catapp/index.html

<html>

<head>

  <title>CatApp</title>

  <script type="application/dart" src="catapp.dart">

  </script>

  <script src="http://dart.googlecode.com/svn/branches/bleeding_edge/dart/client/dart.js"></script>

</head>

<body>

</body>

</html>

~/catapp/catapp.dart

#library('catapp');

#import('dart:html');

#import('package:catpic/catpic.dart');

#import('package:frame/frame.dart');

#import('package:widget/widget.dart');

main() {

  var root = new Widget();

  for (var i = 0; i < 20; i++) {

    root.children.add(new Frame(new Catpic(i)));

  }

  root.runApp();

}

Your "catapp" directory here defines a package. From the imports, you can tell it's using three other packages: catpic, frame, and widget. So you need to tell pub how to find those. In the same catapp directory, you'll create this pubspec:

~/catapp/pubspec

dependencies:

  catpic:

    git: git://github.com/munificent/catpic.git

  frame:

    git: git://github.com/munificent/frame.git

  widget:

    git: https://bitbucket.org/munificent/widget.git

Right now, pub only knows how to get packages from one place: git. Here it lists each dependency and then tells pub how to find it. When pub.dartlang.org is up and running, you won't have to explicitly give it a path like this.

Now you just need to tell pub to work its magic:

$ cd ~/catapp # i.e. run pub from your app's directory

$ pub install

If you have the dart-sdk installed and its bin directory on your PATH, you should be able to just invoke pub like that. When you do that, you should see a bunch of output from git pulling down packages.

Now if you look in /catapp, you should see:

~/catapp/packages/

In there, you'll see directories for catpic, widget, and frame. You'll also have a kittens directory. That's a transitive dependency: catpic uses it, so when you install catpic, pub will install its dependencies too.

Now if you open index.html in Dartium, you should see some glorious pictures of cats.

This is an absolute bare-bones walkthrough of what works now. We're trying to get a couple of more core pieces (the pub.dartlang.org site and versioning) in place, and then we'll spend a good chunk of time writing some (I hope!) really solid docs.

But if you want to start playing with it now, please go for it! We welcome any feedback you have.

I'm using unittest and currently as unittest library is included in
dart-sdk my tests contains import lines such as:
#import('../../../dart/dart-sdk/lib/unittest/unittest.dart');

Pub has some rudimentary support for finding packages in the SDK, but it needs some more work. (In particular, it needs to support reading an environment variable you set so it knows where you have the SDK installed.) That will come along soon.

If you'd like to get a feel for what a hypothetical pub package could look like using the git stuff, take a look at: https://github.com/munificent/catpic. The README there shows exactly what you need to re-use a package: how to add it to your pubspec and how to #import() it.

Cheers!

- bob

[Vadim Tsushko]

Thank you very much, Bob.
I'll give it a try,
Vadim