I already tried the exact thing you propose, see
https://github.com/Ladicek/dpm. It works just fine. I'll let Pub people
comment on why they didn't choose this way -- it's likely that the decisive
factor was usage from the browser.

LT

What exactly prevents the use of this technique in the browser?
Please detail.
I do not think that the browser problem.
Even if it is not Chromium.

суббота, 8 сентября 2012 г., 18:55:08 UTC+6 пользователь Ladislav Thon
написал:

Packages directory is like a "virtual" directory.
Alias for the convenience of addressing.
The virtual machine independently seeking a package in the directory.
The browser will include this VM.
VM first searches the local repository and then globally.
Local repositories are searched from top to bottom in a directory hierarchy.
The first found repository will stop search.

For example.
may_app/
  .dartpackages/
    pkg1/
    pkg2/
      .dartpackages/
      pkg4/        
    pkg3/
  test/
   .dartpackages/

In this way we not realy need these repositories
may_app/pkg2/.dartpackages
may_app/test/.dartpackages

This is enough "may_app/.dartpackages" for all packages.
If we run my_app/test/test.dart than local repo wil be found at
"may_app/.dartpackages".

Package manager may support operation to move all higher packages in one
directory.
Ie from "may_app/pkg2/.dartpackages" and "may_app/test/.dartpackages" to
"may_app/.dartpackages".
Like "pub optimize".

You also may tell package manager to move all dependent packages into local
repository from global repo.
Like "pub install --deployment"

I believe that many problems can be solved.

P.S.
When I say..
..The first found repository will stop search.
I wanted to say..
The first found LOCAL repository (NOT PACKAGE) will stop search NEXT
LOCAL repository.
If package not found in local repo the search continues in global repo.

Simple modification in VM, Chromium and Editor source code.

==pseudocode==
#import('package:pkg1/pkg1.dart', version: "1.0.0");

get_package_dir_for_script(script_file, pkg_spec_from_import_directive) {
  local_repo_dir = get_local_repo_dir_for_script(script_file);
  package_dir = null;
  if(local_repo_dir != null) {
    package_dir = get_package_dir_in_repo(local_repo_dir,
pkg_spec_from_import_directive);
  }

  if(package_dir != null) {
    return package_dir;
  }

  global_repo_dir = get_global_repo_dir();  
  if(global_repo_dir != null) {
    package_dir = get_package_dir_in_repo(global_repo_dir,
pkg_spec_from_import_directive);
  }

  return package_dir;

get_local_repo_dir_for_script(script_file) {
  cur_dir = get_dir_where_script_located(script_file);
  while(true) {
    if(cur_dir == null) {
      break;
    }

    if(!cur_dir.contains(PACKAGE_REPO_DIRNAME)) {
      cur_dir = get_lower_dir(cur_dir );
      continue;
    }

     return cur_dir;
  }

==eof==

In Chromium in not "hosted mode" we can simple assume that application is
deployed and all packages preinstalled and located in
LOCAL_REPO_DIRNAME_OF_APPLICATION.
Otherwise, it is considered that the application is damaged.

In browser requests this may be solved like this solution.

This import...
#import('package:cool_package/cool_package.dart', version: '>=1.2.3
<2.0.0');

Translated to...
http://example.com/my_app/packages/cool_package/ge_1_2_3__lt_2_0_0/co...

Where uri segment "ge_1_2_3__lt_2_0_0" is a symlink on sever to local or
global package with appropriate version for criteria '>=1.2.3 <2.0.0'.
This symlink created by package manager.
Browser not need to resolve appropriate version of package. It simply
request package by criteria.
All work must be done before request by the package manager.

There is not right solution from previous reply.
The easiest way, without the need to create symbolic links.
It can do so.
In local repository should be a file for package version resolving.
==.versions==
cool_package >=1.2.3 <2.0.0: 1.5.6
cool_package: 3.0.0
==eof==
version 1.5.6 suitable for ">=1.2.3 <2.0.0".
And version 3.0.0 is latest current version.
Browser VM download this file then analize it and resolve versions of
imported packages.
Then request appropriate url for installed versions.
No other unnecessary movements.
Simple and clear.
And not a single symlink.

SUMMARY
Packages installed into global repo.
From global repo cache possibly install packages into local cache.
Only one local cache on the apllication (for browser packages requests
comptability and other reasons).

Packages cache structures (global and local):
========================
pkg/
  package1/
    1.2.3/
      package1.dart
    1.3.0/
      package1.dart
    2.0.0/
      package1.dart
  package2/
    2.0.0/
      package2.dart
  package3/
    1.0.0/
      package3.dart
========================

Package cache does not contain inside package directories another caches.
All packages and applications used one the same cache (global or local).

REMARK:
++++++++++
If you want to develop or modify package then you need use package as
application.
Copy package in your project directory and then develop your project.
Inside new application at first time you need run "pub install";
++++++++++

Global repo consists of
1. Packages cache
2. References info

Repo structure:
========================
global_repo_dir/
  pkg/
  referenses
========================

Where file "references" is.
========================
package1 1.2.0: C:\Users\jack\dart\cool_app1\pkg
package1 1.2.0: C:\Users\jack\dart\cool_app2\pkg
package2 2.0.0: D:\Project\dart\cool_app3\pkg
========================

It is not necessary. Only if you want to support the safe removal of
packages.
Package manager may check references before "remove" package and warn if
any usage ref exists.
Like this "pub remove --name package2 --version 2.0.0 --force".

Local cache is the directory where you copied packages from a global cache.
With the same structure.
========================
pkg/
========================
You can copy packages into local cache like this.
"pub install --locally".

The local cache is required for correct operation of the browser with
packages not in "hosted mode".
Browser request the package with required version through "lookup table".
VM, Editor and Third Party Tools also use this approach.
This simplifies the implementation of this approach and to maintain code.
"Lookup table" created only by pub manager or corrected manualy if needs.
To swim quickly through the water needs to prepare everything on the
waterside.

"Lookup table" is file with resolved package versions translation.
==VERSIONS==============
cool_package >=1.2.3 <2.0.0: 1.5.6
cool_package: 3.0.0
another_package: 1.0.0
========================
This file (by convention) can be in application root directory.

To realize this needs to be extended import directive syntax.
#import('packages/cool_package/cool_package.dart', version: '>=1.2.3
<2.0.0');

It would also be useful to extend library syntax.
#library('cool_package', version: '1.2.3');
But it is not necessary.

How VM, Editor and Third Party Tools search and import packages with
appropriate version?
Local cache are searched from top to bottom in a directory hierarchy.
The first found cache directory will stop search next cache directory.

If package not found in local cache the search continues in global cache.
Example.
==pkg/cool_package/2.0.0/lib/somefile.dart
#import('packages/unittest/unittest.dart', version: '<=1.0.0');
==
Search steps.
pkg/cool_package/2.0.0/lib/pkg/
pkg/cool_package/2.0.0/pkg
pkg/cool_package/pkg
pkg/cool_package
pkg/
Local cache found here.

Search file VERSION and if found (Not found? run pub install).
Lookup appropriable version through LOOKUP TABLE.
========================
cool_package >=1.2.3 <2.0.0: 1.5.6
cool_package: 3.0.0
unittest <=1.0.0: 0.0.1
unittest: 0.0.1
========================

Check if exists directory "pkg/unittest/0.0.1".
If exists then done.
Otherwise search in global cache "pkg/unittest/0.0.1" directory.
If exists then done.
Otherwise no package installed or something broken.
Recomendtion is to run "pub install" or check pubspec.yaml configuration.

Browser searches easier.
==www.example.com/my_cool_apps/my_first_app/folder2/webgui .html==
<script type="application/dart" src="webgui.dart"></script>
==
Search steps.
www.example.com/my_cool_apps/my_first_app/folder2/VERSIONS/
www.example.com/my_cool_apps/my_first_app/VERSIONS/
Lookup table should be found here by convention.
If not found there is no packages lookup table used in application (I don't
know why..).

Next search steps.
www.example.com/my_cool_apps/my_first_app/folder2/pkg/
www.example.com/my_cool_apps/my_first_app/pkg/
Packages should be here by convention.
If not found there is no packages used in application (May be broken or not
deployed app?).

Then request package.
www.example.com/my_cool_apps/my_first_app/pkg/unittest/0.0.1/unittest...

====================================
To Dart Team.
If you do not like it then let's ask the users.
Need it or not.
As an experiment, it can be implemented as an alternative in parallel.
For example by new import directive.
#import('repo:packagename/path_to_file.dart', [version: 'version']);

In the VM to implement the algorithm search is not difficult. It is very
simple.
In the Editor, I think it will do the same is not difficult.
In the browser, it seems to me, too, it is not difficult to add a few
search request.
The compiler is also simple.

Is Dart Open Source or Not?

Some mistake from me.
To be consistent with the above file "versions transaltion table lookup"
should be located in the local cache.
If he is in the root directory of the appliction (package) then it will
break the search algorithm.
It must be created and updated by package manager.

Because files can be in apllication folders and in the local and global
cache folders.
Here is local cache search algorithm for concrete file.

findLocalCacheDir(String filepath) {
  if(isPathInGlobalCache(filepath)) {
    return null; // No local cache for file that in global cache
  }

  var curDir = filepath.Dir;

  while(true) {    
    if(curDir == null) {
      return null;
    }

    if(!curDir.containsDir('pkg')) {
      curDir = curDir.parentDir();
      continue;
    }

    retrun curDir;
  }

It turns out that I was talking to myself.
I guess I'm crazy.

Hi Andrew,

Please keep a couple of things in mind:

1. We don't usually work weekends, so if you send something out on
Saturday, you likely won't get a response until the next Monday. Also, we
are often not on the same time zone as you, so even though it may still be
business hours for you, we may not be here.

2. A giant email, even if it's perfectly brilliant and full of wonderful
ideas still takes a lot of time to read, process, understand, and reply to.
If you want your thoughts to get full consideration, then it will take time
for us (and the many other people on this mailing list) to consider them.
Working on open source requires patience: all communication is asynchronous.

3. A giant email followed in quick succession by a flurry of revisions
makes it seem like you're still thinking things through. That's fine, but
it sends a signal that maybe the ideas in it are still being worked on. In
that case, many people won't read it and will wait until it seems more
complete. If you have a large proposal, it's probably better to create a
separate doc (Google Docs, a github repo, whatever you prefer) and work on
it there. Then, once you feel it's solid, send it to the list.

4. A wall of text and code with little formatting is hard to read and
understand. Suggestions and ideas (even big ones!) are always welcome, but
the bigger the change, the more work will be required on your side to
clarify it, communicate it, and make it comprehensible.

Now, onto to your proposal. There is a lot of code there and I don't fully
understand what you're getting at, but let me try to respond. At the very
least, I can give some context which is not apparent but explains some of
why Pub is designed the way it is.

Packages install to a global repository of packages.

Yes, pub does that now. But, it's important to realize that this only
happens *on developer machines* (and possibly production servers). Dart
application end users are not pub users. They do not have package
repositories.

We considered this, but we are intentionally going out of our way to
*not*put version numbers in import directives. This is one of the main
reasons
pubspec.yaml exists: it gives you a place to put version constraints
outside of Dart code.

There are two main reasons for this:

*1. Putting a concrete version number in there leads to version
lock.*Let's say I have package foo and it does:

  #import('package:bar/bar.dart', version: '1.2.3');

Then in my app, I import foo. But I'm also using bar, but a different
version:

  #import('package:bar/bar.dart', version: '1.5.2');

That's bad. We don't want to bake specific details about how a dependency
is loaded (in this case "specific details" means "concrete version")
because it doesn't play nice with transitive dependencies. We want the app
developer to have control over how all of its dependencies are located, *even
transitive ones*.

So, the idea is that dependencies are "abstract" in your Dart code. You say
*what* you want ("some package named bar") but not *how to get it* (some
physical URL or concrete version number).

*2. We don't want to bake package management details into the language
specification. *We want to keep pub decoupled from the Dart language
itself. The language moves much more slowly than pub does, so it would be
painful if every change to it (like what you're proposing here) required
changing the language specification itself.

By having a layer of abstraction here, we can iterate on the package
manager (or replace it entirely!) without interfering with the language.

This means that we can't bake details of how it works into the Dart import
syntax. Things like the version constraint syntax are specific to pub and
not something we want a Dart implementation to have to care about.

Let the virtual machine will load the correct version.

Yes, but where does it load the correct version *from?*

Even when you *do* have the Dart VM in a browser, it needs to load
dependent packages *from the application's server*, not from the local file
system. If some user goes to myawesomeapp.com and the Dart app there
depends on some_package, that package isn't going to be on the user's local
file system. It's going to be on myawesome.com. How does your propsal
handle that?

How does it work with dart2js where there is no Dart VM at all?

Cheers,

- bob

Am 10.09.2012 22:35 schrieb "Bob Nystrom" <rnyst...@google.com>:

Wow, the now following four points could easily be transformed into some
dev-list netiquette! (or did they come from such?).

On Mon, Sep 10, 2012 at 8:31 PM, Dirk Detering <mailto...@googlemail.com>wrote:

Nope, pulled them off the top of my head. I think it says really fantastic
things about our community that we haven't needed to have any such list.
Everyone here is awesome.

(And, just to be clear, Andrew is too. He's put a lot of thought and effort
into his proposal, so it's natural to be frustrated when it seems that's
met with silence.)

- bob

Server and Browser VM do not know anything about "pubspec.yaml" files.
That is correct.
But what if I need to configure my library(es)?
How can I tell VM about that?

I propose to have a configuration file.
One file per library.
Like this example.

==mylibrary.config.yaml==
# This file is not a replacement of "pubspec.yaml".
# But "pub manager" may use this file for it own purpose.
# This file is library configuration file.
# You can not create this file. It will work without it.
# But if you need special settings then the file will be useful to you.

name: mylibrary
#
# Name must be the same as library file name.
#

version:
#
# If not specified it is assumed that the version 0.0.0
#

url: "examples"
#  
# This is an aplliction relative url.  
#  
# If this is your own inside application/package sub-library.
# By example. examples/example_how_it_works.dart
# We tell about that VM.
# If we run this library VM will know where main library (root) located.
#
# If we run this library directly than VM correctly
# search local cache by url at "../packages".
#
# In another case, the VM searches the local cache at default location;

import:

  remote_package1:
    url: http://www.awesome.com/packages
    #
    # url: http://www.awesome.com/packages/remote_package1/current
    #

  remote_package2:  
    url: http://www.awesome.com/packages
    url_type: repository
    #
    # url: http://www.awesome.com/packages/remote_package2/current
    #

  remote_package3:  
    url: http://www.awesome.com/packages
    version: 5.0.0
    #
    # Repository is default value for url_type
    # And url will be combined from "url" and "package_name" and "version".
    # url: http://www.awesome.com/packages/remote_package1/5.0.0
    #

  remote_package4:
    url: http://www.awesome.com/packages/remote_package4/6.0.0
    url_type: direct
    #
    # url: http://www.awesome.com/packages/remote_package4/6.0.0
    #

  physical_location1:
    url: file:///home/jack/dart/development/physical_location1
    url_type: direct
    #
    # url: file:///home/jack/dart/development/physical_location1
    # Not use url "file://" in production mode.
    # "Pub manager" may detect this and warn if run "pub deploy".
    #

  installed_package1:
    version: 1.0.0
    #
    # We want concrete version and we don't specify url.
    # Default location.
    #
    # Sever VM - local and global cache.
    #
    # Browser VM - baseUrl/packages/installed_package1/1.0.0
    #
    # dart2js compile this into one file.

  installed_package2:
    #
    # This is not necessarily. There is no useful information.
    #

==mylibrary.config.yaml==

http://ajax.googleapis.com/ajax/libs/jquery/1.4.4/jquery.min.js
http://dart.googleapis.com/dart/packages/unittest/1.0.0/unittest.dart
Compare this.

Only needs for version resolving this file.
http://dart.googleapis.com/dart/packages/unittest/versions.txt
==versions.txt==
1.0.0
1.2.0
1.8.0
==versions.txt==

It is a remote cache.
Local and global cache on a machine have the same structure.
You can open your global cache for others through internet.
Cache is not a "pub.dartlang.org".
Cache is similar to CDN (Content Delivery Network).

1. If you get access through internet for your global cache.
2. And add file "packages.txt".
http://yourdomain.com/dart/packages/packages.txt
==packages.txt==
unittest
awesome
==packages.txt==
REM: "Pub manager" update this file for you when installs packages.

3. Then add folder "packed" in each package folder.
REM: "Pub manager" by default stored downloaded packages in this folder.

http://yourdomain.com/dart/packages/unittest/packed/1.0.0/unittest.zip
http://yourdomain.com/dart/packages/unittest/packed/1.2.0/unittest.zip

You get lite version of package repository.
We call this "CDN repository".

If all Dart VM's will use CDN-like cache natively it is this will only
benefit.

==code.dart==
import(package:unittest/unittest.dart);
==code.dart==

==library.config.yaml==
import:
  unittest:
    # Later we change url and it is still works.
    # url: http://dart.googleapis.com/dart
    # cache structure the same
    # Only <scheme>://<authority><path> changed.

==library.config.yaml==

P.S.
Last in list element is a current version.
==versions.txt==
1.0.0
1.2.0
1.8.0
==versions.txt==

Andrew,

I'm having a very hard time following this thread. There's so much text I
can't tell if it's interesting and if I should read it all or not. Can you
summarize?

Cheers,
  Justin

All the replies can not read.
Read the last three reply. After Bob Nystrom reply.
VM configuration file is not a main sense.
It is only a possible product of this proposal.

=======================
SUMMARY:
The basic meaning of the unification method of organization of the package

cache.
I propose to use CDN-like style cache.
Both for server and web appications.

Like this.
http://ajax.googleapis.com/ajax/libs/jquery/1.4.4/jquery.min.js

http://dart.googleapis.com/dart/packages/unittest/1.0.0/unittest.dart
file///home/jack/dart/global_package_cache/packages/unittest/1.0.0/unittest .

dart

CDN-like style cache with some minor enhancements to turning this cache
into package repository.

1. With support listing installed packages in cache.

Possibly like this:
http://dart.googleapis.com/dart/packages/PACKAGES.TXT
==PACKAGES.TXT==
unittest
awesome
==PACKAGES.TXT==

2. With support listing of versions of installed packages in cache.
http://dart.googleapis.com/dart/packages/unittest
==VERSIONS.TXT==
1.0.0
1.1.0
1.2.0
1.8.0
==VERSIONS.TXT==

3. Maintenance will be the responsibility of the package manager.
Any manager not only "pub". If desired, even by hand or other usefull tools.
Not package manager dictates the rules how organize the cache localy,
globay or remotely.
One style (structure) cache native used by all VM's.
=======================

Lot of writing code to implementation this?
I do not agree.

среда, 12 сентября 2012 г., 4:58:32 UTC+6 пользователь Justin Fagnani
написал:

At current time, the package manager does two jobs.
Ours and theirs.
1. Ours
Finds and installs the packages.

2. Theirs.
Dictates the rules of integration packages in the application.

VM do not care where the package located.
It is stupid and hungry.
VM relies on the work of the package manager. It depends on him.

The package manager must adapt to the VM.

At the same time, the VM must be progressive and use current technology.
Main advantage of Dart is a web applications.
CDN is not applicable to web applications?
When I'm talking about, I do not mean CDN speed and other advantages.
No. I'm talking about the compatibility of the method to search and access
data.
In this case, search and access to the packages.

If I want the latest version of the package.
What should I do now?
If this is the localy that I will update the package.

But what is my app run on other machines?
I cannot update packages on all other machines.

But if VM use CDN-like cache there is no problem.
All other machines will get from CDN-like cache latest current versions of
packages.

1. VM request list of packackes from cache.
2. If package found then request list of versions.
3. Get last version from he list.
3. Then reference data by the respective protocol. In this case 'http://'.

On local mashine all also only another protocol 'file://'.
Simple install new version in global cache and all done.
VM independently find the latest version.

Sorry, I never got a chance to reply to this last week.

If I understand you, you're proposing:

1. That all Dart implementations (including Dartium) support downloading
and caching Dart packages. Sort of like the Global Assembly Cache for .NET.

2. That pub and pub.dartlang.org act as a CDN for packages so that end user
web apps will request packages directly from Google instead of from
whatever web host the app developer is using.

Assuming I have that right, both of those ideas have been considered,
though we haven't put much thought into either one. There are a couple of
problems:

1. Caching and reusing packages on a user's local browser can be a major
security hole. Let's say someone makes a wildly popular Dart packaged
called "underbone". Hundreds of applications are using it. If an attacker
can get a compromised copy of it on a CDN, then any application using it
can be broken. This isn't an entirely unsolvable problem, but it's a very
hard one.

2. While Google has lots of bandwidth and storage, being the world's Dart
CDN is still a pretty expensive proposition. We want Dart to succeed, but I
don't know if we want to fund the entire world's infrastructure for it.

3. Web developers are very careful about which parts of their production
infrastructure they want to outsource. If they rely on some Google package
manager to serve up their Dart code, when that site goes down, their app is
down and there's nothing they can do about it. Uptime is a big deal, which
is why things like AppEngine and EC2 have like SLAs and guarantees and
stuff like that. The Dart team doesn't have the staff or expertise to do
that.

4. Most production web apps are minified before deployment. That minimizes
the number of network roundtrips you do, and shrinks code size. That's
usually done at the application level: you minify the entire app and all of
its dependencies into a single giant blob of code. Reusing packages at
runtime doesn't work in that model. Likewise, I think snapshotting works at
the isolate, not the library granularity.

Both of your ideas are interesting, and at some point in the web's future I
can see it going that way, but I think it's probably too ambitious for the
Dart team to take on right now. Just getting a new language in everyone's
browser is hard enough!

Cheers,

- bob

Almost but not quite so well.

There is no such thing as a cache.
Any cache is a repository for its functionality.
Local repository in the program directory.
Local repository on a disk.
CND repository.

pub.dartlang.org is a vendor of packages. (Soft) warehouse.
You use 'pub' to install package into your local repository.
Git is a vendor of packages.

Each package (even application) MUST HAVE A VERSION.
We assume that the default version is 0.0.0

The repository structure is the same. Functionality varies.
You can not add a package to a remote CDN.
You can not delete a package form a remote CDN.
Remote repository is a no support for version constraint.

But they are all standard repository.
All VMs are able to do at least one operation.
This operation is quickly to resolve (transform) the url of dependent
package in the repository.
Special file for the VM only specified "host" of repository and possibly
version of dependency.
This special file managed by "pub" or manualy.

==dart.code==
import('package:mypackage1/mypackage1.dart');
import('package:mypackage2/mypackage2.dart');
import('package:mypackage3/mypackage3.dart');
==dart.code==

This file fot VM
==dependencies.config.yaml==
mypackage1.host=http://dart.google.org/cdn/packages // Not good idea
request from CDN. But why not?
mypackage1.version=1.2.0

mypackage2.version= >=2.0.0 <=3.0.0 // Only version constraint specified.

//mypackage2=
// I am already on machime. Look me at 'app repo' and 'user repo'.
// if I am a 'web app' then look me at
'http://awesome.com/my_cool_app1/packages'
// Well, I would venture to the new version?
==dependencies.config.yaml==

VM transform this into follwing urls.
For remote:
http://dart.google.org/cdn/packages/mypackage1/1.2.0

For local request used search paths.
[APPLICTION_ROOT]/pub-cache/packages
[USERPROFIFE]/pub-cache/packages

Limitation of the remote repository is a no support for version constraint.

mypackage1.host=http://dart.google.org/cdn/packages
mypackage1.version=>=2.0.0 <=3.0.0 // Error: version constraint not allowed
for remote repository.

You ask and how to be a web application?
Because it is essentially a remote application.
Will help you 'pub'.

He will correct 'vm.config.file' (dependencies file) for your Web
application.
He will adjust the 'vm.config.file' beforehand given the installed packages.

Limitation of web application it needs 'vm.config.file' with no versions
constrait.
This is a limitation of Browser VM.

You can not adjust concrete versions by 'pub' for Standalone VM.
Standalone VM works corretly with version constraits because VM can search
files in local repo.

You run 'pub install'.
Pub does not create symlinks.
He install packages in repository.
Pub create correct 'vm.config'.

Pub do everything for you.
configure the VM

=='local.vm.config'==
[vm.options]
memory_size=768M
[vm.options.development]
option1=value
[packages]
mypackage1.version=1.0.0
mypackage2.version=5.0.0
=='vm.config'==

Is VM config file bad solution?