Pub cache, packages directories & symlinks

[Thomas Schranz]

Question

Is there a way to tell pub to create 'relative symlinks' instead of 'absolute symlinks' to reference its cache?

Context

It seems like pub is storing the cached packages in ~/ or in the directory specified by the PUB_CACHE environment variable.

Pub is using absolute symlinks which cause a challenge with my vagrant setup.

Vagrant & Symlinks

I'd like to use vagrant (http://www.vagrantup.com/) which provides a convenient way to manage development environments as virtual machines.

One common way to use vagrant is to have your source code repository on your host machine.

Then you do 'vagrant up' which creates/starts a virtual machine and mounts your host directory (usually your code) in the virtual machine.

If I use IntelliJ or Dart Editor on my host machine I want them to know about my code base and be able to navigate through it

so I run 'pub get' on my host.

This creates the cache and also symlinks to the cache. Now my initial problem was that the cache was outside of my source code

repository and therefore was not mounted into the virtual machine which means that the symlinks pointed to a place that did not exist

from the point of view of the virtual machine.

I wanted to work around this by manually setting PUB_CACHE on my host machine as well as in my virtual machine

to the root of my source code repository (/path/to/my/project/.pubcache).

While this is a way to have my pubcache available on my host as well as within the virtual machine it seems like the symlinks from the packages directories

to the pubcache are absolute symlinks.

If they were relative symlinks I could use the IDEs on my host and at the same time run build scripts

within the virtual machine without having to regularly run pub get or rewire the symlinks manually when I switch context.

Workaround?

I will try to re-wire the symlinks manually since I don't run pub get all the time (just need to remember it)

and maybe do a wrapper script that runs pub get and re-wires the symlinks automatically for me.

[Thomas Schranz]

Information about the system cache:

https://www.dartlang.org/tools/pub/glossary.html#system-cache

Information about the PUB_CACHE environment variable:

https://www.dartlang.org/tools/pub/cmd/pub-get.html

[Günter Zöchbauer]

As far as I know the symlinks should go away.

It seems I do not understand your requirements but for deployment you can use dart2dart (see https://code.google.com/p/dart/issues/detail?id=19978#c1) to create a deployable that doesn't rely on symlinks or pubcache (not ready for production yet AFAIK)

[Thomas Schranz]

Thanks for the pointer Günter, dart2dart is really useful and I use it a regularly for deployment-like scenarios.

In my specific case I would like to do 'development' inside and outside of the virtual machine (I don't want to run heavy tools like IDEs inside the VM)

so I need the package symlinks to work on the host system as well as inside of the vm (where the code is mounted in a different path).

Right now every time I run pub get within the virtual machine it causes the IDE on the host to complain that it does not see the packages anymore.

If I then run pub get using the IDE on the host the symlinks are host specific so running the apps within the virtual machine does not work anymore

without running 'pub get' before that.

Another workaround I was thinking about is to define PUB_CACHE on the host and within the virtual machine to use the 'same' path.

(eg /lib/dart-packages/pubcache or something)

That way running pub get on either the host or the virtual machine will not break the symlinks.

Questions about pub serve

I think going forward pub serve might solve my problem if it allows me to not depend on symlinks at all (?)

If I understand correctly pub serve does/will make the packages accessible to the IDE by serving it from a webserver (?)

This way I could run pub serve in the virtual machine and have the host IDE connect to it (?)

Unfortunately as far as I understand I can't use pub serve right now until we've migrated all web ui stuff to polymer

but we're on it :)

[Günter Zöchbauer]

I think the scenario is more clear now but I have no idea how to solve this with methods currently available.

[Bob Nystrom]

On Mon, Aug 25, 2014 at 2:56 PM, Thomas Schranz <tho...@blossom.io> wrote:

Question

Is there a way to tell pub to create 'relative symlinks' instead of 'absolute symlinks' to reference its cache?

Currently, no. But supporting that would be super easy.

 

It seems like pub is storing the cached packages in ~/ or in the directory specified by the PUB_CACHE environment variable.

Pub is using absolute symlinks which cause a challenge with my vagrant setup.

Right and right. We use absolute symlinks here deliberately because they mean that if you move your package directory, the symlinks don't all break.

 

This creates the cache and also symlinks to the cache. Now my initial problem was that the cache was outside of my source code

repository and therefore was not mounted into the virtual machine which means that the symlinks pointed to a place that did not exist

from the point of view of the virtual machine.

I wanted to work around this by manually setting PUB_CACHE on my host machine as well as in my virtual machine

to the root of my source code repository (/path/to/my/project/.pubcache).

Yup, that makes perfect sense. It's one of the reasons the cache directory is configurable.

 

If they were relative symlinks I could use the IDEs on my host and at the same time run build scripts

within the virtual machine without having to regularly run pub get or rewire the symlinks manually when I switch context.

Yeah, having to constantly re-get is a total drag.

 

Workaround?

I will try to re-wire the symlinks manually since I don't run pub get all the time (just need to remember it)

and maybe do a wrapper script that runs pub get and re-wires the symlinks automatically for me.

That's the workaround I'd consider too. The other thing you could do is replace all of your dependencies with path dependencies that point to the packages stored at a location your VM can reach to. In other words, basically vendor your dependencies into your app and then tell pub to use path dependencies to find them. Path dependencies do support relative symlinks so this will then work in your host OS and in the VM. You'll probably need to use dependency overrides to deal with transitive deps. But this will be a pain, and just fixing the symlinks is probably easier in the short term.

Can you file a bug to support relative symlinks to the cache? We're trying to move away from symlinks in general, but that's a long slow process. In the meantime, supporting opt-in relative symlinks to the cache is an easy change. (The usual caveat: that this won't work at all on Windows.)

I'll note that pub build/serve/run don't require these symlinks at all. As long as the cache is in a directory that the VM can see (like you did), they should work both in the VM and on your host machine.

The Dart Editor is also moving (has moved? I'm not sure about the status) to going through pub to resolve "package:" URLs. That means it won't require those symlinks either and should Just Work™. I'm not sure about IntelliJ and the command-line analyzer.

- bob

[Thomas Schranz]

On Tuesday, August 26, 2014 7:33:19 PM UTC+2, Bob Nystrom wrote:

On Mon, Aug 25, 2014 at 2:56 PM, Thomas Schranz <tho...@blossom.io> wrote:

Question

Is there a way to tell pub to create 'relative symlinks' instead of 'absolute symlinks' to reference its cache?

Currently, no. But supporting that would be super easy.

 

It seems like pub is storing the cached packages in ~/ or in the directory specified by the PUB_CACHE environment variable.

Pub is using absolute symlinks which cause a challenge with my vagrant setup.

Right and right. We use absolute symlinks here deliberately because they mean that if you move your package directory, the symlinks don't all break.

+1 great default, makes a lot of sense 

 

This creates the cache and also symlinks to the cache. Now my initial problem was that the cache was outside of my source code

repository and therefore was not mounted into the virtual machine which means that the symlinks pointed to a place that did not exist

from the point of view of the virtual machine.

I wanted to work around this by manually setting PUB_CACHE on my host machine as well as in my virtual machine

to the root of my source code repository (/path/to/my/project/.pubcache).

Yup, that makes perfect sense. It's one of the reasons the cache directory is configurable.

 

If they were relative symlinks I could use the IDEs on my host and at the same time run build scripts

within the virtual machine without having to regularly run pub get or rewire the symlinks manually when I switch context.

Yeah, having to constantly re-get is a total drag.

 

Workaround?

I will try to re-wire the symlinks manually since I don't run pub get all the time (just need to remember it)

and maybe do a wrapper script that runs pub get and re-wires the symlinks automatically for me.

That's the workaround I'd consider too. The other thing you could do is replace all of your dependencies with path dependencies that point to the packages stored at a location your VM can reach to. In other words, basically vendor your dependencies into your app and then tell pub to use path dependencies to find them. Path dependencies do support relative symlinks so this will then work in your host OS and in the VM. You'll probably need to use dependency overrides to deal with transitive deps. But this will be a pain, and just fixing the symlinks is probably easier in the short term.

Can you file a bug to support relative symlinks to the cache? We're trying to move away from symlinks in general, but that's a long slow process. In the meantime, supporting opt-in relative symlinks to the cache is an easy change. (The usual caveat: that this won't work at all on Windows.)

I've added a ticket to the tracker:

https://code.google.com/p/dart/issues/detail?id=20708

I hope the description makes sense. 

I'll note that pub build/serve/run don't require these symlinks at all. As long as the cache is in a directory that the VM can see (like you did), they should work both in the VM and on your host machine.

Awesome to know @ pub build/serve/run.

Does that mean we'll eventually be to tell pub to not even create the packages directories?

Or might that even become a default? :)

 

The Dart Editor is also moving (has moved? I'm not sure about the status) to going through pub to resolve "package:" URLs. That means it won't require those symlinks either and should Just Work™. I'm not sure about IntelliJ and the command-line analyzer.

This sounds amazing. Now I need a time machine :) 

- bob

[Bob Nystrom]

On Wed, Aug 27, 2014 at 4:23 AM, Thomas Schranz <tho...@blossom.io> wrote:

Does that mean we'll eventually be to tell pub to not even create the packages directories?

Yes!

 

Or might that even become a default? :)

https://code.google.com/p/dart/issues/detail?id=15103

That's the goal. We have a bunch of things out of the way. The main things holding us back now are other tools that rely on those symlinks being there.

- bob 

[Thomas Schranz]

Starred :D

http://giphy.com/gifs/TZxu5rrTK9NKw