Extensible snapshots & local paths
6 views
Skip to first unread message
Jan von Löwenstein
Jul 15, 2017, 2:16:45 PM7/15/17
to haskell-stack
Hi,
I was excited to read the blog post about this new feature but was disappointed that exactly my main use case for it was explicitly blocked.
I get the argument about snapshots being supposed to be immutable. However, I need to ship the code and dependencies over to the customers side for compliation without internet access. Right now my approach is to `stack build --prefetch --dry-run` and tar up the resulting STACK_ROOT. On customer side I can create a STACK_ROOT and compile without connection. Unfortunately, the tarball is bigger than I'd like. I hoped to build an immutable - yet local path based - custom snapshot that is about the size the dependencies I really have.
Given this use case, would you reconsider if local paths could be allowed? I imagine this is technically easy, you just don't advise having mutable snapshots. I would happily take the responsibility for making sure my snapshots are immutable.
I hope my use case is clearly stated and gets at least some consideration. If you have any further questions or advice how to achieve the same without local paths in custom snapshots that would be appreciated as well.
Best regards
Jan
I was excited to read the blog post about this new feature but was disappointed that exactly my main use case for it was explicitly blocked.
I get the argument about snapshots being supposed to be immutable. However, I need to ship the code and dependencies over to the customers side for compliation without internet access. Right now my approach is to `stack build --prefetch --dry-run` and tar up the resulting STACK_ROOT. On customer side I can create a STACK_ROOT and compile without connection. Unfortunately, the tarball is bigger than I'd like. I hoped to build an immutable - yet local path based - custom snapshot that is about the size the dependencies I really have.
Given this use case, would you reconsider if local paths could be allowed? I imagine this is technically easy, you just don't advise having mutable snapshots. I would happily take the responsibility for making sure my snapshots are immutable.
I hope my use case is clearly stated and gets at least some consideration. If you have any further questions or advice how to achieve the same without local paths in custom snapshots that would be appreciated as well.
Best regards
Jan
Michael Snoyman
Jul 15, 2017, 2:23:03 PM7/15/17
to Jan von Löwenstein, haskell-stack
On mobile, apologies for brevity.
Your use case makes sense. I can think of two workarounds.
1. Allow an explicit opt in to say "allow file paths, I know what I'm doing"
2. Support local tarballs, especially if we include a sha256 of the tarball contents (a feature I'm planning to add next week)
--
You received this message because you are subscribed to the Google Groups "haskell-stack" group.
To unsubscribe from this group and stop receiving emails from it, send an email to haskell-stac...@googlegroups.com.
To post to this group, send email to haskel...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/haskell-stack/16b16642-2997-466a-b7ef-cfe509260bae%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Jan von Löwenstein
Jul 16, 2017, 8:34:04 AM7/16/17
to Michael Snoyman, haskell-stack
Mobile as well so I might not have looked up or tried out things before I ask...
The second option sounds like exactly what I am looking for. That is awesome. Some questions I have though are:
1. Is there an equivalent of `stack unpack` that just downloads the tarballs? I would issue it for everything `stack list-dependencies` gives me.
1.a. Do I recall correctly that `stack list-dependencies` includes those from ghc (i.e. global package db)? How would I exclude those?
2. Can I start the tarball snapshot from system ghc? I ship a specific ghc (matching the one that my stack project uses).
My project is setup like this.
Local development:
This is a multi-package project pointing to `.` and two git submodules.
I'd like to keep local development based on a stackage snapshot so that adding dependencies is straight forward.
Delivery:
- my project dependencies: https://github.com/SAP/bosh-kubernetes-cpi-release/blob/master/packages/bosh_kubernetes_cpi/packaging#L13
- assembled in: https://github.com/SAP/bosh-kubernetes-cpi-release/blob/master/dependencies/prefetch.sh
- this is the part I'd like to use custom snapshot for as the dependencies tarball now has ~300Mb. Most of it is apparently the metadata of the package database (twice because of git). Only a fraction is my actual dependencies.
- my project packages: https://github.com/SAP/bosh-kubernetes-cpi-release/blob/master/packages/bosh_kubernetes_cpi/packaging#L18
Looking forward to see custom snapshots evolve to supporting my use case. I would offer help, but I am still learning Haskell. This is actually my first Haskell based project and while I think I learned a lot since November last year there is still so many things ahead...
Best
Jan
Michael Snoyman
Jul 16, 2017, 11:34:31 PM7/16/17
to Jan von Löwenstein, haskell-stack
I've added a new PR that adds the local file path support and cryptographic hashes.
On Sun, Jul 16, 2017 at 3:33 PM, Jan von Löwenstein <jan.loe...@gmail.com> wrote:
Mobile as well so I might not have looked up or tried out things before I ask...The second option sounds like exactly what I am looking for. That is awesome. Some questions I have though are:1. Is there an equivalent of `stack unpack` that just downloads the tarballs? I would issue it for everything `stack list-dependencies` gives me.
No, but you can just download with wget. You can download from Hackage at:
Or use the FP Complete mirror, same thing.
1.a. Do I recall correctly that `stack list-dependencies` includes those from ghc (i.e. global package db)? How would I exclude those?
I don't know, I haven't worked on list-dependencies.
2. Can I start the tarball snapshot from system ghc? I ship a specific ghc (matching the one that my stack project uses).
I'm not quite sure what you're asking, but I _think_ the answer is: just put `system-ghc: true` in your stack.yaml. You cannot configure this at the snapshot level.
To unsubscribe from this group and stop receiving emails from it, send an email to haskell-stack+unsubscribe@googlegroups.com.
To post to this group, send email to haskel...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/haskell-stack/16b16642-2997-466a-b7ef-cfe509260bae%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "haskell-stack" group.
To unsubscribe from this group and stop receiving emails from it, send an email to haskell-stack+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/haskell-stack/CAJtP4w6DJp1jLu1nPm371Va-07H4x0%3DPw%2BBxKcYBRQRwg6MYvg%40mail.gmail.com.
Jan von Löwenstein
Jul 18, 2017, 2:02:04 PM7/18/17
to Michael Snoyman, haskell-stack
Great to see. I'll probably start to play with a version build from master soon. Any idea when the next release is planned?
1.4.0 was released in March, wasn't it? Release before that was December. Not sure if that already makes a three month release schedule... ;)
To unsubscribe from this group and stop receiving emails from it, send an email to haskell-stac...@googlegroups.com.
To post to this group, send email to haskel...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/haskell-stack/16b16642-2997-466a-b7ef-cfe509260bae%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "haskell-stack" group.
To unsubscribe from this group and stop receiving emails from it, send an email to haskell-stac...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/haskell-stack/CAJtP4w6DJp1jLu1nPm371Va-07H4x0%3DPw%2BBxKcYBRQRwg6MYvg%40mail.gmail.com.
--
You received this message because you are subscribed to the Google Groups "haskell-stack" group.
To unsubscribe from this group and stop receiving emails from it, send an email to haskell-stac...@googlegroups.com.
To post to this group, send email to haskel...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/haskell-stack/CAKA2JgK-dLEoatvX7qYttv9OFQZsioVFUA5cyeWopbMvizRxyg%40mail.gmail.com.
Michael Snoyman
Jul 18, 2017, 2:33:56 PM7/18/17
to Jan von Löwenstein, haskell-stack
Manny is currently working on the 1.5.0 release, but it will not include extensible snapshots. We wanted to give plenty of time to test on master before releasing.
Jan von Löwenstein
Jul 18, 2017, 3:03:27 PM7/18/17
to Michael Snoyman, haskell-stack
Fair enough. Is there a CI build of master? Ideally producing binaries for Linux and Mac.
Otherwise I will have to include 1.4.0, the source and dependencies of master. Could turn out I would not safe a bit right now.
Anyway, great to see extensible snapshots will come and will include the possibility to ship dependencies as tarballs.
To view this discussion on the web visit https://groups.google.com/d/msgid/haskell-stack/CAKA2Jg%2BLOO6E%3D3eZzYkYuYjMur%3Dnv5RooG%3DOEohKAyA0WjzfSQ%40mail.gmail.com.
Michael Snoyman
Jul 19, 2017, 8:06:21 AM7/19/17
to Jan von Löwenstein, haskell-stack
No, we don't publish any CI builds of the executables from master.
To view this discussion on the web visit https://groups.google.com/d/msgid/haskell-stack/CAJtP4w68bt5zSfKHigfGDEdovUucssbZ6%2Bd7RQHov%3DQAVWy1uQ%40mail.gmail.com.
Reply all
Reply to author
Forward
0 new messages