Why did Stack stop using Shake?

[Edward Yang]

This is not meant as a leading question, I am just curious and would like to know more.

According to https://github.com/commercialhaskell/stack/wiki/Stack's-origins Stack used to use Shake. Looking at the dependencies of Stack, it no longer does so. What caused you to rebuild the functionality of Stack into Shake?

[Michael Snoyman]

It wasn't made for deep reasons, it was very likely done for bad reasons driven by Chris and my lack of experience with Shake when working on the initial Stack code. We had a lot of trouble getting reliably coordination between Stack's, Cabal's, and (to a lesser extent) GHC's dependency tracking, and adding in an extra layer we didn't have full understanding of made it worse. I have no idea if using Shake would have been better or worse in the end, but doing it manually - a way Chris and I had a lot of experience with - was the less risky path.

I _will_ say that on other customer projects, we use Shake extensively, so don't take the lack of usage in Stack as an aversion to Shake. It was just a pragmatic decision when we were trying to churn out a first version quickly.

On Sat, Jul 9, 2016 at 3:00 AM, Edward Yang <ezy...@mit.edu> wrote:

This is not meant as a leading question, I am just curious and would like to know more.

According to https://github.com/commercialhaskell/stack/wiki/Stack's-origins Stack used to use Shake. Looking at the dependencies of Stack, it no longer does so. What caused you to rebuild the functionality of Stack into Shake?

--
You received this message because you are subscribed to the Google Groups "haskell-stack" group.
To unsubscribe from this group and stop receiving emails from it, send an email to haskell-stac...@googlegroups.com.
To post to this group, send email to haskel...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/haskell-stack/5c46f622-a261-487b-96fd-3c9a86edb4ce%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Michael Sloan]

Yeah, one thing I've wanted shake integration for is providing is collection and display of performance stats - a coarse impression of where time is spent.

Thing is, we really aren't dealing with a make-like build system involving a variety of different interdependent tasks.  This is shake's killer app / intended domain.  Instead, we've just got some cabal builds to do.  The implementation of concurrently executing such tasks is very simple: https://github.com/commercialhaskell/stack/blob/master/src/Control/Concurrent/Execute.hs

So I'm also not sure if it's the right decision, but I'm not sure how much it matters.  Stack is a lot more than just the "do build steps" bit, though that is certainly the core.

It could be interesting to consider how difficult it would be to switch some choice bits over to shake, but I'd want to see a couple convincing advantages, and I'd hope for the code impact to be low.  Reuse of the performance stats stuff is one potential advantage.

-Michael

To view this discussion on the web visit https://groups.google.com/d/msgid/haskell-stack/CAKA2Jg%2BdzK7rBp9cJ6aLF1UbkBRkntFDWpnxa4KGbQ2VXCgVhQ%40mail.gmail.com.

[Dan Burton]

As I understand, shake is good at untangling "I need this to do that, I need that to do the other" sorts of tasks. Stack has a lot of these things, e.g. "I need to read the stack.yaml to...", "I need to download the resolver build plan from stackage.org to...", and so forth. A deep dive into using shake pervasively may prove beneficial, although I have practically no experience with shake so I could be completely off base here.

-- Dan Burton

To view this discussion on the web visit https://groups.google.com/d/msgid/haskell-stack/CAEYHaY52y5QuFu1QL3fTcgKE57F4hTzT5Au9Vc3Nr6gAstoixA%40mail.gmail.com.

[Edward Z. Yang]

No Dan, I think you're exactly right.

I have a similar feeling when working with the cabal-install codebase.
Without an API like Shake, you manually construct and run the graph of
operations you want to do. Maybe it is simple, but there are so many
other things which you could be caching / parallelizing more aggressively.

Edward

Excerpts from Dan Burton's message of 2016-07-11 10:20:13 -0700:

> >>> <https://groups.google.com/d/msgid/haskell-stack/5c46f622-a261-487b-96fd-3c9a86edb4ce%40googlegroups.com?utm_medium=email&utm_source=footer>
> >>> .

> >>> For more options, visit https://groups.google.com/d/optout.
> >>>
> >>
> >> --
> >> You received this message because you are subscribed to the Google Groups
> >> "haskell-stack" group.
> >> To unsubscribe from this group and stop receiving emails from it, send an
> >> email to haskell-stac...@googlegroups.com.
> >> To post to this group, send email to haskel...@googlegroups.com.
> >> To view this discussion on the web visit
> >> https://groups.google.com/d/msgid/haskell-stack/CAKA2Jg%2BdzK7rBp9cJ6aLF1UbkBRkntFDWpnxa4KGbQ2VXCgVhQ%40mail.gmail.com

> >> <https://groups.google.com/d/msgid/haskell-stack/CAKA2Jg%2BdzK7rBp9cJ6aLF1UbkBRkntFDWpnxa4KGbQ2VXCgVhQ%40mail.gmail.com?utm_medium=email&utm_source=footer>
> >> .

> >>
> >> For more options, visit https://groups.google.com/d/optout.
> >>
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "haskell-stack" group.
> > To unsubscribe from this group and stop receiving emails from it, send an
> > email to haskell-stac...@googlegroups.com.
> > To post to this group, send email to haskel...@googlegroups.com.
> > To view this discussion on the web visit

> > https://groups.google.com/d/msgid/haskell-stack/CAEYHaY52y5QuFu1QL3fTcgKE57F4hTzT5Au9Vc3Nr6gAstoixA%40mail.gmail.com
> > <https://groups.google.com/d/msgid/haskell-stack/CAEYHaY52y5QuFu1QL3fTcgKE57F4hTzT5Au9Vc3Nr6gAstoixA%40mail.gmail.com?utm_medium=email&utm_source=footer>
> > .

[Neil Mitchell]

Very interesting comments from all.

In my opinion, Shake excels at a few domains.

* If you have lots of complex interdependencies, Shake lets you manage
them nicely. That's not really the case here, but is in large
heterogeneous build systems, e.g. the GHC build system.

* If you are writing things quickly, Shake lets you manage
exceptions/retries/robustness quickly. For a project which has the
effort invested that Stack does, that's less important, but for things
like MinGHC (something Stack killed), it was critically important
because no one cared enough to do all this nasty engineering.

* If you are experimenting, Shake provides a lot of pieces (resources,
parallelism, storage) that help explore the problem space without
having to do lots of work at each iteration. That might mean Shake is
more of a benefit at the start of a project than in a mature project.

FWIW, the concurrent execution stuff in the middle of Stack actually
looks fairly hairy! I see a dependency based build system, with
features like staunch mode and parallelism. It's almost certainly an
O(n^2) algorithm in several places, although I suspect n is typically
small enough. I guess it assumes topological sorting and no dead links
- but I would assume all those are guaranteed in advance. esFinalLock
gives me a gentle shiver. esInAction looks like a space leak, unless
forced by actionDo. ActionContext worries me a bit, is it meant to
list all actions that aren't this one? That seems a very weird
argument.

All those comments about Execute.hs are not intended to persuade you
to move to Shake, but I do see plenty of resemblances of Shake in
there. In many ways you have written a mini-Shake.

Thanks, Neil

> To view this discussion on the web visit https://groups.google.com/d/msgid/haskell-stack/1468257939-sup-6083%40sabre.

[Neil Mitchell]

To tie the knot I summarised this as a blog post:
http://neilmitchell.blogspot.co.uk/2016/07/why-did-stack-stop-using-shake.html

Thanks, Neil