Pier: yet another build tool, written using Shake

[Judah Jacobson]

Hi all,

This message is to announce the release of pier-0.1.0.0, a Haskell build tool written using Shake:

http://hackage.haskell.org/package/pier

https://github.com/judah/pier

Pier is similar in purpose to Stack; it uses *.cabal files for package configuration, and uses Stackage for consistent sets of package dependencies. However, Pier attempts to address some of Stack's limitations by exploring a different approach:

1) Pier invokes tools such as ghc directly, implementing the fine-grained Haskell build logic from (nearly) scratch. In contrast, Stack relies on a separate framework to implement most of its build steps (i.e., Cabal's Distribution.Simple), giving it a more coarse control over the build.

2) Pier layers its Haskell-specific logic on top of a general-purpose library for hermetic, parallel, "forward-defined" builds. That library is motivated by tools such as Nix and Bazel, and itself implemented using Shake.  For details, see:

http://hackage.haskell.org/package/pier-0.1.0.0/docs/Pier-Core-Artifact.html

(Interestingly, Stack originally did depend on Shake, but appears to have stopped using it in part due to added complexity from the extra layer of Cabal build logic.)

Pier is already able to build most of the packages in Stackage (specifically, 90% of those in lts-10.3).  Notably, packages with custom Setup scripts are not supported.

The intermediate build layer may be of interest to readers of this list; I'm particularly curious in whether there's interest in splitting it off into a separate library and/or using it in any other projects.

Best,

-Judah

[Neil Mitchell]

Hi Judah,

Awesome! I once went down this route, I didn't get anywhere (lack of time), other than a name (shark 🦈). I think it's a good thing to have. A few questions and remarks:

* you should include the readme in the package so people can see it on Hackage

* why can't you read the subset of stack.yaml if there is no pier.yaml?

* do you use ghc --make or compile each file individually? Compiling individually is much slower. 

* do you actually use Shake.Forward? Is your artefact an alternative to that? Do you think it's something that should replace that?

* General advice is only split it out after you have used it for a while and polished it using your experience. 

* you can't do packages with custom setup. Does that mean if any of my transitive dependencies have a custom setup i have problems?

* Shake is going to get cloud caching soon. I think pier with just reading from a cloud cache (even just inside a company intranet) would have massive value. Hopefully that will be easy.

Really cool to see - I will try it out and link to it from the Shake home page.  

Thanks, Neil

[Judah Jacobson]

On Thu, May 3, 2018 at 9:30 AM, Neil Mitchell <ndmit...@gmail.com> wrote:

 

* you should include the readme in the package so people can see it on Hackage

Good suggestion, thanks!  I think I should also separate the "user manual" part of the Readme into a separate document.

 

* why can't you read the subset of stack.yaml if there is no pier.yaml?

Originally it acted that way, but I suspect the two specs may drift over time so I'm being cautious for now.  Meanwhile you can still use "pier build --pier-yaml=stack.yaml".

 

* do you use ghc --make or compile each file individually? Compiling individually is much slower.

 

I use ghc --make; it's also easier since you don't need to parse out the import statements.  (Though --make's semantics around finding hi-boot files is...annoying.)

 

* do you actually use Shake.Forward? Is your artefact an alternative to that? Do you think it's something that should replace that?

 

Artifact is an alternative to Shake.Forward and doesn't use that module.  I think it *could* replace it, though the API is different (and opinionated) enough that I'm cautious about claiming as such until it's been battle-tested.

 

* you can't do packages with custom setup. Does that mean if any of my transitive dependencies have a custom setup i have problems?

If there's a custom setup for any package (including transitive deps), pier will ignore it and try to build the package anyway.  So it's fine for something like "lens" where it's just used to install haddocks, but it can't install a package like "gtk" (nor its reverse dependencies) since it uses Setup.hs to locate C libraries.

 

* Shake is going to get cloud caching soon. I think pier with just reading from a cloud cache (even just inside a company intranet) would have massive value. Hopefully that will be easy.

That's really exciting to hear!  I look forward to trying it out.

 

Thanks,

-Judah

[Neil Mitchell]

>> * why can't you read the subset of stack.yaml if there is no pier.yaml?

> Originally it acted that way, but I suspect the two specs may drift over
time so I'm being cautious for now. Meanwhile you can still use "pier
build --pier-yaml=stack.yaml".

I would suggest defaulting to being brave. Stack is the incumbent - you are
more likely to get traction if for simple things pier already works out of
the box.

>> * do you actually use Shake.Forward? Is your artefact an alternative to
that? Do you think it's something that should replace that?

> Artifact is an alternative to Shake.Forward and doesn't use that module.
I think it *could* replace it, though the API is different (and
opinionated) enough that I'm cautious about claiming as such until it's
been battle-tested.

Forward has not been battle tested at all - it's merely a proof that it's
possible. I'm certainly open to moving your Artifact into Shake proper,
replacing Forward, if it makes sense - once you have given it some more
battle testing.

> If there's a custom setup for any package (including transitive deps),
pier will ignore it and try to build the package anyway. So it's fine for
something like "lens" where it's just used to install haddocks, but it
can't install a package like "gtk" (nor its reverse dependencies) since it
uses Setup.hs to locate C libraries.

Great, sounds like it's not such a big deal in practice.

Thanks, Neil

[Evan Laforge]

On Thu, May 3, 2018 at 9:30 AM, Neil Mitchell <ndmit...@gmail.com> wrote:

> * do you use ghc --make or compile each file individually? Compiling
> individually is much slower.

Wow, I somehow totally missed that ghc --make has -j now, and it seems
to actually work. How about that. I guess I should switch away from
per-file now! I guess I just toss the *.hs -> [*.hs.o, *.hi] rule
entirely. But I still have to somehow tell shake that ghc --make
Something will update the whole giant set of .o and .hi files implied
by that, which seems hard... unless you can parse ghc output.
Otherwise it's back to the .PHONY days and just run ghc --make every
time, and hope it's quick no notice no changes are needed. If you use
shake with ghc --make, what is your approach?

I assume I still need to figure out dependent C files that are
"imported" via foreign import, but as far as I know that has to be
some kind of crazy hack anyway, e.g. magic comment, or use foreign
import "something.h" and assume something.h has a corresponding
something.c.

But... I suppose this is going to run into ghci's timestamp-only
recompilation, right? Presumably ghc's recompile avoidance would skip
them, which means that ghci would never want to load them. But it
might be an easier sell to say ghc should at least touch the .o file
when it determines it doesn't have to compile it. Actually viewed
this way it seems like an internal inconsistency between ghc --make
and ghci.

[Neil Mitchell]

> Wow, I somehow totally missed that ghc --make has -j now, and it seems
> to actually work. How about that. I guess I should switch away from
> per-file now! I guess I just toss the *.hs -> [*.hs.o, *.hi] rule
> entirely. But I still have to somehow tell shake that ghc --make
> Something will update the whole giant set of .o and .hi files implied
> by that, which seems hard... unless you can parse ghc output.
> Otherwise it's back to the .PHONY days and just run ghc --make every
> time, and hope it's quick no notice no changes are needed. If you use
> shake with ghc --make, what is your approach?

I asked and answered a Stack Overflow question on exactly how to
compile Haskell, see https://stackoverflow.com/a/50724825/160673. I
also wrote some code in the Shake test suite at
https://github.com/ndmitchell/shake/blob/dd6a446626f67ee053232cef979d5bdcf4a3663a/src/Test/SelfMake.hs.

Running ghc --make on every run is a bit brutal - it can take up to
40s in some cases on Windows machines (it runs all the preprocessors
each time from scratch) - but using ghc -M you can do a lot better.
But if you have dependencies through .dep/.deps files you can still
reuse that logic just as before. I just converted the test suite, and
it took a few minutes at most, with no use of phony etc.

> I assume I still need to figure out dependent C files that are
> "imported" via foreign import, but as far as I know that has to be
> some kind of crazy hack anyway, e.g. magic comment, or use foreign
> import "something.h" and assume something.h has a corresponding
> something.c.

Yeah, hacks are still required...

> But... I suppose this is going to run into ghci's timestamp-only
> recompilation, right? Presumably ghc's recompile avoidance would skip
> them, which means that ghci would never want to load them. But it
> might be an easier sell to say ghc should at least touch the .o file
> when it determines it doesn't have to compile it. Actually viewed
> this way it seems like an internal inconsistency between ghc --make
> and ghci.

Yep, I think ghc --make and ghci should match at least!

Thanks, Neil