Combining stack and shake

[Arnaud Bailly]

Hello,

We have a project with multiple packages built using stack, that also contains or depends on non-Haskell software. I have written a shake file on top of that to orchestrate building docker images for all our services and running system tests. This shake file contains rules for triggering build of images that depend on source files. The build itself uses stack inside docker containers (not using stack's docker mode but it could be the case...). The dependency graph looks like that

source files

   ^   ^

   |   | 

   |   +-- service exe <--- service docker image <---+

   |                                                 +-- tests execution

   + ----- tester exe  <--- tester docker image  <---+ 

The problem I am facing is that changing any source file triggers a full rebuild of the 2 containers, although some of the source files are not relevant to one or the other.

I don't know how to solve this issue. I thought of leveraging stack's capabilities to extract a dependency graph and manually check which project to rebuild depending on which source files changed but this seems pretty much redundant.

I would appreciate any insight on how best to handle this kind of configuration.

Regards,

Arnaud Bailly

[Luke Murphy]

Hi Arnaud,

Interesting setup you have there.

So, Shake triggers a build, and you fire up a container and run Stack
inside that? I assume that it does a full rebuild because it is a fresh
container - regardless of Stack managing .stack and project changes.

Perhaps you can look into directly using the built in docker support:

https://docs.haskellstack.org/en/stable/docker_integration/#usage

From the usage notes:

> The project directory and ~/.stack are volume-mounted into the
> container, so any build artifacts are "permanent" (not deleted with
> the container).

That *sounds* like you'd get incremental builds then!

Best,

Luke

> --
> You received this message because you are subscribed to the Google Groups
> "haskell-stack" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [1]haskell-stac...@googlegroups.com.
> To post to this group, send email to [2]haskel...@googlegroups.com.
> To view this discussion on the web visit
> [3]https://groups.google.com/d/msgid/haskell-stack/39b78d6c-5a07-4e05-a4ab-67fe147a9dc0%40googlegroups.com.
> For more options, visit [4]https://groups.google.com/d/optout.
>
>References
>
> Visible links
> 1. mailto:haskell-stac...@googlegroups.com
> 2. mailto:haskel...@googlegroups.com
> 3. https://groups.google.com/d/msgid/haskell-stack/39b78d6c-5a07-4e05-a4ab-67fe147a9dc0%40googlegroups.com?utm_medium=email&utm_source=footer
> 4. https://groups.google.com/d/optout

[Arnaud Bailly]

Hi Luke,

Thanks for your answer!

On Tue, Dec 19, 2017 at 2:31 PM, Luke Murphy <luk...@riseup.net> wrote:

Hi Arnaud,

Interesting setup you have there.

I am not sure how I should take this statement :)

 

So, Shake triggers a build, and you fire up a container and run Stack
inside that? I assume that it does a full rebuild because it is a fresh
container - regardless of Stack managing .stack and project changes.

Actually no, I mount the full project's directory into the container so I (usually) get incremental builds. 

What's bothersome is that I don't have stack's dependencies in shake so I sometimes rebuild too much, e.g. I rebuild the service container even if only the tester's code changes.

I was looking at `stack dot` which produces a dependency graph, but it seemed ridiculous to parse a graphviz file to reconstruct a dependency graph in my haskell code, so I had a look at stack's code itself but it's really not obvious how to get that graph directly.

Perhaps you can look into directly using the built in docker support:

   https://docs.haskellstack.org/en/stable/docker_integration/#usage

Yes, that I will do. It will probably simplify our CI pipeline too... but it will require some work

[Luke Murphy]

Ah, I see. I also saw you are already posting on the Shake list ;)

Also, another thing to consider:

https://docs.docker.com/engine/userguide/eng-image/multistage-build/

Splitting your container into a build/service one and then pay less time
building, if you can't get Shake to recognize when to build and not build.

Hopefully someone else can weigh in with some pointers. I'll stop guessing!

Best,

Luke

On 0, Arnaud Bailly <arnaud...@gmail.com> wrote:

> Hi Luke,
> Thanks for your answer!

> On Tue, Dec 19, 2017 at 2:31 PM, Luke Murphy <[1]luk...@riseup.net> wrote:
>
> Hi Arnaud,
>
> Interesting setup you have there.
>
> I am not sure how I should take this statement :)
>
>
> So, Shake triggers a build, and you fire up a container and run Stack
> inside that? I assume that it does a full rebuild because it is a fresh
> container - regardless of Stack managing .stack and project changes.
>
> Actually no, I mount the full project's directory into the container so I
> (usually) get incremental builds.
> What's bothersome is that I don't have stack's dependencies in shake so I
> sometimes rebuild too much, e.g. I rebuild the service container even if
> only the tester's code changes.
> I was looking at `stack dot` which produces a dependency graph, but it
> seemed ridiculous to parse a graphviz file to reconstruct a dependency
> graph in my haskell code, so I had a look at stack's code itself but it's
> really not obvious how to get that graph directly.
>
> Perhaps you can look into directly using the built in docker support:
>

>    [2]https://docs.haskellstack.org/en/stable/docker_integration/#usage

>
> Yes, that I will do. It will probably simplify our CI pipeline too... but
> it will require some work
>

>References
>
> Visible links
> 1. mailto:luk...@riseup.net
> 2. https://docs.haskellstack.org/en/stable/docker_integration/#usage