Sharing build artefacts with absolute paths

[Jose Iborra López]

I want to share the Shake build artefacts across multiple nodes, but unfortunately my build rules contain absolute paths which are node dependent, so the naive strategy of using rsync to copy build artefacts over to all the nodes doesn't work.

After having a though I can see the following options:

1. Rewrite the paths after syncing the Shake database - not possible with the current database api.

2. Remove the absolute paths from my rules by changing the working directory before invoking Shake - not  the best solution, as the paths to input artefacts (e.g. source files) will now become absolute.

I am secretly hoping that the new --share flag will provide a better solution. The reason for the absolute paths in these rules is to share the artefacts within the same node. 

In case it helps we have the following topology:

N_i = node i

R_i = repository i

S_i = shared folder i

N_1

  - S_1

  - R_1 pointing to S_1

  - R_2 pointing to S_1

  - R_3 pointing to S_ 1

  - R_4 ...

N_2

  - S_2 copied from N_1->S_1 but on a different local path in N_2

  - R_1 pointing to S_2, will rebuild everything

N_3

  - S_3 copied from N_1->S_1 but on a different local path in N_3

  - R_1 pointing to S_3, will rebuild everything

Thanks

Pepe

[Neil Mitchell]

Hi Pepe,

You might be interested that Cloud Shake support is starting to arrive
now. If you pass `--share=location` then it will store a shared cache
at the file location. The original plan was to add a server mode,
where you could point Shake at a server to store the results, but
given recent discussions and experience I'm moving towards the idea of
just having the file approach, and telling people to use NFS/rsync or
similar to turn the shared drive support into a server mode. The
advantage is that shared drives can be mounted via S3, file systems
and many more besides, and have traditional properties like
read-only/read-write, simplifying the Shake side and the
administration.

The GHC build system Hadrian is now able to build with a shared cache:
https://gitlab.haskell.org/ghc/ghc/merge_requests/317. I'm still
writing down the invariants around what a build system has to say to
support --share, but it doesn't seem to have been too onerous for
Hadrian.

> The reason for the absolute paths in these rules is to share the artefacts within the same node

It's really much easier with relative paths. It sounds like --shared
might make it easier to share within a node, and fix the problem of
absolute paths at the same time.

I'm slightly surprised that having R_1..R_4 all pointing at S_1 works
that well. If they have differ, won't R1 build everything, R2 clobber
a bunch of that, then R3 put things back like R1 and also end up doing
more work? Or maybe the answer is that it doesn't work super well, and
--shared saves you a lot of pain there.

Finally, I should warn that --shared is pretty new, so you might find
bugs in it.

I'm also keen to get feedback from anyone who says that a full
client/server is valuable over-and-above the --shared flag.

Thanks, Neil

[Jose Iborra López]

Awesome to hear, —share=location should allow us to drop the absolute paths and rely in the shared cache instead of sharing the build folder.

The reason this worked well for us is that R_1 .. R_n very rarely differ. Even if they do, the amount of sharing is rather large and the collisions nil. Briefly, R_i is a list of Hackage packages and S_i is a hash tree of package databases.

One concern over switching to —share is space usage, as the S_i can become rather expensive over time. Surely —prune is the answer to that, and rebuilding old versions of S_i should be a non issue with —share.

But then the shared cache can grow to make rsync unpractical, specially when throwing new ghc versions into the mix. Are there any provisions to prune the —share cache ?

[Neil Mitchell]

> The reason this worked well for us is that R_1 .. R_n very rarely differ. Even if they do, the amount of sharing is rather large and the collisions nil. Briefly, R_i is a list of Hackage packages and S_i is a hash tree of package databases.
>
> One concern over switching to —share is space usage, as the S_i can become rather expensive over time. Surely —prune is the answer to that, and rebuilding old versions of S_i should be a non issue with —share.

It's actually a bit worse than that - currently each copy of R_1..R_n
will have a copy of the relevant pieces for it, so you are likely to
pay a lot more disk space. I would claim that in 2019 worrying about
local disk space shouldn't happen anymore, but I have known companies
where that isn't yet true. I plan to introduce symlink support, which
should reduce space usage, but on Windows you need extra privileges to
symlink, which might be incompatible with your environment.

> But then the shared cache can grow to make rsync unpractical, specially when throwing new ghc versions into the mix. Are there any provisions to prune the —share cache ?

Not yet - but it shouldn't be too hard. Using things like
last-accessed time should be sufficient.

Thanks, Neil

[Neil Mitchell]

Shake v0.17.6 just released switches to Symlinks.

Thanks, Neil

[Jose Iborra López]

Thanks for the symlinks release, we just finished upgrading this build system to use --share

Minimal changes were required to the build rules, consisting of:

1. Adding `produces` to one rule to capture all the outputs

2. Adding `historyDisable` to one target that downloads a big binary

The notes on modifying the build system for `--share` [1] were very helpful in the process.

Building from scratch with a warm cache takes around 5 seconds, saving us up to 2 hours. Not bad !

[1] - https://github.com/ndmitchell/shake/blob/master/docs/Cloud.md