Cloud Shake - shared drive or network server

[Neil Mitchell]

Hi all,

Cloud Shake - the project to allow a shared multi-machine cache of
build objects - is underway.
I've just released Shake v0.17.6, which includes improvements to the
--shared feature. It's definitely not ready for prime time yet, but it
does work, see https://github.com/ndmitchell/shake/blob/master/docs/Cloud.md
for some details.

The original plan was to add a feature called History to Shake. An
entry in the history describes a key, it's dependencies, and it's
outputs. The plan was to provide two ways to satisfy the history:

* Via a shared file area, which records history entries in files.
* Via a web server, with a custom protocol.

That was the plan, but then Simon Peyton Jones suggested that actually
the shared drive approach is sufficient. You can map a drive over the
internet, use NFS, rsync, you can control permissions, you can explore
it with the file system. The only purpose of a web server was to
provide a more efficient communication pipe, but is it actually more
efficient? And it is worth the cost? These are open questions.

At the moment the shared file area approach is implemented, and works.
It hasn't been tested on a remote shared drive, and isn't particularly
well designed to support that, although it might work, and could
certainly be adapted. The web server approach hasn't been finished, so
doesn't work at all. We're left with the question - push forward with
the shared file area and delete the web server, or push forward with
the web server.

I'm keen to get information, such as:

* The shared drive approach is a real pain on some CI system on some
OS because <specific flaw>.
* The shared drive can't work because of <fundamental flaw>.
* The network approach isn't necessary because <a way to emulate them>.

Any feedback welcome! I'm definitely out of my comfort zone.

Thanks, Neil

[Jose Iborra]

The shared drive approach via sshfs seems limited to:
- Unix with root privileges
- Windows 10

The rsync approach doesn’t have the limitations above, but at the cost of increased network bandwidth and space usage. Also some complexity in the build script, since presumably rsync needs to be used before and after calling shake.

Have you considered a file system over HTTP approach? Starting up an HTTP server can be done from user land, so neither the client nor the server require root access. And it would not have the downsides of the rsync approach either.

More guidance is needed on the invariants required for a build system to work well with the cache. Ideally a specific lint pass with helpful diagnostic messages.

Thanks for your continued work on Shake!
Pepe

[Neil Mitchell]

> The shared drive approach via sshfs seems limited to:
> - Unix with root privileges
> - Windows 10
>
> The rsync approach doesn’t have the limitations above, but at the cost of increased network bandwidth and space usage. Also some complexity in the build script, since presumably rsync needs to be used before and after calling shake.
>
> Have you considered a file system over HTTP approach? Starting up an HTTP server can be done from user land, so neither the client nor the server require root access. And it would not have the downsides of the rsync approach either.

When you mean a file system over HTTP, do you mean that Shake sees a
file system, or Shake sees an HTTP server?

> More guidance is needed on the invariants required for a build system to work well with the cache. Ideally a specific lint pass with helpful diagnostic messages.

Agreed. Working on it. The procedure in
https://github.com/ndmitchell/shake/blob/master/docs/Cloud.md#modifying-your-build-system
seems to work in practice, but it's far from the final answer.

Thanks, Neil

[Jose Iborra López]

Sorry, I should have been clearer. By file system over HTTP I meant something like WebDAV but simpler, e.g. see:

https://github.com/mafintosh/expose-fs

So Shake sees an HTTP server, not a filesystem.