On-demand remote loading of files?

[Joachim Breitner]

Hi,

Crazy, hypothetical ideas ahead :-)

I’m thinking if I can run a shake build on travis and use its deploy
feature. Assume I do not want a full build on every run (otherwise I
wouldn’t use shake). Also assume I also do not want to download a
complete build tree for somewhere, run shake as usual, and upload the
resulting tree.¹

So assume the tree lives somewhere else, and I can tell shake how to
selectively fetch individual files from there, and also how to push
changed files. The actual source files are all locally present. So after
fetching shake’s DB, would it be possible to run shake in a way that it
would only fetch what’s required to build those files whose dependencies
have changed?

Greetings,
Joachim


¹ I have not yet checked, but this simple solution is probably the way
to go.

--
Joachim “nomeata” Breitner
ma...@joachim-breitner.de • http://www.joachim-breitner.de/
Jabber: nom...@joachim-breitner.de • GPG-Key: 0xF0FBF51F
Debian Developer: nom...@debian.org

[Neil Mitchell]

Hi Joachim,

> I’m thinking if I can run a shake build on travis and use its deploy
> feature. Assume I do not want a full build on every run (otherwise I
> wouldn’t use shake). Also assume I also do not want to download a
> complete build tree for somewhere, run shake as usual, and upload the
> resulting tree.¹
>
> So assume the tree lives somewhere else, and I can tell shake how to
> selectively fetch individual files from there, and also how to push
> changed files. The actual source files are all locally present. So after
> fetching shake’s DB, would it be possible to run shake in a way that it
> would only fetch what’s required to build those files whose dependencies
> have changed?

I assume it would be possible in Shake, but I don't have a fully
worked example, so there may be caveats. One way to implement such a
feature would be at the file system level, so the file system is a
user file system where files which haven't been built are fetched from
the remote side when necessary. Assuming you use modtime checks, and
that the metadata is fully sync'd, that would be simple and
transparent to the build system.

You can probably push a similar approach into Shake (which is probably
necessary as Travis frowns on you mounting different file systems, and
on Windows none of that is really feasible). After you've finished
building, the best thing to do is a rsync to get the results on to the
server, so that doesn't really involve Shake. On the Shake side, you
could change from using `need` as is, to using a custom `need` which
checks a cache of modtimes on the server, and if the file doesn't
exist after (e.g. you didn't build it) then you download it. It's not
a trivial build rule to write, but I don't think it's any more
complicated than your git rules (which I'm still trying to figure out
how to simplify).

I can't think of any complications, but I'd be very keen to see the
result if you do try it.

Note that the Google build system,
http://google-engtools.blogspot.co.uk/2011/08/build-in-cloud-how-build-system-works.html,
is based around this idea, and is very wired into the file system. I
haven't checked if the build system they open sourced does that too or
not.

Thanks, Neil