Shake within Shake

[ryan....@gmail.com]

I have this use case: I have SRC files, INT(ermediate) files, and a final OUT(put) file. I'd like to
run shake to get the output file, and once checking that everything looks good, delete the INT files and not have it trigger a rebuild when I run shake again.

I think I can achieve this by having a rule SRC -> OUT that calls its own separate instance of shake (making sure the database is separate from the parent database) with rules to build the INT files. These files won't be tracked by the parent shake and can be safely deleted without triggering a global rebuild.

I would like to know if there are any problems with this solution, or if there is a better way?

Thanks for any help,
Ryan

[Neil Mitchell]

Hi Ryan,

What's the reason for wanting to delete the INT files? Are they
particularly huge? Or is it some other reason to delete them.

While you might be able to fake this pattern by running two copies of
Shake, it's certainly not desirable. I'm currently working on making
file rules easier, and after I've finished that change, you should be
able to define your own file rule that doesn't check INT actually
exists. The mechanism is also likely to be required for dynamic
builds in the cloud, as per
https://blogs.ncl.ac.uk/andreymokhov/cloud-and-dynamic-builds/

Thanks, Neil

[ryan....@gmail.com]

Hi Neil,

Yes the INT files can be huge -- I'm transitioning our neuroimaging pipeline to Shake, so not only can build outputs be a few gigs in size, they can also take a day or so to generate.

I've already implemented a custom build rule, something like this:

instance (ShakeKey k, BuildNode k) => Rule k [Double] where
storedValue _ q = do
exists <- traverse IO.doesFileExist $ paths q
if not (and exists) then return Nothing # replace this with the stored database value?
else fmap Just $ traverse getModTime $ paths q
equalValue _ _ old new = if old == new then EqualCheap else NotEqual

Your answer suggests I could make a separate rule that returns a key's stored database
value if the file doesn't exist? That way it'll only rebuild if its dependencies are out of date but not if the file doesn't exist. That would be what I want. And I guess storedValue could
find the database by looking at its first argument (shakeOpts). I'll see how you implement the new file rules.

Thanks for link, I was already thinking on how to run distributed cloud based builds.

Cheers,
Ryan

On Wednesday, November 9, 2016 at 10:34:21 AM UTC+11, Neil Mitchell wrote:
> Hi Ryan,
>
> What's the reason for wanting to delete the INT files? Are they
> particularly huge? Or is it some other reason to delete them.
>
> While you might be able to fake this pattern by running two copies of
> Shake, it's certainly not desirable. I'm currently working on making
> file rules easier, and after I've finished that change, you should be
> able to define your own file rule that doesn't check INT actually
> exists. The mechanism is also likely to be required for dynamic
> builds in the cloud, as per
> https://blogs.ncl.ac.uk/andreymokhov/cloud-and-dynamic-builds/
>
> Thanks, Neil
>