Non-file targets and dependencies

[David Turner]

Hi,

Is it possible to use Shake to build things that aren't files, depending on other things that aren't files?

The goal of our build pipeline is, ultimately, to build a bunch (~10) of AMIs on AWS. At the moment we achieve this with an assortment of shell scripts orchestrated through Jenkins, and essentially do a full rebuild of everything on each push. This is nice from the point of view of being sure everything is cleanly rebuilt, but it's quite inefficient (in time and money) to do this even though most pushes don't affect more than one AMI at a time, and it's only going to get worse.

We also have dependencies on non-file things as there's one 'base' AMI that all the other AMIs are derived from this. The base one rarely needs rebuilding but we currently rebuild it frequently as we've not got a better way of ensuring it is fresh.

AMIs are, I believe, file-like enough that this question makes sense: they have a name and a last-modified date, at least.

I'd like to move to a more nuanced build pipeline and Shake's monadic dependencies would definitely be helpful for us, but I can't immediately see a way to have rules that target or depend upon things like AMIs that don't appear in the file system. Is this possible?

Apologies if this is completely obvious and I've missed something when reading the docs, and thanks in advance for your advice,

David

[Neil Mitchell]

Hi David,

The great news is that Shake is not file-specific in any way, so
depending on things that aren't files is no harder. Your use case
seems to make sense for Shake. There are two different approaches to
take:

1) Have each AMI also have some file on disk, containing whatever you
think is the "representative value" (e.g. a timestamp). Now you depend
on files, and use files to represent AMIs. The key things here are
that Shake allows you to have a rule run and not update the file, plus
the monadic dependencies, makes this properly equivalent to really
encoding AMI values.

2) Write your own type of rule that really does talk about AMI values.
The module for writing your own rules is
https://hackage.haskell.org/package/shake-0.15.5/docs/Development-Shake-Rule.html.
You basically define a type for AMI keys (names I guess?) and AMI
values (modified timestamps, hashes - whatever changes whenever an AMI
itself changes). You define a Rule instance which gives you a way to
calculate a storedValue. It's then typical to write some
strongly-typed wrappers over rule and apply, which give you the
interface most people are used to.

If your rule type is a bit simpler, you might even get away with Oracles.

Thanks, Neil

[David Turner]

Hi Neil,

That's great news.

We had considered option (1), mirroring our AMI state in the filesystem, on the grounds that it seemed likely to work even if a bit inelegant. Synchronising the local state with that on the cloud seemed the sort of thing that's likely to go wrong.

On the other hand option (2) looks exactly like what we want, and somehow I'd just missed the Development.Shake.Rule module. On reflection it's a bit more complicated than using names as AMIs are immutable and get a randomly generated identifier on creation, but that seems surmountable.

In fact, looking at how general it really is, I wonder if we can even do most of a deployment with Shake. It'd be awfully cool if we could use this to update the instances running in the soon-to-be-live environment without having to do a full relaunch.

Thanks,

David

[Neil Mitchell]

Hi David,

> Synchronising the local state with that on the cloud seemed the sort of
> thing that's likely to go wrong.

Yep, that's certainly a likely failure mode.

> On the other hand option (2) looks exactly like what we want, and somehow
> I'd just missed the Development.Shake.Rule module.

Great. The reason the Rule module is split off separately is that
defining Rules is slightly complex (not very, but not trivial either),
and most users don't need it. I should also confess that the Rule
module isn't as documented as I think it should be, so please do yell
if you have problems.

> In fact, looking at how general it really is, I wonder if we can even do
> most of a deployment with Shake. It'd be awfully cool if we could use this
> to update the instances running in the soon-to-be-live environment without
> having to do a full relaunch.

That would be awesome. If you ever get a chance to blog your
experiences, or share what your custom Rule instance looks like, I'd
be very interested.

Thanks, Neil

[Joachim Breitner]

Hi,

Am Samstag, den 06.02.2016, 17:03 +0000 schrieb David Turner:
> On the other hand option (2) looks exactly like what we want, and
> somehow I'd just missed the Development.Shake.Rule module. On
> reflection it's a bit more complicated than using names as AMIs are
> immutable and get a randomly generated identifier on creation, but
> that seems surmountable.

I have once created Rule instances that reflect git references and git
objects, and the code is at
https://github.com/nomeata/gipeda/blob/master/src/Development/Shake/Gitlib.hs

If you replace the git SHA by the AMI’s last-modified timestamp, this
might help you get started.


Greetings,
Joachim

-- 
--
Joachim “nomeata” Breitner
ma...@joachim-breitner.de • http://www.joachim-breitner.de/
Jabber: nom...@joachim-breitner.de • GPG-Key: 0xF0FBF51F
Debian Developer: nom...@debian.org

[David Turner]

Thanks Joachim, it's useful to see that kind of thing. As a very first proof-of-concept, I've managed to get something that creates some subnets, according to a spec:

http://lpaste.net/151844

This seems to do what I think it should:

- if there is no VPC, it creates one first.

- it is idempotent.

- if the spec for one of the subnets changes, it creates another one matching the new spec.

- it discovers the IDs of things based on their specs, rather than having to have them hard-coded.

Not terribly pretty code, but this looks promising!

One question that came up: what's the Maybe for in this?

rule :: Rule key value => (key -> Maybe (Action value)) -> Rules ()

Does Nothing signify failure or that another rule should be tried? I'm kind of guessing the latter (and that is the sense that rules are 'disjoint' which is mentioned in the docs) but it'd be useful to know for sure.

Cheers,

[Neil Mitchell]

Nothing signifies that this rule doesn't match. Shake picks the highest priority rule that matches, and that rule must not have any overlaps. I'll clarify the docs. 

Thanks, Neil