Creating Workers

[Dustin Mitchell]

I'm working on bug 1519892. At the moment, I'm trying to get docker-worker
running in staging, just to know that it can work in a non-taskcluster.net
environment. Proof:
https://taskcluster-staging.net/tasks/L8dHcPnSQ4-yLbv52yly1g

In so doing, I'm running into a bunch of parameters I want to adjust when
building the worker:

* what cloud to build for
* what account in that cloud to use
* what regions, etc. to build for
* what version of the worker implementation to include
* what instance types to build for (hvm vs. pv isn't a thing anymore, but
that idea)
* whether to include CoT keys and which keys
* how to get client credentials (aws-provisioner, literal credentials in
userdata, or soon worker-manager)
* whether to use stateless dns
* where to report log data and whether to use statsum / signalfx
* which features/sidecar containers to include (e.g., relengapi)
* probably other stuff I haven't thought of..

Some of this can be determined from userdata at startup, but some cannot.

All of this has me yearning for some `tc-worker-builder worker-spec.yml`
which would build the required images based on the spec. It would need to
pull in secrets from somewhere else -- env vars, maybe, like we do for
terraform? Then we could store those specs out-of-tree as necessary, and
get the secrets from wherever makes sense (SOPS or passwordstore or
whatever someone outside of Mozilla wants to use).

This seems like a natural next step for docker-worker's `deploy.sh` and
generic-worker's `worker_types/worker_type.sh`. And it makes a nice user
experience: set up your TC environment, then create worker images for it
with this tool, then set those up in your worker-manager. The last step
could probably be automated, too (again similar to the existing tools).

Thoughts?

Dustin