CI/release/deploy strategies

[Eric Hauser]

We are currently in the process of setting up a monorepo with Bazel with the intent of it eventually hosting all of our applications. I’m doing some explorations around CI, release, and deployment strategies the repository. There is some solid prior art of setting up a CI system for Bazel monorepo (buildci.py/ci.sh) and a derivative script running on Google Cloud Build has yielded positive results.

For deployments, we have a number of different deployment targets in our applications - Docker pushes, Kubernetes deployments, serverless functions, uploading binaries to a cloud bucket, Ansible scripts, etc. - and over time would like to bring them all into the main repository. I’ve started prototyping with the idea that deployments are defined as a Bazel target:

deploy_bundle(

 deps = [

   :all_build_labels,

   :all_test_labels.

 ],

 run = [

   :first_run_target,

   :second_run_target

 ],

)

For applications that are deployed on merge to master, each “dirty” deployment bundle from the CI run would be scheduled as its own Cloud Build job in a different project (in order to have separate permissions for the service account) i.e.:

for label in bazel query 'kind("deployment_bundle", rdeps(//..., set(dirty files))’

 gcloud gcloud builds submit ... --substitutions DEPLOY_BUNDLE=${label} --config deploy.yaml

We would also provide a release script as some apps will want specific release branches and/or manual deploys. Infrastructure changes have their own Terraform based workflow, so I'm excluding those from the scope here for now.

I haven’t seen anything in open source that accomplishes similar goal. After playing around with this a bit, I did find the chapter in the SRE book on Rapid which seems to have a similar workflow. Any thoughts, references, or in flight initiatives that would be helpful here? Thanks.

[markus....@ecosia.org]

We only deploy kubernetes currently, which makes things a bit easier, but aiming for a similar approach to having these defined as bazel targets. Currently we are running bazel test //... on each commit and utilizing the remote cache, and then we simply run bazel run //:deploy-all which is a k8s_objects rule (from rules_k8s) mapping to all of our services and apps. This strategy is relying on the fact that k8s only updates configs when they have changed and was very simple to set up. We are however slowly starting to reach its limits as we are migrating more and more apps to the monorepo, because the bazel cache downloads of all targets do take quite some time and sending dozens of configuration files to k8s is now also starting to take its time. We will soon have to change the approach, to just building/testing and deploying the targets of modified files and their rdeps only (using git and bazel query based on the ci.sh script in the bazel repo).

In fact we are already doing this for our per-branch deployments and the strategy there is very similar, the only difference is that we are using sky-query, so:
bazel query --keep_going --universe_scope=//... --order_output=no "kind(_run_all, allrdeps(set($modified_files)))

then filter the resulting targets to ones containing "per-branch-deploy.apply" and bazel run all the resulting targets.

At some point I would also like to include our kops and terraform manifests in ci/cd and any non-k8s resources (of which we have very few), but that will still be quite some time away so have not thought about any strategies there. Also currently we only have one set of permissions for ci, so everything is getting released in the same script.