Alerting people of dangerous deployments

[Peter Bengtsson]

dmajor pinged me and we chatted about coordinating the socorro dev world
with Stability...

Recently we've had some unfortunate fluctuations with stability on
Crashstats. E.g. correlations not working, or monitoring of our cron jobs.
These are mostly due to the long tail of teething problems of moving to the
cloud.

Also, sometimes there's code changes that surprise our users who haven't
had a chance to have an super-active participation on the development.

How about this; we the developers highlight "dangerous code changes" by
including something in the commit message of our code. Kinda like you can
type in "fixes bug 1234567 - bla bla".
(remember, commit messages can have multiple lines and be more verbose
after a line break)

A special keyword would trigger an email to the Stability list.
This could alert the Stability list readers about something that might
affect them in upcoming releases or deployments going out "right now".

It'd be up to the developer (including dev-ops on the socorro-infra
project) and code reviewer to take heed of this and make sure it's a
useful/helpful message.

We can send one email when something goes into stage and one email when
something goes into production. Not sure what's preferred.

Please please please, share your thoughts with me here or directly.

--
Peter Bengtsson
Mozilla Web Engineering