info
Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss
improving our dev process, reducing hot fixes, and being more perfect
5 views
Skip to first unread message
Lloyd Hilaiel
Apr 26, 2012, 1:57:55 PM4/26/12
to dev-id...@lists.mozilla.org
James Bonacci, our QA hero, recently noted that each successive train brings more "hotfixes". More out of band fixes for critical blocking issues to code that is branched for final testing 2 weeks before it goes live into production. The hotfixes suck because they are extremely costly, require much more dev work to execute, require ops intervention, and reset the QA process.
We got the core team together today to talk about the problem, and brainstorm steps we can take to get higher quality code out with less cost. Here are the conclusions we reached, along with names of people on the hook for making sure stuff gets done:
* (lloyd) testing automation - we already have basic automation figured out, we need to accelerate the implementation of this so that a full suite of browser tests runs at every commit, and on demand. We'll put development resources into this to assist.
* (lloyd/ozten) respect the l10n lockdown - we've the monday before our release is when we "lock down" final locales for production. We need to get everyone involved conscious of that date, what it means, how it works.
* (lloyd/ozten) patterned l10n testing - to reduce QA cost for vetting the final inclusion of shipping locales doesn't break things, we need tools that can automate a majority of this vetting.
* (ozten) RPM breakout - we should investigate and implement a separation of strings and code to further reduce l10n risk.
* (lloyd) Hotfix ownership and playbook - We need a very clear process for making a decision about a high priority bug - hotfix or derail - and we need to ensure everyone knows the precise set of sets to create, integrate, and document a hotfix so that ops can deploy.
* (ozten) Development Process improvements - We should consider a very tight integration of dev and QA the first days of a train cycle, maybe QA (and the community) test directly against our locked down dev environment to find and fix critical issues *before* they hit the staging environment. Ozten's idea, he'll air it out separately.
* (all devs) Frontload Risky Changes - The dev team needs to work harder to land risky patches early in a dev cycle, and excercise restraint later. Each member should challenge late landing patches of high risk.
thoughts? suggestions? stuff missing?
lloyd
We got the core team together today to talk about the problem, and brainstorm steps we can take to get higher quality code out with less cost. Here are the conclusions we reached, along with names of people on the hook for making sure stuff gets done:
* (lloyd) testing automation - we already have basic automation figured out, we need to accelerate the implementation of this so that a full suite of browser tests runs at every commit, and on demand. We'll put development resources into this to assist.
* (lloyd/ozten) respect the l10n lockdown - we've the monday before our release is when we "lock down" final locales for production. We need to get everyone involved conscious of that date, what it means, how it works.
* (lloyd/ozten) patterned l10n testing - to reduce QA cost for vetting the final inclusion of shipping locales doesn't break things, we need tools that can automate a majority of this vetting.
* (ozten) RPM breakout - we should investigate and implement a separation of strings and code to further reduce l10n risk.
* (lloyd) Hotfix ownership and playbook - We need a very clear process for making a decision about a high priority bug - hotfix or derail - and we need to ensure everyone knows the precise set of sets to create, integrate, and document a hotfix so that ops can deploy.
* (ozten) Development Process improvements - We should consider a very tight integration of dev and QA the first days of a train cycle, maybe QA (and the community) test directly against our locked down dev environment to find and fix critical issues *before* they hit the staging environment. Ozten's idea, he'll air it out separately.
* (all devs) Frontload Risky Changes - The dev team needs to work harder to land risky patches early in a dev cycle, and excercise restraint later. Each member should challenge late landing patches of high risk.
thoughts? suggestions? stuff missing?
lloyd
0 new messages