Udai Gupta <mailtoud@gmail.com>
Proposal: Stable development for frequent releases
20 messages
Udai Gupta <mailtoud@gmail.com> Fri, Sep 10, 2010 at 6:47 PM
Hi,

I have created a proposal for more stable development. You can vote
1-10 and help me to improve this proposal to make it a 10. I would be
bringing this as a topic in Engineering meeting.

Udai
stable_branching_proposal.txt
3K
Keith Woodlock <keithwoodlock@gmail.com> Fri, Sep 10, 2010 at 10:07 PM
To: Udai Gupta <mailtoud@gmail.com>
Udai,

 I would give it a 5 at the moment.

What I liked:

I liked the goal.

>> Problem: We want to come up with a better way of managing our development and release process so that -
>> - We can release stable code more frequently
>> - We can have better and stable development life cycle
>> - We have more reliable QA

>> Current practice:
>> - We follow a release/development cycle where everything is developed and committed to the trunk incomplete and bleeding edge state, then QA comes later at the release time.

>> Proposed practice:
>> - We keep a pre-integration(PI) branch, and trunk(head-master) as our moving release branch which will always be stable. Stable features will be >> tested on the PI and merged into the trunk after QA process. We can cut a release any point of time from our moving release branch (trunk)

What I don't like:

I don't like the proposed practice as I don't think the proposal with
solve the problem stated.

There are other issues that may be more towards the root cause:

To improve the cycle time from concept to delivery we must identify
where the constraints are. I would say
# QA/Testing seems to be to a big one (not only one) at present, by why

1) Too much work in progress per release, why?
# Releases timelines are too long, I would like to see 1.x + MMF
(Minimal marketable features) = 1.y -> Release IT.
-> Shorter release -> less WIP -> less features/stories to test
-> Align upcoming released features with sales and marketing

2) Stories for new work or refactoring should be written in a way that
its easy to assess when its done. Its done when Dev + QA/Tester +
Product Owner says its done... Then it can be moved from its 'feature
branch' into 'release branch'
-> As a result, we need to keep stories reasonably small as we don't
want feature branches moving to far way from latest 'release branch'

3) There is one QA/Tester and now lots of developers
-> developers more accountable/responsilbe for testing
-> another full time tester?

4) The way we organise branching will also be part of the solution but
on its own will only move the current bottle neck of QA from trunk to
'pre-integration' branch.

Regards,
KeithW.
Udai Gupta <mailtoud@gmail.com> Fri, Sep 10, 2010 at 10:58 PM
To: Keith Woodlock <keithwoodlock@gmail.com>
Hi Keith,

Your concerns and suggestions are pretty valid,

I am not proposing How we can remove QA bottleneck.

I am not proposing How we manage stories (small or large) or feature
development.

In simple words I am proposing that every new feature can be released
as trunk will be our moving stable release branch, so it's a different
problem how we can speed up the development of features.

Also I am proposing pre-integration branch which will be stable so
that others don't get affected by unstable changes (not QAed).

Thanks,
Udai
Kojo Gambrah <kgambrah@gmail.com> Fri, Sep 10, 2010 at 11:43 PM
Udai I appreciate your clear separation of the issues. A fundamental problem is that not all acceptance tests are automated but that is quite separate from the branch hierarchy. There are now 1.5 testers - me being the 0.5th. As we do not have a QA staff strength to meet the developer staff strength, it may be prudent to have QA develop test cases and some of the test automation tasks taken on by developers. That way developers are not doing testing as such - they are still writing code.  This perhaps should be a separate proposal. 

Back to Udai's proposal - I am afraid we may still not be able to cut a release from the trunk without additional QA effort because of the partial automation problem. What I mean is that because the trunk is stable does not mean all tests have passed because not all of them are automated.

I give the proposal an 8. The goal of "More reliable QA" in my view is somewhat external to the branch strategy and not addressed by the proposal. The proposal shifts the timing of the QA effort but does not change how much QA or the nature of the QA effort.

Regards,
Kojo
Adam Feuer <afeuer@grameenfoundation.org> Sat, Sep 11, 2010 at 3:35 AM
Udai,

I rate this an 8 out of 10.

I liked:

* that the plan is clearly explained overall
* that the document says why we want to do this
* that the steps to commit are listed clearly

To be a 10:

* Put the plan completely in an email message, not an attachment
* Call the integration branch "Integration" rather than "Pre-integration"
* Specifically address KeithW's point about pushing the problem to QA - for
instance by clarifying that this proposal is only about branch stability,
not about solving the deeper problems that Keith mentions.

-adam
--
Adam Feuer
afeuer@grameenfoundation.org
http://mifos.com
Grameen Foundation's Mifos Initiative
Creating a poverty-free world
Adam Feuer <afeuer@grameenfoundation.org> Sat, Sep 11, 2010 at 3:37 AM
To: Keith Woodlock <keithwoodlock@gmail.com>, Udai Gupta <mailtoud@gmail.com>
On 9/10/10 9:37 , "Keith Woodlock" <keithwoodlock@gmail.com> wrote:
> What I don't like:

Protocol check. When giving perfection information, only state what you
like, and what it will take to make it a 10. :-)

Would you be willing to restate your perfection following the protocol?

cheers
adam
--
Adam Feuer
afeuer@grameenfoundation.org
http://mifos.com
Grameen Foundation's Mifos Initiative
Creating a poverty-free world
Jeff Brewster <jbrewster@grameenfoundation.org> Sat, Sep 11, 2010 at 5:27 AM
Hi Udai,
I give this a 6

What I liked was the idea - having a branch that is more stable.

To be a 10:
- add another goal of new practice - We have more flexibility if we decide to temporarily stop work on a feature
- change name to integration branch (agree with AdamF)
- make this more clear to me:  "Any issue that will be reported by developers apart from feature will be a real bug that we might skip or ignore in development over unstable trunk."
- Add a "dev box" testing step before promoting to the release branch.
- Include TW team on proposal for their ideas and buy in.
- add more clarity about on-the-fence cases - feature is mostly done but has several P2, P3 bugs.  What is threshold for pass/fail?  ( I can try to help with this)

Jeff
Keith Woodlock <keithwoodlock@gmail.com> Sat, Sep 11, 2010 at 6:53 PM
To: Adam Feuer <afeuer@grameenfoundation.org>
Udai, (restating)

 I would give it a 5 at the moment.

What I liked:

I liked the goal.

To be a ten,

# I would like if you reworded the proposal to be clear that you are
proposing changing our branching strategy
# I would like to see a picture/diagram that more clearly shows the
different branches and how dev, releases and maintenance will be
handled (someting like http://nvie.com/git-model)
# being pedantic with this one but could we drop the term 'QA' and
replace with 'Testing/Tester' (others might not agree)

Regards,
Keith.
Keith Woodlock <keithwoodlock@gmail.com> Sat, Sep 11, 2010 at 6:57 PM
Theres also this chapter from git pro book http://progit.org/book/ch3-4.html
Udai Gupta <mailtoud@gmail.com> Sun, Sep 12, 2010 at 9:56 AM
To: Jeff Brewster <jbrewster@grameenfoundation.org>
> To be a 10:
> - add another goal of new practice - We have more flexibility if we decide to temporarily stop work on a feature
Added

> - change name to integration branch (agree with AdamF)
Okay, let's call it "integration" branch

> - make this more clear to me:  "Any issue that will be reported by developers apart from feature will be a real bug that we might skip or ignore in development over unstable trunk."
When there is a WIP in trunk it's hard for others who are not aware
about WIP that weather it's bug or just a side effect. So, if they
report that it's a bug then it might be resolved as invalid (or fixed)
as the other party will complete the work. Now this can result in a
list of lot's of bugs which were not actually a bug but just a side
effect and pollute our bug tracker. (Will add this in new revision of
proposal)

> - Add a "dev box" testing step before promoting to the release branch.
Done

> - Include TW team on proposal for their ideas and buy in.
Sure

> - add more clarity about on-the-fence cases - feature is mostly done but has several P2, P3 bugs.  What is threshold for pass/fail?  ( I can try to help with this)
Great, I would like to know what you have in your mind to improve this. :)
Let's talk on Monday to get this clear. Next revision we can post this
proposal to the dev mailing list.

Udai
Udai Gupta <mailtoud@gmail.com> Sun, Sep 12, 2010 at 10:53 AM
> # I would like if you reworded the proposal to be clear that you are
> proposing changing our branching strategy

I will make it clear in next revision that it's about "More
stability", It's not just branching strategy, it's also how the
features will be developed and brought into the integration and then
master branch.

> # I would like to see a picture/diagram that more clearly shows the
> different branches and how dev, releases and maintenance will be
> handled (someting like http://nvie.com/git-model)

Great, I will add that.

> # being pedantic with this one but could we drop the term 'QA' and
> replace with 'Testing/Tester' (others might not agree)

To me it's just a process where the feature gets accepted and merged
into the master (moving release branch)

Thanks,
Udai
Udai Gupta <mailtoud@gmail.com> Sun, Sep 12, 2010 at 12:26 PM
To: Kojo Gambrah <kgambrah@gmail.com>
Thanks Kojo, I understand the concern here and it's something that
everyone wants to solve.

By "reliable" I meant more stable development, and it's a solved
problem, by adding layer of code integration to make release process
or QA more reliable.

At the point of QA resource being bottleneck, I agree but it's just
another problem that we need to solve. I did not used faster QA or
testing. Right now the release is only made when the all feature
completes. I want every feature(or bug fix) to be able to release and
it only possible if we keep a stable release branch.

Keith has given some links which also shows that it's a standard
practice to feature release.

As an open source project we would have to change the our attitude and
consider other possibilities to bring or increase QA resources.

For e.g. I think that developers can do QA also which is a part of XP
practice. But to get this accepted I think we need a discussion and
some experiments.

It is a possible that even with 1/10 QA/developer ratio can release
stable products in XP environment.
http://www.jrothman.com/Papers/ItDepends.html
http://riceconsulting.com/home/index.php/Testing-Metrics/the-elusive-tester-to-developer-ratio.html
www.kiberle.com/pnsqc1/estimate.doc

Udai
Kojo Gambrah <kgambrah@gmail.com> Mon, Sep 13, 2010 at 9:22 PM
To: Udai Gupta <mailtoud@gmail.com>
Hi Udai,
I completely agree on the integration branch strategy. A similar (if not same) strategy was used for the development of the non-sequential database upgrades. I developed in a feature branch - nonSequentialDatabaseUpgrades, wrote unit and integration tests. I was given a normalised client data for user acceptance testing. Then Adam Monsen came in to do the review and merging. If I remember correctly, he created a new branch and pulled in changes from the nonSequentialDatabaseUpgrades branch and once everything was okay, he merged the code into the master branch. 

This worked awesomely well in my view. I think you could use this case to support your proposal. 
Can you throw more light on the release process for me? 

Regards 
Kojo

PS: Are you going to be at the training in Bangalore? It would be great to meet you in person. 
Udai Gupta <mailtoud@gmail.com> Mon, Sep 20, 2010 at 10:17 AM
I have simplified the proposal, this should not change process of
Continuous Integration much.

Problem: We want to come up with a better way of managing our
development process so that -
- We can release stable code more frequently
- We can have stable development branches

Current practice: Everything is developed and committed to the MASTER
incomplete and bleeding edge state. Which can result in
   *    Blocking release as feature in master is not complete
   *    Result in redundant/Bogus bug because of WIP
   *    Can block development of some other feature.

Proposed practice:
- Merge only stable changes which should not block release.
- Cut next release branch immediately after every release. (maintenance branch)
- Merge changes from MASTER to release branch only after QA.
- The decision of development on a branch and frequency of continuous
integration from that branch depends on balance of risk and release
plan.

Some initial thoughts on flow to decide how the development should
happen (Branching and Continuous Integration).
http://tinyurl.com/37mpqfs (zoom if not visible)

Udai
Adam Monsen <amonsen@grameenfoundation.org> Tue, Sep 21, 2010 at 3:28 AM
To: Udai Gupta <mailtoud@gmail.com>
I give it a 6/10.

I like that it is succinct, and clearly divided into what's wrong and
how we'll fix it.

To make it a 10, I suggest the following changes:

> - We can have stable development branches

* Ger rid of that bullet, the previous one covers it.

Alternatively, collapse "Problem" into:

       We want to come up with a better way of managing our
        development process so that we always have a release-ready
       stable branch in version control.

> Result in redundant/Bogus bug because of WIP

* instead of WIP, say "WIPs (works in progress)". ie:

       Which can result in...
       * redundant/bogus bugs due to WIPs (works in progress)

> - Merge only stable changes which should not block release.

* if you mean "into master", say so. I suggest:

       Only merge stable changes into master, e.g.: those which will
       not block the next release.

> - Cut next release branch immediately after every release.
> (maintenance branch)

* Instead, I'd say:

       The master branch represents the next release. As soon as a
       stable change for a future release must be committed, cut a
       release branch, then commit on master.

Example:
* version 1.0 is released: built from the tip of the 1.x release branch
* 2.x release branch does not yet exist
* joe wants to commit a change for version 3.0. It relies on framework
that will only be part of version 3.0.
* joe creates a 2.x release branch off the tip of master
* joe commits to master

> - Merge changes from MASTER to release branch only after QA.

I'm confused about this. Generally we do bugfixes on release branches,
then merge or port them back to master, yes?

I say strike that bullet. I think development is clear/slow/isolated
enough on release branches that fixes and QA for a release can happen
together on the release branch.

> - The decision of development on a branch and frequency of continuous
> integration from that branch depends on balance of risk and release
> plan.

Clarify what "frequency of continuous integration" means.

> Some initial thoughts on flow to decide how the development should
> happen (Branching and Continuous Integration).
> http://tinyurl.com/37mpqfs (zoom if not visible)

* remove the first part of the flow chart around "development type", it
doesn't provide actionable information
* clarify "No CI to master" or use a different phrase

One last thing, I agree with Kay that examples would help. However, I
don't think they necessarily have to be part of this proposal, but
should be part of documentation we maintain about how we're going to
manage release branches, master, etc. in version control. This might
just mean updating the "Mifos Version Control Guide" on the wiki.
signature.asc
1K
Vivek Singh <vsingh@thoughtworks.com> Tue, Sep 21, 2010 at 2:57 PM
To: Udai Gupta <mailtoud@gmail.com>
Cc: "Keith Woodlock (contact)" <keithwoodlock@gmail.com>, Adam Feuer <afeuer@grameenfoundation.org>, Jeff Brewster <jbrewster@grameenfoundation.org>, Van Mittal-Henkle <vanmh@grameenfoundation.org>, Adam Monsen <amonsen@grameenfoundation.org>, Emily Tucker <ETucker@grameenfoundation.org>, "John Woodlock (contact)" <john.woodlock@gmail.com>, Jakub Sławiński <jslawinski@soldevelo.com>, Kojo Gambrah <kgambrah@gmail.com>, TWteam <teamgrameen@thoughtworks.com>
I would rate this 5/10. In order to get it to 10 we need to look at the requirements.

"Always release ready stable branch", feels more like solution than a requirement. If we ask the question why we can come up with real requirements, and better solution may be. For example the requirements can be:
- We don't want the release to be compromised because of features not important to this release.

When we come to choosing the solution we would need to be explicit about the trade-offs. Like in the current solution we are trading:
Stability of branch with Cost of development/testing.

In any case, I would also like to propose a different approach for branching, explain via diagram here.

I am hoping more people are involved in this, because this is very important.
--
Vivek Singh | +91 98452 32929 | http://sites.google.com/site/petmongrels
Vinod C John <johnvic@thoughtworks.com> Tue, Sep 21, 2010 at 6:17 PM
To: Vivek Singh <vsingh@thoughtworks.com>
Cc: Udai Gupta <mailtoud@gmail.com>, "Keith Woodlock (contact)" <keithwoodlock@gmail.com>, Adam Feuer <afeuer@grameenfoundation.org>, Jeff Brewster <jbrewster@grameenfoundation.org>, Van Mittal-Henkle <vanmh@grameenfoundation.org>, Adam Monsen <amonsen@grameenfoundation.org>, Emily Tucker <ETucker@grameenfoundation.org>, "John Woodlock (contact)" <john.woodlock@gmail.com>, Jakub Sławiński <jslawinski@soldevelo.com>, Kojo Gambrah <kgambrah@gmail.com>, TWteam <teamgrameen@thoughtworks.com>
As per the proposal a feature can be graduated (merged to release branch) when all stories of that feature are done.

QA can be done per story on the feature branch, so QA won't be a bottle neck for feature graduation.

The volume of conflicts that can arise during feature graduation mainly depends on the number of active feature branches competing for graduation, order in which features move to release branch (first feature to graduate will have least merge issues) etc. 

After feature graduation, regression testing is required on the release branch to validate conflict resolution and merge.

I consider the additional effort required for conflict resolution and QA as the cost worth spending to maintain a release ready branch and wait for business to confirm this requirement.

Vivek's reply to the original proposal and http://martinfowler.com/bliki/FeatureBranch.html are relevant to the new proposal also and in added to this thread below.

regards,
--vinod



---------- Forwarded message ----------
From: Vivek Singh <vsingh@thoughtworks.com>
Date: Tue, Sep 14, 2010 at 4:38 PM
Subject: Re: Proposal: Stable development for frequent releases v0.2
To: Udai Gupta <mailtoud@gmail.com>
Cc: Keith Woodlock <keithwoodlock@gmail.com>, Adam Feuer <afeuer@grameenfoundation.org>, Jeff Brewster <jbrewster@grameenfoundation.org>, Kay Chau <kchau@grameenfoundation.org>, Van Mittal-Henkle <vanmh@grameenfoundation.org>, Adam Monsen <amonsen@grameenfoundation.org>, Emily Tucker <ETucker@grameenfoundation.org>, John Woodlock <john.woodlock@gmail.com>, Jakub Sławiński <jslawinski@soldevelo.com>, Kojo Gambrah <kgambrah@gmail.com>, TWteam <teamgrameen@thoughtworks.com>


Thanks Udai for putting it down. This is very important decision and its important that we get this right.

There are three ways of being release ready.
a) Feature Branching
This is what Udai has explained. Martin has explain the pros and cons quite well here: http://martinfowler.com/bliki/FeatureBranch.html
[Although I feel in such a model we should do CI on the team branches as well and get rid of one (either master or integration) branch.]

b) Disabling features
Build the features in a way so that they can be disabled in a simple fashion. There are two important things here:
- sometimes it might be hard to disable features
- it might lead a trail of switches
Facebook believes that frequent releases is business complexity and hence it alright for the switches to be present in code. Also, once it becomes business complexity you find good ways to disable code it becomes a design issues than a tactical choice.

c) Develop next release only features
This is basically solving the problem by taking it away completely. This is Lean based approach building only what is must have. Since this relies on pull it needs discipline and higher involvement from business.

Personally I have used combination of (c) and (b), in that order, in a really large project which was doing monthly releases. They cover 95% of your scenarios and for rest 5% you go with (a). So yes, I do not believe there is one silver bullet, thats why we need humans for developing software.
-- 
Vivek Singh | +91 98452 32929 | http://sites.google.com/site/petmongrels
Vinod C John <johnvic@thoughtworks.com> Tue, Sep 21, 2010 at 9:52 PM
To: Vivek Singh <vsingh@thoughtworks.com>
One more point to add from developers perspective,

The merge conflicts that can arise during a feature graduation can span across multiple stories of a feature possibly done by different developers. These developers will have to work together for the resolution as they know their parts of the code better.

Pessimism apart, we will see lots of seamless feature graduations on good days :)

-vinod
Udai Gupta <mailtoud@gmail.com> Wed, Sep 22, 2010 at 2:48 PM
To: Vinod C John <johnvic@thoughtworks.com>
Cc: Vivek Singh <vsingh@thoughtworks.com>, "Keith Woodlock (contact)" <keithwoodlock@gmail.com>, Adam Feuer <afeuer@grameenfoundation.org>, Jeff Brewster <jbrewster@grameenfoundation.org>, Van Mittal-Henkle <vanmh@grameenfoundation.org>, Adam Monsen <amonsen@grameenfoundation.org>, Emily Tucker <ETucker@grameenfoundation.org>, "John Woodlock (contact)" <john.woodlock@gmail.com>, Jakub Sławiński <jslawinski@soldevelo.com>, Kojo Gambrah <kgambrah@gmail.com>, TWteam <teamgrameen@thoughtworks.com>
Hi Folks,

I am taking back the proposal for now, because it's becoming more of a
discussion.

We will try to figure out this in a meeting.

I will schedule that sometime.

Thanks for all your suggestions, it really helped me to understand the
challenges and strategies that we need to think before coming to a
decision.

Cheers,
Udai