how to develop, test, and deliver in parallel?

[Yegor Bugayenko]

Hi all,

We're developing a web application. We have a team of programmers and two groups of manual/functional testers. Every change is made in its own Git branch, and is merged into `master` branch automatically after 1) being reviewed by an architect and 2) passing automated unit/integration tests. Jenkins automatically deploys `master` to the testing server after each merge. The first group of testers is trying to break the product at the testing server. They report defects into our bug tracking system. Programmers are fixing them in order of importance and merge into master.

Every few weeks, when master has an acceptable amount of unsolved blocking defects, we branch it into rc-123 (rc stands for release candidate and 123 for its ordinal number). Jenkins automatically deploys the most recent rc-? branch to our staging server. The second group of testers is trying to find critical defects in the product at the staging server. They are focusing on blocking issues mostly. They report defects into the bug tracking system and programmers fix them before everything else. We merge these fixes into the rc-123 branch and into master.

When the release candidate has an acceptable amount of unsolved blocking defects we branch rc-123 into release-1.3.9 (for example) and Jenkins automatically deploys it to the production server.

What do you think about this approach?

Yegor Bugayenko

[Jez Humble]

Hi Yegor

It's not a process that I would describe as continuous delivery, but if it works well for you, then that's great. For it to be continuous delivery:

* Developers would be checking straight into master, with reviews happening post-check-in.

* The team would prioritize keeping master releasable over delivering new features, which means this can't happen: "Every few weeks, when master has an acceptable amount of unsolved blocking defects"

Thanks,

Jez.

--
Jez Humble

Co-author, Continuous Delivery

http://continuousdelivery.com/

http://jezhumble.net/

[Yegor Bugayenko]

Hi Jez,

The key problem with one-step continuous delivery ("releasable master" as you said) is that business can't trust unit tests. No matter how good is our test code coverage and end-to-end testing automation - we still may have fatal errors which shouldn't be automatically deployed to production. Especially when financial transactions are involved.

That's why there should be a buffer, called "release candidate" in our case. This RC is continuously delivered to the staging platform. And there is another buffer, called "release", which is continuously delivered to production.

Thus, we have three parallel continuous delivery channels. What do you think?

—
Yegor Bugayenko

[Jez Humble]

Hi Yegor

I think that you should improve your automated tests. I have worked in investment banks where billions of dollars are at risk - and I know people who work in PCI-DSS regulated environments - and in this situation the best way to reduce risk is to put really good automated tests in place, not to rely on error-prone manual regression testing.

I have no way to stop you describing your process as "three parallel continuous delivery channels", but in my opinion, it's not continuous delivery.

Why is it important for you to describe your process as "continuous delivery"? CD isn't for everybody, and just because you're not doing CD, doesn't make your process in some way wrong or bad (although as I say, you shouldn't be relying on manual testing if there is money at stake).

Thanks,

Jez.

[Jez Humble]

By the way - I should clarify - there is absolutely a place for manual validations in continuous delivery - exploratory testing, user testing and so forth - but discovering critical bugs in this way should be the exception, not the rule, and when such bugs are discovered the answer is to put an automated test in place to prevent regressions in future.

What stops me describing your process as continuous delivery is not the presence of manual validations, but the fact that your master branch spends long periods of time not being in a releasable state, and the fact that developers have a manual validation - architectural review - as a gate to check-in.

Thanks,

Jez.

[Steve Smith]

Hi Yegor

Jez has hit the nail on the head, yet your "business can't trust unit tests" comment intrigues me. I would politely suggest that if business owners consider unit testing as an "untrustworthy" practice then you have a bigger, much earlier challenge than implementing Continuous Delivery. Do you have automated acceptance tests, that operate at a much higher level and use the language of your business owners? Do you subject each major defect found in test and Production environments to a root cause analysis session to ensure it cannot re-occur?

If you gradually replace manual regression tests with automated acceptance tests, you will be in a better place.

Regards

Steve