BDD is an out dated approach in a CD/CI environment

[nilofar nissa]

Greetings!

Thanks for all the discussions here. I have benefited tremendously through BDD and discussions here.  

I am going to list out some of the most common push backs or criticisms  or crazy opinions or discomforts or irritations against BDD that I have faced. Can you share your experience on how you would approach these questions. Also, feel free to add you own experiences or frustrations.

1. BDD is an out dated approach and slows down continuous integration and continuous flow.

2. BDD is just another form of unit tests  or extension of unit tests. 

3. Combination of code coverage (Unit tests/integration tests) and UI tests are more effective for CI/CD over BDD.

4. Many teams extend their BDD scenarios in to BDD-tests and not fully taking advantage of verification techniques like integration testing and system testing.

5. BDD adds more burden on Devs and process. Lets Devs write development code and tester add test code.

6. BDD introduces confusion between technology faced testing (are we building the system right?) Vs business faced testing (are you building the right system?). 

7. Lets business take care of business faced testing. Tech team should focus only on technology and rework based on the feedback.

8. How come you have zero automation tests outside unit tests+BDD and have no bug leakage ? What happens to the well known code verification automation techniques post development?

9. BDD forces more UI based testing approach.

Best Regards,

Nilofar

[Liz Keogh]

Hi Nilofar,

This looks like fun. I've got a go-to phrase for these kinds of conversations. See if you can spot it!

1. BDD is an out dated approach and slows down continuous integration and continuous flow.

Can you define what you understand to be "BDD"? Because if you say "testing", BDD isn't about testing. More on that later.

 

2. BDD is just another form of unit tests  or extension of unit tests. 

BDD isn't about testing. Though, arguably, unit testing as it's normally done isn't about testing either. It's about coming up with examples of how to use a system, or a class, to illustrate the behaviour you're after. Conversations around the examples let us explore whether we have a common understanding of what the system should do. 

 

3. Combination of code coverage (Unit tests/integration tests) and UI tests are more effective for CI/CD over BDD.

BDD isn't about testing. Though you might use scenarios to supplement other forms of tests. I think of scenarios and unit-level examples as living documentation, and the CI / CD pipeline as the thing which keeps those up-to-date. If you make the code easy to understand and keep the documentation alive, there are normally fewer bugs anyway.

Other than that, I fully agree with the idea that you should have a lot of unit tests, fewer integration tests and far fewer end-to-end scenarios. Most teams do get the balance wrong IMO.

 

4. Many teams extend their BDD scenarios in to BDD-tests and not fully taking advantage of verification techniques like integration testing and system testing.

BDD isn't about testing. Having said that, see my point above.

 

5. BDD adds more burden on Devs and process. Lets Devs write development code and tester add test code.

BDD isn't about testing. I normally ask that the devs write the scenarios down in English, at least, and then I don't mind who actually automates them. I've found a lot of benefit in having scenarios written and automated before writing the code that makes those scenarios pass; it provides a level of focus and quick feedback that really helps me as a dev.

A common pattern I've seen is that testers pair with devs on the scenarios. That works really well IMO, especially when testers are still learning to code.

 

6. BDD introduces confusion between technology faced testing (are we building the system right?) Vs business faced testing (are you building the right system?). 

BDD isn't about testing.

 

7. Lets business take care of business faced testing. Tech team should focus only on technology and rework based on the feedback.

BDD isn't about testing.

 

8. How come you have zero automation tests outside unit tests+BDD and have no bug leakage ? What happens to the well known code verification automation techniques post development?

BDD isn't about testing. And you will have bugs. Even Amazon's system went down earlier this year. There are always scenarios we didn't think of.

There's a lot of ways to mitigate that, and I really love the DevOps movement and its focus on automated deployment and recoverability. But, you know, if you've got some other techniques that help you, BDD isn't a silver bullet and was never intended to be. Use the tools where they make sense.

9. BDD forces more UI based testing approach.

BDD isn't about testing. It's just examples of how your code works. Konstantin and Aslak have both done huge amounts of work around automation at different levels on the stack... and the first BDD tool actually started off as a replacement for JUnit; you can use it as an alternative to TDD as well (TDD is also not about testing).

So, just to recap: BDD isn't about testing. If anything, it's an analysis technique. It might produce things you call tests as a nice by-product, but if those are the things you're focusing on, you're probably doing it wrong.

The one thing that was missing here is the conversational side; the fact that talking through the scenarios is a really lightweight way to analyze requirements and uncover misunderstandings or make discoveries around what else might be needed.

I recommend doing this with at least the three main roles involved: someone who has a problem or a need they want resolved (PO, analyst, etc.), someone who's going to solve that problem (dev, dev pair) and someone who can spot problems we missed (tester). That's the 3 amigos (where, as Matt Wynne puts it, 3 can be somewhere between 3 and 7).

If you're just automating tests, and not having conversations, you're not doing BDD.

Cheers,

Liz.

--

Liz Keogh
l...@lunivore.com
http://lizkeogh.com
http://lunivore.com

[Erlis Vidal]

Liz,

Thanks for taking the time to write such detailed response. You should write a blog with it,  if you haven’t already.

Best regards,

Erlis

--
You received this message because you are subscribed to the Google Groups "Behaviour Driven Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to behaviordrivendeve...@googlegroups.com.
To post to this group, send email to behaviordriv...@googlegroups.com.
Visit this group at https://groups.google.com/group/behaviordrivendevelopment.
For more options, visit https://groups.google.com/d/optout.

[nilofar nissa]

Hi Liz,

Thanks for your detailed, but simple explanations. I am thinking of making a T shirt that says "Scenarios are not Test".

On Sunday, October 22, 2017 at 5:38:11 AM UTC-4, Liz Keogh wrote:

Hi Nilofar,

This looks like fun. I've got a go-to phrase for these kinds of conversations. See if you can spot it!

1. BDD is an out dated approach and slows down continuous integration and continuous flow.

Can you define what you understand to be "BDD"? Because if you say "testing", BDD isn't about testing. More on that later.

I go one step further, and ask them to write a spc. Based on my experience, you have to do it wrong first to evolve in to proper BDD thinking.

 

 

BDD isn't about testing. Though, arguably, unit testing as it's normally done isn't about testing either. It's about coming up with examples of how to use a system, or a class, to illustrate the behaviour you're after. Conversations around the examples let us explore whether we have a common understanding of what the system should do. 

 

 

3. Combination of code coverage (Unit tests/integration tests) and UI tests are more effective for CI/CD over BDD.

BDD isn't about testing. Though you might use scenarios to supplement other forms of tests. I think of scenarios and unit-level examples as living documentation, and the CI / CD pipeline as the thing which keeps those up-to-date. If you make the code easy to understand and keep the documentation alive, there are normally fewer bugs anyway.

Other than that, I fully agree with the idea that you should have a lot of unit tests, fewer integration tests and far fewer end-to-end scenarios. Most teams do get the balance wrong IMO.

Can we say BDD scenarios may play a role in this balance ?  I have seen our friends in this group stated that, they are able to manage with scenarios and v few post BDD tests.

Dev opps mentality argues that, instead of spending time up in front, I am willing to do re-work based on the feedback loop, powered by CI/CD. 

 

8. How come you have zero automation tests outside unit tests+BDD and have no bug leakage ? What happens to the well known code verification automation techniques post development?

BDD isn't about testing. And you will have bugs. Even Amazon's system went down earlier this year. There are always scenarios we didn't think of.

There's a lot of ways to mitigate that, and I really love the DevOps movement and its focus on automated deployment and recoverability. But, you know, if you've got some other techniques that help you, BDD isn't a silver bullet and was never intended to be. Use the tools where they make sense.

9. BDD forces more UI based testing approach.

BDD isn't about testing. It's just examples of how your code works. Konstantin and Aslak have both done huge amounts of work around automation at different levels on the stack... and the first BDD tool actually started off as a replacement for JUnit; you can use it as an alternative to TDD as well (TDD is also not about testing).

So, just to recap: BDD isn't about testing. If anything, it's an analysis technique. It might produce things you call tests as a nice by-product, but if those are the things you're focusing on, you're probably doing it wrong.

"It might produce things you call tests" >>>> I have started saying this more frequently these days. But it takes years to feel this. 

 Agree on all other comments. Simple but powerful explanations. 

Thanks,

Nilofar 

[Dan North]

Hi Nilofar,

To help emphasise Liz’s excellent reply, here is a talk I gave last year about exactly this topic (requires free sign-up):

https://skillsmatter.com/skillscasts/8633-bdd-is-not-about-testing

tl;dr: BDD is not about Testing (but it has an interesting relationship with testers).

Thanks,

Dan

[aslak hellesoy]

I agree with everything that others have said in this thread.

It sounds as though you're working in an organisation where there is a fair amount of resistance against BDD. Some of this resistance is probably based on a poor understanding of what BDD is, but telling people they're wrong might cause further resistance.

In situations like this I've always found that the best way to convince people is to just show them. Can you find some like-minded people who are willing to try it out? Can you try it out on a small problem, perhaps a small user story and see how it works out? If it does, you'll have something concrete to point to, and that might spark people's enthusiasm.

A good place to start is Example Mapping [1]. If you can get some people to do that, you'll have a good foundation for trying out the technical practices, which essentially is TDD. Write a test (scenario), make it pass, clean up your mess. Rinse and repeat.

[1] https://cucumber.io/blog/2015/12/08/example-mapping-introduction 

--

You received this message because you are subscribed to the Google Groups "Behaviour Driven Development" group.

To unsubscribe from this group and stop receiving emails from it, send an email to behaviordrivendevelopment+unsub...@googlegroups.com.
To post to this group, send email to behaviordrivendevelopment@googlegroups.com.

[Michael Mayer]

Dear Liz,

The original intentions of BDD are well documented. Sticking our heads in the sand and say "you're doing it wrong" doesn't help. Observed that many times in the agile community. Instead of coaching we do teaching and the others have to listen. Let me share three observations:

1) Nothing stops you from having conversations and writing great specs with real world examples. Not the invention of BDD. Some people think specs are not agile though. I consider that a major issue we should work on.

2) I haven't come across a single software project where stakeholders used BDD tools to collaborate on specs as originally intended (you can blame my poor selection of customers if it helps). Instead I observed that many teams started doing "BDD system tests" (running the tests against an API or UI) as a replacement for unit testing when they figured their architecture wasn't good for TDD and they didn't want to write unit tests anyway. As you wrote, many teams get the balance wrong and IMHO it starts with getting infected by a hype like "BDD" or "TDD is dead".

3) Frameworks hiding test code in deeply nested blocks of Describe, Context, It and Expect are the immortal remnants of BDD. See https://onsi.github.io/ginkgo/ and many others. For my part, I would prefer developers to write classic unit tests based on the specification and additional edge cases as I find them more readable and concise (but I know, that's just my opinion).

Michael

[Liz Keogh]

Hi Michael,

 

The original intentions of BDD are well documented. Sticking our heads in the sand and say "you're doing it wrong" doesn't help.

The only thing that I tend to mandate as "doing BDD" is having conversations around examples. And honestly, that's not very difficult. I put an "if" in there, and hopefully everyone involved in BDD is doing at least that... and if they're not, I'm not ashamed of the wake-up call.

The more common pattern I see is that people are doing various forms of testing - integration, maybe exploratory, maybe monitoring and security and other kinds of things - and getting confused because it doesn't fit into the BDD "pattern", and they're not finding that "Given, When, Then" language helpful. So they're doing it right... but it isn't BDD. BDD isn't about testing.

 

1) Nothing stops you from having conversations and writing great specs with real world examples. Not the invention of BDD. Some people think specs are not agile though. I consider that a major issue we should work on.

The thing that I really love about BDD's focus on scenarios and examples is that it lets you do easy exploration, before specification. And, sure, that's not unique to BDD; Dan describes BDD as "second-generation" because it's derived from other practices. Not using the word "test" is the biggest difference.

I think that when Gojko wrote his really popular book that people forgot there was anything before the specs. The conversations remain the most important bit, and often those include a lot of scenarios which don't make the cut. That's the difference between specification and exploration.

Here's the (slightly whimsical, sorry!) post that I wrote about that. Here's the post on the conversational patterns I use for exploration.

 

2) I haven't come across a single software project where stakeholders used BDD tools to collaborate on specs as originally intended (you can blame my poor selection of customers if it helps).

The best collaboration usually happens in the conversations; those teams tend to use the tools just to record the result. This is one of the reasons I keep banging on about the conversations.

I'm seeing a trend towards just writing things down instead of having the conversations, particularly missing out the people (devs) who are actually going to do the work. At that point it's just documentation being handed over the wall. I really like it when those "three amigos" sessions happen just as the work is being picked up by the devs, with the devs who are going to do it.

Another thing I see is focus on using BDD's scenarios for "accurate estimates"; quite aside from being an oxymoron, it's not what BDD was designed for. Most teams can come up with similar estimates just with the titles of scenarios ("The one where..."), rather than the full-fleshed steps, and I recommend that as a first tiny step for moving to more meaningful conversations if you find yourself in this place.

 

Instead I observed that many teams started doing "BDD system tests" (running the tests against an API or UI) as a replacement for unit testing when they figured their architecture wasn't good for TDD and they didn't want to write unit tests anyway.

Sure. In a world where lots of software is just plugging things together, that might be a useful thing to do. Again, it only really counts as BDD if there are conversations involved, and sometimes what people want is just smoke tests to check that everything's wired up, which might be the right thing to do, but which isn't BDD.

A good check is to see if there are multiple whens and thens; this is pretty typical of longer journeys. I wrote a Stack Overflow answer on doing this kind of end-to-end journey well. (And you can see how I rephrase things to get rid of that word, "test", too.) Note that this still isn't BDD, even when it's a good thing to do.

As you wrote, many teams get the balance wrong and IMHO it starts with getting infected by a hype like "BDD" or "TDD is dead".

It might not be useful in particular situations. I'm OK with that. Lots of people have found it useful, and I ask those who haven't to approach with curiosity and find out what they did that was different.

 

3) Frameworks hiding test code in deeply nested blocks of Describe, Context, It and Expect are the immortal remnants of BDD. See https://onsi.github.io/ginkgo/ and many others. For my part, I would prefer developers to write classic unit tests based on the specification and additional edge cases as I find them more readable and concise (but I know, that's just my opinion).

Unit-level was where BDD (most especially JBehave 1.0) started. And I'm not a fan of those kind of speccy indented frameworks either, but you know what? If it works, sure. I've not used it, so I can't tell if it's easy to change and maintain. I don't find it all that easy to read, but if it's only got a technical audience and they like it, whatever works.

I tend to write both scenarios (which might just be in a DSL rather than a BDD tool) and unit-level examples (which I might call "XXXBehaviour" rather than "XXXTest" just because I can). I just put the "Given, When, Then" in comments at the unit level. Here's an example.

And I go through the UI for my higher-level scenarios, but I know some people don't, and that might be OK too, especially if the UI is unimportant compared to the domain model / data integrity (an admin page for example). Again, look at Aslak and Konstantin's work on this.

The most important thing for me is that people are having conversations, and looking at concrete examples of what the system should do, even if those conversations have to be with a rubber duck.

Cheers,

Liz.