Re: [GOOS] Slow acceptance tests

[Dale Emery]

Hi James,

> Developing an application outside-in tends to lead to the growth of an extensive acceptance test suite. These tests are often slow to run. A slow build means delayed feedback and makes development painful. How do people tackle this? Which of the following tactics would you recommend using and which would you caution against? Are there any other useful tactics worth considering?
>
> A. Optimise acceptance tests e.g. setup domain objects directly rather than driving the admin UI; share setup between tests; etc.

My first preference is to avoid the UI altogether /except/ for the (relatively) few cases where the UI is part of what I'm testing.

For most acceptance tests, I want to bypass the UI. For this to be a reasonable test of the (nearly) whole system, I have to make sure the UI that I'm bypassing does very, very little work. If possible, I want the UI to do two things: 1. Translate user gestures into a single method call on a controller. 2. Draw stuff on the screen when notified that some relevant information has changed.

And even when I'm testing something that is /specifically about the UI, I want to use the UI for only those things where the UI is essential to the purpose of the test. I think of the test in three parts:
1. Establish some conditions.
2. Invoke some system responsibility.
3. Verify the results that the system produced.

Now I can write my test to use the UI only for those parts that matter to the responsibility I'm testing. If I want to verify that the UI displays the results correctly, I may not need to use the UI establish conditions or invoke the responsibility.

I generally /do not/ want to share setup, or to have one test rely on conditions established by another. Those dependencies always cause trouble. I can't run a single test. I can't run tests in parallel. The dependent test does not describe the entire responsibility it is testing--in particular, the dependent test's code omits a clear description of one of the "givens," one of the conditions that matter to the test.

So my general principle for optimizing is: Bypass any technology that is not /essential/ to the responsibility I'm testing.

> B. Only run a subset of all acceptance tests within the developer build; run the rest less often e.g. only on continuous integration server, or only when core test suite passes in a "build pipeline".

Yes, I often arrange for expanding suites of tests that cover ever-larger scopes. And there are two dimensions that seem most helpful for organizing suites. One dimension is technology. Some tests test a little bit of the system technology, and others test more or nearly all. Another dimension is functionality. Some suites test a little bit of functionality. Other suites cover more.

Functionally sliced suites seem more useful to help developers quickly run acceptance tests that are (presumably) most likely to be affected by their current work. Technology-slced suites seem most useful for deciding whether to promote code to the next stage of delivery—developer into common codebase, codebase into a test environment, tested system into production.

> C. Move tests for particular behaviour further down the Test Pyramid [1] e.g. bypassing the UI.

This, too. The trick (and it's tricky) is that the more of the system you include, the more you're testing the actual system. The less of the system you include, the more you rely on an assumption that the things you bypassed or mocked do not affect the value of the tests.

One caution with this: When you move tests further down the pyramid, you have only the /illusion/ that you're testing the same behavior. Or perhaps a more accurate phrasing is: Each time you move a test down the pyramid, you are testing a smaller system.

Dale

--
Dale Emery
Consultant to software teams and leaders
http://dhemery.com

[Seb Rose]

It's also worth remembering that when you are /specifically/ testing the UI you may be able to do this without testing the rest of the system.

Put another way: you can *unit* test your UI

--

---
You received this message because you are subscribed to the Google Groups "Growing Object-Oriented Software" group.
To unsubscribe from this group and stop receiving emails from it, send an email to growing-object-oriente...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
ACCU - Professionalism in Programming - http://accu.org

http://www.claysnow.co.uk
http://twitter.com/#!/sebrose
http://uk.linkedin.com/in/sebrose

[Colin Vipurs]

In addition - parallelise your tests and get them running concurrently, spreading across multiple hardware if you need to on the CI server.

--

---
You received this message because you are subscribed to the Google Groups "Growing Object-Oriented Software" group.
To unsubscribe from this group and stop receiving emails from it, send an email to growing-object-oriente...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Maybe she awoke to see the roommate's boyfriend swinging from the chandelier wearing a boar's head.

Something which you, I, and everyone else would call "Tuesday", of course.

[Matt Wynne]

Hi James,

On 19 May 2013, at 18:00, James Mead <floeh...@gmail.com> wrote:

Developing an application outside-in tends to lead to the growth of an extensive acceptance test suite. These tests are often slow to run. A slow build means delayed feedback and makes development painful. How do people tackle this? Which of the following tactics would you recommend using and which would you caution against? Are there any other useful tactics worth considering?

A. Optimise acceptance tests e.g. setup domain objects directly rather than driving the admin UI; share setup between tests; etc.

B. Only run a subset of all acceptance tests within the developer build; run the rest less often e.g. only on continuous integration server, or only when core test suite passes in a "build pipeline".

C. Move tests for particular behaviour further down the Test Pyramid [1] e.g. bypassing the UI.

For me, a top-heavy testing pyramid (or testing ice-cream cone as some people call it) is a sign of problems. Here are some of my thoughts on the subject:

http://blog.mattwynne.net/2013/03/04/optimising-a-slow-build-youre-solving-the-wrong-problem/

Also some more buried in here:

http://skillsmatter.com/podcast/scala/why-your-step-definitions-should-be-one-liners-and-other-pro-tips

TL;DR: My view is that top-heavy testing pyramids usually indicate you're not doing enough design, because otherwise it would be easier and more obvious how to test the same behaviour at lower layers. (Rails just loves to get you into this hole). They can also indicate you have a trust issue: defining behaviour in business-readable acceptance tests can help to build trust with non-technical people, but ultimately you need to build this trust to the point where you can decide the most appropriate layer to write a test.

James.

----

http://jamesmead.org

[1] http://martinfowler.com/bliki/TestPyramid.html

--
 
---
You received this message because you are subscribed to the Google Groups "Growing Object-Oriented Software" group.
To unsubscribe from this group and stop receiving emails from it, send an email to growing-object-oriente...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

cheers,

Matt

--
http://mattwynne.net || https://twitter.com/mattwynne || http://the-cucumber-book.com || http://bddkickstart.com || http://www.relishapp.com

[Kevin Rutherford]

Hi,

On Mon, May 20, 2013 at 9:37 AM, Colin Vipurs <zodi...@gmail.com> wrote:
> In addition - parallelise your tests and get them running concurrently,
> spreading across multiple hardware if you need to on the CI server.

I would definitely NOT do this.

If the tests are painfuly slow, that pain is telling you a lot about
your design. Keep the tests slow until the design is fixed. And if you
absolutely must have some very slow full-stack tests, push them into a
batch job that runs asynchronously to your TDD cycle; if any of these
very slow tests ever fails, it means you're missing a fast test
somewhere that could have caught it sooner.

See also http://silkandspinach.net/2011/08/08/why-i-dont-use-spork/
from my blog...

Cheers,
Kevin
--
http://xpsurgery.com -- remote one-to-one tutoring in TDD and OO
http://kevinrutherford.co.uk -- software development coaching
http://refactoringinruby.info -- Refactoring in Ruby, the book

[Anthony Green]

On Sunday, 19 May 2013 18:00:39 UTC+1, James Mead wrote:

Developing an application outside-in tends to lead to the growth of an extensive acceptance test suite. These tests are often slow to run. A slow build means delayed feedback and makes development painful. How do people tackle this? Which of the following tactics would you recommend using and which would you caution against? Are there any other useful tactics worth considering?

A. Optimise acceptance tests e.g. setup domain objects directly rather than driving the admin UI; share setup between tests; etc.

B. Only run a subset of all acceptance tests within the developer build; run the rest less often e.g. only on continuous integration server, or only when core test suite passes in a "build pipeline".

C. Move tests for particular behaviour further down the Test Pyramid [1] e.g. bypassing the UI.

Like most things, it depends. 

On your current understanding of their individual purpose and value.
On how that understanding is different today to what it was yesterday.
As your heuristical sense of design and test smells matures (a process that never ends BTW) you'd get better at picking the best choice for that moment.
Sometimes it's a worthwhile lesson to push practices, like pushing tests down, to the limits solely to discover what shapes form.

[Samuel Edwin]

Dale,

When you're bypassing the UI, how do you assert the output of the test? Do you still assert from the UI like checking the text in a text field? Is it okay to check no-UI end to end test with a mock object?

[Dale Emery]

Hi Samuel,

> When you're bypassing the UI, how do you assert the output of the test? Do you still assert from the UI like checking the text in a text field? Is it okay to check no-UI end to end test with a mock object?

I verify through the UI /only/ if I specifically care how the UI renders things.

If I'm focusing on functionality, I rarely care about the UI. What I care about are the results produced by the system. There are three primary ways I can observe results:

- If the result is returned by the method that I called to invoke the system's responsibility, I can verify that with a normal assertion, such as an assertThat() method with a matcher.

- If the result is a message sent by the system to something outside the system (either through the UI or through some other channel), I replace the UI or the channel or the message receiver with a test double, and verify that the test double received the desired message.

- If the result is a change in the system's internal state, I might query the system state and verify that, or verify that some subsequent responsibility (that depends on state) does the right thing. I don't do a lot of databasey things. If I did, I might consider replacing the data store with a test double for some tests. For that, I'd want to make sure that my test double mimics the data store with sufficient fidelity.

Bypassing the UI (to set up conditions, to invoke the responsibility, or to verify results) is okay when you don't care about the UI. For some tests, you care about the UI—what it displays, maybe something about how it displays it, whether the UI is wired up correctly, and so on. So the question for me is: Does the intention of this test /necessarily/ involve some element of the UI? If not, my preference is to bypass the UI for the entire test. Don't even construct it.

Another consideration: The more processing the UI does, the more nervous I become about bypassing it. (Also, the more nervous I become about the processing that the UI does.)

[Samuel Edwin]

Thanks for your reply Dale :)

I'm only concerned about the data produced by the system, so I'll be using test doubles rather than using the UI. Time to tweak my design.

[Samuel Edwin]

Looks like I'm a little bit lost, need some help.

I finally replaced my UI by using mock objects. I've done a few end to end tests, and it feels good, my test looks much less brittle than before, and asserting the states is easy, put some expectations and we're done. However, I sense that I'll be doing many duplications. My end to end test and my unit test will have many lines of code that doing exactly the same things: setting expectations, creating objects that will be the arguments of my expected objects.

The purpose of end to end tests is to make sure everything works together from the very start. However, I find that some of the complex operations that is related with UI will also be replaced with test doubles, and my end to end tests will give me less confidence. I don't know if that operation will work well with the rest of my code. How do I handle this kind of problem?

[Josue Barbosa dos Santos]

>>However, I find that some of the complex operations that is related with UI will also be replaced with test doubles, 

>>and my end to end tests will give me less confidence. I don't know if that operation will work well with the rest of my code. 

>>How do I handle this kind of problem?

It is a trade-off

Tests from ui are:

- more slow;

- more brittle to construct.

but:

- the confidence that everything works together is higher. You test in an environment much more closer to the production environment.

And when you bypass the UI:

- faster tests;

- easier to construct tests.

but:

- the confidence that everything works together is lower. You test in an environment far from the production environment.

Personally, I prefer to build both to have the best from the two. I execute with more frequency bypassing the UI but the UI ones are also executed with a less frequency.

--
 
---
You received this message because you are subscribed to the Google Groups "Growing Object-Oriented Software" group.
To unsubscribe from this group and stop receiving emails from it, send an email to growing-object-oriente...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
Abraços,
Josué
http://twitter.com/josuesantos

[Samuel Edwin]

I see, thanks for your reply Josué.

How much effect the UI testing brittleness cause to you when requirements change? Writing UI automation scenario tends to make me scratch my head when I'm reading it for the next few months.

To unsubscribe from this group and stop receiving emails from it, send an email to growing-object-oriented-software+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Matteo Vaccari]

It depends on how you write your UI tests.  If you write your tests in terms of what the user is trying to do, you can avoid most mentions of UI details in the text of the test.  That makes your easier to maintain.

Matteo

To unsubscribe from this group and stop receiving emails from it, send an email to growing-object-oriente...@googlegroups.com.

[Josue Barbosa dos Santos]

Matteo already responded. For example (more or less Java) to give an idea:

....

@Injected

Application app;

public void shouldTransferWhenOriginAccountHasFunds(){

    app.createAccount(ID_FROM, 100.00);

    app.createAccount(ID_TO, 100.00);

    app.transfer(ID_FROM, ID_TO, 100.00);

    Account from = app.getAcount(ID_FROM);

    Account to = app.getAcount(ID_TO);

    AssertEquals(from.getBalance(), 0.0);

    AssertEquals(to.getBalance(), 200.0);

}

interface Application {

    createAccount(int idCount, double balance);

    transfer(int idFromAccount, int idToAccount, double value);

    getAcount(idAccount);

}

DirectCallTest implements Application {//bypassing UI

....

    transfer(int idFromAccount, int idToAccount, double value){

        transferService.transer(idFromAccount, idToAccount, value); //bad smell, service in name

    }

...

}

WebTest implements Application {

....

    transfer(int idFromAccount, int idToAccount, double value){

        selenium.type("fromAccount", idFromAccount);

        selenium.type("toAccount", idToAccount);

        selenium.type("value", value);

        selenium.click("transfer");

    }

...

}

It is approximately what I do.

-- 
Abraços,
Josué
http://twitter.com/josuesantos

[James Richardson]

But not really with doubles... :-)

James