Thoughts on end-to-end testing through the UI

285 views
Skip to first unread message

Kent

unread,
Jan 12, 2010, 4:25:07 PM1/12/10
to Growing Object-Oriented Software
Hi guys,

First off - loving the book. The idea of 'end-to-end' meaning truly
end-to-end, including packaging and deployment to a production-like
environment, was quite profound to me. We have 'end-to-end' testing at
work (using Selenium), but on a test environment that differs from
production in too many ways, so we almost always encounter issues
deploying to production. I love Steve & Nat's concept of end-to-end as
"edge-to-edge".

Anyway, I write a lot of unit-tests but I've always struggled with end-
to-end tests. While reading the book, I was getting excited about end-
to-end tests through the UI again, but then I read Michael Feathers'
blog entry "UI Test Automation Tools are Snake Oil":
http://blog.objectmentor.com/articles/2010/01/04/ui-test-automation-tools-are-snake-oil

Michael (and Robert Martin) seem to be saying that UI tests should
test only the UI - not business logic. This seems contrary to what
Steve & Nat are doing in their Auction Sniper example. I'm curious to
hear peoples' opinions on this (or have I misinterpreted things?).
It's very satisfying when an end-to-end test finally turns green, but
do they provide long-term value in peoples' experiences?

Thanks,
Kent

philip schwarz

unread,
Jan 12, 2010, 6:32:38 PM1/12/10
to Growing Object-Oriented Software
Brian Marick partly blogged on the subject last week:
http://www.exampler.com/blog/2010/01/08/some-preliminary-thoughts-on-end-to-end-testing-in-growing-object-oriented-software/

On Jan 12, 9:25 pm, Kent <kent.skin...@gmail.com> wrote:
> Hi guys,
>
> First off - loving the book. The idea of 'end-to-end' meaning truly
> end-to-end, including packaging and deployment to a production-like
> environment, was quite profound to me. We have 'end-to-end' testing at
> work (using Selenium), but on a test environment that differs from
> production in too many ways, so we almost always encounter issues
> deploying to production. I love Steve & Nat's concept of end-to-end as
> "edge-to-edge".
>
> Anyway, I write a lot of unit-tests but I've always struggled with end-
> to-end tests. While reading the book, I was getting excited about end-
> to-end tests through the UI again, but then I read Michael Feathers'

> blog entry "UI Test Automation Tools are Snake Oil":http://blog.objectmentor.com/articles/2010/01/04/ui-test-automation-t...

Nat

unread,
Jan 15, 2010, 10:13:51 AM1/15/10
to Growing Object-Oriented Software
Michael and I have obviously had quite different experiences with UI
automation. I lean heavily towards end-to-end system testing and
testing through the GUI because I've seen bad failures caused by
inadequate end-to-end testing.

One extreme example: a rich client app performed significant
functionality in response to focus-in/focus-out events in the GUI.
When the user tabbed through the fields in the expected order, the app
worked as expected. One day a user filled in a field at the bottom of
the screen first and then tabbed round to the top of the screen to
fill in the rest, and the application unexpectedly sent tens of
millions of dollars to another company. This bug was caused by the way
the GUI controls were connected to the non-GUI objects. An automated
"soap-opera" test would have detected when this bug was introduced.

Another example (not the GUI): an app used a fake implementation of
the JMS API in its automated acceptance tests. When deployed, it
didn't work at all because the program did not called "start" on the
JMS connection. The fake JMS implementation did not exactly mirror
the behaviour of the real implementation. An end-to-end test, even a
basic smoke test, would have caught this straight away.

I think that Michael Feathers is talking about UI automation tools
such as Mercury or QTP. I've used these and don't like them either.
They don't work well in a TDD process because they either use a custom
scripting language that doesn't integrate into the development tools
and codebase of the app itself, or they use a "record/playback"
metaphor that requires the app to exist before you can create test
cases.

In my experience, there's a big difference between those tools (which
are used for after-the-fact testing) and using a library for GUI
automation within a TDD process. If you use an automation library
written in the same language as the system itself, you can share
constants and functions for generating component identifiers between
the GUI code and the tests. As you refactor the GUI code, it's easy
to keep your tests in synch. This applies much more to languages that
have modern IDEs. In languages that you have to program by editing
text, rather than performing program transformations, using a
different language makes less difference and propagating changes
becomes more costly, so the cost/benefit trade-offs are different.

We also didn't write our tests in terms of the GUI but in terms of the
goals that the user wants to achieve, and the activities they perform
to achieve those goals. We then had a layer that implements those
activities in terms of UI navigation & gestures. (In the HCI
literature these are termed goals/activities/tasks, but I prefer the
terms goals/activities/gestures because I always get confused between
activities and tasks). This isolated our tests from changes to the
GUI.

Michael is right that timing/asynchrony makes testing more difficult.
But if you're writing end-to-end tests for distributed systems, you
need to cope with these problems anyway. If your tests cope with the
asynchronous nature of the system itself, they cope with the
asynchronous nature of the GUI for free. We describe some techniques
we use to cope with asynchrony in the last chapter of the book.

Most significantly to me, however, is the difference between "testing"
end-to-end or through the GUI and "test-driving". A lot of people who
are evangelical about TDD for coding do not use end-to-end tests for
driving design at the system scale. I have found that writing tests
gives useful design feedback, no matter what the scale. If the GUI is
volatile, maybe it's too complicated. We found that test-driving the
GUI pushed us towards simpler user interfaces and away from lots of
buttons/fields/dialogs (I can't put it better than this:
http://stuffthathappens.com/blog/2008/03/05/simplicity/). Test-driving
from system tests forced us to introduce ways to synchronise the tests
with system behaviour and fake out external services. This led us to
architect the app as simple cooperating services and put simple web
services between our app and the organisations data services that we
used. It turned out this made it really easy to add management &
monitoring functions and scale the app up when needed. We found that
just as TDD of code naturally guides code to a style that is easier to
maintain, so TDD of systems naturally guided our architecture to a
style that is easier to support.

Gosh... I've written a lot. That's enough for now!

--Nat

On Jan 12, 9:25 pm, Kent <kent.skin...@gmail.com> wrote:

> Hi guys,
>
> First off - loving the book. The idea of 'end-to-end' meaning truly
> end-to-end, including packaging and deployment to a production-like
> environment, was quite profound to me. We have 'end-to-end' testing at
> work (using Selenium), but on a test environment that differs from
> production in too many ways, so we almost always encounter issues
> deploying to production. I love Steve & Nat's concept of end-to-end as
> "edge-to-edge".
>
> Anyway, I write a lot of unit-tests but I've always struggled with end-
> to-end tests. While reading the book, I was getting excited about end-
> to-end tests through the UI again, but then I read Michael Feathers'

> blog entry "UI Test Automation Tools are Snake Oil":http://blog.objectmentor.com/articles/2010/01/04/ui-test-automation-t...

Kent

unread,
Jan 25, 2010, 8:46:07 PM1/25/10
to Growing Object-Oriented Software
Thanks for the detailed reply, Nat. I also enjoyed Steve's reply to
Brian Marick: http://www.m3p.co.uk/blog/2010/01/17/responding-to-brian-marick/.

I have been attempting to use end-to-end tests to drive design at the
system scale, as you put it, with some success; but it has also raised
some more questions. I'm still unsure about the ratio of system tests
to unit-tests; for example, do you explore every nook and cranny with
end-to-end tests, or only main-line scenarios? When a bug is
discovered, do you always write a failing system test before fixing
the bug, or is a failing unit-test sufficient? I suspect the answer is
"it depends" :)

I'm also curious about what people do as far as data seeding for end-
to-end tests. One approach would be to do no data-seeding: all data is
entered by manipulating the UI. But this leads to many tests running
the same code over and over (e.g. for an e-commerce site, almost every
test begins by registering a customer, even if that's incidental to
the purpose of the test). I have written my end-to-end tests to seed
the database directly with some common data, with certain tests
seeding additional data for their purposes. The downside to this is
that I have to look in two locations (the test and the common data-
seeder) to know what entities exist in the database, which makes me
uncomfortable for the reasons discussed by Jim Newkirk here:
http://jamesnewkirk.typepad.com/posts/2007/09/why-you-should-.html. On
the other hand, having every test explicitly insert lists of
countries, states, etc. seems silly. Perhaps an exception can be made
for static (or mostly static) data.

Cheers,
Kent

Ben Butler-Cole

unread,
Feb 4, 2010, 5:14:11 AM2/4/10
to Growing Object-Oriented Software
Kent said:
> I'm also curious about what people do as far as data seeding for end-
> to-end tests.

I find having common seed data unacceptable for the reason you
outline. Specifically I find that that this duplication leads to
bloated fixtures, duplication and a lack of understanding amongst the
team of what really is needed. I prefer every test to be self-
sufficient, so that its implications can be understood in isolation
and the amount of setup kept to a minimum.

As you say, driving this setup through the UI means that some code
gets executed repeatedly. This can make the tests slow (especially if
they are running in a browser) and brittle (e.g. a simple change to
the login process breaks many tests). I prefer to define each test's
fixture declaratively and minimally. The layer underneath that which
actually does the setup can take short-cuts, share code with the
application, be optimized and intelligently make decisions about
exactly what does and doesn't need setting up. When you are reading
the test (and when writing new ones once the application is mature),
you need only be concerned with the intrinsic details of the fixture.

Ben

Kent

unread,
Feb 8, 2010, 6:07:06 PM2/8/10
to Growing Object-Oriented Software
Thank you Ben. I have followed your advice, moved almost all of the
setup code into the individual test fixtures (as declaratively and
minimally as I can), and I now feel more comfortable about getting the
system into a known state by using some carefully applied short-cuts.
I'm learning a lot.

Thanks again,
Kent

Reply all
Reply to author
Forward
0 new messages