Chapter 14 - An Agile Test Automation Strategy - "Agile Testing" by Crispin & Gregory
266 views
Skip to first unread message
Steve Faber
Mar 14, 2012, 4:45:31 PM3/14/12
to Provo Agile & Scrum User Group
Since I did not make time to read the whole book, I took the "agile"
approach and decided to do a chapter report instead of a book report.
We're in the third sprint of establishing a java automation framework
for our product and converting tests to this environment so I read
this chapter searching for some validation for our efforts. I found
some points in chapter 14 that seem to validate what we are doing and
found other things to keep in mind as we carry the automation effort
forward:
1- The "Agile Testing Quadrants" model seems to be a good way to
categorize tests, especially when considering who is critiquing the
product (customers) and the business facing aspects of the product
(product owners). A decent pic of the model can be found at
http://lisacrispin.com/wordpress/category/agile-testing-quadrants/.
2- The "Test Automation Pyramid" is another good way to look at
prioritizing automated tests. Like many teams, I think our team has a
bit of an inverted pyramid with perhaps more automated GUI tests than
automated unit / component tests. I found what is probably a
plagiarized pic of the pyramid and the "cloud" of manual tests here:
http://blog.goneopen.com/2010/08/test-automation-pyramid-review/
(scroll down to the one that says "Crispin"). The idea is that the
foundation layer of the pyramid is the automated unit / component
tests. The API tests layer build on those and the GUI tests layer
build on those, all of which seem to be smaller in either scope/size/
emphasis/priority according to the model.
To further illustrate the model, Patrick Wilson-Welsh adds the "Three
Little Pigs" element describing the bottom layer as bricks, the middle
layer as sticks and the top layer as straw. Given the volatility of
the GUI being changed by the direction of the wind at times, the
analogy of "straw" to the GUI seems fitting.
3- Agile principles can be applied to test automation. I summarize
just a few:
a) Keep it Simple: "do the simplest thing that could possibly
work." We're not trying to automate everything at once. We're
tackling pieces of the framework first. We broke up the framework
epic into 6 stories. After we get the framework done and some of the
critical tests converted, we'll focus on the automated test technical
debt.
b) Iterative feedback: Even though we're into our automation
framework 3 sprints, we should be complete with the framework after 9
weeks. If it falls completely on it's face, it's not like we've
wasted 6 months. We can take what we've learned and start something
else if we need to.
c) Whole team approach: Although going to java is a learning curve
for some of our guys, and some of them resisted initially, it's the
language most of our dev's know and therefore, when the framework is
ready, they'll be more willing to "play" when contributing to
automated tests.
d) Taking time to do it right: Our test engineers want reassurance
that the work they've spent on creating the framework will not go to
waste by not having time during upcoming sprints to burndown test
automation technical debt. The way we plan on achieving this is to
take a bit of a hit on team velocity and allocate time for the team
each sprint to burn down these automation tasks.
Some questions that were raised:
Q: Do the authors advocate automating a certain percentage of tests?
A: Not that I can see. They do clearly call out some tests that
should not be automated, such as usability tests, corner cases,
exploratory testing or ad-hoc testing. These are the kinds of tests
that our test team wishes they had time for at the beginning of each
sprint. The more they can automate, the more they can have time for
these customer facing type tests.
Q: In relation to the priority of testing the GUI based on the Test
Automation Pyramid model, do the authors advocate less UI testing?
A: No. Some level of test automation must exercised on the GUI. But,
remember the "Big Bad Wolf". One quote from the book resonates here -
"make sure the programmers name their objects or assign IDs to them.
If they rely on system-generated identifiers, then every time a new
object is added to the page, the IDs will change, requiring changes to
the tests."
My plan is to read and report on another chapter during the next
iteration :) Hope you found something useful.
Steve
approach and decided to do a chapter report instead of a book report.
We're in the third sprint of establishing a java automation framework
for our product and converting tests to this environment so I read
this chapter searching for some validation for our efforts. I found
some points in chapter 14 that seem to validate what we are doing and
found other things to keep in mind as we carry the automation effort
forward:
1- The "Agile Testing Quadrants" model seems to be a good way to
categorize tests, especially when considering who is critiquing the
product (customers) and the business facing aspects of the product
(product owners). A decent pic of the model can be found at
http://lisacrispin.com/wordpress/category/agile-testing-quadrants/.
2- The "Test Automation Pyramid" is another good way to look at
prioritizing automated tests. Like many teams, I think our team has a
bit of an inverted pyramid with perhaps more automated GUI tests than
automated unit / component tests. I found what is probably a
plagiarized pic of the pyramid and the "cloud" of manual tests here:
http://blog.goneopen.com/2010/08/test-automation-pyramid-review/
(scroll down to the one that says "Crispin"). The idea is that the
foundation layer of the pyramid is the automated unit / component
tests. The API tests layer build on those and the GUI tests layer
build on those, all of which seem to be smaller in either scope/size/
emphasis/priority according to the model.
To further illustrate the model, Patrick Wilson-Welsh adds the "Three
Little Pigs" element describing the bottom layer as bricks, the middle
layer as sticks and the top layer as straw. Given the volatility of
the GUI being changed by the direction of the wind at times, the
analogy of "straw" to the GUI seems fitting.
3- Agile principles can be applied to test automation. I summarize
just a few:
a) Keep it Simple: "do the simplest thing that could possibly
work." We're not trying to automate everything at once. We're
tackling pieces of the framework first. We broke up the framework
epic into 6 stories. After we get the framework done and some of the
critical tests converted, we'll focus on the automated test technical
debt.
b) Iterative feedback: Even though we're into our automation
framework 3 sprints, we should be complete with the framework after 9
weeks. If it falls completely on it's face, it's not like we've
wasted 6 months. We can take what we've learned and start something
else if we need to.
c) Whole team approach: Although going to java is a learning curve
for some of our guys, and some of them resisted initially, it's the
language most of our dev's know and therefore, when the framework is
ready, they'll be more willing to "play" when contributing to
automated tests.
d) Taking time to do it right: Our test engineers want reassurance
that the work they've spent on creating the framework will not go to
waste by not having time during upcoming sprints to burndown test
automation technical debt. The way we plan on achieving this is to
take a bit of a hit on team velocity and allocate time for the team
each sprint to burn down these automation tasks.
Some questions that were raised:
Q: Do the authors advocate automating a certain percentage of tests?
A: Not that I can see. They do clearly call out some tests that
should not be automated, such as usability tests, corner cases,
exploratory testing or ad-hoc testing. These are the kinds of tests
that our test team wishes they had time for at the beginning of each
sprint. The more they can automate, the more they can have time for
these customer facing type tests.
Q: In relation to the priority of testing the GUI based on the Test
Automation Pyramid model, do the authors advocate less UI testing?
A: No. Some level of test automation must exercised on the GUI. But,
remember the "Big Bad Wolf". One quote from the book resonates here -
"make sure the programmers name their objects or assign IDs to them.
If they rely on system-generated identifiers, then every time a new
object is added to the page, the IDs will change, requiring changes to
the tests."
My plan is to read and report on another chapter during the next
iteration :) Hope you found something useful.
Steve
Mike Nauta
Mar 15, 2012, 1:29:13 PM3/15/12
to provo-agile-sc...@googlegroups.com
Thank you Steve for the summary of your presentation at yesterday's
meeting. One of my take aways was to provide a compilation of
information on the PMI-ACP exam that I used in preparing for the exam.
You should be able to find the zip file here:
meeting. One of my take aways was to provide a compilation of
information on the PMI-ACP exam that I used in preparing for the exam.
You should be able to find the zip file here:
http://dl.dropbox.com/u/15482450/PMI%20ACP.zip
We can review the contents in detail at our next meeting.
The meetings have become a very valuable source of information and
sharing for me each month. Thanks to everyone that attended. I hope
to see you all again next month.
Mike Nauta
--
Thank you,
Michael Nauta
http://www.linkedin.com/in/michaelnauta
http://twitter.com/nauta_moto
http://www.nauta-moto.com/
Reply all
Reply to author
Forward
0 new messages