By William Manley. 08 Sep 2015
In the previous blog post we discussed the Page Object pattern and how it could be used to improve the readability of your test scripts. In this blog post we introduce the Frame Object pattern, a specialisation of the Page Object pattern that can be especially effective in stb-tester’s style of black-box UI testing1. Frame Objects can reduce the maintenance cost of your test-packs, particularly when the UI-under-test changes.
Based on Martin Fowler’s diagram for Page Objects in HTML UI testing
A Frame Object is a class that extracts information from a frame of video,
typically by calling stbt.ocr()
or stbt.match()
. All the rest of your
testing code then uses these objects. A Frame Object translates from the
vocabulary of low-level image processing functions and regions (like
stbt.ocr(region=stbt.Region(213, 23, 200, 36))
) to the vocabulary of
high-level features and user-facing concepts (like programme_title
).
Here’s an example of a Frame Object (GuideFrame
) and a test-case that uses it:
class GuideFrame(object):
def __init__(self, frame=None):
if frame is None:
frame = stbt.get_frame()
self.frame = frame
def __nonzero__(self):
return self.is_visible
def __repr__(self):
if self.is_visible:
return ("GuideFrame(is_visible=True, programme_title=%r, "
"current_time=%r)") % (
self.programme_title, self.current_time)
else:
return "GuideFrame(is_visible=False)"
@property
def is_visible(self):
return bool(stbt.match('guide-header.png', frame=self.frame))
@property
def programme_title(self):
return stbt.ocr(
region=stbt.Region(213, 23, 200, 36), frame=self.frame)
@property
def current_time(self):
m = stbt.match('clock.png', frame=self.frame)
return stbt.ocr(
region=m.region.extend(x=20, right=100), frame=self.frame)
def open_guide():
stbt.press('KEY_GUIDE')
guide = stbt.wait_until(GuideFrame)
assert guide
return guide
def test_that_guide_displays_the_correct_time():
guide = open_guide()
assert guide.current_time == datetime.datetime.now().strptime('%H:%M')
This will seem very much like a Page Object – it acts as a layer of abstraction on top of the low-level image processing functions. Unlike a Page Object it contains no behaviours or any means to stimulate the device under test:
stbt.get_frame()
(except in the constructor). We pass
self.frame
into every call of stbt.match()
, stbt.ocr()
or any other
image-processing function.stbt.wait_until()
or time.sleep()
- this object
can’t change so there is no point waiting.stbt.press()
or anything else that affects the device
under test.These may seem like arbitrary or unreasonable restrictions but they provide some great benefits:
Testability: Unlike full Page Objects, Frame Objects can be tested very cheaply. All you need is a corpus of example screenshots. This is because Frame Objects behave in an entirely deterministic way based on the parameters provided to the constructor (typically the only parameter is a frame of video). Most significantly, Frame Objects don’t depend on the dynamic behaviour of the device-under-test which is far more difficult to capture than a few screenshots.
By providing a __repr__
method you can test these objects using
doctests. This opens up other possibilities, which we will cover in a
future blog post.
Agility: Because of their improved testability, Frame Objects are cheap to maintain and easy to evolve. If the cosmetics of the application under test change, or you find a bug in your Frame Object, just capture a screenshot of that situation and add it to your test corpus. You can then fix it with confidence:
Consistency: Calling stbt.match
or stbt.ocr
multiple times to extract
information from a frame can lead to confusing inconsistencies if each
function sees a different frame. The Frame Object Pattern avoids this by
capturing a frame exactly once.
Performance: Frame Objects extract required information from the frame when needed, rather than in advance. This can avoid doing a lot of expensive image matching or OCR when it’s not necessary. Because the object is immutable and each property or method on the object is deterministic you can apply memoisation, allowing you to write tests in a natural manner without a performance penalty.
So that’s all well and good, but what about the behaviours that you removed? After all they are usually the thing you want to test in the underlying device. It turns out that the Page Object and the Frame Object patterns are actually complementary rather than conflicting. Where necessary you can still have page objects, but they use Frame Objects to extract the information from the underlying frames.
Two classes rather than one sounds complicated? In our experience the page object is no longer necessary in most cases and you can get away with just using helper functions or using the Frame Objects directly from test cases. But when you have more complex interactions and your helper functions need to store state you can still create page objects. These page objects will use Frame Objects: the additional overall complexity is more than offset by the reduction in complexity in the more expensive to maintain page objects - an overall win for maintainability.
For example here’s a helper function that works in conjunction with the above Frame Object:
def select_next_programme(self, direction=1):
key = {-1: 'KEY_LEFT', 1: 'KEY_RIGHT'}[direction]
original_title = GuideFrame().programme_title
for _ in range(10):
stbt.press(key)
if wait_until(lambda: GuideFrame().programme_title != original_title,
timeout_secs=3):
return GuideFrame()
def test_that_we_can_scroll_back_and_forth_through_programmes_in_the_guide():
guide = open_guide()
original_title = guide.programme_title
assert original_title
new_title = select_next_programme(direction=1).programme_title
assert new_title
assert select_next_programme(direction=-1).programme_title == original_title
So: the look of your device is captured by frame objects and the behaviour of the device is captured by helper functions and page objects. This works out very well as in our experience the look of a UI will tend to change a lot during development as the developers tweak according to what looks nice, whereas the underlying behaviour changes at a much slower rate.
Frame Objects checklist:
__repr__
method printing as much data from the frame as possible
to enable use with doctests and logging.__nonzero__
method dictating whether the current frame is
suitable - for use with stbt.wait_until
.We’re considering adding specific support for Frame Objects in stb-tester to help reduce the amount of boilerplate required
Footnotes:
By “black-box UI testing” we mean that you’re testing what the user sees, by capturing a video stream of the rendered UI and using image processing to check it. ↩