Blog post: Improve black-box testing agility: meet the Frame Object pattern

[Will Manley]

This is a copy of a blog post published here: http://stb-tester.com/blog/2015/09/08/meet-the-frame-object-pattern

 

I'm reposting to the mailing list to facilitate commentary and conversation.

 

Improve black-box testing agility: meet the Frame Object pattern

By William Manley. 08 Sep 2015

In the previous blog post we discussed the Page Object pattern and how it could be used to improve the readability of your test scripts. In this blog post we introduce the Frame Object pattern, a specialisation of the Page Object pattern that can be especially effective in stb-tester’s style of black-box UI testing¹. Frame Objects can reduce the maintenance cost of your test-packs, particularly when the UI-under-test changes.

Based on Martin Fowler’s diagram for Page Objects in HTML UI testing

A Frame Object is a class that extracts information from a frame of video, typically by calling stbt.ocr() or stbt.match(). All the rest of your testing code then uses these objects. A Frame Object translates from the vocabulary of low-level image processing functions and regions (like stbt.ocr(region=stbt.Region(213, 23, 200, 36))) to the vocabulary of high-level features and user-facing concepts (like programme_title).

Here’s an example of a Frame Object (GuideFrame) and a test-case that uses it:

class GuideFrame(object):
    def __init__(self, frame=None):
        if frame is None:
            frame = stbt.get_frame()
        self.frame = frame

    def __nonzero__(self):
        return self.is_visible

    def __repr__(self):
        if self.is_visible:
            return ("GuideFrame(is_visible=True, programme_title=%r, "
                    "current_time=%r)") % (
                    self.programme_title, self.current_time)
        else:
            return "GuideFrame(is_visible=False)"

    @property
    def is_visible(self):
        return bool(stbt.match('guide-header.png', frame=self.frame))

    @property
    def programme_title(self):
        return stbt.ocr(
            region=stbt.Region(213, 23, 200, 36), frame=self.frame)

    @property
    def current_time(self):
        m = stbt.match('clock.png', frame=self.frame)
        return stbt.ocr(
            region=m.region.extend(x=20, right=100), frame=self.frame)


def open_guide():
    stbt.press('KEY_GUIDE')
    guide = stbt.wait_until(GuideFrame)
    assert guide
    return guide


def test_that_guide_displays_the_correct_time():
    guide = open_guide()
    assert guide.current_time == datetime.datetime.now().strptime('%H:%M')

This will seem very much like a Page Object – it acts as a layer of abstraction on top of the low-level image processing functions. Unlike a Page Object it contains no behaviours or any means to stimulate the device under test:

• There are no implicit dependencies on time or external stimulus once constructed.
• There no calls to stbt.get_frame() (except in the constructor). We pass self.frame into every call of stbt.match(), stbt.ocr() or any other image-processing function.
• We perform no image-processing or other expensive work in the constructor - this object is very cheap to construct.
• The object is immutable – there are no methods to change the externally visible state of the object. If you want an updated view of the system-under-test’s state you can construct a new object for each frame.
• There are no calls to stbt.wait_until() or time.sleep() - this object can’t change so there is no point waiting.
• There are no calls to stbt.press() or anything else that affects the device under test.

These may seem like arbitrary or unreasonable restrictions but they provide some great benefits:

• Testability: Unlike full Page Objects, Frame Objects can be tested very cheaply. All you need is a corpus of example screenshots. This is because Frame Objects behave in an entirely deterministic way based on the parameters provided to the constructor (typically the only parameter is a frame of video). Most significantly, Frame Objects don’t depend on the dynamic behaviour of the device-under-test which is far more difficult to capture than a few screenshots.

  By providing a __repr__ method you can test these objects using doctests. This opens up other possibilities, which we will cover in a future blog post.

• Agility: Because of their improved testability, Frame Objects are cheap to maintain and easy to evolve. If the cosmetics of the application under test change, or you find a bug in your Frame Object, just capture a screenshot of that situation and add it to your test corpus. You can then fix it with confidence:

  • Confidence that the updated Frame Object fixes the issue – you can verify this by applying your new Frame Object to the new screenshots added to your corpus.
  • Confidence that you haven’t introduced regressions with your fixes – you can see that your Frame Object behaves the same when run against your existing corpus of screenshots.
  • Confidence that you haven’t changed your expectations of the behaviour of the device-under-test – the Frame Object contains no behavioural expectations.

• Consistency: Calling stbt.match or stbt.ocr multiple times to extract information from a frame can lead to confusing inconsistencies if each function sees a different frame. The Frame Object Pattern avoids this by capturing a frame exactly once.

• Performance: Frame Objects extract required information from the frame when needed, rather than in advance. This can avoid doing a lot of expensive image matching or OCR when it’s not necessary. Because the object is immutable and each property or method on the object is deterministic you can apply memoisation, allowing you to write tests in a natural manner without a performance penalty.

So that’s all well and good, but what about the behaviours that you removed? After all they are usually the thing you want to test in the underlying device. It turns out that the Page Object and the Frame Object patterns are actually complementary rather than conflicting. Where necessary you can still have page objects, but they use Frame Objects to extract the information from the underlying frames.

Two classes rather than one sounds complicated? In our experience the page object is no longer necessary in most cases and you can get away with just using helper functions or using the Frame Objects directly from test cases. But when you have more complex interactions and your helper functions need to store state you can still create page objects. These page objects will use Frame Objects: the additional overall complexity is more than offset by the reduction in complexity in the more expensive to maintain page objects - an overall win for maintainability.

For example here’s a helper function that works in conjunction with the above Frame Object:

def select_next_programme(self, direction=1):
    key = {-1: 'KEY_LEFT', 1: 'KEY_RIGHT'}[direction]
    original_title = GuideFrame().programme_title
    for _ in range(10):
        stbt.press(key)
        if wait_until(lambda: GuideFrame().programme_title != original_title,
                      timeout_secs=3):
            return GuideFrame()


def test_that_we_can_scroll_back_and_forth_through_programmes_in_the_guide():
    guide = open_guide()
    original_title = guide.programme_title
    assert original_title

    new_title = select_next_programme(direction=1).programme_title
    assert new_title

    assert select_next_programme(direction=-1).programme_title == original_title

So: the look of your device is captured by frame objects and the behaviour of the device is captured by helper functions and page objects. This works out very well as in our experience the look of a UI will tend to change a lot during development as the developers tweak according to what looks nice, whereas the underlying behaviour changes at a much slower rate.

Frame Objects checklist:

1. Take a frame in the constructor.
2. Provide properties describing the contents of that frame.
3. Contain no actions or behaviours.
4. Include a __repr__ method printing as much data from the frame as possible to enable use with doctests and logging.
5. Should include a __nonzero__ method dictating whether the current frame is suitable - for use with stbt.wait_until.

We’re considering adding specific support for Frame Objects in stb-tester to help reduce the amount of boilerplate required

Footnotes:

1. By “black-box UI testing” we mean that you’re testing what the user sees, by capturing a video stream of the rendered UI and using image processing to check it. ↩

 

Thanks

 

Will

---

William Manley

stb-tester.com Ltd.