Blog post: Handling UI elements of non-fixed size

[Will Manley]

This is a copy of a blog post published here: http://stb-tester.com/blog/2015/08/25/handling-ui-element-of-non-fixed-size

I'm reposting to the mailing list to facilitate commentary and conversation.

Handling UI elements of non-fixed size

By William Manley. 25 Aug 2015

In this blog post we use the OpenCV cv2.floodFill function to improve the robustness of test helper functions for dynamic UIs.

Recently we have been writing tests involving extracting information from BBC iPlayer on some smart TVs.

When you just start playing content iPlayer shows an overlay providing some information about the content being played:

We wanted to read the title of the current piece of content so we can make some assertions about it. For example:

assert overlay.title == "The One Show"

We need to know the location of the title on screen before we can perform OCR. For most static UIs we can hard-code it.

Naturally we use the page object pattern to improve the self-testability and clarity of our tests. The implementation for this screen looks like:

class IPlayerContentOverlay(object):
    def __init__(self, frame=None):
        if frame is None:
            frame = stbt.get_frame()
        self.frame = frame

    @property
    def title(self):
        info_box_region = stbt.Region(x=194, y=437, width=763, height=173)
        title_region = info_box_region.replace(height=36)
        return stbt.ocr(region=title_region, frame=self.frame)

We test this against the screenshot we captured above and it works:

>>> overlay = IPlayerContentOverlay(frame=screenshot1)
>>> overlay.title
The One Show

Great. But wait! not all iPlayer overlays are identical. Some are taller than others depending on the content and some are in a different position:

In the previous example we hard-coded the location of the overlay with info_box_region = Region(x=194, y=437, width=763, height=173). This will cause our title helper function to produce the wrong title for this screenshot.

Fortunately the iPlayer overlay is nonetheless fairly regular:

• It is always rectangular
  
• It is opaque and always has a constant background colour
  
• It is always near the bottom of the screen
  
• It is always roughly centred
  

We can take advantage of these facts to locate it.

Fortunately OpenCV (the library stb-tester uses for image processing under the hood) provides a function for finding connected areas of constant colour: cv2.floodFill. Among other things it takes an image and a point and returns the bounding rectangle of the connected areas.

Using this we can create a helper function to integrate it with our stbt scripts:

def find_contiguous_region(point, frame=None):
    """
    This function can be used to find contiguous regions of a single colour.
    This is useful when a UI element is in a constant position but has
    non-constant size.

    Args:
        frame (numpy.ndarray): An image
        point (tuple): A 2-tuple (x, y) coordinates of the location in the image
            within the region you want to find.

    Returns
        stbt.Region: The bounding box of contiguous colour from the given point.
    """
    import cv2
    if frame is None:
        frame = stbt.get_frame()
    _, rect = cv2.floodFill(
        frame, None, point, None,
        flags=cv2.FLOODFILL_FIXED_RANGE | cv2.FLOODFILL_MASK_ONLY)
    region = stbt.Region(*rect)
    stbt.debug("find_contiguous_region(..., %s) -> %s" % (point, str(region)))
    return region

We know that the pixel at (640, 608) is always a background pixel within the overlay so we will use that as the point parameter. Our IPlayerContentOverlay class is modified like so:

class IPlayerContentOverlay(object):
    def __init__(self, frame=None):
        if frame is None:
            frame = stbt.get_frame()
        self.frame = frame

    @property
    def title(self):
        info_box_region = find_contiguous_region((640, 608), self.frame)
        title_region = info_box_region.replace(height=36)
        return stbt.ocr(region=title_region, frame=self.frame)

Note that we are now dynamically working out the location of the overlay with info_box_region = find_contiguous_region((640, 608), self.frame).

And now it works for both screenshots:

>>> overlay = IPlayerContentOverlay(frame=screenshot2)
>>> overlay.title
The One Show

We are currently considering adding find_contiguous_region or some generalisation of it to stb-tester proper.

Thanks

 

Will

---

William Manley

stb-tester.com Ltd.