A/B testing and Selenium automation?

[David]

Should Selenium automation cover/handle A/B testing? Or should it only cover one of the two flows (control/normal flow, or test/new flow)?

If should cover, what are suggested best practice approaches to handle the A/B tests (differences in UI flow & difference in UI element locators).

For example, what if the business group (ideally) wants automated test coverage of both A/B flows?

By default, most test automation frameworks don't offer A/B testing support out of the box, so you have to built it in somehow if supporting it. 

And this is easier for A/B tests covering small changes to a site but for major UI changes via A/B tests, that can get a bit complex to architect and maintain in Selenium automation.

[David]

Doing a google search, found one post

http://stackoverflow.com/questions/5269027/how-do-you-handle-testing-with-selenium-when-you-are-running-ab-tests

not sure if there are others.

However, that post's answer doesn't cover the case in which you want to cover both flows...

[David Lai]

Personally, I like using abstract factory pattern for this.

Here's a short write up I wrote a while ago on how to do it using webdriver.

http://engineeringquality.blogspot.com/2012/06/creating-generic-advanced-page-factory.html

[David Lai]

One thing I didn't mention in the writeup that I should of, was using that technique, you can easily create variations on the same page by just extending the the other PageObject class, then overriding the methods in where they differ.

[David]

Thanks David, useful info. A colleague recommended a similar approach but for something else (not A/B testing).

Just wondering, what are your thoughts about A/B tests that are moderately complex in differing A/B flow but that one flow will eventually replace the other flow. Of course the timeline of when that happens is up to the business/executive group of the company.

In such a scenario and with your approach, is it even worthwhile to attempt to cover 2 flows in automation? I guess if the test duration is long enough in terms of months (maybe weeks), then perhaps ok, but for a 1-2 week run, that might be too much work with some of the code eventually thrown away, unless (your suggested) framework was already in place to facilitate the task, in this case not yet.

[David Lai]

For my case, I didn't get to going around major flow changes for A/B testing.  These were mainly different page arrangements or page UI redesigns, then after some period they'll eventually prune the variations out.  So in that case, I can prioritize testing, by passing in a pageset param, 1 being the most use, to X.. being the least used.  It'll test using the most used (better scoring) flows first.  Usually by the 3rd or 4th pagesets, we almost don't bother testing those as they'll be pruned out within the next month.

The abstract factory approach is a good approach for testing mobile/desktop/tablet variations on the same page.  For A/B tests, of single PageObjects, you can quickly extend the original pageset, then override where they differ, then prune it out later when that pageset is removed.

For A/B tests of entire flows, it's annoying and you end up having to bite the bullet if it spans multiple sprints and have a separate set of tests.  In terms of organizing my test structure, I use 2 layers of abstraction between separating my high level tests from low level details.  The first level I call Pages/Screens, these are Page Objects, or Objects representing screens in the application.  The other layer I call flows, these encompass any reusable steps that spans 2 or more screens/pages.  At the test layer itself, the test will read almost like pure english.  (Ideally I'd like to use a BDD framework like cucumber at this layer, however I opted not to as it's hard to extend those frameworks with custom test runners for massive parallelization vs. a simple unit test framework)