Automated Browser Testing

[Matt Farmer]

Hey folks,

So I’ve been spending some of my cycles at Elemica recently working on improving our automated browser testing. We’ve historically tried to use Selenium and in general nobody likes interacting with the code we need to work with to make it work and it takes an incredibly long time to run. At first, I thought this was largely a symptom of our API choice. Though I think I found a better API through webdriver.io (a Node library) – we still ran headlong into a lot of problems.

Some issues we’ve run into center around timing actions for AJAX requests by either inserting really long pauses or injecting JS into the browser to check the jQuery active request counts and the fact that the Selenium architecture itself doesn’t seem to be really reliable. It’s regularly ignoring custom script timeouts I send it, tests written that pass in Chrome don’t pass in Phantom, the speed that Phantom has boasted just doesn’t hold water when run through webdriver, Firefox appears to have been broken with Selenium since January, and Safari just kind of ignores your requests to start a webdriver session.

All in all, I’m not confident that Selenium is a reliable platform for developing cross-browser functional tests. We have historically become very disheartened by the unreliability of tests which kills any internal motivation to maintain them effectively.

As a result we’ve started investigating other solutions to see what’s out there. I tried using Casper with Phantom directly, which really wouldn’t answer the question of cross-browser compatibility but would be interesting to check, and it experienced some of the same inherent problems that we had with Selenium. Additionally, we found a tool called Sahi that’s almost good enough to make writing these tests a point and click operation. Some things don’t work quite so well for a Lift application, but we’ve reached out to them to ask them whether or not there are configuration settings that might be able to fix those rough edges.

Anyway, I’m interested in asking: What is everyone using for this need? Has everyone else just been suffering with Selenium’s problems or are there other solutions that you guys have found to meet this need?

Would love to hear your thoughts.

Cheers,

—
Matt Farmer | Blog | Twitter

[Joe Barnes]

Hi, my name is Joe and I too suffer from Selenium.

So... if you had asked me to describe my team's experience with Selenium, you would have taken the words out of my mouth.  We so badly need a good solution to this problem, but the team can never buy it wholeheartedly because it simply cannot be relied on.  We have learned our way around some of issues you mention related to AJAX requests (in short, we use Scalatest's eventually for that) and have gotten good at writing tests which are not fundamentally flaky with race-conditions.

However, Selenium flat out screws up often enough you cannot build any reliable machinery around it.  You can see where I have also utilized Selenium to give me full coverage of lift-ng.  I run an entire test suite exercising over every feature of lift-ng against four different Scala/Lift version configurations.  How do I rely on it?  I give it 5 mulligans.  Each suite gets 5 chances to pass all the way through.  I NEVER... let me repeat... NEVER see it pass all four the first time.  This is a dead-simple stupid project that doesn't do anything meaningful.  Up the ante to my project at work and you have a team of sharp developers who reluctantly write Selenium tests because they have no faith that it will create a reliable test suite.

I have no idea how to make it better at this point.  The only alternative which has ever been offered is the Squish GUI Tester from froglogic.  One of the teams we have on board via acquisition used this to test their web UI.  However, it was more of a canvas-based GUI, and I got the impression it's less ideal for plan ol' DOM.

Unfortunately, I can only add frustration and have no solutions to offer you, Matt.

Joe

[John Walker]

Hi Matt
Are you only looking at open source?
Who is writing / maintaining the tests, devs or testers?

I have been using a framework built on TestComplete (using JScript and leaning heavily on underscore.js)  for a number of years now and don't suffer from the issues you are facing.

TestComplete is a very different tool to Selenium but from the sounds of things that might not be too much of a problem to you.

John