Test randomly failing when running more than X at a time

[Stephen Stchur]

I'm using xUnit as my test framework.  The way xUnit works, tests in different classes will be run in parallel, but tests within the same class will run in serial.  This is in fact, what I'm seeing -- essentially, for every test class I have, a new webdriver instance launches.  But I have a problem in that the test seem very flaky.  I'll run my entire test suite, and a hand full of tests will randomly fail.  If I go back and manually run just those that failed, they will work.  After a LOT of testing, it seems that when too many tests are run at once, I get random failures.

To eliminate my own app from the equation, I create a very simply test that loads bing.com and searches for "The Avett Brothers" and then verifies that the search worked.  I copied this test 5 times and gave each test a different name (exact same code though).  I then copied the class 10 times.  Thus, I have 11 total test classes, each running 5 tests.  If I run any of these tests alone, it ALWAYS succeeds.  If I run them all, 11 browsers get launches (I'm using chromedriver incognito every time) and MOST of the tests pass, but a few will occasionally fail.

Anyone have any clue what could cause this?  I've tried increasing the delay when finding elements on that page to 30sec (from 10) but honestly, 10 should be WAY more than enough.  There doesn't seem to be any specific number of tests running in parallel that cause the problem -- just "more than a few" in parallel and I seem to get random failures (in my real project).

Here's what the test looks like:

// 10 more classes just like this named BingTest1 - 10 which will all run in parallel

public class BingTests : TestBase

{

public BingTests(WebDriverFixture fixture) : base(fixture) { }

[Fact]

public void SearchBingForTheAvettBrothers()

{

IWebElement searchBox = Driver.Wait(ExpectedConditions.ElementIsVisible(By.Id("sb_form_q")));

searchBox.SendKeys("The Avett Brothers" + Keys.Enter);

var link = Driver.Wait(ExpectedConditions.ElementIsVisible(By.CssSelector("a[href='http://theavettbrothers.com']")));

Assert.True(link.Displayed);link.Click();

var image = Driver.Wait(ExpectedConditions.ElementIsVisible(By.CssSelector("img[src='/templates/default/images/logo.svg']")));

Assert.True(image.Displayed);

}

// 4 more tests just like above named SearchBingForTheAvettBrothers2 - 5 which will all run in serial

}

public class TestBase: IClassFixture<WebDriverFixture>, IDisposable

{

protected IWebDriver Driver { get; private set; }

public TestBase(WebDriverFixture fixture)

{

var options = new ChromeOptions();

options.AddArguments("--no-sandbox", "--window-size=1366,768");

options.AddArgument("--incognito");

var codeBaseUrl = new Uri(Assembly.GetExecutingAssembly().CodeBase);

var codeBasePath = Uri.UnescapeDataString(codeBaseUrl.AbsolutePath);

Driver = new ChromeDriver(Path.GetDirectoryName(codeBasePath), options);

}

public void Dispose()

{

Driver.Quit();

}

}

and Driver.Wait is just an extension method that does something like this:

var wait = new WebDriverWait(driver, TimeSpan.FromSeconds(timeoutInSeconds));

return wait.Until(condition);

[Mike Hetzer]

I don't use xUnit but my guess is that with the parallel tests - you are getting some overlap from subsequent tests using the same driver for a little bit before new drivers are created.
Try initializing a new driver in and for each test for this theory . .

Or is there a way in xUnit to force run them in different threads?

Also, I hope you arent being forced to use xUnit - it's very primitive.
It seems like you are using C# - I'd recommend NUnit.

[Stephen Stchur]

Thanks Mike!  I will give this a try, and no I'm not forced to use xUnit.  We're using dot net core and my understanding was that xUnit was the "latest" and what runs automatically with "dotnet test."

But I was actually wondering if xUnit might be part of the problem.  I will look into nUnit and others.