Shared tests using different withFixture

863 views
Skip to first unread message

None

unread,
Jun 16, 2011, 2:05:01 PM6/16/11
to scalatest-users
Hello all,

First of all, i'm very new to this list so i'm sorry for any stupid
question. =P

Well, currently I have a scenario where i'm not sure what is the best
solution to test it.

I want to create a test suite for a managed interface (interface with
start and close methods, like a stream). So, reading scalatest
documentation I saw the 'withFixture(OneArgTest)' method thought that
would be the way to go.

So, at this point my test suite would looks like:

[code]
class MyTestSuite extends FixtureFunSuite {

override def withFixture(test: OneArgTest) {
val input = createInput;
try {
test(input)
} finally {
input.close
}
}

test("testing example...") { input =>
// performing tests on input
}

test("another testing example...") { input =>
// performing tests on input
}

}
[/code]

So far so good.

Now I would like to perform all those tests over more than one
implementation of my managed interface (like as i had a data source
interface and want to test it connecting to more than one database).
How would I do that?

I read 'Shared tests' section of FunSuite's documentation. But that
example doesn't seem to fit to my needs, since I need to perform my
tests on a managed implementation, and example is testing a stack that
is just "thrown away" at the end of each test.

Any suggestion?

Bill Venners

unread,
Jun 16, 2011, 2:14:22 PM6/16/11
to scalate...@googlegroups.com
Hi Fabio,

You message came though with from name of "None", but I'm guessing
from the rest of your email address you're actually Some(Fabio).
Please correct me if I'm wrong.

Can you elaborate on what you mean by managed object? If I understand
correctly, the input you're creating implements some interface that
has start() and a close() method, which are like lifecycle methods. So
you'd want to maybe start() it before each test and close() it after?
Is that what you mean by managed object?

Then you have multiple implementations of that interface that you need
to run the same tests across? If that's correct then I think either
shared tests or using a one-column table should work. Another
possibility, which I didn't consider before, is to call the test
function multiple times from within withFixture. I think that would
also work, and it could possibly be the place to use the table.

Please let me know if I understand your requirements correctly first,
and then I'll try and propose a solution for you. Also, which version
of ScalaTest are you using?

Thanks.

Bill

> --
> You received this message because you are subscribed to the Google
> Groups "scalatest-users" group.
> To post to this group, send email to scalate...@googlegroups.com
> To unsubscribe from this group, send email to
> scalatest-use...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/scalatest-users?hl=en
> ScalaTest itself, and documentation, is available here:
> http://www.artima.com/scalatest
>

--
Bill Venners
Artima, Inc.
http://www.artima.com

Fabio Cechinel Veronez

unread,
Jun 16, 2011, 2:52:14 PM6/16/11
to scalate...@googlegroups.com
Hi Bill,

Thanks for your prompt response, and yes, I'm Some("Fabio"). =P

Well, when I say managed objects I mean objects like Input/Output
Streams, java.sql.Connection, etc, which have 'close' method that
should be called after using them.

In my case I'm working with Selenium WebDrivers [1], which contain a
method 'quit' that, from what i know, are expected to be called after
each test. So, in this way, I have a new WebDriver for each test.

WebDriver is the interface, and selenium provides one implementation
of WebDriver for each browser it supports (IE, Firefox, Chrome..).

So, my goal here is: write a test suite for a given web page and run
it using a set of WebDriver implementations.

Currently i'm using scala-test 1.6.1 and scala 2.9.0.

I have already made some tries using your suggestion of calling test
function multiple times within withFixture method but I found two not
so good things about it:

1- Report tool only record test being called once and I have no way to
tell what drivers were tested.
2- When a test fail for one driver all other subsequent drivers will
be ignored (since exceptions are used to register failure).

Is my points correct?

About using shared tests I guess that is not the better solution since
tests would have to manually close (quit in my case) my resource
(webdrive), wouldn't they?

[1] http://code.google.com/p/selenium/wiki/GettingStarted
[2] http://selenium.googlecode.com/svn/trunk/docs/api/java/org/openqa/selenium/WebDriver.html

--
Fabio Cechinel Veronez

Bill Venners

unread,
Jun 16, 2011, 3:02:46 PM6/16/11
to scalate...@googlegroups.com
Hi Fabio,

On Thu, Jun 16, 2011 at 11:52 AM, Fabio Cechinel Veronez
<fabio....@gmail.com> wrote:
> Hi Bill,
>
> Thanks for your prompt response, and yes, I'm Some("Fabio"). =P
>
> Well, when I say managed objects I mean objects like Input/Output
> Streams, java.sql.Connection, etc, which have 'close' method that
> should be called after using them.
>
> In my case I'm working with Selenium WebDrivers [1], which contain a
> method 'quit' that, from what i know, are expected to be called after
> each test. So, in this way, I have a new WebDriver for each test.
>
> WebDriver is the interface, and selenium provides one implementation
> of WebDriver for each browser it supports (IE, Firefox, Chrome..).
>
> So, my goal here is: write a test suite for a given web page and run
> it using a set of WebDriver implementations.
>
> Currently i'm using scala-test 1.6.1 and scala 2.9.0.
>
> I have already made some tries using your suggestion of calling test
> function multiple times within withFixture method but I found two not
> so good things about it:
>
> 1- Report tool only record test being called once and I have no way to
> tell what drivers were tested.
> 2- When a test fail for one driver all other subsequent drivers will
> be ignored (since exceptions are used to register failure).
>
> Is my points correct?
>

Yes, you're correct.

> About using shared tests I guess that is not the better solution since
> tests would have to manually close (quit in my case) my resource
> (webdrive), wouldn't they?
>

No I think that can be done via withFixture. I'll code up a quick
example to make sure.

Bill

Derek Williams

unread,
Jun 16, 2011, 3:08:10 PM6/16/11
to scalate...@googlegroups.com
I have been playing around with doing something similar, and maybe
there is a better way of accomplishing this, but maybe this would work
for you?

class MyClassSpec extends WordSpec with MustMatchers {

"my object" when {

"doing this" must {
behave like goodObject { test =>
myObj = setupMyObject()
try {
test(myObj)
} finally {
myObj.close()
}
}
}

"doing that" must {
behave like goodObject { test =>
myObj = setupMyObject()
try {
myObj.doThat
test(myObj)
} finally {
myObj.close()
}
}
}

"being bad!" must {
behave like badObject { test =>
myObj = setupMyObject()
try {
myObj.doBad
test(myObj)
} finally {
myObj.close()
}
}
}
}

def goodObject(f: (MyObject => Unit) => Unit) {
"good behaviour 1" in { f((myObj) => myObj must be ('good)) }
"good behaviour 2" in { f((myObj) => myObj must not be ('bad)) }
}

def badObject(f: (MyObject => Unit) => Unit) {
"bad behaviour 1" in { f((myObj) => myObj must not be ('good)) }
"bad behaviour 2" in { f((myObj) => myObj must be ('bad)) }
}
}

--
Derek

Bill Venners

unread,
Jun 16, 2011, 3:50:34 PM6/16/11
to scalate...@googlegroups.com
Hi Fabio,

Here's how I'd recommend you do this:

abstract class WebDriver {
private var quitHasBeenCalled = false
def quit() {
quitHasBeenCalled = true
}
override def toString = getClass.getName
}

class IEWebDriver extends WebDriver
class ChromeWebDriver extends WebDriver
class FirefoxWebDriver extends WebDriver

import org.scalatest.fixture.FixtureFunSuite

class WebDriverSuite extends FixtureFunSuite {

type FixtureParam = WebDriver

def withFixture(test: OneArgTest) {

// Extract browser name from the test name
val browserName = test.name.takeWhile(_ != ':')

// Create the appropriate driver for the browser
val webDriver = browserName match {
case "IE" => new IEWebDriver
case "Chrome" => new ChromeWebDriver
case "Firefox" => new FirefoxWebDriver
}

// Test with the driver fixture, ensuring it is cleaned up after
try {
test(webDriver)
}
finally {
webDriver.quit()
}
}

for (browser <- List("IE", "Chrome", "Firefox")) {
test(browser + ": test one") { driver =>
info("Testing using " + driver)
}
test(browser + ": test two") { driver =>
info("Testing using " + driver)
}
test(browser + ": test three") { driver =>
info("Testing using " + driver)
}
test(browser + ": test four") { driver =>
info("Testing using " + driver)
}
test(browser + ": test five") { driver =>
info("Testing using " + driver)
}
}
}

What I did is encode the browser name at the beginning of each test
name, using the for loop. So each pass through the for loop you
register a batch of tests for one browser. But the driver will be
passed to each test. To make sure it passed the appropriate driver,
withFixture extracts the browser name from the front of the test name,
then creates the appropriate driver, and passes that into the test
function. That way you get one of each test you write per browser.

When I run this from the interpreter it looks like:

setups-MacBook-Pro-2:sharedt bv$ scala -cp
.:scalatest-1.6.1/scalatest-1.6.1.jar
Welcome to Scala version 2.9.0.final (Java HotSpot(TM) 64-Bit Server
VM, Java 1.6.0_24).
Type in expressions to have them evaluated.
Type :help for more information.

scala> import org.scalatest._
import org.scalatest._

scala> run(new WebDriverSuite)
WebDriverSuite:
- IE: test one
+ Testing using IEWebDriver
- IE: test two
+ Testing using IEWebDriver
- IE: test three
+ Testing using IEWebDriver
- IE: test four
+ Testing using IEWebDriver
- IE: test five
+ Testing using IEWebDriver
- Chrome: test one
+ Testing using ChromeWebDriver
- Chrome: test two
+ Testing using ChromeWebDriver
- Chrome: test three
+ Testing using ChromeWebDriver
- Chrome: test four
+ Testing using ChromeWebDriver
- Chrome: test five
+ Testing using ChromeWebDriver
- Firefox: test one
+ Testing using FirefoxWebDriver
- Firefox: test two
+ Testing using FirefoxWebDriver
- Firefox: test three
+ Testing using FirefoxWebDriver
- Firefox: test four
+ Testing using FirefoxWebDriver
- Firefox: test five
+ Testing using FirefoxWebDriver

Let me know if you have any questions or concerns about this approach.

By the way in the process of answering your question I found a bug in
my shared tests example that I think came in at Scala 2.8. I need to
change +: to +=: in push in the Stack class.

Bill

Fabio Cechinel Veronez

unread,
Jun 16, 2011, 4:32:57 PM6/16/11
to scalate...@googlegroups.com
Hello Bill,

Thank you very much.

Very interesting this approach of sending information encoded at test's name.

Is there any other way to send this king of information? Maybe using configMap.

Btw, is there any reason why test's tags info are not present at OneArgTest?

Bill Venners

unread,
Jun 16, 2011, 5:07:17 PM6/16/11
to scalate...@googlegroups.com
Hi Fabio,

On Thu, Jun 16, 2011 at 1:32 PM, Fabio Cechinel Veronez
<fabio....@gmail.com> wrote:
> Hello Bill,
>
> Thank you very much.
>
> Very interesting this approach of sending information encoded at test's name.
>

Test names all have to be unique in a ScalaTest suite, and to make
them unique when using shared tests you need to encode something about
the fixture in there anyway. So may as well pull it out in withFixture
in your case.

> Is there any other way to send this king of information? Maybe using configMap.
>

The configMap passed to withFixture is done by runTest, so if you
override runTest you could also add things to the configMap before you
pass it into withFixture. But that would be a bit of work.

> Btw, is there any reason why test's tags info are not present at OneArgTest?
>

Just that my intention for tags was to allow filtering of tests to run
and not run. So once a test survives filtering, withFixture would be
called and it should probably just run the test. If you want to use a
test's tags in withFixture, though, you could do so by calling tags
and looking up the test's tags by its test name (which is passed to
withFixture).

Jack Chi

unread,
Oct 3, 2013, 5:37:00 PM10/3/13
to scalate...@googlegroups.com
Hi Some(Fabio) & Bill, 

I am still confused on how to share integration tests across multiple browsers (Chrome / Firefox / Safair / IE) without recreating these tests. 

I am using FlatSpec with ScalaTest 2.0RC1 & setting up TestServer in PlayFramework 2.2.0. Is this not able to accomplished in FlatSpec?

I've read the sharing fixtures Stack example and looked at the iterative approach but still do not know where to start? For example, just running through checking HTML BODY element to show up in the correct language for internationalization messages. 

thanks, 
Jack Chi

Bill Venners

unread,
Oct 3, 2013, 6:14:59 PM10/3/13
to scalate...@googlegroups.com
Hi Jack,

One way is to use the Driver trait. You'll need to use 2.0.RC1. There's a terse example on its Scaladoc:

http://www.artima.com/docs-scalatest-2.0.RC1/#org.scalatest.selenium.Driver

Just stick your tests you want to "share" between browsers into traits that have WebBrowser with Driver as their self type. Then make concrete classes for each Browser you want to use.

I believe another way to do it would be to use a table of browsers and run tests forAll (browsers). I'll try and work up an example of that today and, if it works, add it to the documentation.

Bill


--
You received this message because you are subscribed to the Google
Groups "scalatest-users" group.
To post to this group, send email to scalate...@googlegroups.com
To unsubscribe from this group, send email to
scalatest-use...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/scalatest-users?hl=en
ScalaTest itself, and documentation, is available here:
http://www.artima.com/scalatest
---
You received this message because you are subscribed to the Google Groups "scalatest-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scalatest-use...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Jack Chi

unread,
Oct 3, 2013, 7:49:43 PM10/3/13
to scalate...@googlegroups.com
Hi Bill, 

In my tests, I'm using FlatSpec / Matchers / BeforeAndAfterAll, if I include it as such, I get an illegal inheritance compilation error

trait MyBrowserTests {
    this: WebBrowser with Driver with FlatSpec with Matchers with BeforeAndAfterAll

     val server = TestServer(3333)
     // my tests...
}

class MyBrowswerTestsWithChrome extends MyBrowserTests with Chrome
...

If I leave FlatSpec / Matchers / BeforeAndAfterAll out, it'll compile but I cannot use my tests. 

Jack


On Thursday, June 16, 2011 11:05:01 AM UTC-7, None wrote:

Jack Chi

unread,
Oct 3, 2013, 7:53:06 PM10/3/13
to scalate...@googlegroups.com
Ah silly me. 

Use this inheritance pattern. 

trait MyBrowserTests extends FlatSpec with Matchers with BeforeAndAfterAll {
  this: WebBrowser with Driver =>
 // tests
}


Jack Chi 

Bill Venners

unread,
Oct 3, 2013, 8:14:50 PM10/3/13
to scalate...@googlegroups.com
Hi Jack,

Yup. That should do it. Sorry for the confusion. I'll clarify the documentation.

Bill


--
You received this message because you are subscribed to the Google
Groups "scalatest-users" group.
To post to this group, send email to scalate...@googlegroups.com
To unsubscribe from this group, send email to
scalatest-use...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/scalatest-users?hl=en
ScalaTest itself, and documentation, is available here:
http://www.artima.com/scalatest
---
You received this message because you are subscribed to the Google Groups "scalatest-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scalatest-use...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Jack Chi

unread,
Oct 3, 2013, 8:41:04 PM10/3/13
to scalate...@googlegroups.com
No sorry for being a total noob. Would like to help with documentation or examples if you need me. 

Jack

Jack Chi

unread,
Oct 4, 2013, 5:45:03 PM10/4/13
to scalate...@googlegroups.com
I find that if I use a trait instead of a class, the BeforeAndAfterAll () cases gets ran twice:

trait AuthenticatorSuites extends FlatSpecLike with BeforeAndAfterAll with Matchers{
  this: Suite =>

  override def beforeAll() {
    Logger.debug("AuthenticatorSuites before")
    super.beforeAll()
  }

  override def afterAll() {
    super.afterAll()
    Logger.debug("AuthenticatorSuites after")
  }

  behavior of "test"
  it should "see this test " in {
    1 shouldBe (1)
  }

  it should "als see this test" in {
    "Jack" shouldEqual ("Jack")
  }
}

@RunWith(classOf[JUnitRunner])
class AuthenticatorOne extends AuthenticatorSuites {
  Logger.info("trace")

Jack Chi

unread,
Oct 4, 2013, 8:11:46 PM10/4/13
to scalate...@googlegroups.com
I got it! Do not mark your test with JUnitRunner. 

Another question - Is there a way to detect the existence of the different web browsers available on the system and choose to run those tests?


On Thursday, June 16, 2011 11:05:01 AM UTC-7, None wrote:

Bill Venners

unread,
Oct 4, 2013, 8:45:53 PM10/4/13
to scalate...@googlegroups.com
Hi Jack,


On Fri, Oct 4, 2013 at 2:45 PM, Jack Chi <jc...@wirelessglue.com> wrote:
I find that if I use a trait instead of a class, the BeforeAndAfterAll () cases gets ran twice:

trait AuthenticatorSuites extends FlatSpecLike with BeforeAndAfterAll with Matchers{
  this: Suite =>

  override def beforeAll() {
    Logger.debug("AuthenticatorSuites before")
    super.beforeAll()
  }

  override def afterAll() {
    super.afterAll()
    Logger.debug("AuthenticatorSuites after")
  }

  behavior of "test"
  it should "see this test " in {
    1 shouldBe (1)
  }

  it should "als see this test" in {
    "Jack" shouldEqual ("Jack")
  }
}

@RunWith(classOf[JUnitRunner])
class AuthenticatorOne extends AuthenticatorSuites {
  Logger.info("trace")
}

Looks like you found a solution already. What I think may be going on is that JUnit runs each test in its own instance, so because you have two tests, you're seeing two invocations of the beforeAll and afterAll. I'll give that a try and if that's the case, document that as well. You're definitely helping me find areas where I can improve the documentation!

Bill

Chee Seng Chua

unread,
Oct 5, 2013, 12:11:03 AM10/5/13
to scalate...@googlegroups.com
Hi Jack, 


>>Another question - Is there a way to detect the existence of the different web browsers available on the system and choose to run those tests?

The following try/catch trick used here might help you to detect if a web browser is available:


may be a good candidate for enhancement.

Hope it helps.

Cheers, 
Chee Seng


--
You received this message because you are subscribed to the Google
Groups "scalatest-users" group.

Jack Chi

unread,
Oct 7, 2013, 4:05:12 PM10/7/13
to scalate...@googlegroups.com
Hi Chua Chee Seng, 

Thanks for pointing me to org.scalatest.ChromeDriver / FirefoxDriver / etc. It does help with the try { ... } catch block. But I'd need to structure my tests a bit differently. Right now I'm performing all the tests in my : 

trait MyBrowserTests extends FlatSpec with Matchers with TestServerSuite { 
... // shared tests

class LoginTestsWithHTMLUnit extends LoginTests with HtmlUnit
class LoginTestsWithFirefox extends LoginTests with Firefox
class LoginTestsWithChrome extends LoginTests with Chrome
class LoginTestsWithSafari extends LoginTests with Safari
class LoginTestsWithIE extends LoginTests with InternetExplorer

Is there a way to do my method plus the try new WebDriver detection?

I am thinking that there is an implicit webDriver in trait Driver, that I can use from : 

trait MyBrowserTests { 
    this: WebBrowser with Driver =>
}

thanks, 
Jack Chi

On Thursday, June 16, 2011 11:05:01 AM UTC-7, None wrote:

Jack Chi

unread,
Oct 7, 2013, 9:05:01 PM10/7/13
to scalate...@googlegroups.com
HI Bill & Chee Seng, 

Also - do you know if I can put FireFoxProfile into a different language preference?

Thanks, Jack

Jack Chi

unread,
Oct 7, 2013, 9:50:22 PM10/7/13
to scalate...@googlegroups.com
trait JapaneseFirefox extends Firefox {
  firefoxProfile.setPreference("intl.accept_languages", "ja,en-us,en" );
  override val webDriver = new FirefoxDriver(firefoxProfile)
}

But I cannot get just one instance of the FirefoxDriver to show up. 

Chee Seng Chua

unread,
Oct 8, 2013, 1:19:08 AM10/8/13
to scalate...@googlegroups.com
Hi Jack, 

Unfortunately as of now I can't see good way of doing it.  Though I think it is still possible if you define your custom WebBrowser and use withFixture to auto-cancel tests when the browser is not avaiable, like the following: 

import org.openqa.selenium.WebDriver
import org.openqa.selenium.chrome.ChromeDriver
import org.scalatest._
import selenium._

trait AutoCancelChrome extends WebBrowser with Driver with Suite {
  
  val (driverSupported, chromeDriver: Option[WebDriver]) = 
     try {
       (true, Some(new ChromeDriver))
     }
     catch {
        case e: Throwable => (false, None)
     }
  implicit val webDriver = chromeDriver getOrElse (null)

  override protected def withFixture(test: NoArgTest): Outcome = {
    if (driverSupported)
      test()
    else
      cancel
  }
}

Hope it helps.

Cheers, 
Chee Seng

--

Jack Chi

unread,
Oct 9, 2013, 4:32:39 PM10/9/13
to scalate...@googlegroups.com
This pattern is helpful, I'm able to generalize an AutoCloseBrowser, now in this expression:

 implicit val webDriver = chromeDriver getOrElse (null)

If I am using BeforeAndAfterAll to do the tear down like such:

override protected def afterAll() {
      server.stop
       close
}

I'd get a NullPointerException because there is no implicit WebDriver in place

Thanks, Jack 

Jack Chi

unread,
Oct 9, 2013, 4:38:26 PM10/9/13
to scalate...@googlegroups.com
This implicit val outta do it

implicit val driverSupported : Boolean

  override protected def afterAll() {
    if (driverSupported)   {
      println("supported")
      close()
    }
    else
      println("not supported")
    super.afterAll()
Reply all
Reply to author
Forward
0 new messages