New feature: simpler factories

[Cédric Beust ♔]

Hi everyone,

I recently added a new feature that simplifies the @Factory functionality. As you probably know, a factory is usually a method that returns an array of objects, each of which expected to be an instance of a test class.

You can now save yourself the trouble of creating a method that will return an array of objects and instead, put the @Factory annotation directly on the constructor but with a data provider. Here is an example:

public class FactoryDataProviderSampleTest {

  @Factory(dataProvider = "dp")

  public FactoryDataProviderSampleTest(int n) {

    super(n);

  }

  @DataProvider

  static public Object[][] dp() {

    return new Object[][] {

      new Object[] { 41 },

      new Object[] { 42 },

    };

  }

}

In the code above, TestNG will create two instances of the class FactoryDataProviderSampleTest, one initialized with "41" and the other with "42".

Thanks to Stephen "Joda" Colebourne for convincing me that this was a worthy addition to TestNG!

Try it and let me know what you think.

--
Cédric

[Tomek Kaczanowski]

nice!

--
Regards / Pozdrawiam
Tomek Kaczanowski

2011/4/26 Cédric Beust ♔ <ced...@beust.com>:

> --
> You received this message because you are subscribed to the Google Groups
> "testng-users" group.
> To post to this group, send email to testng...@googlegroups.com.
> To unsubscribe from this group, send email to
> testng-users...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/testng-users?hl=en.
>

[Mark Derricutt]

Nice indeed.

I've been thinking also on a pattern of test I seem to be writing more and more often lately:

@DataProvider(name="something")

public Object[][[] provideSomething() {

...

... process a bunch of data - ie csv, jdbc, or other, finding all "failing data"

}

@Test

public void fail(SomeObject dataThatFails fail) {

  fail("Bad data found " + fail);

}

( the test sometimes has some actual tests in it tho )

I've thought of maybe doing this as a @Factory that returns instances of an anonymous inner class at times, but thought something like:

@Test

public void testSomeData(IProvidedTest auto) {

  for (Object each : something ) {

      auto.callback(new Object[][] { xxx. xxx };

  }

}

where IProvidedTest is some form of TestNG generated meta-test,   which kinda inverts things.  I've not really thought too much on a better way to write it tho.  

The other idea I had for tests with a few variations was:

@Test(data=[

  @Data("one", "two", three"),

  @Data("four", "five", "six")])

public void testit(String first, String second, String third) {

  

}

Ideas worth pondering?

Mark

--
"Great artists are extremely selfish and arrogant things" — Steven Wilson, Porcupine Tree

[Cédric Beust ♔]

Hi Mark,

I'm confused by all your ideas except one :-)

On Wed, Apr 27, 2011 at 1:04 AM, Mark Derricutt <ma...@talios.com> wrote:

Nice indeed.

I've been thinking also on a pattern of test I seem to be writing more and more often lately:

@DataProvider(name="something")

public Object[][[] provideSomething() {

...

... process a bunch of data - ie csv, jdbc, or other, finding all "failing data"

}

@Test

public void fail(SomeObject dataThatFails fail) {

  fail("Bad data found " + fail);

}

Invoking a test method that will always fail? Not sure what the point of that would be(?).

 

( the test sometimes has some actual tests in it tho )

I've thought of maybe doing this as a @Factory that returns instances of an anonymous inner class at times, but thought something like:

@Test

public void testSomeData(IProvidedTest auto) {

  for (Object each : something ) {

      auto.callback(new Object[][] { xxx. xxx };

  }

}

where IProvidedTest is some form of TestNG generated meta-test,   which kinda inverts things.  I've not really thought too much on a better way to write it tho.  

That seems intriguing but I can't think of a use case, can you?

 

The other idea I had for tests with a few variations was:

@Test(data=[

  @Data("one", "two", three"),

  @Data("four", "five", "six")])

public void testit(String first, String second, String third) {

  

}

This would be very limited since Java annotations can only contain constants...

-- 

Cédric

[Mark Derricutt]

Cedric,

I'm confused by all your ideas except one :-)

I've often heard that :)  It's also maybe a sign  my ideas are crazy, and warrant being shot down.  Either that or I've just not explained myself clearly enough - quite likely the former tho.

 

Invoking a test method that will always fail? Not sure what the point of that would be(?).

One scenario where I've done this is as part of our integration test suite. First the setup - the integration test starts an embedded ApacheDS LDAP server, along with an embedded Apache Felix OSGi container and our RESTful application.  This runs thru a series of fairly standard tests doing HTTP calls with content checking return types, LDAP validations etc. etc.

As part of our internal auditing we store in the database a record of all HTTP calls to the API, and I have a "test" which attempts to give us a form of resource coverage in the form of DataProvider that first queries the server for available REST resources, then queries the request audit log, and finds any resource/http-method combinations that were not called during the test run.

That DataProvider then feeds those "missing requests" to the test method which records a test failure for any resource that was not tested.

Another scenario I have for this is also at the tail end of our integration tests in which a data provider scans the generated webserver log over the course of the integration test, pulling out any stacktrace and passing that over to a test to record an "unhandled exception" in the server.

For these I could use the DataProvider/Test approach or a Factory.  The DataProvider/Test approach seems to work (mentally to me) better in that theres only ever a single test involved.

One of the scenario's I've thought of but haven't yet implemented is more a "system health test", where a Factory/DataProvider inspects the database for bad data and reports a test failure in each case ( eg.  Customer X has a country of Australia but has billing details for New Zealand Dollars ).

The other idea I had for tests with a few variations was:

@Test(data=[

  @Data("one", "two", three"),

  @Data("four", "five", "six")])

public void testit(String first, String second, String third) {

  

}

This would be very limited since Java annotations can only contain constants...

True - but I've done things similar to this in the past where I could only provide String values/constants.  In the case of Strings, parameters would be checked via reflection for having a valueOf(String) method ( String, Integer, Double, Float, etc. have these ).

The current flow has a static factory method returning the test object instances itself, and the more I think about it, I think a variation of the Factory setup could for all these scenarios.  Basically have a new IFactoryGenerator interface with a #yield(Object) method on it, if an @Factory method returns void, and takes IFactoryGenerator as an argument then do something like:

  https://skitch.com/talios/rhs9e/untitled-quick-sequence-diagram-editor

Maybe I'm just crazy :)

Mark

 

[Santo Pfingsten]

I've been trying to come up with a way to use this feature together with parameters(from the xml) and I've come up with this little hack:

    boolean single;
    @Parameters({"single"})
    public DemoTest(@Optional("true") boolean single) {
        this.single = single;
    }
    @Factory(dataProvider = "createInstances")
    public DemoTest(String name) {
        ....
    }
    @DataProvider
    public Object[][] createInstances() {
        if(single) {
            return new Object[][] { new Object[] { "single" } };
        }
        return new Object[][] {
            new Object[] { "first" },
            new Object[] { "second" },
        };
    }

The first constructor will only be called to create a temporary object to call the non-static DataProvider.
The second one will be the one actually used. It would be nice if there was a better way to do something like this, but this works good so far.

Hint: This way you can run only one instance using rightclick->test file and test all instances when using the xml file (when building for example).

---

View this message in context: Re: New feature: simpler factories
Sent from the testng-users mailing list archive at Nabble.com.

[Cédric Beust ♔]

Hi Santo,

That's a nice hack to get two different behaviors depending on whether you launch from a testng.xml or from the the class directly.

Clever idea to combine @Parameters and @Optional :-)

-- 

Cédric

[Scal]

Cool!

[Iakiv Kramarenko]

Hi Cédric,

I have a problem with this feature if i put a @Factory annotation on a constructor of TestClas that has its BaseClass.

Architecturally I have:

BaseTest

   protected WebDriver driver;

   //no constructor

  @BeforeSuite

       //here i setup a driver, etc.

Test extends BaseTest
   private String mode;

   @Factory(....)

   public Test(String mode) 

       //here I set mode from dataprovider passed to factory

   @BeforeClass

       //here I set up everything I need for all tests, creating pageobjects, etc...

   @BeforeMethod

      //here I get the page being a start-point for all tests

      // !!! AND HERE I GET A NulllPointerException

Seems like something wrong with this organisation... Because this is what testng do:

1 Create 1st instance of Test (according to first value from dataprovider passed to factory-constructor)

   // this instance will have the 'driver' field null

2 call @BeforeSuite in BaseTest

3 Create 2nd instance of Test (according to second value from dataprovider...)

   // this instance will have the 'driver' field initialized because initialization of the driver already was made on step 2

4. Call @BeforeClass in Test (for the 1st instance)

5. Call @BeforeMethod in Test (for the 1st instance)

   // and here I get NullPointerException because driver is null....

What did I mainly wrong? How can I adjust my architecture of tests to deal with @Factory (according to what i need: 1. having base setup somewhere 'outside' of a test class like in base test class, 2. having a test class setup 3. having per method setup 4. having several 'modes' for test class)

Big thanks forward,

Yakiv

Вівторок, 26 квітня 2011 р. 01:40:26 UTC+3 користувач Cedric написав:

[Iakiv Kramarenko]

Hi again...

Seems like I found my 'weak' place...

I have changed 

BaseTest

   protected WebDriver driver;

   //no constructor

  @BeforeSuite

       //here i setup a driver, etc.

to 

BaseTest

   protected WebDriver driver;

   //no constructor

  @BeforeClass

       //here i setup a driver, etc.

And now all work...

You would ask, why did I use @BeforaClass in BaseTest did not work somehow.... It was called, but @BeforeClass in Test was missed, and execution was passed to @BeforeMethod in Test leading to crash... After That I only added factory and constructor to the Test class, and now @BeforeClass works... Mystery... Or maybe I did something wrong, fixed it and don't remember what... 

In all other cases, is my architecture of tests ok?

I would appreciate any links on something similar... To learn more from others experience... 

I am also thinking about moving this 'factory constructor' to the BaseTest, because other test class also may need the same 'modes' to be run with... Can there be any pitfalls? Do you have any good Ideas how to do this? 

--
You received this message because you are subscribed to a topic in the Google Groups "testng-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/testng-users/nCGwcAmQSu8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to testng-users...@googlegroups.com.

To post to this group, send email to testng...@googlegroups.com.

Visit this group at http://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Jeff]

I got a little lost with your final explanation, so if I have any input it would be that WebDriver/Selenium will NOT work if using a single instance that is initialized in @BeforeSuite unless the following conditions are met:

• Your entire Suite is SINGLE_THREADED 
  
• You only have one test
• You go to great lengths to synchronize access to the WebDriver instance (this would nullify any concurrency)
• You are using multiple tabs within a single browser window (it's possible but painful and still prone to issues as tests fight for their tab to receive focus)

Additionally, @BeforeClass is also tricky with WebDriver and could cause problems as well with concurrency unless the following conditions are met:

• Your classes are set SINGLE_THREADED
  
• You only have one test per class instance
• You go to great lengths to synchronize access to the WebDriver instance (this would nullify any concurrency...same as single-threaded)
• You are using multiple tabs within a single browser window (again, it's possible but likely painful)

If you think about it, anything initialized in @BeforeSuite will be akin to a global SINGLETON.  If you have more than one test method running on more than one thread all trying to manipulate the same browser, you will have issues.

If you have more than one test method in a class and your web driver was initialized using @BeforeClass and you are running multiple threads, you will potentially have issues.

It's like having your kids all run to the computer and try to type and fight for control of the mouse at the same time....it won't work :-).

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to testng-users...@googlegroups.com.

To post to this group, send email to testng...@googlegroups.com.
Visit this group at http://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--

Jeff Vincent
See my LinkedIn profile at:
http://www.linkedin.com/in/rjeffreyvincent

[Iakiv Kramarenko]

Hi Jeff,

Thanks for the detailed answer... It's really good to know about such details.

Yeah, Now i see that my last post is not quite good understandable. It was a deep night, and I was too tired. What resulted in missing some words in sentences:) 

I wanted to say that somehow the day before, the following configuration:

BaseTest

   protected WebDriver driver;

   //no constructor

   @BeforeClass

       //here i setup a driver, etc.

Test extends BaseTest

   @BeforeClass

       //here I set up everything I need for all tests, creating pageobjects, etc...

   @BeforeMethod

      //here I get the page being a start-point for all tests

      // !!! AND HERE I GET A NulllPointerException

Did not work (the execution order was 1. BaseTest->@BeforeClass method... 2. Test->@BeforeMethod method).  I had no any multiple threads, I guess... At least I did not configure them explicitly... 

Nevertheless, with factory, now all works... 

The question I have now - can I move this @Factory constructor from Test class to BaseTest ? If not, or if there is other way - how can I organize my classes so I can use one @Factory (constructor) for more than one test class?

Big thanks forward!

Iakiv

[Iakiv Kramarenko]

Hi Jeff, 

Thanks a lot:) I've found your answer to my last question in a separate discussion:) - "[testng-users] @Factory execution order"

By the way, if you have any blog, or one place containing your knowledge shared, I would appreciate to get a link to it;)

Yakiv