JVM Cucumber Integration with Arquillian

[Logan McGrath]

I'm working on integrating Cucumber-JVM with the Arquillian test framework and I've found a couple significant differences:

• Arquillian requires each test executed to be in the structure of test class > test instance > test method. The API and SPI depend on this structure.
  
• Cucumber-JVM organizes tests roughly by feature > scenario > step and reflects this with its JUnit integration.
  

While working with JBehave, I found there was really no granular test structure. You put a story or set of stories in a singular test case, which means all scenarios and steps are effectively part of the same test. Running JBehave tests with JUnit basically requires packaging up the dependencies for a particular story or set of stories into your test case and executing them all as a single test method. If you look at this particular arquillian-testrunner-jbehave project you'll see it repeats this structure for Arquilllian tests.

My question is, given the very different structure of Cucumber-JVM tests and Arquillian tests, does it make sense to package up the dependencies of a single feature into a dedicated test fixture? Basically, I would create a basis for packaging a feature as an Arquillian test and running it similarly to the main method of the cucumber.cli.Main class.

Any thoughts or pointers are greatly appreciated.

[Rex Hoffman]

Ran in to the same problem when working on testNG integration.   Cucumber jvm is an object based testing tool, and reports are based on objects as well...  TestNG internals are heavily coupled to the class based model for testing and reporting.   It's reporting is particularly problematic.

I've pretty much dumped testNG as a result.  Used TestNG for years due to feature set and extended it a bit... but once I saw the inherent coupling of the system, I realized the damage it does to the way we write tests to think of them -- as class based rather than object based.  TestNG's reporting is starting to fall apart as a result if you use some of there extended features.... ObjectFactory in particular....

Check out:

http://junit.sourceforge.net/doc/cookstour/cookstour.htm

To understand why junit integration was sensible and (simple/easy).

That said we need a way to run/report on scenarios as a thread-safe unit, which I'm starting on now (after doing some catch up with the community).

Hope this helps.  

Rex

Almost all my work is based around the Main class right now (including maven test executions).  Decomposing cucumber-jvm in to it's constituent parts to do interesting things (tm) is pretty difficult right now, especially multithreaded stuff.

Any thoughts or pointers are greatly appreciated.

Don't know Arquillian well enough to give you more than that.... hope it helps. 

 

[Logan McGrath]

Is that interesting things (tm) as in "trademark"? lulz

Any thoughts or pointers are greatly appreciated.

Don't know Arquillian well enough to give you more than that.... hope it helps. 

 

Arquillian is very much stuck in the class-based model... The reading does help a lot though in understanding the reasons for JUnit's architecture and why it's a perfect fit for Cucumber-JVM. I'm thinking I'll wrap or reimplement the Cucumber test runner and stick that in a super class. It's an ugly solution, especially considering JUnit's intended design, but it'll work.

Otherwise, the remainder of the cucumber-java module should integrate relatively easily with Arquillian: just use the active Arquillian test enrichers on all the objects created on an ObjectFactory implementation.

Here's what I'm thinking:
* A super class named "ArquillianCucumber" which is an abstract test fixture
** It should have single test method named "run" which creates and runs a single feature
* A reimplementation of the ResourceIterator which is capable of reading resources from archives deployed in container (MultiLoader fails all over itself on this)
* Running only feature files who's names match the name of the test fixture (same package, name in underscores or something)
* The subclass is responsible for packaging the test deployment with all the resources necessary to execute the associated feature.

It's very limiting compared to what Cucumber is capable of doing in JUnit alone, but it at least gets you to a spot where you can execute the test in a container environment.

The other task is to execute Cucumber scenarios as a client against a test deployment. Ideally, this should deploy the test to the container once then execute all features against it, but the architecture differences are likely going to keep it in that one-feature-per-test-case box.

[Rex Hoffman]

On Wednesday, July 11, 2012 4:44:31 PM UTC-7, Logan McGrath wrote:

On Thursday, July 5, 2012 7:05:14 PM UTC-5, Rex Hoffman wrote:

Ran in to the same problem when working on testNG integration.   Cucumber jvm is an object based testing tool, and reports are based on objects as well...  TestNG internals are heavily coupled to the class based model for testing and reporting.   It's reporting is particularly problematic.

I've pretty much dumped testNG as a result.  Used TestNG for years due to feature set and extended it a bit... but once I saw the inherent coupling of the system, I realized the damage it does to the way we write tests to think of them -- as class based rather than object based.  TestNG's reporting is starting to fall apart as a result if you use some of there extended features.... ObjectFactory in particular....

Check out:

http://junit.sourceforge.net/doc/cookstour/cookstour.htm

To understand why junit integration was sensible and (simple/easy).

That said we need a way to run/report on scenarios as a thread-safe unit, which I'm starting on now (after doing some catch up with the community).

Hope this helps.  

Rex

On Tuesday, July 3, 2012 1:55:34 PM UTC-7, Logan McGrath wrote:

I'm working on integrating Cucumber-JVM with the Arquillian test framework and I've found a couple significant differences:

• Arquillian requires each test executed to be in the structure of test class > test instance > test method. The API and SPI depend on this structure.
  
• Cucumber-JVM organizes tests roughly by feature > scenario > step and reflects this with its JUnit integration.
  

While working with JBehave, I found there was really no granular test structure. You put a story or set of stories in a singular test case, which means all scenarios and steps are effectively part of the same test. Running JBehave tests with JUnit basically requires packaging up the dependencies for a particular story or set of stories into your test case and executing them all as a single test method. If you look at this particular arquillian-testrunner-jbehave project you'll see it repeats this structure for Arquilllian tests.

My question is, given the very different structure of Cucumber-JVM tests and Arquillian tests, does it make sense to package up the dependencies of a single feature into a dedicated test fixture? Basically, I would create a basis for packaging a feature as an Arquillian test and running it similarly to the main method of the cucumber.cli.Main class.

Almost all my work is based around the Main class right now (including maven test executions).  Decomposing cucumber-jvm in to it's constituent parts to do interesting things (tm) is pretty difficult right now, especially multithreaded stuff.

Is that interesting things (tm) as in "trademark"? lulz

Gotta have fun with this stuff, I've made this my pet project at work, and am using it as a lever to teach 40 or so java devs functional programming concepts, design by interface, AOP, monads, etc  -- by leaning on spring (beans) as an object factory and a declarative programming language for application state, but any DI would have done for this. :)

 

Shockingly, It seems to be working 

Any thoughts or pointers are greatly appreciated.

Don't know Arquillian well enough to give you more than that.... hope it helps. 

 

Arquillian is very much stuck in the class-based model... The reading does help a lot though in understanding the reasons for JUnit's architecture and why it's a perfect fit for Cucumber-JVM. I'm thinking I'll wrap or reimplement the Cucumber test runner and stick that in a super class. It's an ugly solution, especially considering JUnit's intended design, but it'll work.

 

True.  It'll do the job... but I think you now have a responsibility to inform your users, fellow coders working on Arquillian, etc,  of the limitations of the system.

We will get to the point at my current employment where we are doing very interesting thing with multiple scoped spring contexts sharing resources, gherkin files, etc, but running -- and being cleanly reported on, in a single cucumber run.  Think test scope (thread local -- specific to a test), configuration scope ( which app url it's hitting, desktop/mobile version, brewer versions, etc), and base singleton scope ( web driver factories, other thread safe service that is non-configurable).

This is possible because I plan on using the context as a factory of objects, and those objects as tests.

[Note: I'm not advocating running all these possible scenarios on every check-in, just ensuring by devs/business owners can on their desktops, if I have the computational power, of course I'd love to run them, but I won't let that interfere with developer productivity  ]

Object v Class based is an incredibly powerful concept, and one I didn't realize I had sacrificed for feature set with TestNG -- which worked fine, until it I tried to do more with it and was limited by it's internal model.  Please help your users at least be conscious of the trade-offs they are implicitly making.  I know it's not great for the business model, but I think we owe the open source community and out users that much, and if we're honest with our design deficiencies we are more likely to correct them.

My 2 cents.  Only following up because you seemed to generally appreciate what I pointed out in my last reply.