[cucumber] ideas on how to test multiple cucumber suites across different envs

[jh]

I'm sure others are struggling with this, and I can't find much discussion for strategies.

Basically we're looking to build up a meta cucumber suite:

1. Top level meta cucumber instance for delegating to other cucumber instances
2. Cucumber instances will be on different servers and can be any language (ruby, java, scala, etc)
3. Teams build up their cucumber suite independently 
1. Possibly provide top level features for top level meta tests ?
4. Meta tests can/will cut across instances to test lots of services
5. Test results are aggregated at the top level (it is a simple cucumber run)

I'm sure we could write gherkin that evaluates to http calls at other nodes, and possibly some kind of rest wrapper around a cucumber instance for running features and collecting results.

Why not just build up a massive, single instance, in a single language cucumber suite?  exactly.

Anyone doing this sort of thing?

Thanks

-John

[Paolo Ambrosio]

On 5 Nov 2015 02:35, "jh" <hwh...@gmail.com> wrote:
>
> I'm sure others are struggling with this, and I can't find much discussion for strategies.
>
> Basically we're looking to build up a meta cucumber suite:
> Top level meta cucumber instance for delegating to other cucumber instances
> Cucumber instances will be on different servers and can be any language (ruby, java, scala, etc)

Can't say I've heard anything like this before, but it looks like a solution rather than a problem statement, and I can't understand what is the issue you are trying to solve. Can you be more specific?

> Teams build up their cucumber suite independently 

Yep

> Possibly provide top level features for top level meta tests ?
> Meta tests can/will cut across instances to test lots of services

What are "top level features" and "top level meta tests"?

Do you mean that you have the same feature implemented by multiple products? The Cucumber TCK is an example of this.

> Test results are aggregated at the top level (it is a simple cucumber run)
> I'm sure we could write gherkin that evaluates to http calls at other nodes, and possibly some kind of rest wrapper around a cucumber instance for running features and collecting results.

It looks like an over-engineered solution and it will prevent developers from running their test suite in local as part of the build...

> Why not just build up a massive, single instance, in a single language cucumber suite?  exactly.

...that brings up the question: what is wrong with CI servers that run those tests?

Paolo

[Matt Wynne]

This reads like a solution to me, but I'm unclear what exactly the problem is. Can you describe more about the problem you're trying to solve?

[jh]

Here's the problem(s):

- teams have developed cucumber tests for their domain (service) and they run in jenkins
- teams each have chosen diff language implementation for cucumber like ruby, java, and scala
- goal is to build cross-service tests but to leverage some of each team's existing gherkins

The problem is how to build CI/CD smoke tests that touch all the services (each service is complex with difficult test setup/scaffolds and each team can iterate/change at their own rate) and can utilize the knowledge/existing tests of the teams in aggregate. There's some pushing to require java-uber-alles for cucumber as the solution but that's lame for today.

There's also an element of release management / qa team(s) wanting to develop special tests in order to release with their own level of confidence.

Granted I've only just noticed the cucumber wire protocol stuff and it looks promising but not so much supported.

Each team can provide simple gherkin statements but they need to run on their own test implementations that are not the same language/env, ideally their own step definitions with all the configuration scaffold et al ..

[George Dinwiddie]

JH,

On 11/18/15 10:00 PM, jh wrote:
> Here's the problem(s):
>
> - teams have developed cucumber tests for their domain (service) and they run in jenkins
> - teams each have chosen diff language implementation for cucumber like ruby, java, and scala
> - goal is to build cross-service tests but to leverage some of each team's existing gherkins
>
> The problem is how to build CI/CD smoke tests that touch all the
> services (each service is complex with difficult test setup/scaffolds

This sounds like your underlying problem, rather than the mix of
languages. Difficult setup hints at design issues.

> and each team can iterate/change at their own rate) and can utilize
> the knowledge/existing tests of the teams in aggregate. There's some
> pushing to require java-uber-alles for cucumber as the solution but
> that's lame for today.
>
> There's also an element of release management / qa team(s) wanting to
> develop special tests in order to release with their own level of
> confidence.
>
> Granted I've only just noticed the cucumber wire protocol stuff and
> it looks promising but not so much supported.
>
> Each team can provide simple gherkin statements but they need to run
> on their own test implementations that are not the same language/env,
> ideally their own step definitions with all the configuration
> scaffold et al ..

If you want to run the same gherkin across multiple services, I'd think
you'd need glue code specific for doing so. And I'd think that this glue
code would be quite different from service to service.

The running of the multiple test suites seems quite easy with your CI
server. I wouldn't press Cucumber into service for that.

Or maybe I misunderstand the problem.

- George


--
----------------------------------------------------------------------
* George Dinwiddie * http://blog.gdinwiddie.com
Software Development http://www.idiacomputing.com
Consultant and Coach http://www.agilemaryland.org
----------------------------------------------------------------------

[jh]

Complexity is an issue, each team manages it away in their cukes daily.  By "setups" I'm basically talking about scaffold or gherkin backgrounds to prep a test scenario.

In an enterprise env, eg. with lots of various domains + service APIs across multi teams, you don't get simple setups and certainly not a single integration test suite. 

Instead you get test suites which each team writes to cover their project concerns, and yes those all run individually on jenkins as CI. Yes they do call out against other services too (which can be brittle btw). 

But sometimes you want to write cross-domain concerns at a higher level. 

So if you want to do higher level tests, you have to figure out how to orchestrate, delegate? .. and leverage the existing tests (+ setups) that the teams publish in their cuke suites which they use to get acceptance for release. 

I can imagine many cucumber nodes, each handles a specific service, and an orchestration cucumber node which runs features that evaluate to step_defs, or feature files, or tags that run on the specific cucumber node (and gathers back the results).

So it would look like this: 

Given I have a new account at service X  

And I have a some device registered at service Y

And the account tries to register at service Z using the some device

[George Dinwiddie]

JH,

Does the end user really care about your implementation into different
services? I suggest writing your cukes from an
implementation-independent point of view. That might help you get away
from the problem you're having. It might also get you away from having
users register separately for different services.