Execution order is wrong when run on Linux

[gu...@gopro.net]

Hey everyone!

I am using Cucumber with Java to create some test suites. One of the test suites is testing a Rest API we have, mostly CRUD operations, so scenarios and features are highly coupled (I know this is not ideal, but in this situation it seemed to be justified).
The structure of feature files can be seen below, the glue and implementation files follow suite.

What happens is this (4 scenarios):
1. Running with maven test on Windows: Execution order is normal.
2. Running as a jar file on Windows: Execution order is normal.
3. Running with maven test on Linux: Execution order is NOT normal. Delete tests of the CRUD operations, which should be last, are done first. Meaning that it tries to delete files that are not there, then starts all the other CRUD operations and then leaves the test data on the server.
4. Running as a jar file on Linux: Execution order is normal.

I get no errors or anything indicating why this is happening and searching for a solution online has been a dead end. Anyone seen this kind of behavior before - and maybe knows how to fix this?

Best regards,
Gulli

[Andrew Premdas]

You have made a mistake coupling your scenarios together. There is nothing special about CRUD operations for a REST api that requires/demands coupled scenarios. You have a couple of options

1. Uncouple your scenarios

2. Use a different test tool

3. Find a way to force Cucumber to run scenarios in the order you want on all platforms

I suspect you will want to do 3. My advice is to do 1. I'm afraid I don't know how to do 3 outside of running one scenario at a time

Note: Cucumber is not misbehaving on Linux. The expected execution order for Cucumber is to run scenarios randomly. Clearly Cucumber java doesn't do a very good job of doing this.

Hopefully someone else will have  some more info to help with maven.

All best

Andrew

 

--
Posting rules: http://cukes.info/posting-rules.html
---
You received this message because you are subscribed to the Google Groups "Cukes" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cukes+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

------------------------

Andrew Premdas

blog.andrew.premdas.org

[gu...@gopro.net]

Hi!

I wasn't thorough enough in my opening post, it's not the scenarios that are not executing in the right order (I didn't even know they were random), but it's the features that are running out of order, only when run on Linux and only with Maven. As can be seen in the picture, all scenarios run in the order they are supposed to (as I've learned it is alphabetically) - except when run with maven on Linux. Then the last feature (in package testsuite.test_9.services) seems to run before all the others (which then execute in the right order).

I'm already on 3), I'm going to test to use one runner and use @Cucumber.Options(features= and see if that works.

Thanks!

To unsubscribe from this group and stop receiving emails from it, send an email to cukes+un...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Eric Kessler]

On Monday, January 9, 2017 at 6:36:45 AM UTC-5, apremdas wrote:

On 9 January 2017 at 09:51, <gu...@gopro.net> wrote:

Hey everyone!

I am using Cucumber with Java to create some test suites. One of the test suites is testing a Rest API we have, mostly CRUD operations, so scenarios and features are highly coupled (I know this is not ideal, but in this situation it seemed to be justified).
The structure of feature files can be seen below, the glue and implementation files follow suite.

What happens is this (4 scenarios):
1. Running with maven test on Windows: Execution order is normal.
2. Running as a jar file on Windows: Execution order is normal.
3. Running with maven test on Linux: Execution order is NOT normal. Delete tests of the CRUD operations, which should be last, are done first. Meaning that it tries to delete files that are not there, then starts all the other CRUD operations and then leaves the test data on the server.
4. Running as a jar file on Linux: Execution order is normal.

I get no errors or anything indicating why this is happening and searching for a solution online has been a dead end. Anyone seen this kind of behavior before - and maybe knows how to fix this?

The order in which directory structures are traversed (and thus the order in which feature files are found by programs doing any kind of searching) can differ across operating systems. On several occasions, I have had to explicitly sort arrays of files before using them because the Windows and Linux results will be the opposite of each other.

 

Best regards,
Gulli

You have made a mistake coupling your scenarios together. There is nothing special about CRUD operations for a REST api that requires/demands coupled scenarios. You have a couple of options

1. Uncouple your scenarios

Do this.

 

2. Use a different test tool

Nah, we can deal with the problem.

 

3. Find a way to force Cucumber to run scenarios in the order you want on all platforms

Create a file that explicitly orders your features and then tell Cucumber to run the tests in that file.

File:

--my_ordered_tests.txt--

b.feature

a.feature

c.feature

----

Command:

cucumber @my_ordered_tests.txt

This is how running tests using the rerun formatter works but in this case we are creating the list ourselves.

 

I suspect you will want to do 3. My advice is to do 1. I'm afraid I don't know how to do 3 outside of running one scenario at a time

Note: Cucumber is not misbehaving on Linux. The expected execution order for Cucumber is to run scenarios randomly. Clearly Cucumber java doesn't do a very good job of doing this.

Hopefully someone else will have  some more info to help with maven.

All best

Andrew

 

--
Posting rules: http://cukes.info/posting-rules.html
---
You received this message because you are subscribed to the Google Groups "Cukes" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cukes+un...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Andrew Premdas]

On 9 January 2017 at 13:34, <gu...@gopro.net> wrote:

Hi!

I wasn't thorough enough in my opening post, it's not the scenarios that are not executing in the right order (I didn't even know they were random), but it's the features that are running out of order, only when run on Linux and only with Maven. As can be seen in the picture, all scenarios run in the order they are supposed to (as I've learned it is alphabetically) - except when run with maven on Linux. Then the last feature (in package testsuite.test_9.services) seems to run before all the others (which then execute in the right order).

Then you've made a different but similar mistake, but the same solutions apply. Its a really bad idea to couple features and a really simple thing to uncouple them. The only cost is some extra execution time for the features. The benefits will only be really apparent when you have bugs and broken tests and start getting cascading errors as you try and fix things, but then it may be a bit late to fix things.

Anyhow good luck with whatever you do.

To unsubscribe from this group and stop receiving emails from it, send an email to cukes+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Paolo Ambrosio]

Cucumber-JVM for filesystem resources uses the File.listfiles method that does not guarantee any specific order. I didn't look at the native method implementation in the JVM itself, but I suspect it uses the system-specific C library calls: readdir on Linux and FindNextFile on Windows. None of those guarantees a specific order for the entries [a] [b].

Relying on feature execution order for Cucumber though is just wrong (won't sugar-coat it like Andrew did). I'm sure you can understand why you don't do that for unit testing, and the same considerations are true for Cucumber features and scenarios. If not for anything else, it forces you to run the entire suite (or at least part of it) whenever you want to run a scenario, so probably you are using Cucumber for something that is not BDD.

Cheers,

Paolo

Notes:

[a] https://utcc.utoronto.ca/~cks/space/blog/unix/ReaddirOrder

[b] https://msdn.microsoft.com/en-us/library/windows/desktop/aa364428(v=vs.85).aspx

To unsubscribe from this group and stop receiving emails from it, send an email to cukes+unsubscribe@googlegroups.com.