Technical Difficulties with Rerun.txt tests
Jason Smiley
I am having some issues using the rerun.txt as input to my maven tests.
cucumberTest.java:
@RunWith(Cucumber.class)
@Cucumber.Options(tags={"@test"}, format = {"pretty", "rerun:rerun.txt","json:target/json-report.json", "junit:target/junit-report.xml", "html:target/html-report"})
public class CucumberTest {
}
Scenarios in feature testRerun.feature:
@test
Scenario Failing test
Given I fail this test
@test
Scenario Passing test
Given I pass this test
Commands run:
1) mvm clean test #[Runs tests with 1 pass and one failure]
2) mvm clean -Dcucumber.options”@rerun.txt” test #[Gives me the error below.]
Error:
java.lang.IllegalArgumentException: Not a file or directory: /Users/jason/Desktop/github/test-scripts/testRerun.feature
It seems that the feature file location is incorrect (should be /Users/jason/Desktop/github/test-scripts/features/src/test/features/testRerun.feature.
How do I fix this? I feel like there is a value I am not setting that needs to be set.
Björn Rasmusson
Assuming that the rerun.txt contains "testRerun.feature:<line>", what should happen is
- that testRerun.feature is tried to be loaded relative to the current directory (it seems to be /Users/jason/Desktop/github/test-scripts in this case),
- when testRerun.feature is not found there, it is tired to loaded from the classpath,
- if also that fails the error from the first try "Not a file or directory /Users/jason/Desktop/github/test-scripts/testRerun.feature" is reported.
The error message above make sense if the directory /Users/jason/Desktop/github/test-scripts/features/src/test/features/ was on the classpath during the first run (when the rerun.txt was created), but not during the second run (when the rerun.txt was used).
Best Regards
Björn
Jason Smiley
I am not explicitly mentioning class path in either case so I don't understand what I need to do. Its the same machine too, so, I guess I am not understanding why its set for one run but not the other.
Björn Rasmusson
@ Björn, How do I fix this?
I am not explicitly mentioning class path in either case so I don't understand what I need to do. Its the same machine too, so, I guess I am not understanding why its set for one run but not the other.Is "file input" runner different than the "tag input" runner or something?
I recommend that you clone the cucumber/cucumber-java-skeleton project and try there.
When executing mvn test a file target/rerun.txt with the content "skeleton/belly.feature:3" is created.
Then executing mvn -Dcucumber.options="@target/rerun.txt" test that scenario is executed again.
(that's what happens for me)
It is not much of a test, the only scenario fails, and then the same scenario is executed again using @target/rerun.txt,
But maybe it's best to get that to work, before trying something else.
Best Regard
Björn
Jason Smiley
1) When I ran mvn test I get the expected results you described.
Björn Rasmusson
I am able to cause this same error in cucumber-java-skeleton.The bug(?) is related to mvn clean test.My Steps:1) When I ran mvn test I get the expected results you described.2) When I ran mvn -Dcucumber.options="@target/rerun.txt" test I got the results you described.3) When I run mvn clean test, I get the expected results above.4) When I run mvn clean -Dcucumber.options="@target/rerun.txt" test, I got my initial error again.
(so the complaint will be on that .../cucumber-java-skeleton/target/rerun.txt does not exist).
To be able to use clean, first move the target/rerun.txt to rerun.txt and run
mvn clean -Dcucumber.options="@rerun.txt" test
This works for me.
Best Regards
Björn
Jason Smiley
cucumberTest.java:
@RunWith(Cucumber.class)
@Cucumber.Options(tags={"@test"}, format = {"pretty", "rerun:rerun.txt","json:target/json-report.json", "junit:target/junit-report.xml", "html:target/html-report"})
public class CucumberTest {}
New Code:
cucumberTest.java:
@RunWith(Cucumber.class)
@Cucumber.Options(format = {"pretty", "rerun:rerun.txt","json:target/json-report.json", "junit:target/junit-report.xml", "html:target/html-report"})
public class CucumberTest {}
Not sure if this was intended, but, I can't have tags in my cucumber.options Annotation when using the rerun.txt. It makes sense that you couldn't have tags and a rerun.txt input though since they are conflicting.
I would ask that we MABYE make it so file input for tests overwrite any tag settings, but, I can see my issue as more of a user issue than anything else. If you say no I will accept this lol.
Björn Rasmusson
That wasn't 100% accurate but I was able to figure things out based on what you told me.Original Code:cucumberTest.java:
@RunWith(Cucumber.class)
@Cucumber.Options(tags={"@test"}, format = {"pretty", "rerun:rerun.txt","json:target/json-report.json", "junit:target/junit-report.xml", "html:target/html-report"})
public class CucumberTest {}
New Code:
cucumberTest.java:
@RunWith(Cucumber.class)
@Cucumber.Options(format = {"pretty", "rerun:rerun.txt","json:target/json-report.json", "junit:target/junit-report.xml", "html:target/html-report"})
public class CucumberTest {}
Not sure if this was intended, but, I can't have tags in my cucumber.options Annotation when using the rerun.txt. It makes sense that you couldn't have tags and a rerun.txt input though since they are conflicting.
I would ask that we MABYE make it so file input for tests overwrite any tag settings, but, I can see my issue as more of a user issue than anything else. If you say no I will accept this lol.
In this case this error message is hidden, because from the rerun-file the feature files are tried to be loaded for relative from the current directory and then from the classpath, and if both fails, the first error message is presented (which made sense at the time if the input is "path/file.feature" an error message saying the "classpath:path/file.feature" is missing would be confusing). In this case though it hides the fact that the load from the classpath does not fail due to a missing file, but to that the fact that both tag-filters (from the @CucumberOptions) and line-filters (from the rerun-file) are used, which is even more confusing.
To cancel out the tag-filters from the @CucumberOptions when using reading from the rerun-file makes sense, since it otherwise the "inconsistent filters" error always occurs if using tags in the @CucumberOptions, and since it even though it possible to override the tag-filters from the @CucumberOptions with -Dcucmber.options, it is not possible to cancel them (the trick using -Dcucumber.options="--tags ~@NoTag" to "cancel the tag-filters" and run all features does not work because there still is a tag filter even though it does not filter out anything).
Best Regards
Björn
Jason Smiley
Björn Rasmusson
Just out of curiosity, I see the "--format" option adds to the format @Cucumber.Options instead of over writting.If I wanted to get two reports, one for the 1st run and one for the rerun.txt run, do I have to clear my cucumber.otpions annotation? Or is there another way to do this?
So you would have to specify the reports in -Dcucumber.options, for instance
First run: mvn -Dcucumber.options="--format json:file1.json" test
Second run: mvn -Dcucumber.options="--format json:file2.json @rerun.txt" test
Then you would have file1.json with the results from the first run, and file2.json with the result from the second run.
On Monday, June 30, 2014 2:24:24 PM UTC-4, Jason Smiley wrote:I am having some issues using the rerun.txt as input to my maven tests.
Error:
java.lang.IllegalArgumentException: Not a file or directory: /Users/jason/Desktop/github/test-scripts/testRerun.feature
It seems that the feature file location is incorrect (should be /Users/jason/Desktop/github/test-scripts/features/src/test/features/testRerun.feature.
How do I fix this? I feel like there is a value I am not setting that needs to be set.
Samuel S
aslak hellesoy
Unfortunately the inbuilt rerun formatter has 2 limitations: 1. no way to instantly re-rerun a fail (has to wait till end). 2. The test results outputted do not accurately take into account the results from the first test run--they either show if the rerun passed or failed.
On Wednesday, July 2, 2014 at 6:48:06 AM UTC-7, Björn Rasmusson wrote:Jason Smiley wrote:Just out of curiosity, I see the "--format" option adds to the format @Cucumber.Options instead of over writting.If I wanted to get two reports, one for the 1st run and one for the rerun.txt run, do I have to clear my cucumber.otpions annotation? Or is there another way to do this?The formatters specified in @CucumberOptions would overwrite the files create during the first run, when the rerun.txt run is executed.
So you would have to specify the reports in -Dcucumber.options, for instance
First run: mvn -Dcucumber.options="--format json:file1.json" test
Second run: mvn -Dcucumber.options="--format json:file2.json @rerun.txt" test
Then you would have file1.json with the results from the first run, and file2.json with the result from the second run.
On Monday, June 30, 2014 2:24:24 PM UTC-4, Jason Smiley wrote:I am having some issues using the rerun.txt as input to my maven tests.
<snip>Error:
java.lang.IllegalArgumentException: Not a file or directory: /Users/jason/Desktop/github/test-scripts/testRerun.feature
It seems that the feature file location is incorrect (should be /Users/jason/Desktop/github/test-scripts/features/src/test/features/testRerun.feature.
How do I fix this? I feel like there is a value I am not setting that needs to be set.
--
Posting rules: http://cukes.info/posting-rules.html
---
You received this message because you are subscribed to the Google Groups "Cukes" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cukes+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
aslak hellesoy
On Sat, May 9, 2015 at 7:21 AM, Samuel S <smers...@gmail.com> wrote:Unfortunately the inbuilt rerun formatter has 2 limitations: 1. no way to instantly re-rerun a fail (has to wait till end). 2. The test results outputted do not accurately take into account the results from the first test run--they either show if the rerun passed or failed.Limitations are good. My bike is limited - I can't transport suitcases with it.Everybody who wants tests to re-run automatically have problem: Their tests or system under test are non-deterministic.Running a non-deterministic test over and over again doesn't prove anything, and it erodes confidence in your tests. How many times do you have to toss a coin before you get tails?What prevents you from making the tests/system deterministic?Also see this discussion: https://groups.google.com/d/msg/cukes/NCtmpE8brDc/rkPg-EhLxHQJ
Ajay Jaiswal
I am getting following error with Java 1.8 for project available in https://github.com/cucumber/cucumber-java-skeleton