Serenity report incomplete, missing expected title and tabs

[Tony Tubbs]

It seems we've done something that is preventing the Serenity reports for generating correctly, but we are not getting any errors or warnings or other information that would help indicate what we've done wrong.

• serenity.project.name - is set in serenity.properties file, but not appearing on home page.  Only shows up when we drill down into one of the test result pages.
  
• serenity.requirement.types - is set for three levels, I'm pretty sure these three levels were appearing as tabs on the home page at one time, but all we get now is "Overall Test Results" and "Requirements".  Again, if we drill down into a test result page, three tabs appear for "Epic", "Theme" and "Feature", but when we click on them we get a "page not found" error reported in the browser.
  
• There are groups for Epics, Features and Themes on the home page in the list of "Related Tags" beside the pie chart though - so there is some indication parts of the report are finding them.

Following are snippets of the logs if that helps..

• Loads the properties file
  
• Is it a problem that project.build.directory and project.reporting.OutputDirectory are null and OutpoutDirectory defaults?  (this is different in the report plugin, see below)

-------------------------------------------------------

 T E S T S

-------------------------------------------------------

Running com.MyShop.MyGroup.MyAppFeatureTest

INFO : net.thucydides.core.util.PropertiesFileLocalPreferences - serenity.requirements.dir=src/test/resources/com/MyShop/MyGroup/MyApp

INFO : net.thucydides.core.util.PropertiesFileLocalPreferences - serenity.logging=VERBOSE

INFO : net.thucydides.core.util.PropertiesFileLocalPreferences - webdriver.chrome.driver=target/drivers/chromedriver.exe

INFO : net.thucydides.core.util.PropertiesFileLocalPreferences - webdriver.base.url=http://localhost:8080/

INFO : net.thucydides.core.util.PropertiesFileLocalPreferences - chrome.switches=--disable-extensions

INFO : net.thucydides.core.util.PropertiesFileLocalPreferences - serenity.project.name=MyApp Automated Tests

INFO : net.thucydides.core.util.PropertiesFileLocalPreferences - serenity.requirement.types=theme, epic, feature

INFO : net.thucydides.core.webdriver.SystemPropertiesConfiguration - project.build.directory : null

INFO : net.thucydides.core.webdriver.SystemPropertiesConfiguration - project.reporting.OutputDirectory : null

INFO : net.thucydides.core.webdriver.SystemPropertiesConfiguration - OutputDirectory : C:\dev\MyGroup\MyGroup_MyApp_trunk\MyGroupMyAppIT\target\site\serenity

Only running one test, but when that starts "Loading requirements..." happens

• The project.build.directory, project.reporting.OutputDirectory and OutputDirectory are still the same at this step.
  
• We get many of these "...Loading requirements from {0}" messages - not terribly helpful, but indicating it's finding something.
  

|_   _| ____/ ___|_   _| / ___|_   _|/ \  |  _ \_   _| ____|  _ \ 

  | | |  _| \___ \ | |   \___ \ | | / _ \ | |_) || | |  _| | | | |

  | | | |___ ___) || |    ___) || |/ ___ \|  _ < | | | |___| |_| |

  |_| |_____|____/ |_|   |____/ |_/_/   \_\_| \_\|_| |_____|____/ 

TEST STARTED: Inactivity timeout logs out user

-------------------------------------------------------------------

INFO : net.thucydides.core.webdriver.SystemPropertiesConfiguration - project.build.directory : null

INFO : net.thucydides.core.webdriver.SystemPropertiesConfiguration - project.reporting.OutputDirectory : null

INFO : net.thucydides.core.webdriver.SystemPropertiesConfiguration - OutputDirectory : C:\dev\MyGroup\MyGroup_MyApp_trunk\MyGroupMyAppIT\target\site\serenity

DEBUG: net.thucydides.core.requirements.FileSystemRequirementsTagProvider - Loading requirements from {0}

DEBUG: net.thucydides.core.requirements.FileSystemRequirementsTagProvider - Loading requirements from {0}

DEBUG: net.thucydides.core.requirements.FileSystemRequirementsTagProvider - Loading requirements from {0}

After the test passes, reports are generated

• We see many Requirement ancestors for: messages in this step - they all look like they correctly identify the correct Theme, Epic and Feature
  
• We don't see any messages about any report being unable to be generated here

  _     \ \  |_   _| ____/ ___|_   _| |  _ \ / \  / ___|/ ___|| ____|  _ \ 

 (_)_____| |   | | |  _| \___ \ | |   | |_) / _ \ \___ \\___ \|  _| | | | |

  _|_____| |   | | | |___ ___) || |   |  __/ ___ \ ___) |___) | |___| |_| |

 (_)     | |   |_| |_____|____/ |_|   |_| /_/   \_\____/|____/|_____|____/ 

        /_/                                                                

TEST PASSED: Inactivity timeout logs out user

----------------------------------------------------------------------------

DEBUG: net.thucydides.core.reports.ReportService - Reporting formats: [JSON, XML, HTML]

DEBUG: net.thucydides.core.reports.junit.JUnitXMLOutcomeReporter - GENERATING JUNIT REPORTS

DEBUG: net.thucydides.core.reports.junit.JUnitXMLOutcomeReporter - GENERATING JUNIT REPORT SERENITY-JUNIT-6563d98750da0f614e1b25a72a3e0469cd69a1042b80a8f884c5f03ffcbe140f.xml using temporary file target\site\serenity\SERENITY-JUNIT-6563d98750da0f614e1b25a72a3e0469cd69a1042b80a8f884c5f03ffcbe140f.xml061e26ae-db56-4dc6-b4a9-5330adbee2ed

DEBUG: net.thucydides.core.reports.xml.XMLTestOutcomeReporter - Generating XML report for Inactivity timeout logs out user to file C:\dev\MyGroup\MyGroup_MyApp_trunk\MyGroupMyAppIT\target\site\serenity\6563d98750da0f614e1b25a72a3e0469cd69a1042b80a8f884c5f03ffcbe140f.xml (using temp file C:\dev\MyGroup\MyGroup_MyApp_trunk\MyGroupMyAppIT\target\site\serenity\6563d98750da0f614e1b25a72a3e0469cd69a1042b80a8f884c5f03ffcbe140f.xml465ed1cc-0b5b-4a2f-aa8b-88ddeecb651e)

DEBUG: net.thucydides.core.reports.xml.XMLTestOutcomeReporter - XML report generated (C:\dev\MyGroup\MyGroup_MyApp_trunk\MyGroupMyAppIT\target\site\serenity\6563d98750da0f614e1b25a72a3e0469cd69a1042b80a8f884c5f03ffcbe140f.xml bytes) 2510

DEBUG: net.thucydides.core.reports.junit.JUnitXMLOutcomeReporter - GENERATING JUNIT REPORTS

DEBUG: net.thucydides.core.reports.junit.JUnitXMLOutcomeReporter - GENERATING JUNIT REPORT SERENITY-JUNIT-6563d98750da0f614e1b25a72a3e0469cd69a1042b80a8f884c5f03ffcbe140f.xml using temporary file target\site\serenity\SERENITY-JUNIT-6563d98750da0f614e1b25a72a3e0469cd69a1042b80a8f884c5f03ffcbe140f.xml1ba0d1fc-2e7b-4d17-af05-64535e9cb840

DEBUG: net.thucydides.core.reports.json.JSONTestOutcomeReporter - Generating JSON report for Inactivity timeout logs out user to file C:\dev\MyGroup\MyGroup_MyApp_trunk\MyGroupMyAppIT\target\site\serenity\6563d98750da0f614e1b25a72a3e0469cd69a1042b80a8f884c5f03ffcbe140f.json (using temp file C:\dev\MyGroup\MyGroup_MyApp_trunk\MyGroupMyAppIT\target\site\serenity\6563d98750da0f614e1b25a72a3e0469cd69a1042b80a8f884c5f03ffcbe140f.json9bf20980-8fae-41ea-a3f6-c8be79655b53)

DEBUG: net.thucydides.core.reports.junit.JUnitXMLOutcomeReporter - GENERATING JUNIT REPORTS

DEBUG: net.thucydides.core.reports.junit.JUnitXMLOutcomeReporter - GENERATING JUNIT REPORT SERENITY-JUNIT-6563d98750da0f614e1b25a72a3e0469cd69a1042b80a8f884c5f03ffcbe140f.xml using temporary file target\site\serenity\SERENITY-JUNIT-6563d98750da0f614e1b25a72a3e0469cd69a1042b80a8f884c5f03ffcbe140f.xml438f2c92-03a6-4f94-a034-0bb2f7138070

DEBUG: net.thucydides.core.requirements.RequirementsTagProvider - Reading requirements from net.thucydides.core.requirements.FileSystemRequirementsTagProvider@64193c70

DEBUG: net.thucydides.core.requirements.RequirementsTagProvider - Reading requirements from net.thucydides.core.requirements.PackageAnnotationBasedTagProvider@4db20b84

DEBUG: net.thucydides.core.requirements.RequirementsTagProvider - Requirement ancestors for:Requirement{name='Core', type='theme' parent = 'null', cardNumber='null'} = [Requirement{name='Core', type='theme' parent = 'null', cardNumber='null'}]

DEBUG: net.thucydides.core.requirements.RequirementsTagProvider - Requirement ancestors for:Requirement{name='Login', type='epic' parent = 'Core', cardNumber='null'} = [Requirement{name='Core', type='theme' parent = 'null', cardNumber='null'}, Requirement{name='Login', type='epic' parent = 'Core', cardNumber='null'}]

DEBUG: net.thucydides.core.requirements.RequirementsTagProvider - Requirement ancestors for:Requirement{name='Internal user login', type='feature' parent = 'Login', cardNumber='null'} = [Requirement{name='Core', type='theme' parent = 'null', cardNumber='null'}, Requirement{name='Login', type='epic' parent = 'Core', cardNumber='null'}, Requirement{name='Internal user login', type='feature' parent = 'Login', cardNumber='null'}]

DEBUG: net.thucydides.core.requirements.RequirementsTagProvider - Requirement ancestors for:Requirement{name='Logout', type='epic' parent = 'Core', cardNumber='null'} = [Requirement{name='Core', type='theme' parent = 'null', cardNumber='null'}, Requirement{name='Logout', type='epic' parent = 'Core', cardNumber='null'}]

DEBUG: net.thucydides.core.requirements.RequirementsTagProvider - Requirement ancestors for:Requirement{name='There is a way to logout', type='feature' parent = 'Logout', cardNumber='null'} = [Requirement{name='Core', type='theme' parent = 'null', cardNumber='null'}, Requirement{name='Logout', type='epic' parent = 'Core', cardNumber='null'}, Requirement{name='There is a way to logout', type='feature' parent = 'Logout', cardNumber='null'}]

DEBUG: net.thucydides.core.requirements.RequirementsTagProvider - Requirement ancestors for:Requirement{name='Timeout', type='epic' parent = 'Core', cardNumber='null'} = [Requirement{name='Core', type='theme' parent = 'null', cardNumber='null'}, Requirement{name='Timeout', type='epic' parent = 'Core', cardNumber='null'}]

DEBUG: net.thucydides.core.requirements.RequirementsTagProvider - Requirement ancestors for:Requirement{name='User logged out after inativity', type='feature' parent = 'Timeout', cardNumber='null'} = [Requirement{name='Core', type='theme' parent = 'null', cardNumber='null'}, Requirement{name='Timeout', type='epic' parent = 'Core', cardNumber='null'}, Requirement{name='User logged out after inativity', type='feature' parent = 'Timeout', cardNumber='null'}]

DEBUG: net.thucydides.core.requirements.RequirementsTagProvider - Requirement ancestors for:Requirement{name='Examples', type='theme' parent = 'null', cardNumber='null'} = [Requirement{name='Examples', type='theme' parent = 'null', cardNumber='null'}]

DEBUG: net.thucydides.core.requirements.RequirementsTagProvider - Requirement ancestors for:Requirement{name='Browser support', type='epic' parent = 'Examples', cardNumber='null'} = [Requirement{name='Examples', type='theme' parent = 'null', cardNumber='null'}, Requirement{name='Browser support', type='epic' parent = 'Examples', cardNumber='null'}]

DEBUG: net.thucydides.core.requirements.RequirementsTagProvider - Requirement ancestors for:Requirement{name='Use chrome browser', type='feature' parent = 'Browser support', cardNumber='null'} = [Requirement{name='Examples', type='theme' parent = 'null', cardNumber='null'}, Requirement{name='Browser support', type='epic' parent = 'Examples', cardNumber='null'}, Requirement{name='Use chrome browser', type='feature' parent = 'Browser support', cardNumber='null'}]

DEBUG: net.thucydides.core.requirements.RequirementsTagProvider - Requirement ancestors for:Requirement{name='Use configured browser', type='feature' parent = 'Browser support', cardNumber='null'} = [Requirement{name='Examples', type='theme' parent = 'null', cardNumber='null'}, Requirement{name='Browser support', type='epic' parent = 'Examples', cardNumber='null'}, Requirement{name='Use configured browser', type='feature' parent = 'Browser support', cardNumber='null'}]

DEBUG: net.thucydides.core.requirements.RequirementsTagProvider - Requirement ancestors for:Requirement{name='Use firefox browser', type='feature' parent = 'Browser support', cardNumber='null'} = [Requirement{name='Examples', type='theme' parent = 'null', cardNumber='null'}, Requirement{name='Browser support', type='epic' parent = 'Examples', cardNumber='null'}, Requirement{name='Use firefox browser', type='feature' parent = 'Browser support', cardNumber='null'}]

DEBUG: net.thucydides.core.requirements.RequirementsTagProvider - Requirement ancestors for:Requirement{name='Use internet explorer browser', type='feature' parent = 'Browser support', cardNumber='null'} = [Requirement{name='Examples', type='theme' parent = 'null', cardNumber='null'}, Requirement{name='Browser support', type='epic' parent = 'Examples', cardNumber='null'}, Requirement{name='Use internet explorer browser', type='feature' parent = 'Browser support', cardNumber='null'}]

DEBUG: net.thucydides.core.requirements.RequirementsTagProvider - Requirement ancestors for:Requirement{name='Use phantom browser', type='feature' parent = 'Browser support', cardNumber='null'} = [Requirement{name='Examples', type='theme' parent = 'null', cardNumber='null'}, Requirement{name='Browser support', type='epic' parent = 'Examples', cardNumber='null'}, Requirement{name='Use phantom browser', type='feature' parent = 'Browser support', cardNumber='null'}]

DEBUG: net.thucydides.core.requirements.RequirementsTagProvider - Requirements found:[Requirement{name='Core', type='theme' parent = 'null', cardNumber='null'}, Requirement{name='Examples', type='theme' parent = 'null', cardNumber='null'}]

And lastly build the aggregate report

• Notice project.build.directoy IS NOT null in this case.  Typically this path would end with "\target".  Not sure how this got set.  Is this a problem?
  
• Notice that OutputDirectory still defaults to the same place (still inside target even though that wasn't indicated in the build dir property).
  
• Notice the Generating HTML Story Reports from and to are the same location.  Is that a problem?
  
• Then the most 'wrong' looking thing to us is the "Merging requirements = []" business.  So, even though requirements where found above, they are gone in this step.  Why?
  
• Seems the "Generating release reports for: []" may be a hint as to why we are not seeing the report name on the home page too.  
  

[INFO] --- serenity-maven-plugin:1.1.36:aggregate (serenity-reports) @ MyGroupMyAppIT_runner ---

[INFO] project.build.directory : C:\dev\MyGroup\MyGroup_MyApp_trunk\MyGroupMyAppIT

[INFO] project.reporting.OutputDirectory : null

[INFO] OutputDirectory : C:\dev\MyGroup\MyGroup_MyApp_trunk\MyGroupMyAppIT\target\site\serenity

[INFO] current_project.base.dir: C:\dev\MyGroup\MyGroup_MyApp_trunk\MyGroupMyAppIT

Generating HTML Story Reports from C:\dev\MyGroup\MyGroup_MyApp_trunk\MyGroupMyAppIT\target\site\serenity

Generating HTML Story Reports to C:\dev\MyGroup\MyGroup_MyApp_trunk\MyGroupMyAppIT\target\site\serenity

[INFO] project.build.directory : C:\dev\MyGroup\MyGroup_MyApp_trunk\MyGroupMyAppIT

[INFO] project.reporting.OutputDirectory : null

[INFO] OutputDirectory : C:\dev\MyGroup\MyGroup_MyApp_trunk\MyGroupMyAppIT\target\site\serenity

Merging requirements = []

Merging requirements = []

Merged requirements set = []

[INFO] REPORTS GENERATED IN C:\dev\MyGroup\MyGroup_MyApp_trunk\MyGroupMyAppIT\target\site\serenity

[INFO] REPORT HOME PAGE: C:\dev\MyGroup\MyGroup_MyApp_trunk\MyGroupMyAppIT\target\site\serenity\index.html

[INFO] Generating release reports for: []

GENERATE CUSTOM REPORTS

Our cukes are in a test only project 

+---src\main\java

|   |       \---com.MyShop.MyGroup.MyApp

|   |              MyAppFeatureTest.java

|   |                           

|   \src\test\java

|       |   \---com.MyShop.MyGroup.MyApp

|       |                   +---common

|       |                   |   +---pages

|       |                   |   +---steplibs

|       |                   |   \---steps

|       |                   |           

|       |                   +---config

|       |                   |       

|       |                   +---core

|       |                   |   +---pages

|       |                   |   +---steplibs

|       |                   |   \---steps

|       |                   |               

|       |                   +---example

|       |                   |   \---steps

|       |                               

|       \---resources

|           |   log4j.xml

|           |   serenity.properties

|           |   

|           +------com.MyShop.MyGroup.MyApp

|           |               |   config.xml

|           |               |   README.txt

|           |               |   

|           |               +---core

|           |               |   +---login

|           |               |   |       internal_user_login.feature

|           |               |   |       

|           |               |   +---logout

|           |               |   |       there_is_a_way_to_logout.feature

|           |               |   |       

|           |               |   \---timeout

|           |               |           user_logged_out_after_inativity.feature

|           |               |           

|           |               +---examples

|           |               |   \---browser_support

|           |               |          use_chrome_browser.feature

|           |               |          use_configured_browser.feature

|           |               |          use_firefox_browser.feature

|           |               |          use_internet_explorer_browser.feature

|           |               |          use_phantom_browser.feature

Our pom to run tests and generate the report

The <dependenciesToScan> bit is because we have a main pom that builds the jar, and this pom that scans that jar and runs the tests it contains

            <plugin>

                <groupId>org.apache.maven.plugins</groupId>

                <artifactId>maven-failsafe-plugin</artifactId>

                <version>${failsafe.version}</version>

                <configuration>

                    <reportsDirectory>${project.build.directory}/failsafe-reports/e2e</reportsDirectory>

                    <summaryFile>${project.build.directory}/failsafe-reports/e2e/failsafe-summary.xml</summaryFile>

                    <forkCount>1</forkCount>

                    <reuseForks>false</reuseForks>

                    <argLine>${failsafe.argLine}</argLine>

                    <failIfNoTests>false</failIfNoTests>

                    <skipITs>false</skipITs>

                    <includes>

                        <include>**/*FeatureTest*</include>

                    </includes>

                    <dependenciesToScan>

                        <dependency>com.MyShop.MyGroup:MyGroupMyAppIT</dependency>

                    </dependenciesToScan>

                </configuration>

                <executions>

                    <execution>

                        <goals>

                            <goal>integration-test</goal>

                            <goal>verify</goal>

                        </goals>

                    </execution>

                </executions>

            </plugin>

            <plugin>

                <groupId>net.serenity-bdd.maven.plugins</groupId>

                <artifactId>serenity-maven-plugin</artifactId>

                <version>${serenity.version}</version>

                <executions>

                    <execution>

                        <id>serenity-reports</id>

                        <phase>post-integration-test</phase>

                        <goals>

                            <goal>aggregate</goal>

                        </goals>

                    </execution>

                </executions>

            </plugin>

        </plugins>

    </build>

Any help would be appreciated,

3T

[John Ferguson SmartSmart]

Hi Tony,

This could be a configuration issue. Would it be possible to post a scaled-down version of your project that illustrates the problem? (For example, you could remove the contents of the test steps so that they don’t actually do anything, but you still generate the report data)?

John.

___________________________________________________
John Smart | Wakaleo Consulting  |  +44 7398 832273
Making smart teams collaborate better

http://johnfergusonsmart.com  | rea...@johnfergusonsmart.com
___________________________________________________

--
You received this message because you are subscribed to the Google Groups "Serenity BDD Users Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to thucydides-use...@googlegroups.com.
To post to this group, send email to thucydid...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Tony Tubbs]

Example project attached - SerenityStudy.zip

• Simple project with a single feature file and only cucumber step definitions to add and subtract numbers (no PageObjects, Step Libraries or other Serenity based stuff.)
  
• The main pom.xml, via mvn clean install, builds a test only jar (and in our real world runs code analyzers and validates feature files in dry-run mode)
  
• The pomIT.xml, via mvn -f pomIT.xml verify, scans the test jar and launches the tests using FailSafe plugin.
  

I am seeing the same issues in this scaled down version I reported previously.

[Tony Tubbs]

I do everything through Eclipse, but downloaded and ran from mvn command line to rule out any issue with Eclipse's m2e plugin.  I get the same incorrect reports from mvn command line too.  Any other ideas I can try?

Thanks

[John Smart]

Hi Tony,

We need to investigate further, but it may be related to the fact that your test classes are in a separate JAR, which will make it hard for Serenity to guess the requirements structure (it uses packages for JUnit tests, or the directory structure for feature files but if they are bundled in a JAR file, the directory structure is effectively lost).

To unsubscribe from this group and stop receiving emails from it, send an email to thucydides-users+unsubscribe@googlegroups.com.
To post to this group, send email to thucydides-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

___________________________________________________
John Smart | Wakaleo Consulting  |  +44 7398 832273
Making smart teams collaborate better

http://johnfergusonsmart.com  |  john....@wakaleo.com
___________________________________________________

We love breaking down silos and helping smart teams collaborate better! Ask about our tailored on-site workshops in Agile Product Planning, BDD Requirements Discovery,  BDD, TDD and Clean Coding, and Advanced BDD Test Automation.

___________________________________________________

[Tony Tubbs]

That did occur to me.  I am using cucumber, so configured the pomIT.xml file to extract the feature files into the target folder.  That is already part of the sample app if that helps with the investigations.  I appreciate the help.

Thanks

[Tony Tubbs]

Have made some progress on this.  I went back to running FailSafe from main pom to get things working w/o scanning a test jar.  Things I determined:

• The serenity.properties file MUST be in the project root
  Mine was in src/test/java, and even though it was picked up and my tests ran, the final report was not formatted correct
  
• I was missing serenity.test.root property
  Here is the important bits of my properties file

serenity.requirements.dir=src/test/resources/com/ttt/serenity/study
serenity.test.root=com.ttt.serenity.study
serenity.requirement.types=theme, epic, feature

Having sorted out the above, all worked as advertised and expected!

The following is what is happening when I go back to using the pomIT.xml and scanning the test jar to launch the tests:

So, I go back to scanning the jar, and I have added an antrun task to extract the *.feature files and Narrative.txt files to target/features.  I update the serenity.requirements.dir to point to target/features instead of src/test/resources, and I lose the Epics in the report.  Instead they show up as another feature.

serenity.requirements.dir=target/features/com/ttt/serenity/study/
serenity.test.root=com.ttt.serenity.study
serenity.requirement.types=theme, epic, feature

I found, if I remove the Narrative.txt from the epic folder, I'll get the epics tab, tags and everything back.  Narrative.txt at the theme level though worked just fine and didn't effect report output.

Any ideas why the inclusion of Narrative.txt in the epic folder when in target/features is causing me an issue when the same downstream file structure works just fine when in src/test/resources?

[Tony Tubbs]

I have gotten a sample app running that does a two step build process.  The first step (pom.xml) to mimic CI check-in time (where you might want to do code analysis and things) and builds jars.  The second step (pomIT.xml) acts like an ad hoc test runner step that scans the test jar built in step one and launches the tests contained within.  To keep Serenity happy, serenity.properties, feature files, narratives, and drivers are extracted from the jar into the target folder so Serenity can find them via the file system.  (The narrative issue I was seeing before just went away, not sure why, I have updated version of serenity, but I rather expect it was a configuration issue on my part, just not sure what change I made fixed that.)

See sample app here:

https://github.com/tbbstny/SerenityCucumberStudy