Converting Geb 3 tests to use testManager

[Nick Bensema]

It looks like Geb development has come a long way, so I'm trying to keep my codebase current, and I've tried a few times to run my tests with geb 4.1 instead of 3.4.1 -- baby steps on the way to 5.1.  So far, it won't even compile, and I need some help.

The first problem I have is that I have several specs, both abstract and concrete, which try to override the `reportFailure` method, which no longer compiles. I know that I'm supposed to use testManager, but it's not quite obvious how to do it; there's very little documentation about it.

First of all, I have a main abstract class that also overrides reportFailure(), because those tests all have two browser windows open, and I want to make sure to screenshot both of them.

But also, sometimes I override reportFailure in a concrete test class, so that I can get screenshots of dialog boxes, and close them.  That method also calls super.reportFailure(), meaning both of my custom methods will run.

So, given a test class that overrides reportFailure(), how do I move that method into a test manager object that the class can use?  and will it support the kind of inheritance I had before?

Example of the kind of thing my method will contain: calling the main report function, doing stuff with page content, and calling the superclass method which might also be an override:

@Override

void reportFailure() {

    if( page instanceOf DesignerPage ) {

        if(preview.displayed) {

            report "preview"

            preview.close()

        }

    }

    super.reportFailure()

}

[Marcin Erdmann]

Hi Nick,

Sorry for a late reply.

So for the reporting on multiple windows you can use a built-in capability - MultiWindowReporter. I don't know when it was introduced but it's definitely available in 4.1 because its usage is described in the manual for that version, see https://gebish.org/manual/4.1/#reporting-on-multiple-windows.

With regards to your second requirement of triggering another report before performing an action and doing the report proper I would suggest implementing another `geb.report.Reporter` and then plugging it in together with MultiWindowReporter via configuration in a similar way to the one reported in https://gebish.org/manual/4.1/#reporting-on-multiple-windows. I think you should be able to get ideas how to implement that reporter from looking at the sources of MultiWindowReporter (https://github.com/geb/geb/blob/master/module/geb-core/src/main/groovy/geb/report/MultiWindowReporter.groovy) but let me know if you struggle with it and I will help out with some code.

Thanks,
Marcin

--
You received this message because you are subscribed to the Google Groups "Geb User Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to geb-user+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/geb-user/d3a5566a-672e-433b-a9d2-322da900e12en%40googlegroups.com.

[Nick Bensema]

Thanks, you've made it a lot clearer for me, and I think I'm getting close. I've created a custom reporter that extends ReporterSupport, with its own writeReport method that does its own thing and calls notifyListeners.  I created a constructor where I can pass the existing reporter, so in my setupSpec I have something like this:

browser.config.reporter = new MyReporter(browser.config.reporter)

I can even call methods, like when I've downloaded the file I want to test, I can call browser.config.reporter.setDownloadedFile(file) and the log statements say it's executing.  But when I test it by introducing a purposeful failure, it still calls the ordinary ScreenshotReporter and skips my code entirely, and once again I get a png of the download page.

So, what am I missing, why might my reporter not be activating?

[Marcin Erdmann]

There is a new instance of browser and config created per test, so modifying config in setupSpec() will only have for the first test method in a spec. You might get more luck by putting that code in setup but there still will be cases where that code might not be applied to a browser instance, like for example when reporting is triggered from cleanupSpec. I would therefore consider registering your custom reporter in your config script like I initially suggested.

Marcin

To view this discussion on the web visit https://groups.google.com/d/msgid/geb-user/7488922e-f26e-445c-a7bc-dc947ad1029an%40googlegroups.com.

[Nick Bensema]

I did indeed fix that problem by moving it to just setup() -- moving it to the config isn't yet appropriate because I want to report different test classes differently; I have different tests for the UI and for the downloads.  Maybe that will work if I can figure out how to modularize this project, but that's another thread...

Now I'm having a different problem with the MultiWindowReporter.  It's writing the HTML and PNG for both windows, but only one of them is showing up in my geb-spock-reports output -- always the same one, the main window rather than the editor window.  If I add an extra report "notFailure" call in my test, both reports show up, but still only the first window.  I went so far as to upgrade to Spock 2.0 and Groovy 3.0, but ended up right back where I started.  For what it's worth, I went from version 1.8.0 to version 2.0-groovy-3.0 of this: https://github.com/AOEpeople/geb-spock-reports

I thought it might have to do with the long window name or the space in the name or something, so I tried writing my own writeWindowReport call. I'd need to do this anyway because "main" and "design" will be more helpful names than the window ID. My reporter correctly determines the correct window label, writes the logging message, and writes both HTML and PNG files for each one, but only the "main" shows up in the HTML report for that test class:

    class DesignerReporter extends MultiWindowReporter {
        public DesignerReporter(Reporter r) {
            super(r)
        }
        void writeWindowReport(ReportState reportState, String windowId) {
            //geb.Browser browser = reportState.browser
            reportState.browser.withWindow(windowId) {
                def windowLabel = "${reportState.label}.main"
                if(isAt(DesignerPage)) {
                    windowLabel = "${reportState.label}.designer"
                }
                log.info "reporting $windowLabel with window $windowId"
                def windowState = new ReportState(reportState.browser, windowLabel, reportState.outputDir)
                backing.writeReport(windowState)
            }
        }

[Nick Bensema]

so, I've looked at the default template in the source code, and I've even added my own listener object to confirm that both "main" and "designer" were being caught by listeners, but still only "main" appeared in the report HTML.  If I suppress the "main" window, then "designer" does show up.  I poked around the source code and think there might be a problem with  GebReportingListener in geb-spock-reports; there's a lot of strange logic in there and I can see how it might not do the right thing if findArtifactByNumber already returns something...

having hit a dead end there, I took another look at how to write a GebTestManager.  I poked around the source code some more and thought I could extend GebTestManager and move my custom reportFailure() method into there, but there's this whole GebTestManagerBuilder thing that populates a private, not protected, singleton, and it doesn't seem to have a mechanism to just override one method. Since getTestReporter()'s return type is GebTestManager, I have to return a subclass of that, and I can't just write a one-method class and @Delegate to call the real test manager for everything else.  And the real test manager is a private static final singleton, and so I'd have to delegate everything to that specific instance.  does @Delegate take precedence over superclass methods or do I have to write a dozen @Override methods to shunt every other method call to the delegate?  Either way it seems fragile, and smells like the wrong way to do it.

so, either I have to find the right way to change reportFailure(), or fix the bug in geb-spock-reports.  am I missing something here?

[Marcin Erdmann]

Nick,

So as far as I'm aware, geb-spock-reports is no longer maintained (https://github.com/AOEpeople/geb-spock-reports/issues/37#issuecomment-950887176) thus trying to use it will become harder and harder over time, especially with newer versions of Geb. I've never been involved with geb-spock-report or has been a user of it so unfortunately won't be able to help with that. I can help with what reports and how are being generated from Geb but not with what geb-spock-reports then does with that.

With regards to extending GebTestManager - it's not designed to be extended but it's designed to be configured via the builder you mention in your email. This is because the old "test framework support is scattered in base test classes" mechanism that was used prior to introduction of GebTestManager lead to duplication. It also meant that users would be encouraged to extend and override code from these base classes which then makes it hard to track all possible use cases and makes it harder to evolve the code of the framework without inadvertently introducing breaking changes. 

Unfortunately GebTestManager currently does not cater for your usecase of having some custom logic for generating reports upon failure but which seems like a completely valid usecase. My idea would be to introduce 

GebTestManagerBuilder withReportFailure(Consumer<GebTestManager> reportOnFailure) {
    ...
}

to GebTestManagerBuilder. Then you would use that in custom base classes in your project, for example

@DynamicallyDispatchesToBrowser
class DesignerPageReportFailureHandlingSpec extends Specification implements ManagedGebTest {

    private final static GebTestManager TEST_MANAGER = new SpockGebTestManagerBuilder()
        .withReportingEnabled(true)
        .withReportFailure { testManager ->
            def page = testManager.browser.page

            if( page instanceOf DesignerPage ) {

                if(page.preview.displayed) {
                    testManager.report "preview"
                    page.preview.close()
                }
            }
           
            testManager.report "failure"
        }
        .build()

    @Override
    @Delegate(includes = ["getBrowser", "report"])
    GebTestManager getTestManager() {
        TEST_MANAGER
    }
}

I believe that would allow you to achieve what you are after. Would that work for you? If that's the case then I can create an issue for this and incorporate this into the upcoming Geb 6.0 release which will be based on Spock 2.1 and WebDriver 4. I know that this would force you to do a big version jump and will be somewhat inconvenient but I unfortunately do not have capacity to backport this.

Thanks for bearing with me on this.

To view this discussion on the web visit https://groups.google.com/d/msgid/geb-user/9892e5ba-6bff-448c-ab50-9c972eb8828fn%40googlegroups.com.

[Nick Bensema]

actually, it looks like geb-spock-reports is the only thing in my way, both for multi-window reporting, and for reporting downloaded files.  Indeed, the custom reporter that saves a ZIP file works great, but the reports show links to nonexistent screenshots.  Since it's not being maintained, maybe I just need to switch to something else.  There's probably no need to add features to Geb for this case.

Can you recommend a reporting extension for the geb-spock stack that includes links to Geb artifacts in the output?

[Marcin Erdmann]

Yeah, makes sense.

I've never used it so have no idea if it's any good but there seems to be something called Allure (https://github.com/allure-framework/allure2) which has Spock 1.3 support (https://github.com/allure-framework/allure-java/tree/master/allure-spock, for Spock 2.0 you probably could use JUnit 5 support) and it has ability to add attachments (https://docs.qameta.io/allure-report/#_attachments). You could probably plug into that via an implementation of geb.report.ReportingListener (https://gebish.org/manual/current/#listening-to-reporting).

Hope this helps,

Marcin

To view this discussion on the web visit https://groups.google.com/d/msgid/geb-user/1737e7e1-95e0-483f-a74f-417456d30e82n%40googlegroups.com.