Can the use of listeners be selective?

[Bear Giles]

I've been trying to selectively use listeners + custom annotations
(e.g., a test method for an HttpServlet test can have a custom
annotation specifying resource files to use as the POST data or as the
expected output for comparison. The listener sets some ThreadLocal
values that are then accessed by the tested method. It's not 100%
transparent but it's close.) I've found one unexpected behavior and am
trying to find a workaround for the second.

The first unexpected behavior was that I had three test classes, each
annotated with a @Listener. The IInvokedMethod listener wasn't invoked
once per tested method though - it was invoked THREE times. Worse, it
was setup(), setup(), setup(), test(), test(), test(), teardown(),
teardown(), teardown() (using the junit names for convenience) - it
was clear that there are three stacked listeners, not just one
listener called three times. This was very surprising since I would
have expected the listeners to only be applied to the classes (and
subclasses) that are explicitly annotated.

I decided the best workaround is to just establish a local policy that
listeners have to be defined in the testng.xml file. That way there
will only be one listener.

The second unexpected behavior is that I can't seem to limit the scope
of the listener. I have a testng.xml file that contains one suite and
two tests:

<suite>
<test name="web-tests">
<listeners><listener name="my.web.listener"/></listeners>
<packages><package name="my.web.classes"/></packages>
</test>
<test name="non-web-tests">
<packages><package name="my.nonweb.classes"/></packages>
</test>
</suite>

The DTD clearly allows <listener> to be nested within <test>. However
when I run the tests I'm still getting the listener executed on the
non-web-test classes - it's acting like the <listener> is defined
under <suite> instead of <test>

In some ways this is a minor issue since the first thing my before()
and after() methods do is check for the presence of the custom
annotation and immediately return if it's not present. But doing the
unnecessary tests (because I know this annotation will never be
present in this package) puts in a little extra computational load and
that sucks when your tests already take 20+ minutes.

[Cédric Beust ♔]

Hi Bear,

Your conclusions are essentially correct.

Listeners are global: they were initially only allowed in testng.xml and I recently allowed them as annotations, which gives you the impression that they are local to the test class they are declared on but they are really not: they will apply to your entire suite.

Same observation regarding the specification of listeners under <test> (I shouldn't have allowed this in the first place, I agree it's counter intuitive).

I am considering adding a scope attribute to the @Listeners annotation, which would be an enum of probably SUITE, TEST and CLASS:

@Listeners(value = MyListener.class, scope = Listeners.Scope.CLASS).

Let me know what you think of this idea.

In the meantime, you can do this in your listener yourself: just take a look at the class of the method or the currentXmlTest() and decide to run the listener or not based on this value.

-- 

Cédric

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to testng...@googlegroups.com.
To unsubscribe from this group, send email to testng-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.

[Bear Giles]

It took me many, many hours to track down that issue since it is so contrary to expectations. The documentation on the main page for @Listener needs to make it extremely clear that it applies at the suite level, not the class level, and in fact I would argue it should be removed entirely.

The recent addition of the annotation explains why the listeners were stacked instead of only using one copy. In the testng.xml multiple references to the same class are weird but not unreasonable. But annotations are easily duplicated so it's more common to have a policy of eliminating duplicates unless there's a clear difference in configuration.

Bear

2011/10/12 Cédric Beust ♔ <ced...@beust.com>

[Cédric Beust ♔]

Hi Bear,

FYI, I just updated the documentation to make this clearer (section 5.18.1).

-- 

Cédric

[Nick]

I would be very interested in seeing the scope attribute to the
listeners field.. would this also apply the the xml?  (ie if i put the
<listener> under the <test> xml it would be used only for that test?)
-Nick

[Cédric Beust ♔]

2011/12/27 Nick <nick.h...@gmail.com>

I would be very interested in seeing the scope attribute to the
listeners field.. would this also apply the the xml?  (ie if i put the
<listener> under the <test> xml it would be used only for that test?)

Yes, that would be the idea, although there is a small potential to break existing clients who might have declared listeners under <test> and suddenly see their listener no longer applied outside of this <test> tag...

-- 

Cédric

[Michael Grove]

Hi,

I came across this thread while troubleshooting why listeners I added to one test base class in my hierarchy were messing up tests that didn't reference that test base class or listener but that were in the same suite.  This thread explains it.

It doesn't appear to me that scope was added, which would have solved my problem - I'd just set the scope to be CLASS anywhere I reference a listener and that would do it.  I see a Retention annotation but I assume that's something different (couldn't find it referenced in the TestNG documentation).

Since my tests are more commonly run as individual classes/methods rather than a suite, putting the listeners in the suite file isn't ideal.  So sounds like my best bet, if there are no plans to add scope, is in every listener inspect the class being tested and based on that analysis (in my case, of the java package and/or instanceof test on the class) either do what is in the listener or skip it.  That sound right?

Thanks.

-mike

[Krishnan Mahadevan]

Michael,

Looks like given the current situation, and the use case that you are calling out, what you state could perhaps be the only way of getting this done.

Since you did state that all of your classes which need a listener intervention are going to be annotated using the @Listeners annotation, perhaps the below logic can be something that you could leverage.

public class MyFunkyListener implements IInvokedMethodListener{
    @Override
    public void beforeInvocation(IInvokedMethod iInvokedMethod, ITestResult iTestResult) {
        Annotation a = iInvokedMethod.getTestMethod().getRealClass().getAnnotation(Listeners.class);
        if (a == null) {
            return;
        }
    }

    @Override
    public void afterInvocation(IInvokedMethod iInvokedMethod, ITestResult iTestResult) {

    }
}

​

Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @ http://wakened-cognition.blogspot.com/

My Technical Scribbings @ http://rationaleemotions.wordpress.com/

--
You received this message because you are subscribed to the Google Groups "testng-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to testng-users...@googlegroups.com.

To post to this group, send email to testng...@googlegroups.com.

Visit this group at http://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.

[Michael Grove]

Thanks for the reply.  I don't think that will work for me, as I have 2 separate base classes for tests in my suite, and the different base classes both use listeners, just different sets of listeners.  So I require a more specific check than "are there listener annotations defined".

If scope, or similar, is added, I will gladly switch to use that.

[Krishnan Mahadevan]

Michael,

You mean something like this ?

@Override
public void beforeInvocation(IInvokedMethod method, ITestResult testResult) {
    Class realClass = method.getTestMethod().getRealClass();
    Listeners annotation = (Listeners) realClass.getAnnotation(Listeners.class);
    if (annotation == null) {
        return;
    }
    List<?> usedAnnotations = Arrays.asList(annotation.value());
    if (! usedAnnotations.contains(this.getClass())) {
        System.err.println("Class doesn't need this listener to be executed");
    }
}

Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @ http://wakened-cognition.blogspot.com/

My Technical Scribbings @ http://rationaleemotions.wordpress.com/

On Wed, Oct 7, 2015 at 9:52 PM, Michael Grove <snow...@gmail.com> wrote:

Thanks for the reply.  I don't think that will work for me, as I have 2 separate base classes for tests in my suite, and the different base classes both use listeners, just different sets of listeners.  So I require a more specific check than "are there listener annotations defined".

If scope, or similar, is added, I will gladly switch to use that.

--

[Michael Grove]

Haven't tried that, but could be a viable option.  That code is more complex for me to read/follow than just calling testResult.getInstance() and then doing an instanceof check, so I'll stick with my approach for now in the absence of scope.

Any chance we'll get scope?