Skipped due to Selenium Grid issues not getting generated in TestNG

[Pavithra]

Cedric,

While running tests in parallel on many browser combination, in few tests - say some methods ran and while running next method if selenium grid throws 'Error communicating with browser" or "session timeout" or any kind of grid communication issues - test results for that particular test set is not getting generated in TestNG report.

We run tests from Hudson and in hudson console I could clearly see that few methods ran successfully under one test and one particular failed because of communication issue because of which other methods under that test set is skipped. But the same thing is not coming in TestNG.

Note: I can't conclude any configuration failures doesn't lead to result generation. We use Before method to signin to the application and after method to signout. So, when there is failure during signin or signout it skips remaing tests and it generate in the report. Similarly I expect that failure during teardown (while quitting the browser) also should generate the test result in report - Hope I am correct ?


Below is the error from Hudson Console:

org.openqa.selenium.WebDriverException: Session [1340636615158] was terminated due to CLIENT_STOPPED_SESSION
Command duration or timeout: 250 milliseconds
Build info: version: '2.23.0', revision: '17137', time: '2012-06-07 09:08:12'
System info: os.name: 'Windows XP', os.arch: 'x86', os.version: '5.1', java.version: '1.6.0_24'
Driver info: driver.version: RemoteWebDriver
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
    at java.lang.reflect.Constructor.newInstance(Unknown Source)
    at org.openqa.selenium.remote.ErrorHandler.createThrowable(ErrorHandler.java:188)
    at org.openqa.selenium.remote.ErrorHandler.throwIfResponseFailed(ErrorHandler.java:145)
    at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:458)
    at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:462)
    at org.openqa.selenium.remote.ExecuteMethod.execute(ExecuteMethod.java:44)
    at org.openqa.selenium.remote.AddTakesScreenshot$1.invoke(AddTakesScreenshot.java:36)
    at org.openqa.selenium.remote.Augmenter$CompoundHandler.intercept(Augmenter.java:264)
    at org.openqa.selenium.remote.RemoteWebDriver$$EnhancerByCGLIB$$8af380d1.getScreenshotAs(<generated>)
    at com.aol.webmail.TestListener.captureScreenShot(TestListener.java:55)
    at com.aol.webmail.TestListener.onConfigurationFailure(TestListener.java:124)
    at org.testng.internal.Invoker.runConfigurationListeners(Invoker.java:1770)
    at org.testng.internal.Invoker.handleConfigurationFailure(Invoker.java:323)
    at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:226)
    at org.testng.internal.Invoker.invokeMethod(Invoker.java:613)
    at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:846)
    at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1170)
    at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:125)
    at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:109)
    at org.testng.TestRunner.runWorkers(TestRunner.java:1147)
    at org.testng.TestRunner.privateRun(TestRunner.java:749)
    at org.testng.TestRunner.run(TestRunner.java:600)
    at org.testng.SuiteRunner.runTest(SuiteRunner.java:317)
    at org.testng.SuiteRunner.access$000(SuiteRunner.java:34)
    at org.testng.SuiteRunner$SuiteWorker.run(SuiteRunner.java:351)
    at org.testng.internal.thread.ThreadUtil$CountDownLatchedRunnable.run(ThreadUtil.java:147)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.lang.Thread.run(Unknown Source)
Caused by: org.openqa.grid.common.exception.GridException: Session [1340636615158] was terminated due to CLIENT_STOPPED_SESSION
    at org.openqa.grid.internal.ActiveTestSessions.getExistingSession(ActiveTestSessions.java:104)
    at org.openqa.grid.internal.Registry.getExistingSession(Registry.java:423)
    at org.openqa.grid.web.servlet.handler.RequestHandler.getSession(RequestHandler.java:234)
    at org.openqa.grid.web.servlet.handler.RequestHandler.process(RequestHandler.java:116)
    at org.openqa.grid.web.servlet.DriverServlet.process(DriverServlet.java:84)
    at org.openqa.grid.web.servlet.DriverServlet.doGet(DriverServlet.java:62)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
    at org.seleniumhq.jetty7.servlet.ServletHolder.handle(ServletHolder.java:565)
    at org.seleniumhq.jetty7.servlet.ServletHandler.doHandle(ServletHandler.java:479)
    at org.seleniumhq.jetty7.server.session.SessionHandler.doHandle(SessionHandler.java:225)
    at org.seleniumhq.jetty7.server.handler.ContextHandler.doHandle(ContextHandler.java:1031)
    at org.seleniumhq.jetty7.servlet.ServletHandler.doScope(ServletHandler.java:406)
    at org.seleniumhq.jetty7.server.session.SessionHandler.doScope(SessionHandler.java:186)
    at org.seleniumhq.jetty7.server.handler.ContextHandler.doScope(ContextHandler.java:965)
    at org.seleniumhq.jetty7.server.handler.ScopedHandler.handle(ScopedHandler.java:117)
    at org.seleniumhq.jetty7.server.handler.HandlerWrapper.handle(HandlerWrapper.java:111)
    at org.seleniumhq.jetty7.server.Server.handle(Server.java:349)
    at org.seleniumhq.jetty7.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:452)
    at org.seleniumhq.jetty7.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:47)
    at org.seleniumhq.jetty7.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:884)
    at org.seleniumhq.jetty7.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:938)
    at org.seleniumhq.jetty7.http.HttpParser.parseNext(HttpParser.java:634)
    at org.seleniumhq.jetty7.http.HttpParser.parseAvailable(HttpParser.java:230)
    at org.seleniumhq.jetty7.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:66)
    at org.seleniumhq.jetty7.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:254)
    at org.seleniumhq.jetty7.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
    at org.seleniumhq.jetty7.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
    at java.lang.Thread.run(Thread.java:636)


===============================================
    Read Smoke Test On Win7-FF9
    Tests run: 11, Failures: 2, Skips: 0
    Configuration Failures: 2, Skips: 0
===============================================

[edwolb]

This sounds very similar to the issue I was running into as well.  One of GRID nodes had a Java heap issue, and for some reason, the results for the test it was supposed to run did not get included in test results.  Not the xml, nor the HTML nor the ReportNG result I had, but it was caught by the Screenshot listener I use.  I was unable to replicate it, but if it can be replicated by a browser issue on a grid node, it should be somewhat easy to reproduce?

--

Chris

[Pavithra]

Cedric,

Could you please check for the above? Is this the bug or expected behavior?

 Note: TestNG Version: 6.0.1

Pavithra

[Cédric Beust ♔]

I don't really see anything TestNG specific in your question above, can you isolate the problem in a small test case that doesn't use Selenium that I could build?

-- 

Cédric

Pavithra

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To view this discussion on the web visit https://groups.google.com/d/msg/testng-users/-/Y48Gp6p0r00J.
To post to this group, send email to testng...@googlegroups.com.
To unsubscribe from this group, send email to testng-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.

[Ian Blackburn]

Hi,

I know this thread is a little old, but I am encountering the same issue when tests are terminated by Selenium Grid due to timeout.

When trying to recreate the issue in a test using only TestNG, I found that the timeout was ignored when running the tests in parallel (set at suite level in testng.xml using parallel="tests"). I believe this has already been logged as an issue here.
I also found that the reports are only missing when running the tests in parallel.
I can provide you the testNG-only test if you still require it, but it doesn't show the issue due to the fact that I can't get the test to terminate due to timeout. Is there some other way to mimic the termination of the tests that is triggered by Selenium Grid timing out?
Alternatively, I could also provide full source of a minimal Selenium test (including set-up for Grid on localhost) - this could work with minimal tweaks to testng.xml and gridnode.json as long as you have at least 2 browsers installed on the local machine.

All I can work out for myself at the moment is that the reports on these terminated tests are only missing if I use a custom listener (defined in the testNG.xml file under the two others required to utilise reportNG reports) AND when running the tests in parallel.
When leaving the custom listener out of the equation but still running in parallel, the terminated tests do appear in the reports, but @AfterSuite is skipped.
When running in sequence, there are no issues with or without the custom test listener.

I initially saw this in testng 6.8.7, using reportng1.1.4, and am still experiencing in testng 6.8.21

Thanks and regards,

Ian

[Ian Blackburn]

For anyone arriving at this page because of missing test results when using Selenium + TestNG + ReportNG or just Selenium + TestNG, this issue goes away as soon as you wrap any attempts to capture screenshots with a try & catch to catch any RunTimeExceptions.

I ran into this very issue because I was taking screenshots and reading console log errors in a custom TestListener (which was extending TestNG's TestListenerAdapter) in my override of TestListenerAdapter.onTestFailure function.

If the browser has been closed, this will throw a WebDriverException, and when this is not caught the end result is that these test results are not included in the generated HTML reports or testng-failed.xml.

If you do catch this exception, any tests that have failed due to timeout (or having the browser closed for any other reason) will appear in your reports, as well as being added to the testng-failed.xml correctly.

This is not a bug with TestNG, ReportNG or Selenium's Grid as far as I can tell.