On single test failure remaining WebDriver tests skip without any dependencies

[fmtjatt]

Hi All,

I have been fighting this problem from last few months. Basically, I have 50 odd tests created using webdriver and often if one test fails then rests are usually skipped. It also marks rest of the setup/tear methods as failed.
Here what happened today. One test failed and then reason was page did not load within timeout window. I expect the tear down to close the browser but it left the browser open. After that ALL tests were skipped even though they did not depend on this test. These failures happen at random spots and for random reason. Another reason, I have seen is that if unexpected model dialogue shows up then same thing happens and all tests fails. It is really hard to pinpoint the problem to one test. Any help or guidance is greatly appreciated.

• Windows 7
  
• IE 8 - 32 bit
  
• Java 1.7
• TestNG 6.8.6
• WebDriver 2.32
• IEDriverServer 2.32.3 - 32 bit
  

Here is the failure which caused all the tests to skip and all the setup/tear down methods to failed. I'm using driver.quit to close the driver so not sure by browser did not shut down.

org.openqa.selenium.TimeoutException: Timed out waiting for page to load. (WARNING: The server did not provide any stacktrace information) Command duration or timeout: 60.49 seconds Build info: version: '2.32.0', revision: '6c40c18', time: '2013-04-09 17:23:22' System info: os.name: 'Windows 7', os.arch: 'amd64', os.version: '6.1', java.version: '1.7.0_21' Session ID: 1fbf200f-1454-4184-ab88-ec3af35972b0 Driver info: org.openqa.selenium.ie.InternetExplorerDriver Capabilities [{platform=WINDOWS, elementScrollBehavior=0, javascriptEnabled=true, enablePersistentHover=true, ignoreZoomSetting=false, browserName=internet explorer, enableElementCacheCleanup=true, unexpectedAlertBehaviour=dismiss, version=8, cssSelectorsEnabled=true, ignoreProtectedModeSettings=false, allowAsynchronousJavaScript=true, requireWindowFocus=false, handlesAlerts=true, initialBrowserUrl=, nativeEvents=true, takesScreenshot=true}] at org.openqa.selenium.remote.ErrorHandler.createThrowable(ErrorHandler.java:187) at org.openqa.selenium.remote.ErrorHandler.throwIfResponseFailed(ErrorHandler.java:145) at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:554) at org.openqa.selenium.remote.RemoteWebElement.execute(RemoteWebElement.java:268) at org.openqa.selenium.remote.RemoteWebElement.click(RemoteWebElement.java:79) at test.java.pageObjects.ManageReportPage.changeStatus(ManageReportPage.java:88) at test.java.test.PercTest.vascularScanDocWorkFlow(PercTest.java:1487) ... Removed 28 stack frames Thanks in advance!

[darrell]

If the driver.quit() is not closing the browser you need to make sure the browser is closed. I had this problem with different frameworks and IE (5.5, 6, 7, 8, etc.). IE hangs if the software is flakey. A lot of times I see javascript which has timing issues. But when a human is using the website they are seldom clicking and filling in things as fast as WebDriver will. This means a human will rarely encounter the timing issues which are always there. WebDriver runs a lot faster than a human, so it will find the problems all the time.

Additionally, if I'm going to a company website, as a real user, and IE hangs or takes too long I would not hesitate to close IE and try again (or switch to Chrome). So what am I doing in these cases? I run through my story (test), my teardown become "if IE has hung, just close the browser. if closing the browser fails, go to task manager and kill it". So when I write my test framework, I literally use, as teardown, Run.Exec to taskkill IE (and IEDriverServer.exe) if driver.quit() does not close it properly. I believe there are utility classes in Selenium which have methods to check if it is Windows and if Windows kill IE.

Thus if you end the test properly, the subsequent tests should work fine. You should also code the framework such that, after you run it once, re-run all failed tests again in case some failed doe to a flakey IE.

[kapil aggarwal]

Can you please post your Setup and TearDown method

[fmtjatt]

Darrell,

Thank you for your detailed response. It has been a nightmare for me as in one run all tests would pass and in another majority of them would skip. Unfortunately, our application only supports IE 8, so I'm stuck with that browser. I will look into adding the KILL command to my tear-down method so it can kill IE if it is still open. Can you please share details on how to make all the failed tests run again automatically?

Thanks,

[fmtjatt]

  Setup Method    
       @BeforeMethod
        @Parameters({"environment", "browser"})
        public void setup(String environment, String browser) throws InterruptedException, FileNotFoundException, IOException {
           
            File file = new File("C:/automation/EPMS/InternetExplorerDriver/IEDriverServer.exe");
            System.setProperty("webdriver.ie.driver", file.getAbsolutePath());
           
            DesiredCapabilities capabilities = new DesiredCapabilities();
            capabilities.setCapability(CapabilityType.ACCEPT_SSL_CERTS, true); 
            capabilities.setCapability("requireWindowFocus", true);
            capabilities.setCapability(CapabilityType.UNEXPECTED_ALERT_BEHAVIOUR, UnexpectedAlertBehaviour.DISMISS); //Accept unexpected alerts
           
                   
            //Open browser based on what is being passed as parameter in testng.xml file
            if(browser.contains("IE")){
                driver = new InternetExplorerDriver();
            }else if(browser.contains("FF")){
                driver = new FirefoxDriver();
            }else{
                boolean invalidBrowser = true;
                Assert.assertFalse(invalidBrowser, "Browser type must be IE or FF");
            }
       
           
            //Navigate to the test url by what is being passed as parameter in testng.xml file           
            if (environment.contains("branch")){
                baseUrl = "https://abc/login.jsp";
                Reporter.log("Performing tests on "+environment+ " environment");
            }else if (environment.contains("autotrunk")){
                baseUrl = "https://xyz/login.jsp";
                Reporter.log("Performing tests on "+environment+ " environment");
            }else if(environment.contains("production")){
                baseUrl = "https://dcf.com/login.jsp";
                Reporter.log("Performing tests on "+environment+ " environment");
            }else if(environment.contains("trunk")){
                baseUrl = "https://xxxabc/login.jsp";
                Reporter.log("Performing tests on "+environment+ " environment");
            }else if(environment.contains("staging")){
                baseUrl = "https://abc123.com/login.jsp";
                Reporter.log("Performing tests on "+environment+ " environment");
            }else{
                baseUrl=null;
                Reporter.log("Please provide valid test environment");
                Assert.assertNotNull(baseUrl, "Please provide valid test environment");
            }
           
            //Navigate to URL, Maximize the window, Setup page and object timeout to 30 seconds
            driver.get(baseUrl);
            driver.manage().window().maximize();
            driver.manage().timeouts().implicitlyWait(60, TimeUnit.SECONDS);
            driver.manage().timeouts().pageLoadTimeout(60, TimeUnit.SECONDS);
           
            if(!environment.contains("production"))    {       
                //In IE if Security certificate message is displayed then click continue to website link
                driver.navigate().to("javascript:document.getElementById('overridelink').click()");
                Thread.sleep(3000);
            }
          }

Tear Down
public void tearDown() throws IOException{
            try{
                driver.quit();
            }catch (Exception e){
                System.out.println("exception caught while closing driver" +e);
            }finally{
                driver.quit();

[kapil aggarwal]

Just one thing I believe your tearDown method in decorated with @AfterMethod ?

Because Generally skipping of non dependent test cases happen if there is exception in Configuration Method in TestNG

--
You received this message because you are subscribed to a topic in the Google Groups "webdriver" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/webdriver/8dfO6V_C0To/unsubscribe.
To unsubscribe from this group and all its topics, send an email to webdriver+...@googlegroups.com.
To post to this group, send email to webd...@googlegroups.com.
Visit this group at http://groups.google.com/group/webdriver.
For more options, visit https://groups.google.com/groups/opt_out.

[darrell]

Re-running failed tests would be outside the scope of WebDriver. That would be something I would code in the tool which launches the tests. For example, I might be using gradle to launch a JUnit suite. At the end of the execution I could code a loop that checks for failed tests. If failure_count > 0 and current_failure_count < previous_failure_count (the number of failures are going down) then run the failed tests again.

I would also imagine that certain build tools (Jenkins, TeamCity, etc.) might have some programmability as well.

The kill sequence would be:

    import static org.openqa.selenium.os.WindowsUtils.*;

    driver.quit();

    if(thisIsWindows()) {

      tryToKillByName("iexplore.exe");

    }

Before you do this, kapil has a point. Are you annotating your tearDown method? If you don't have the appropriate annotation it might be that the tearDown just isn't running.

[fmtjatt]

Hi Darrel,

Thanks for your reply. I will look into gradle to re-run failed tests. And Yes, I do have @AfterMethod annontation for the teardown method. Few days ago, I also added to logic kill the IEDriverServer.exe process if there is some exception that happens while running driver.quit(). That did not help at all as today again on very first failure everything else skipped. Even though teardown method has the code to kill the IEDriverProcess.exe it did not kill it and left IE8 open. I'm totally lost as to what might be causing these unexpected failures/skips.

Thanks,

[Bill Ross]

You could use AutoIt or the like to close IE. Does IEDriverServer log anywhere? Does IEDriverServer ever die, or do you need to reboot? Can you get the code and add your own traces?

Bill

--
You received this message because you are subscribed to the Google Groups "webdriver" group.
To unsubscribe from this group and stop receiving emails from it, send an email to webdriver+...@googlegroups.com.

[darrell]

If you aren't using gradle to launch your tests you might want to look into whatever you are using to launch your tests and see if you can add some logic to it. I mention gradle just because that is what I'm using on my current project. I could just as easily be using Bourne shell, ruby, etc. to run my test suite. If you are using something which allows for programmability to launch tests right now, look into using it. Gradle uses groovy and groovy is Java-like. So I can use plugins and libraries to add programming to my gradle tasks.

If you are launching your tests from say Jenkins and Jenkins is simply running a ruby script then you could add to the ruby script. Alternatively, if you are launching the tests using the JUnit core runner and Java command line instructions you could write a Java program which uses JUnit APIs to launch the tests and adds a loop capability.

The options are endless. It just depends on what technology you already know or are willing to learn.

The key to all this is figuring out why IE is not closing properly. I remember when I was supporting IE6 there was a know bug with PNGs (page never finished loading occasionally) and it leaked memory. So it would crash if you didn't close it and flush the DLLs. Took me a while to figure that out but once I knew the problem, worked around the PNG bug so the page would load and wrote a counter to make IE close and flush memory after a certain number of test runs.

Maybe look into turning on debugging for IEDriverServer.exe and see if it gives you any clues. The best trick I have is doing different things so I can eliminate one thing each time. For example, I might write different tests which do different things and watch the Performance Monitor to see if something causes the problem. Hopefully something will jump out at me. For example, with IE6 I noticed the IE process memory kept going up and up until the system got flaky. I wasn't looking for a memory leak but it was obvious once I started monitoring the OS. Maybe you can see what is happening at the OS level when you start seeing the tests skipping. Monitor the OS, the JVM running Selenium, the process running your Selenium scripts, the IEServerDriver.exe process, etc.

I should also warn you that I have been doing development, testing, operations, etc. for many years. Even with my experience, pairing with a colleague helps me a lot. You might need to get others to help you regardless of your skill level.

Darrell

P.S. BillR is asking a lot of questions because we'd even ask these questions of ourself in order to track down the problem. Often in an attempt to answer the question others ask me I figure out what the problem is.

[fmtjatt]

Hi Bill,

I'm closing the IE following way, If it tries to use Driver.Quit() and if it fails then it kills it by process name:

       @AfterMethod
        public void tearDown() throws Exception{

            try{
                driver.quit();
            }catch (Exception e){
                System.out.println("exception caught while closing driver" +e);
            }finally{

                Utilities process = new Utilities(driver);
                process.killProcessRunging("IEDriverServer.exe");
                Thread.sleep(5000);
               
            }
        }

IE does not die it just stays open and then I need to kill it manually and then restart the tests. I do not know how to make IEDriverServer log somewhere, any suggestions?

[fmtjatt]

Darrell,

Thanks for your detailed reply. I'm using TestNG to run my tests. Currently my browser closes after every method is executed. I guess I can add additional logic to clear the DLL files as you suggested. Do you know which files needs to be flushed? Unfournately, I'm the only QA person who knows WebDriver so I cannot get help from anyone else. If I ask a developer they are simply not interested or know enough about WebDriver to help.

Thanks

================On Wednesday, August 28, 2013 4:47:59 PM UTC-7, darrell wrote:=========================

[darrell]

Don't ask a developer to help you with WebDriver. Ask a developer to help you understand why IE is crashing. WebDriver is probably just exposing a bug which is always there. Maybe a developer can look at things like the ChromeDriver logs and figure out why IE might be crashing. Also understand that WebDriver is just javascript. A good developer should be able to figure out why javascript code (GreaseMonkey in Firefox for example) is crashing IE.