Unable to properly run tests in parallel (Selenium WebDriver + TestNG + Selenium Grid + ThreadLocal)

[MarkM]

Hi -

This is my very first post on this forum; I am a newbie Selenium User with beginner skill in Java.

To give you a background of our work; we are using Page Object Model, all of our tests uses a single DataProvider method that fetches its data from an ".xlsx" file based on the test case name that calls/uses the DataProvider.

We are unsure however if we have declared our ThreadLocal the way it should be.  And also, we are not sure if the declaration of our getDriver() method is correct.  Another issue is that we are unsure if we should be using "@BeforeTest"/"@AfterTest" or "@BeforeClass"/"@AfterClass" on our setup and tearDown methods.

The following problem is being encountered:
- One test fails, the succeeding test also fails.
- There are times when the test data being fetched is inaccurate (more column data thrown than expected).

Here is our CONFIGTESTBASE class:
public class ConfigTestBase {

    private static ThreadLocal<RemoteWebDriver> threadedDriver = null;
    private static XSSFSheet ExcelWSheet;
    private static XSSFWorkbook ExcelWBook;
    private static XSSFCell Cell;
    private static XSSFRow Row;
    private static final String Path_TestData = GlobalConstants.testDataFilePath;
    private static final String File_TestData = GlobalConstants.testDataFileName;

    @Parameters({"objectMapperFile"})
    @BeforeSuite
    public void setupSuite(String objectMapperFile) throws Exception {
        GlobalConstants.objectMapperDefPath = new File(objectMapperFile).getAbsolutePath();
        new Common().OverrideSSLHandshakeException();
    }

    @Parameters({"browserName"})
    @BeforeClass
    public void setup(String browserName) throws Exception {

        threadedDriver = new ThreadLocal<>();

        DesiredCapabilities capabilities = new DesiredCapabilities();

        if (browserName.toLowerCase().contains("firefox")) {
            capabilities = DesiredCapabilities.firefox();
            capabilities.setCapability(CapabilityType.ACCEPT_SSL_CERTS, true);
            capabilities.setBrowserName("firefox");
            capabilities.setPlatform(Platform.WINDOWS);
        }

        if (browserName.toLowerCase().contains("ie")) {
            System.setProperty("webdriver.ie.driver","C:\\selenium\\IEDriverServer.exe");
            capabilities = DesiredCapabilities.internetExplorer();
            capabilities.setCapability(InternetExplorerDriver.INTRODUCE_FLAKINESS_BY_IGNORING_SECURITY_DOMAINS, true);
            capabilities.setCapability(InternetExplorerDriver.FORCE_CREATE_PROCESS, false);
            capabilities.setBrowserName("internet explorer");
            capabilities.setPlatform(Platform.WINDOWS);
        }

        if (browserName.toLowerCase().contains("chrome")) {
            System.setProperty("webdriver.chrome.driver","C:\\selenium\\chromedriver.exe");
            capabilities = DesiredCapabilities.chrome();
            capabilities.setCapability(CapabilityType.ACCEPT_SSL_CERTS, true);
            capabilities.setBrowserName("chrome");
            capabilities.setPlatform(Platform.WINDOWS);
        }

        if (browserName.toLowerCase().contains("safari")) {
            SafariOptions options = new SafariOptions();
            options.setUseCleanSession(true);
            capabilities = DesiredCapabilities.safari();
            capabilities.setCapability(SafariOptions.CAPABILITY, options);
            capabilities.setBrowserName("safari");
            capabilities.setPlatform(Platform.WINDOWS);
        }

        threadedDriver.set(new RemoteWebDriver(new URL(GlobalConstants.GRIDHUB), capabilities));

    }

    protected static RemoteWebDriver getDriver()
    {
        RemoteWebDriver driver = null;

        try {
            driver = threadedDriver.get();
        }
        catch (Exception e) {
            e.printStackTrace();
        }
        return driver;
    }

    @AfterClass
    public void tearDown() throws Exception {
        getDriver().quit();
    }

    //This method is to set the File path and to open the Excel file, Pass Excel Path and Sheetname as Arguments to this method
    public void setExcelFile(String Path,String SheetName) throws Exception {
        try {
            // Open the Excel file
            FileInputStream ExcelFile = new FileInputStream(Path);

            // Access the required test data sheet
            ExcelWBook = new XSSFWorkbook(ExcelFile);
            ExcelWSheet = ExcelWBook.getSheet(SheetName);
        } catch (Exception e) {
            throw (e);
        }
    }

    //This method is to read the test data from the Excel cell, in this we are passing parameters as Row num and Col num
    @SuppressWarnings("static-access")
    public String getCellData(int RowNum, int ColNum) throws Exception{
        try{
            Cell = null;
            Cell = ExcelWSheet.getRow(RowNum).getCell(ColNum);
            Cell.setCellType(Cell.CELL_TYPE_STRING);
            return Cell.getStringCellValue();
        } catch (Exception e) {
            return "";
        }
    }

    //This method is to write in the Excel cell, Row num and Col num are the parameters
    @SuppressWarnings("static-access")
    public void setCellData(String textValue,  int RowNum, int ColNum) throws Exception    {
        try{
            Row  = ExcelWSheet.getRow(RowNum);
            Cell = Row.getCell(ColNum, Row.RETURN_BLANK_AS_NULL);
            if (Cell == null) {
                Cell = Row.createCell(ColNum);
                Cell.setCellValue(textValue);
            } else {
                Cell.setCellValue(textValue);
            }

            // Constant variables Test Data path and Test Data file name
            FileOutputStream fileOut = new FileOutputStream(Path_TestData + File_TestData);
            ExcelWBook.write(fileOut);
            fileOut.flush();
            fileOut.close();
        } catch (Exception e) {
            throw (e);
        }
    }

    @DataProvider(name="getDataFromFile")
    public Object[][] getDataFromFile(Method testMethod, ITestContext context) throws Exception {

        String[][] tabArray;
        int intCounter;
        int intRowCount = 0;
        int intRowCounter = 0;
        int intColCount = 0;
        int intColCounter;
        int intColDataCount = 0;
        int intColDataCounter = 0;
        String temp;

        String testName = testMethod.getName();
        String banner = context.getCurrentXmlTest().getParameter("banner");
        setExcelFile(Path_TestData + File_TestData, banner);

        //get number of data to be returned
        for(intCounter=0;intCounter<ExcelWSheet.getLastRowNum()+1;intCounter++){
            if(getCellData(intCounter, 1).equals(testName)) {
                if (intColCount == 0) {
                    intColCount = ExcelWSheet.getRow(intCounter).getLastCellNum() - 2;
                }
                intRowCount++;
            }
        }

        if(intRowCount == 0){
            System.out.println("\n*** Data for '" + testName + "' was not found.");
            throw new AssertionError("Data for '" + testName + "' was not found.");
        }

        for(intCounter=0;intCounter<ExcelWSheet.getLastRowNum()+1;intCounter++){
            if(getCellData(intCounter, 1).equals(testName)) {
                for(intColCounter=2;intColCounter<intColCount+2;intColCounter++) {
                    temp = getCellData(intCounter,intColCounter);
                    if(temp != null && !temp.isEmpty()){
                        intColDataCount++;
                    }
                }
                //to exit FOR loop
                intCounter = ExcelWSheet.getLastRowNum()+1;
            }
        }

        //set data array dimension
        tabArray = new String[intRowCount][intColDataCount];

        for(intCounter=0;intCounter<ExcelWSheet.getLastRowNum()+1;intCounter++){
            if(getCellData(intCounter, 1).equals(testName)) {
                intRowCounter++;
                for(intColCounter=2;intColCounter<intColCount+2;intColCounter++) {
                    temp = getCellData(intCounter,intColCounter);
                    if(temp != null && !temp.isEmpty()){
                        tabArray[intRowCounter-1][intColDataCounter] = getCellData(intCounter,intColCounter);
                        intColDataCounter++;
                    }
                }
            }
        }

        return tabArray;

    }

}



Here is a sample test class that we have; extends CONFIGTESTBASE class... one class per test:
public class Google6 extends ConfigTestBase {

    private Generic generic = null;
    private HomePage homePage = null;

    @BeforeMethod
    public void MethodInit(ITestResult result) throws Exception {
        generic = new Generic(getDriver());
        homePage = new HomePage(getDriver());
    }

    @Test(dataProvider="getDataFromFile")
    public void Google6(String execute, String url, String searchString) throws Exception {

        if(execute.toUpperCase().equals("YES")) {

            //navigate to application page
            generic.navigateToURL(url);
           
            //search
            homePage.search(searchString);

        } else {
            generic.log("Execute variable <> 'YES'. Skipping execution...");
            throw new SkipException("Execute variable <> 'YES'. Skipping execution...");
        }

    }
}


And here is our suite file; 'banner' parameter is used for finding specific sheet on ".xlsx" file:
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">

<suite name="Full Test Suite - Firefox" parallel="classes" thread-count="5" verbose="1">
    <parameter name="objectMapperFile" value="mom.objectdef.properties" />

    <test name="Regression - Firefox">
        <parameter name="browserName" value="firefox" />
        <parameter name="banner" value="SAMPLE" />
        <classes>
            <class name="com.company.automation.web.app.testsuites.Google1" />
            <class name="com.company.automation.web.app.testsuites.Google2" />
            <class name="com.company.automation.web.app.testsuites.Google3" />
            <class name="com.company.automation.web.app.testsuites.Google4" />
            <class name="com.company.automation.web.app.testsuites.Google5" />
            <class name="com.company.automation.web.app.testsuites.Google6" />
        </classes>
    </test>

</suite>

[Krishnan Mahadevan]

Mark,

The problem is indeed due to the way in which you have used ThreadLocale.

TestNG doesnt guarantee that your @BeforeClass [ that is where you have taken care of instantiating your webdriver ] and your @Test annotated test methods are all going to be running in the same thread [ This is true especially when you resort to parallel executions ]

You might perhaps want to take a look at this blog post of mine to get an idea on how to go about doing this : https://rationaleemotions.wordpress.com/2013/07/31/parallel-webdriver-executions-using-testng/

Lastly, I think your question is more of TestNG related than Selenium/Webdriver. You might want to post queries related to TestNG on the testng-users google forum.

I would request you to please help keep this forum relevant by posting queries that are directly related to Selenium/WebDriver.

Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @ http://wakened-cognition.blogspot.com/

My Technical Scribbings @ http://rationaleemotions.wordpress.com/

--
You received this message because you are subscribed to the Google Groups "Selenium Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to selenium-user...@googlegroups.com.
To post to this group, send email to seleniu...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/selenium-users/414f7546-4145-4328-b573-6963f1112e16%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Mark Collin]

You will want to write some code to keep track of the threads so that you can ensure you use the same one throughout your test.  Have a look at:

https://github.com/Ardesco/Selenium-Maven-Template/blob/master/src/test/java/com/lazerycode/selenium/DriverFactory.java

Which hands off to:

https://github.com/Ardesco/Selenium-Maven-Template/blob/master/src/test/java/com/lazerycode/selenium/config/WebDriverThread.java

to instantiate the driver object, it should help you out.

Your configTestBase class is very large and could probably do with being broken up into smaller classes that do one thing.

[MarkM]

I've seen that the "driverThread = new ThreadLocal<WebDriverThread>()" code is on the method of @BeforeSuite annotation while on my ConfigTestBase, I've placed it on the method of @BeforeClass annotation.  What's the difference between those two implementations?

[Krishnan Mahadevan]

Like I said before, the thread on which @BeforeSuite runs and the thread that runs @Test annotated test method need not be the same [ same is applicable even for @BeforeClass as well].

The only guarantee provided by TestNG is below [ with respect to running in the same thread ]

@BeforeMethod

@Test

@AfterMethod

beforeInvocation()

@Test

afterInvocation()

Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @ http://wakened-cognition.blogspot.com/

My Technical Scribbings @ http://rationaleemotions.wordpress.com/

On Thu, Jan 21, 2016 at 6:57 AM, MarkM <mmasi...@gmail.com> wrote:

I've seen that the "driverThread = new ThreadLocal<WebDriverThread>()" code is on the method of @BeforeSuite annotation while on my ConfigTestBase, I've placed it on the method of @BeforeClass annotation.  What's the difference between those two implementations?

--

You received this message because you are subscribed to the Google Groups "Selenium Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to selenium-user...@googlegroups.com.
To post to this group, send email to seleniu...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/selenium-users/66cfd52b-d180-4452-9010-76e186b98f63%40googlegroups.com.

[Mark Collin]

You don't have this bit:

{
	@Override
	protected WebDriverThread initialValue() {
	WebDriverThread webDriverThread = new WebDriverThread();
	webDriverThreadPool.add(webDriverThread);
	return webDriverThread;
	}
	};
	}

Which is storing the threads in a synchronised list so that I can keep track of them.  I can then use the @AfterSuite annotation to close down all of the threads cleanly when all the tests have finished.

I'm using @BeforeSuite/@AfterSuite because I use the maven-failsafe-plugin to set my tests to run as parallel methods with a set number of threads.  All my tests then get thrown into a bucket and each thread started up is treated as a new suite.  I don't use @BeforeClass because the documentation talks about it running before methods in the current class are run, The way I'm set up, I don't care what class a specific method is in and I don't want to rely on that.

You are running everything in one class using the same driver instance.  Does each test clear down cookies and set itself up to run correctly no matter where it is on the site you are testing?  One failure could leave you somewhere that you don't expect to be which could then break the next test in that class.  TestNG doesn't guarantee order either so the tests may not run in the order you expect them to run.