SoftAssert.Assertall(); provides invalid results to listener class

[Kasun Herath]

Hi,
Context of Question

I am using SoftAsserts in my test script and I do have a custom listener class which takes screenshot on failure , but when I do the  SoftAsserts    -> assertAll() within @AfterTest or @AfterMethod , it fires up my onTestSuccess method in my custom listener class which means that even though there are failures my test is marked as passed , but if I use assertAll() within the same @Test method it works fine.

What I want is rather than declaring assertAll() inside each test method to put it inside a @AfterTest or @AfterMethod , so that it captures the errors and point it to my listener class with failures

Sample Script

@Listener(MyCustomListener.class)

public class GoogleTest {
    WebDriver driver;
    SoftAssert softAssert;

    @BeforeMethod
    public void setup() {
        // Set the path to the chromedriver executable
        System.setProperty("webdriver.chrome.driver", "path/to/chromedriver");

        // Create a new instance of ChromeDriver
        driver = new ChromeDriver();

        // Initialize SoftAssert
        softAssert = new SoftAssert();
    }

    @Test(description="Testing Soft Asserts")
    public void verifyGoogleTitle() {
        // Navigate to Google
        driver.get("https://www.google.com");

        // Verify the title using SoftAsserts
        softAssert.assertEquals( driver.getTitle() , "Invalid Title");
    }

    @AfterMethod
    public void tearDown() {
        // Perform assertions
        softAssert.assertAll();

        // Quit the driver
        driver.quit();
    }
}

Above script when @AfterMethod is executed even though there are failures listener class takes it as passed.

Requesting support on this issue

Regards
Kasun Herath

[venkatesh iyengar]

SoftAssert doesn't fail the test, it will allow continue, you should use hard Assert which will mark the test as failed. 

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to testng-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/testng-users/5581b8e3-ddc5-4346-a15f-311ab1bd23een%40googlegroups.com.

[Kasun Herath]

Hi Venkatesh ,

Below method is responsible for checking all the softAssert failures at the end and it does fail the tests , problem is it doesn't invoke the listener class correctly to call the onTestFailure method 

 @AfterMethod
    public void tearDown() {
        // Perform assertions

        softAssert.assertAll(); <- Checks all the soft Asserts 

[venkatesh iyengar]

Hi Kasun, 

Assert all does mark it as failed but it is doing in after method instead if it is done @Test it will invoke your listener correctly

Regards, 
Venkatesh

To view this discussion on the web visit https://groups.google.com/d/msgid/testng-users/3b9a232c-8249-47fc-b774-e1f753b50061n%40googlegroups.com.

[⇜Krishnan Mahadevan⇝]

Kasun,

Before|After are configuration methods. Triggering an assertion failure in an AfterXX configuration method is definitely NOT going to fail your test cases. That explains why the TestNG listeners that are responsible for eavesdropping on the test method results arent detecting the failure, because the test method passed, but the assertion failure triggered a failure of an AfterXXX configuration method.

Its not clear as to what exactly you are trying to do.

The fact that the SoftAssert is being instantiated within the "@Test" (It should be done within the "@Test" or else the other test methods if incase they end up using that same object can fail due to failures recorded from different test methods) also means/requires that the assertAll() be invoked from within the Test method. What are the concerns in having it done apart from the "Oh I have to remember to call softAssert.assertAll() as the last line in the method ?

Alternatively you might want to look at IHookable ( https://testng.org/doc/documentation-main.html#ihookable) interface which lets you define a run() method in a base class, and the control would always come to it. Now within the run() method, you can create a SoftAssert() object and pass it to the test method via the ITestResult object. The test would retrieve the SoftAssert object from within the ITestResult object attribute and add assertions to it.

Sample

import java.util.Objects;
import org.testng.IHookCallBack;
import org.testng.IHookable;
import org.testng.ITestResult;
import org.testng.Reporter;
import org.testng.asserts.SoftAssert;

public class AbstractSampleTestCase implements IHookable {

@Override
public void run(IHookCallBack callBack, ITestResult testResult) {
SoftAssert soft = new SoftAssert();
testResult.setAttribute("soft", soft);
callBack.runTestMethod(testResult);
soft.assertAll();
}

protected SoftAssert softAssert() {
Object object = Reporter.getCurrentTestResult().getAttribute("soft");
Objects.requireNonNull(object, "Soft Assertion object cannot be null");
if (object instanceof SoftAssert) {
return (SoftAssert) object;
}
throw new IllegalStateException("Soft Assertion object was not set");
}
}

import org.testng.annotations.Test;

public class SampleTestCase extends AbstractSampleTestCase {


@Test
public void passingTest() {
softAssert().assertEquals(true, true);
}

@Test
public void failingTest() {
softAssert().assertTrue(true);
softAssert().assertFalse(true);
}
}

Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @ http://wakened-cognition.blogspot.com/

My Technical Scribblings @ https://rationaleemotions.com/

To view this discussion on the web visit https://groups.google.com/d/msgid/testng-users/CAHqOtnOKWgUktcnOMEq4CXTMLkick1zpvxQeHgyoSLfk7n44vQ%40mail.gmail.com.

[Kasun Herath]

This is what I exactly wanted to understand  , Thank you for the detailed response Krishnan!

Regards
Kasun Herath