assertEqualsNoOrder bug?

3 views

Skip to first unread message

fixr

unread,

Feb 20, 2026, 3:22:38 AM (10 days ago) Feb 20

to testng-dev

There is a discrepancy in the `assertEqualsNoOrder` behaviour from Object array to Collection implementations. As seen in the test class below using the assertion on List passes while for arrays it fails (as expected).

The issue is both in allowing empty Lists (or any subset) and in allowing duplicate elements -- for arrays these fail with message "Arrays Do not have the same size...".
Is this expected or not; if it is then probably the documentation needs to be more clear.

Else; to fix the issue Assert.class:1672 `public static void assertEqualsNoOrder(Collection<?> actual, Collection<?> expected, String message)` should follow the same implementation of Assert.class:1639 `public static void assertEqualsNoOrder(Object[] actual, Object[] expected, String message)` in checking the collection size and in removing objects one by one from the collection instead of removeAll.

Following is an illustration of the surprising results:

-----

import static org.testng.Assert.assertEqualsNoOrder;

import java.util.List;

import org.testng.annotations.Ignore;

import org.testng.annotations.Test;

public class TestTestNgLists {

@Test

public void testArrayOk() {

assertEqualsNoOrder(

new String[] { "a", "b", "c" },

new String[] { "c", "b", "a" });

}

@Test

@Ignore("This test fails as expected!")

public void testArrayShouldFail() {

assertEqualsNoOrder(

new String[] { },

new String[] { "c", "b", "a" });

}

@Test

@Ignore("This test fails as expected!")

public void testArrayWithDuplicatesShouldFail() {

assertEqualsNoOrder(

new String[] { "c", "c", "b", "a" },

new String[] { "c", "b", "a" });

}

@Test

public void testListOk() {

assertEqualsNoOrder(

List.of("a", "b", "c"),

List.of("c", "b", "a" ));

}

@Test

public void testListShouldFail() {

// NOTE: This test succeeds surprisingly!!

assertEqualsNoOrder(

List.of(),

List.of("c", "b", "a" ));

}

@Test

public void testListWithDuplicatesShouldFail() {

// NOTE: This test succeeds surprisingly!!

assertEqualsNoOrder(

List.of("c", "c", "b", "a"),

List.of("c", "b", "a" ));

}

fixr

unread,

Feb 20, 2026, 9:17:00 AM (10 days ago) Feb 20

to testng-dev

This was using testng 7.5.1 - upgrading to 7.12.0 fixed the issue; and this makes sense since this functionality was rewritten some nine months ago.

The gradle libs.testng gave this old version still (using gradle 8.14.3), for anyone else encountering this using `testImplementation("org.testng:testng:7.12.0")` instead of the libs version fixes the problem.