The problem is that the test.jar used for compilation is from a different version than the one that the plugin uses to run the tests.
The change was made here:
http://github.com/cbeust/testng/commit/d50ccac164dae9b0d657c359aeae90cc9d338880
and then reverted here:
http://github.com/cbeust/testng/commit/49bffd72afdde5c803ed766cfa45ba50dccfd185Cedric, the problem of version compatibility is a tough one. Several approaches are possible:
1) Maintain backwards compatibility at all cost, like in this case, revert the commit or overload the API methods. This means that testNG's public API will be more difficult to evolve (in this case based on user feedback).
2) Does it make sense to compile the tests with one version and run them with another? If I'm not mistaken ANT and Maven pick up the jar from the classpath. However the Eclipse plugin comes bundled with a jar and that one is used, hence the discrepancy with the compilation. This means enhancing the Eclipse plugin, I'm not sure if that's difficult.
3) Add a check on the Eclipse plugin to enforce or warn for a minimum version in the project's build path. This will enforce API compatibility but sometimes the developers cannot choose the testNG version to use (corporate policies, etc).
The public API comprises the asserts, annotations and hookable interfaces. This problem may arise again in the future.
Cedric, what's your opinion? How far should TestNG go to maintain backwards compatibility?
Cheers,
Tomás