TCK/Spec compliance in the light of a faulty/brittle spec?

[Heiko Rupp]

During the work on MP-Metrics we (=team working on MP-Metrics) have discovered some tests that are brittle because they rely on timing. They pass most of the time - and thus we have not identified them before finalising the MP 1.0 and 1.0.1 TCKs

Similar for others where an expected result is something like 499.5 ,but sometimes ends up in 499 or 500.

Or where some CDI rules were "violated".

For all those an implementation may be lucky enough that their JDK does the rounding in the expected way,

CDI-impl may return a bean instead of a proxy or the CI box is fast enough to not trigger the timing issue.

My question is now: how do we deal with that?

Of course we can fix or disable all of those brittle tests for MP-Metrics 1.1.

But can an implementation still claim compatibility with MP-Metrics 1.0 if e.g. CDI returns a proxy and one (ill written) test constantly fails because of it?

We can of course not crack the MP-1.0 release open, disable the tests and then re-release.

I am proposing that specifications can in a case like this add a text file to a release "errata" that lists those problematic tests and thus waives failures of those. Does that make sense?

[Ladislav Thon]

2017-12-07 10:54 GMT+01:00 'Heiko Rupp' via Eclipse MicroProfile <microp...@googlegroups.com>:

We can of course not crack the MP-1.0 release open, disable the tests and then re-release.

Why not release 1.0.2 with TCK fixes/amendments?

LT

[Heiko Rupp]

Am Donnerstag, 7. Dezember 2017 11:03:27 UTC+1 schrieb Ladislav Thon:

Why not release 1.0.2 with TCK fixes/amendments?

Yeah, I had a bit of a brain-fart here, as for me a .z release added more tests, which then hardened the implementation and I forgot that the .z could also fix bugs in the TCK.

As long as it is clear that passing 1.0.2 means being MP-Metrics 1.0 compliant