TCK technologies

[Ken Finnigan]

In a previous MP Architecture call, we briefly discussed the issues raised by [1] around the different technologies used to execute the TCKs of different specifications.

At the time we agreed it required a much wider discussion before we could begin to reach any kind of agreement on how to proceed.

So, I'd like to ask the group what is our "preferred" technology stack for developing TCKs?

Ignore what we have today, we will likely need to adjust most TCKs in some way or the other irrespective of what we actually decide.

For instance, though a lot of TCKs use TestNG I keep hearing questions about whether that's actually the best anymore, or the advent of JUnit 5 has brought that back to the front.

Ken

[1] - https://github.com/eclipse/microprofile/issues/51

[Gordon Hutchison]

Personally, I think the advantage of allowing innovation and improvements in this area outweighs the advantage of having 'one true way'

for MicroProfile. (thinking about TestContainers, and other evolution or experiments etc.)

A second point I'd make is that most vendors also have to run the Jakarta suites so any 'standard' that is different to the TCK's there will still

not be a 'one set of tools to learn' - we should co-ordinate with Jakarta TCK's ideally.

Gordon.

[Stephane Epardaud]

Well, in the case of MP-CP we switched from JUnit to TestNG because we looked at other MP specs and assumed we should imitate. If we're really free to use non-uniform techs if they're better, then we should switch (at least) MP-CP to JUnit ASAP and get it resolved at least for this one. But some official guidance would be nice as to what the policy is, if any :)

--
You received this message because you are subscribed to the Google Groups "Eclipse MicroProfile" group.
To unsubscribe from this group and stop receiving emails from it, send an email to microprofile...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/925c2480-e5e3-4b44-af7a-a84d3013824c%40googlegroups.com.

--

Stéphane Épardaud

[Ken Finnigan]

I'm not suggesting no one should "innovate" and come up with better approaches for the TCKs, but I do see that a common set of "best practices" for creating TCKs is necessary. Anyone can explore alternative test approaches for a given TCK and then propose a change to the "recommended stack" for other specifications to use it as well.

But we need the TCKs to be similar enough that anyone from the community can easily switch to work on different specifications without needing to learn another set of new technologies just to fix some tests. It also complicates TCK execution if you want to run them all in a batch if they're using different technologies.

--

[Stephane Epardaud]

OK, but then it doesn't appear that this discussion has come to any conclusion :)

To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/CAKeeVe5uP3pX%2BKW_EVy3ggDthsap_rQ2uAYk70m6RpsdzxT0%3DA%40mail.gmail.com.

--

Stéphane Épardaud

[Arjan Tijms]

Hi,

While technically not a TCK, Java EE Samples is TCK-like. Initially the project allowed for "innovation", but before long we had tests in groovy, in java, in testng, in junit, using htmlunit, httpunit, hamcrest, spock, various other technologies and various permutations of all of that.

At the end of the day, it wasn't pretty to say the least.

Though I'm not personally a fan on TestNG, it's what the MP TCKs have been using, and it's what the non-Oracle originating Jakarta TCKs (Bean Validation, CDI and Batch) are using as well.

JUnit 5 is quite difficult to use, since Arquillian doesn't support it. This would mean using JUnit 4 anyway.

[Arjan Tijms]

p.s.

I spoke too soon about JUnit 5 not working at all; it partially works, namely for the remote mode (test runs as client): see last messages here https://github.com/arquillian/arquillian-core/issues/137

[Ken Finnigan]

My preference would certainly be to use something a bit more modern, such as JUnit 5, and if necessary work with Arquillian to resolve any remaining issues related to it

--
You received this message because you are subscribed to the Google Groups "Eclipse MicroProfile" group.
To unsubscribe from this group and stop receiving emails from it, send an email to microprofile...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/ef953b90-5730-4540-be77-7eb441d99774%40googlegroups.com.

[Ken Finnigan]

I wanted to reignite this thread to see if we can begin to make some headway.

There is general agreement that it makes a lot of sense to align on the same TCK technology for all MP specifications, which is a great first step.

Next step, the hardest, is deciding what to align to.

Though there might be upstream issues we need to fix, does anyone have major concerns with aligning to JUnit 5 and Arquillian?

[Emily Jiang]

Currently most if not all MP specs use Arquillian with TestNG. I am not aware of any issues we are facing right now. I would say we stay with Arquillian and TestNG and revisit this if there is a need to align with Junit 5.

Having that said, we should document MicroProfile test framework across the board. We need a wiki to state this kind of technical guideline.

Thanks

Emily

To unsubscribe from this group and stop receiving emails from it, send an email to microp...@googlegroups.com.

[Ken Finnigan]

Though most, if not all, specs currently utilize TestNG. It wasn't through explicit choice, but rather following with what others have done. There was never, that I recall, discussion or decisions around what test frameworks a TCK should use.

Given most people I speak with aren't fans of TestNG, I don't think we should retain it just because it became the de facto choice.

Yes, it would require rewriting the TCKs, which isn't a small amount of work, but retaining technical debt because it's "already there" isn't a great approach either.

If there is general agreement that JUnit 5 is a better testing framework, either because of its design, ease of use, customizability, or whatever, we shouldn't prevent change if it makes the most sense for the long term.

I'm happy for us to discuss whether JUnit 5, TestNG, or anything else, is the most appropriate testing framework for the TCKs, but I don't agree with staying with TestNG just because it's already there. If it's discussed and it's determined that TestNG is the best framework, then that's a different matter.

Ken

To unsubscribe from this group and stop receiving emails from it, send an email to microprofile...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/68c56347-014d-425d-9144-fcac703daf48%40googlegroups.com.

[Emily Jiang]

I am not against changes if there is a need. My point is that we should only fix something that needs to be fixed. If there is no issue, why do we need to spend effort to change one thing to another?

but I don't agree with staying with TestNG just because it's already there.

I think you might have misunderstood what I meant. Let me explain better. We are using TestNG and did not find issues raised against it. If there is no valid issues, I don't think we should spend effort to change just because of TestNG having less fans.

I think it is better to look at the detailed issues. The issue Arjan mentioned about JUnit5 not working with Arquillian is also worrying.

My 2cents.

Emily

To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/68c56347-014d-425d-9144-fcac703daf48%40googlegroups.com.

[Ken Finnigan]

On Fri, Feb 7, 2020 at 1:59 PM 'Emily Jiang' via Eclipse MicroProfile <microp...@googlegroups.com> wrote:

I am not against changes if there is a need. My point is that we should only fix something that needs to be fixed. If there is no issue, why do we need to spend effort to change one thing to another?

but I don't agree with staying with TestNG just because it's already there.

I think you might have misunderstood what I meant. Let me explain better. We are using TestNG and did not find issues raised against it. If there is no valid issues, I don't think we should spend effort to change just because of TestNG having less fans.

Not just about less fans, many have complained to me about what's possible with JUnit 5 is a lot harder, or not possible, to do with TestNG. It impacts the design and implementation of the TCK to be restricted by TestNG.

 

I think it is better to look at the detailed issues. The issue Arjan mentioned about JUnit5 not working with Arquillian is also worrying.

I believe there are some options for making JUnit 5 work with Arquillian. Besides that, what kind of open source citizens would we be if we discounted JUnit 5 because there are issues with Arquillian instead of helping to resolve them?

 

To unsubscribe from this group and stop receiving emails from it, send an email to microprofile...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/2581ef05-09f9-4647-87b3-19724c3429f8%40googlegroups.com.

[Mark Little]

+1

To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/CAKeeVe4RzjHqqq5jrSEVUDPLMipj48wdO5TnuykofoqEeBsSyQ%40mail.gmail.com.

[Rüdiger zu Dohna]

If we can't agree on JUnit 5 vs. TestNG, it's probably premature to require one or the other. I personally prefer JUnit 5 because I use it everywhere else; it's a great, mature, and active platform.

I also used Arquillian in all my projects in the past. Recently, Arquillian seems to not get the love it used to. The last release of any of its projects is from 2018. This is okay if it is already mature enough and does everything people need. But the difficulties with JUnit 5 can also be symptomatic. I've used Testcontainers recently, and I currently use it for the Problem Detail sandbox project. Compared to Arquillian, it feels simpler to me.

I'm not promoting JUnit 5 nor Testcontainers, but I'm not sure that we should standardize as long as the choices can not be made clearly.

+1

To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/2581ef05-09f9-4647-87b3-19724c3429f8%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Eclipse MicroProfile" group.

To unsubscribe from this group and stop receiving emails from it, send an email to microp...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/CAKeeVe4RzjHqqq5jrSEVUDPLMipj48wdO5TnuykofoqEeBsSyQ%40mail.gmail.com.

[Roberto Cortez]

Maybe if we can list some things that are possible to do with JUnit 5 that can’t be done with TestNG (or harder to do), could help the case?

To unsubscribe from this group and stop receiving emails from it, send an email to microprofile...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/bd387cbe-4499-4e1d-b06e-5670c052ccf0%40googlegroups.com.

[Emily Jiang]

+1 on creating an issue to list all cons/pros for TestNG vs. Junit! With that, it will answer my questions in my earlier email on what issues we are facing or potentially face with TestNG.

I also think we should revisit our test framework once again with Arquillian and TestContainer as mentioned by Rudiger! TestContainer is getting more and more popular for cloud native apps. In additon to that, MicroShed Testing Tool offers a great integration with MicroProfile and Test Container. Maybe now is a good time to reevaluate the old topic on Testing. Thoughts?

Thanks

Emily

On Monday, February 10, 2020 at 1:14:04 PM UTC, Roberto Cortez wrote:

Maybe if we can list some things that are possible to do with JUnit 5 that can’t be done with TestNG (or harder to do), could help the case?

To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/bd387cbe-4499-4e1d-b06e-5670c052ccf0%40googlegroups.com.

[Ken Finnigan]

Let's not bring MicroShed into things for now.

Initially, it's good to focus on JUnit vs TestNG from the start

To unsubscribe from this group and stop receiving emails from it, send an email to microprofile...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/d2b35101-c16d-4ad7-8087-83d1bb4eb8d9%40googlegroups.com.

[Werner Keil]

As TestNG is currently also used in every Jakarta Specification (well maybe some of the older ones by Oracle could still use something different?) shouldn't especially those MP features that hope to also get a downstream use by Jakarta EE (whichever package name, that is not relevant for this topic ;-D) some time stick with TestNG?

I used both, TestNG to create or significantly contribute to the TCKs for JSRs 354, 363 and 385 (the latter with a Multi-Profile modularity and optionality far beyond CDI although it was inspired by its TCK) and recently help a customer introduce JUnit 5 including Parameterized Tests.

Using some of those extensions (the one with JSON is indeed based on JSON-P ;-) I believe, JUnit 5 could be used in similar ways as TestNG is with XML-based configuration and parameters, but especially for existing TestNG based TCKs a transition would be less than trivial, so if a spec already has TestNG tests or hopes to be accepted into the Jakarta EE platform soon (10ish) I would stick with TestNG.

A new spec that never had a TCK before and may not hope for Jakarta EE inclusion ever or much later, why not give JUnit 5 a try.

I'd be happy to share some of the experience with Parameterized Tests if I have time and someone would like me to help.

Werner

To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/2581ef05-09f9-4647-87b3-19724c3429f8%40googlegroups.com.