Can we decouple TCK releases from spec releases?

[John D. Ament]

Hi all

Putting this out there.  I think Scott's brought it up before, and I know we spoke of it on the last JWT call.  I think we need to come up with a way to split up the specs and the TCKs.  We're undoubtly going to run into cases where TCKs need minor fixes applied, or just in general missing tests.  It would make sense if we could then just release a new version of the TCK as a patch fix release.

John

[Heiko Rupp]

Putting this out there.  I think Scott's brought it up before, and I know we spoke of it on the last JWT call.  I think we need to come up with a way to split up the specs and the TCKs.  We're undoubtly going to run into cases where TCKs need minor fixes applied, or just in general missing tests.  It would make sense if we could then just release a new version of the TCK as a patch fix release.

I see your point and I guess you are absolutely right here. 

How do you envision handling the artefacts? Right now both spec and tck are in one git repo (which I like a lot). Would you split them?

 

[sst...@redhat.com]

Agreed.

I believe they could still be in the same repo and just do minor releases from the just the tck module keeping the api and parent pom version the same as the current release?

I have not tried such a release yet to know if such a structure works with the release plugin.

Otherwise, we would need separate -tck project repos.

[Ondro Mihályi]

We wouldn't need a separate repo even if the release plugin can't do what we want. We can simply remove the TCK maven module from the parent, so that it isn't build recursively from parent. Then the release plugin would omit the TCK module.

From the discussion with Scott, I remember somebody (possibly me :) ) summarizing our options to basically 2 options:

 - either we release API and TCK together and allow fix versions for TCK later

 - or we will release API and TCK separately, with API first and the TCK released after a timeboxed while, e.g. a month

The second option would allow implementors to start implementing (against the API and snapshot TCK), but would also allow additional changes to the TCK. I believe Implementors don't need the final version of the TCK to start implementing. And the second option would also prevent a situation when an implementation passes the initial final version of the TCK but doesn't pass subsequent fix versions. That would put 2 implementations into inequality, depending on which TCK version they were tested against. But this option also requires to abide to more strict release processes to meet a deadline for a TCK so that an API doesn't live without a TCK for too long.

--Ondro

[Emily Jiang]

I will go with releasing API and TCK together and allow fix versions for TCK later.

The main reason is that TCK is the one that can prove whether APIs are useful and efficient.  Without TCKs, we won't have great confidence on whether APIs are good enough. I don't see the real benefit of releasing API early without TCK.

In no doubts, TCKs can be improved and then released after being released with API. I have also thought about the versions for the TCKs. We should increase micro version for any subsequent TCK releases.

e.g. We have released Config API and TCK at the version 1.1. If next week, we found an issue with TCK, we should fix it and release it with the version of 1.1.1 and so on. 

We should also make sure all implementations to use the latest TCK. I think this will have to mandated by the overall MP x release. If we release Config TCK 1.1.1, we should include Config 1.1 with TCK 1.1.1 in MicroProfile 1.3 release. Any runtime certifies against MP 1.3 release, they have to make sure to pass the Config 1.1.1 TCK.

Emily

[sst...@redhat.com]

Ok, ack on simply removing the TCK from the parent if needed. Option 2 is what I was suggesting before, but both could be done. It is a good point that a reasonable TCK exists during the API release to validate it.

[Werner Keil]

+1

That goes along what most JSRs do. If structural API changes come with a MR then the TCK normally gets updated, too. Sometimes the TCK gets updates, e.g. 363 just did, or 107 has a 1.0.1 TCK while the API is still unchanged.

Werner

[donbo...@gmail.com]

I like the idea of allowing fixes and enhancements to the TCKs.  That said, there is some danger that someone could have released a product that complies with an earlier version of the TCK, and is suddenly made non-compliant by a new or modified test, so perhaps we need to consider what we would do in that situation.

As with the discussion on significant PRs, perhaps post-release TCK changes should require at least a week of time for any implementers to try them.  If an implementation couldn't comply with the new test we'd have to discuss whether the test is strictly necessary to prove an implementation is compliant with the spec.

I'm interested to add a few tests to the Metrics TCK for areas where coverage is missing -- should we formally agree on an approach before we proceed?

Don

[Heiko Rupp]

Am Freitag, 22. September 2017 14:39:40 UTC+2 schrieb donbo...@gmail.com:

I like the idea of allowing fixes and enhancements to the TCKs.  That said, there is some danger that someone could have released a product that complies with an earlier version of the TCK, and is suddenly made non-compliant by a new or modified test, so perhaps we need to consider what we would do in that situation.

Well,

if the original TCK is tagged with 1.0.0 and the fixed one has 1.0.1 then the implementor can say that they comply with 1.0.0.

If the TCK change also needs a spec change, then I guess that would both go into 1.1.0 where no one would expect that a 1.0.0 server would pass the 1.1.0 test kit.

[Emily Jiang]

+1 on Heiko!

Don, please see my earlier message with more explination:

The main reason is that TCK is the one that can prove whether APIs are useful and efficient.  Without TCKs, we won't have great confidence on whether APIs are good enough. I don't see the real benefit of releasing API early without TCK.

In no doubts, TCKs can be improved and then released after being released with API. I have also thought about the versions for the TCKs. We should increase micro version for any subsequent TCK releases.

e.g. We have released Config API and TCK at the version 1.1. If next week, we found an issue with TCK, we should fix it and release it with the version of 1.1.1 and so on. 

We should also make sure all implementations to use the latest TCK. I think this will have to mandated by the overall MP x release. If we release Config TCK 1.1.1, we should include Config 1.1 with TCK 1.1.1 in MicroProfile 1.3 release. Any runtime certifies against MP 1.3 release, they have to make sure to pass the Config 1.1.1 TCK.

Basically, you can go ahead to add/modify tests. You can release them at any time. When MP 1.3 releases, you can create a PR against to list the latest released metrics artifacts. HTH

Emily

[donbo...@gmail.com]

Heiko, that approach to versioning makes sense to me.  

I'm hoping to add a few TCK tests shortly, so hopefully that will give us a chance to try it.

Regards,

Don

[Kevin Sutter]

Just catching up on this thread.  +1 to allow the separation of TCK updates from the API.  Key word is *updates* since the API and TCK need to be released together for the main release.

[John D. Ament]

All,

So at this point it seems like there's enough consensus to get this done.  Who wants to give it a shot?

I have one issue that's a TCK only fix for config - https://github.com/eclipse/microprofile-config/issues/248

So how do we do this? Is there an eclipse IP review?  

John

[Kevin Sutter]

John,
No IP review is required.  One thing that I learned from the last go-around with the MicroProfile 1.2 release is that the individual component releases do not need IP reviews.  (I still need to update our wiki for this process.)  Since there is no way to isolate these IP reviews to just the code in the component repo, the Eclipse IP team instructed me to just skip the IP reviews for the component releases.  We will do the IP reviews for the MicroProfile releases instead.  This should help with our overall processing.

I would first post your intent of creating an updated TCK release and call it Config TCK 1.1.1 (rename it after the fact since the initial creation only accepts a numbering scheme):
https://projects.eclipse.org/projects/technology.microprofile/governance

When the release is getting closer, you will still need to request a Release Review, but the IP log should no longer be necessary.  Which also means that no CQ updates will be required.  Of course, this will just add to the overhead associated with the MicroProfile releases, but it's probably more containable at that point.

Let me know if you have further questions.  Every time we do a release, I learn something new about the Eclipse processes.

Thanks, Kevin

[John D. Ament]

Hmmm there's some interesting implications there.  Right now we don't have a case for an off cycle release, but ideally we would head there.  For instance, right now fault tolerance is at 1.0.  Maybe 1.1 doesn't ship in any MP but then a 1.2 does.  Are we in fact saying that the changes for 1.1 never get IP review?

Anyways, thats fine.

I'll get a release entered, but looking moreso for input from Ondrej on how we would actually execute the release.

John

[Kevin Sutter]

John,
We can always make exceptions to the rule.  But, in general, we should all be conscious of the code and dependencies we are pulling into our project.  We shouldn't require an IP review to learn that we pulled in something we shouldn't have.  Of course, things will be missed and that's why we need this check-and-balance.  But, in general, there shouldn't be any surprises.

When we are doing the major MicroProfile releases, then there is definitely no reason to do the IP reviews since all of the code will be reviewed by the MP release process.

If we are doing a component release out-of-band, then we can do what we feel is right.  Common sense.  If there are major changes being introduced and the component wants the comfort of doing an IP review, then go ahead and request one.  Since any IP review brings in all of the source code from the whole project, then that component will need to be ready to resolve all issues across the repos -- even if they are just in a state of development.

Since we plan on having regular releases of MicroProfile, I would suggest that we hold off on doing the IP reviews until we do the MicroProfile releases.  This approach will save everybody a lot of work.  But, as stated, exceptions can be made when necessary.

Thanks, Kevin