Skip to first unread message
David Blevins
Oct 6, 2020, 7:45:23 PM10/6/20
to Micro Profile
A lot of the staging repository capabilities in Nexus Pro were added at the request of Apache. The background there is Apache has always had the concept of holding binaries in a temporary location for a community vote and these features had to be added before Apache could agree to use the software.
Here's an example walkthrough using a recent release (8.0.0-M3) that took a few tries.
# First vote
- "[VOTE] TomEE 8.0.0-M3 (staging-1135)"
https://lists.apache.org/thread.html/f41909414db91fb946d82bfe17045f6a45c46758435b112e4fcf21a2%40%3Cdev.tomee.apache.org%3E
In this email there's a staging repo to `https://repository.apache.org/content/repositories/orgapachetomee-1135/` and another link to a `https://dist.apache.org/repos/dist/dev/tomee/staging-1135/tomee-8.0.0-M3`
The first link is the primary thing we should focus on and is the Nexus Pro staging repository. Nexus Pro will create one of these automatically for you when you start uploading. However, the contents of the staging repo are not made public until the repo is closed. Once closed the staging repository is now immutable.
The reason this is important is that once a set of binaries (a staging repo) goes up for vote it should *not* be modified.
On our side of the fence we've had enough votes fail that we often put the staging repo name in the vote thread from the start. It helps keep people from voting on old vote threads and generally reduce confusion. This particular vote ended up getting withdrawn to address feedback.
# Second vote
- "[VOTE] TomEE 8.0.0-M3 (staging-1136) - take two"
https://lists.apache.org/thread.html/22c2dbea944dbfac78b229389f8b0dc49190558507123c9f2a40db8a%40%3Cdev.tomee.apache.org%3E
In the second vote attempt we have now a new staging repository at `https://repository.apache.org/content/repositories/orgapachetomee-1136/`
The new `orgapachetomee-1136` was created the same way the previous `orgapachetomee-1135` was created; full release build including tagging git. At this point in time the `orgapachetomee-1135` staging repository is still visible and people can refer to it if they want to compare the new proposed `orgapachetomee-1136` binaries against the old `orgapachetomee-1135` binaries. It's also important to reflect and see how the immutability is very nice as if the `orgapachetomee-1135` binaries were still mutable, odds are someone someday would cut a corner and just tweak them, creating some very ambiguous monkey patching scenario. That's not possible since you can only share immutable binaries. There is no question on "how do we tell them apart" or doubt that two people visited the same URL on different days and saw different things.
Now, it's also important to understand that Nexus Pro does have a "virtual" staging repository of sorts that is the aggregate of all staging repositories. This is what we have been using on the Jakarta side. The trick is that everything available at this `virtual` staging repository URL is volatile and transient. As an aggregate of all staging repositories, it's also an aggregate of all failed attempts. What this means is you'll see all of the `orgapachetomee-1135` binaries with the `orgapachetomee-1136` binaries overlaid on top. If `orgapachetomee-1135` had a file it should not have and `orgapachetomee-1136` ensured that file was not present when it was published, the virtual staging directory will still show the deleted file and the only possible clue you'd have that it shouldn't be there is the older date stamp.
You can of course have a policy to delete the old staging repos immediately to avoid confusion around the virtual staging repository, but it does not fix that the virtual staging repository is non-deterministic and volatile. Any new attempts from anyone will keep overlaying and the contents of the virtual staging repository URL will change. Virtual staging repository URLs are mutable.
# That second URL?
You might have wondered what that second "Binaries and Source" URL was that you've seen in the votes. Apache has a binary distribution and mirror system that predates Maven Central and it's a requirement that projects use it.
For example the final TomEE 8.0.0-M3 zip/tar binaries and every zip/tar we've ever released are here: http://archive.apache.org/dist/tomee/tomee-8.0.0-M3/ What lives here will be the select binaries people find most interesting, like the server zips/tars and as well as the full source zip of the entire git or svn tag that built those binaries. It's an Apache requirement that you are able to build the published binaries from a published source zip; note those "-source.jar" files in Maven Central do not cut it as they are not buildable.
When we stage this full-repo source zip and select "interesting" binaries we have a convention to use the Nexus Pro staging repo name in the URL path. So the first and second URLs are distinguishable:
- https://dist.apache.org/repos/dist/dev/tomee/staging-1135/tomee-8.0.0-M3
- https://dist.apache.org/repos/dist/dev/tomee/staging-1136/tomee-8.0.0-M3
It is not Nexus Pro doing this work and these URLs *are* mutable, but in practice we do not change them and treat them just as Nexus Pro would. Though it is an on-your-honor system and requires humans to do the right thing, we find that making the Nexus Pro staging number the universal "our release attempt" number and reflecting it everywhere (vote threads, all staged directories) helps keep everyone on the same page. Where there is less confusion there are fewer mistakes. We have also since automated creating those "out of nexus" staging directories and have a command line tool that will download the select "interesting" binaries from the actual Nexus Pro staging repos and put them into that similarly-named directory.
If we did decide we wanted to push some files to https://download.eclipse.org/microprofile/, my recommendation would be we stage them using the same approach: we stage to Nexus Pro and pull anything that needs to get published to download.eclipse.org in some sort of `https://download.eclipse.org/microprofile/staging/orgeclipsemicroprofile-1234` directory during the vote.
# Publishing a release
Once a vote goes final you just need to got to Nexus Pro, select the staging repo and say "release." When that is done the contents are pushed to Maven Central and the staging repo is deleted. Any other staging repos from past attempts can either get deleted manually or stay there until Nexus expires them automatically after 30 days.
Similarly, the manually handled dist.apache.org staged binaries are moved to the permanent location via a simple copy of all contents inside the `staging-1136/*` directory to the root TomEE dist directory. I.e. we take everything under `staging-1136` and move it to where it needs to go then we delete the now empty `staging-1136` directory.
Hope some of this is useful for our consideration in the MicroProfile Specification Process voting/releasing.
The short version of the idea is that we could put all these staging links into a Github Issue and use that to track progress and prepare the materials for vote. This would include putting the staged repo name/number in the actual ballot.
--
David Blevins
http://twitter.com/dblevins
http://www.tomitribe.com
310-633-3852
Here's an example walkthrough using a recent release (8.0.0-M3) that took a few tries.
# First vote
- "[VOTE] TomEE 8.0.0-M3 (staging-1135)"
https://lists.apache.org/thread.html/f41909414db91fb946d82bfe17045f6a45c46758435b112e4fcf21a2%40%3Cdev.tomee.apache.org%3E
In this email there's a staging repo to `https://repository.apache.org/content/repositories/orgapachetomee-1135/` and another link to a `https://dist.apache.org/repos/dist/dev/tomee/staging-1135/tomee-8.0.0-M3`
The first link is the primary thing we should focus on and is the Nexus Pro staging repository. Nexus Pro will create one of these automatically for you when you start uploading. However, the contents of the staging repo are not made public until the repo is closed. Once closed the staging repository is now immutable.
The reason this is important is that once a set of binaries (a staging repo) goes up for vote it should *not* be modified.
On our side of the fence we've had enough votes fail that we often put the staging repo name in the vote thread from the start. It helps keep people from voting on old vote threads and generally reduce confusion. This particular vote ended up getting withdrawn to address feedback.
# Second vote
- "[VOTE] TomEE 8.0.0-M3 (staging-1136) - take two"
https://lists.apache.org/thread.html/22c2dbea944dbfac78b229389f8b0dc49190558507123c9f2a40db8a%40%3Cdev.tomee.apache.org%3E
In the second vote attempt we have now a new staging repository at `https://repository.apache.org/content/repositories/orgapachetomee-1136/`
The new `orgapachetomee-1136` was created the same way the previous `orgapachetomee-1135` was created; full release build including tagging git. At this point in time the `orgapachetomee-1135` staging repository is still visible and people can refer to it if they want to compare the new proposed `orgapachetomee-1136` binaries against the old `orgapachetomee-1135` binaries. It's also important to reflect and see how the immutability is very nice as if the `orgapachetomee-1135` binaries were still mutable, odds are someone someday would cut a corner and just tweak them, creating some very ambiguous monkey patching scenario. That's not possible since you can only share immutable binaries. There is no question on "how do we tell them apart" or doubt that two people visited the same URL on different days and saw different things.
Now, it's also important to understand that Nexus Pro does have a "virtual" staging repository of sorts that is the aggregate of all staging repositories. This is what we have been using on the Jakarta side. The trick is that everything available at this `virtual` staging repository URL is volatile and transient. As an aggregate of all staging repositories, it's also an aggregate of all failed attempts. What this means is you'll see all of the `orgapachetomee-1135` binaries with the `orgapachetomee-1136` binaries overlaid on top. If `orgapachetomee-1135` had a file it should not have and `orgapachetomee-1136` ensured that file was not present when it was published, the virtual staging directory will still show the deleted file and the only possible clue you'd have that it shouldn't be there is the older date stamp.
You can of course have a policy to delete the old staging repos immediately to avoid confusion around the virtual staging repository, but it does not fix that the virtual staging repository is non-deterministic and volatile. Any new attempts from anyone will keep overlaying and the contents of the virtual staging repository URL will change. Virtual staging repository URLs are mutable.
# That second URL?
You might have wondered what that second "Binaries and Source" URL was that you've seen in the votes. Apache has a binary distribution and mirror system that predates Maven Central and it's a requirement that projects use it.
For example the final TomEE 8.0.0-M3 zip/tar binaries and every zip/tar we've ever released are here: http://archive.apache.org/dist/tomee/tomee-8.0.0-M3/ What lives here will be the select binaries people find most interesting, like the server zips/tars and as well as the full source zip of the entire git or svn tag that built those binaries. It's an Apache requirement that you are able to build the published binaries from a published source zip; note those "-source.jar" files in Maven Central do not cut it as they are not buildable.
When we stage this full-repo source zip and select "interesting" binaries we have a convention to use the Nexus Pro staging repo name in the URL path. So the first and second URLs are distinguishable:
- https://dist.apache.org/repos/dist/dev/tomee/staging-1135/tomee-8.0.0-M3
- https://dist.apache.org/repos/dist/dev/tomee/staging-1136/tomee-8.0.0-M3
It is not Nexus Pro doing this work and these URLs *are* mutable, but in practice we do not change them and treat them just as Nexus Pro would. Though it is an on-your-honor system and requires humans to do the right thing, we find that making the Nexus Pro staging number the universal "our release attempt" number and reflecting it everywhere (vote threads, all staged directories) helps keep everyone on the same page. Where there is less confusion there are fewer mistakes. We have also since automated creating those "out of nexus" staging directories and have a command line tool that will download the select "interesting" binaries from the actual Nexus Pro staging repos and put them into that similarly-named directory.
If we did decide we wanted to push some files to https://download.eclipse.org/microprofile/, my recommendation would be we stage them using the same approach: we stage to Nexus Pro and pull anything that needs to get published to download.eclipse.org in some sort of `https://download.eclipse.org/microprofile/staging/orgeclipsemicroprofile-1234` directory during the vote.
# Publishing a release
Once a vote goes final you just need to got to Nexus Pro, select the staging repo and say "release." When that is done the contents are pushed to Maven Central and the staging repo is deleted. Any other staging repos from past attempts can either get deleted manually or stay there until Nexus expires them automatically after 30 days.
Similarly, the manually handled dist.apache.org staged binaries are moved to the permanent location via a simple copy of all contents inside the `staging-1136/*` directory to the root TomEE dist directory. I.e. we take everything under `staging-1136` and move it to where it needs to go then we delete the now empty `staging-1136` directory.
Hope some of this is useful for our consideration in the MicroProfile Specification Process voting/releasing.
The short version of the idea is that we could put all these staging links into a Github Issue and use that to track progress and prepare the materials for vote. This would include putting the staged repo name/number in the actual ballot.
--
David Blevins
http://twitter.com/dblevins
http://www.tomitribe.com
310-633-3852
Scott Stark
Oct 7, 2020, 3:59:06 PM10/7/20
to MicroProfile
What is the corresponding url in the microprofile staging repo server? If I look at the last release build job:
it shows an orgeclipsemicroprofile-1346 repository id, but I don't see valid repository on the oss.sonatype.org server. For example, the following URL:
results in a 404 error. What would be the correct repository URL?
Regarding https://download.eclipse.org/microprofile/, we use that as a very flat directory currently. There are probably 100 directories in there corresponding to individual and platform spec releases. We should impose some hierarchy there. Given the staging usecase, perhaps:
staging/
releases/
snapshots/
with further subdirectories corresponding to the platform x.y version.
David Blevins
Oct 7, 2020, 7:40:53 PM10/7/20
to Micro Profile
> On Oct 7, 2020, at 12:59 PM, Scott Stark <sst...@redhat.com> wrote:
>
> What is the corresponding url in the microprofile staging repo server? If I look at the last release build job:
> https://ci.eclipse.org/microprofile/job/MicroProfile%20Releases/445/console
>
> it shows an orgeclipsemicroprofile-1346 repository id, but I don't see valid repository on the oss.sonatype.org server. For example, the following URL:
> https://oss.sonatype.org/content/repositories/orgeclipsemicroprofile-1346
>
> results in a 404 error. What would be the correct repository URL?
Once the staging repository is released, the contents are pushed to Maven Central and the staging repository is deleted.
>
> What is the corresponding url in the microprofile staging repo server? If I look at the last release build job:
> https://ci.eclipse.org/microprofile/job/MicroProfile%20Releases/445/console
>
> it shows an orgeclipsemicroprofile-1346 repository id, but I don't see valid repository on the oss.sonatype.org server. For example, the following URL:
> https://oss.sonatype.org/content/repositories/orgeclipsemicroprofile-1346
>
> results in a 404 error. What would be the correct repository URL?
So you have the right URL, just that release process is over.
>
> Regarding https://download.eclipse.org/microprofile/, we use that as a very flat directory currently. There are probably 100 directories in there corresponding to individual and platform spec releases. We should impose some hierarchy there. Given the staging usecase, perhaps:
>
> staging/
> releases/
> snapshots/
> with further subdirectories corresponding to the platform x.y version.
One thing to keep in mind is that if we move things all the URLs will break and there is no way we ourselves can do a redirect. I suspect Eclipse Infra can do it, but it would need to be a coordinated effort.
So perhaps shortest path is to leave everything the way it is and add a `staging/` directory at the root. Long-term we could move everything into a directory structure like you suggest and work to get the right redirects in place. I personally wouldn't be eager to tackle the reorg/redirect, but if someone wanted to do it I'd be ok with it.
-David
Scott Stark
Oct 20, 2020, 8:59:59 AM10/20/20
to MicroProfile
I have updated the https://github.com/eclipse/microprofile/pull/193 PR to be an issue template that can deal with any of the project lifecycle reviews and incorporated the simplifications discussed.
John Clingan
Oct 22, 2020, 3:10:29 PM10/22/20
to MicroProfile
Having gone through MP releases, I don't recall seeing a "Close" option. I could easily be wrong, of course. I think the two options are "remove" and "release". Does "closing a repo and making it public before releasing" require a commercial license?
Also, I think we need a state diagram, LOL.
Scott Stark
Oct 23, 2020, 10:48:36 AM10/23/20
to microp...@googlegroups.com
The next release done in the https://ci.eclipse.org/microprofile/job/MicroProfile%20Releases/ job should leave it in the staged state so that we can investigate its usage. I still have not been able to find an MP repo in a staged state.
--
You received this message because you are subscribed to a topic in the Google Groups "MicroProfile" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/microprofile/azQZccjYTPE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to microprofile...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/93c06ad2-66a2-46a1-83e9-821c2abf6dbao%40googlegroups.com.
Emily Jiang
Oct 29, 2020, 12:47:47 PM10/29/20
to MicroProfile
Further to the call, MP config team took the action of releasing
MicroProfile Config 2.0-RC2 using the approach put forward by David.
However, after the Jenkins release, I tried to click the button "close"
in sonatype, but the button is greyed out. The status says it is closed.
David,
is this to do with the access right? John/Kevin, can you log into
Sonatype to see whether the "close" button is enabled for you.
Thanks
Emily
John Clingan
Nov 2, 2020, 11:05:42 AM11/2/20
to MicroProfile
It looks like it is closed now. Any idea how this was accomplished? Wasn't me :-)
Emily Jiang
Nov 2, 2020, 11:58:35 AM11/2/20
to MicroProfile
John, it is not just you. It says "closed" when I logged in even though I have not clicked any buttons. I think once a Jenkins release is done, the staging repo says closed. We can only "Release" this to maven central by clicking the "Release" button. The issue is that the artifacts are not under the "Staging" subdirectory.
To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/9b8e76df-5b18-4772-8323-ca65a50faa9bo%40googlegroups.com.
--
Thanks
Emily
Emily
John Clingan
Nov 2, 2020, 3:51:08 PM11/2/20
to MicroProfile
Gotcha. IIRC, the last MP 4-RC1 run Edwin, Emerson, and I did ask if we wanted to auto-close. I thought there was a prompt. There was a step in there, IIRC.
To unsubscribe from this group and all its topics, send an email to microprofile+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/9b8e76df-5b18-4772-8323-ca65a50faa9bo%40googlegroups.com.
--Thanks
Emily
Emily Jiang
Nov 2, 2020, 5:24:03 PM11/2/20
to MicroProfile
I didn't see the prompt for auto-close when kicking off Jenkin build. Since we want to "close" the staging repo, would the repo be under the staging directory already based on what David said IIUC. However, the artifacts are not under staging dir.
To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/9b8e76df-5b18-4772-8323-ca65a50faa9bo%40googlegroups.com.
--Thanks
Emily
Roberto Cortez
Nov 5, 2020, 7:39:44 AM11/5/20
to microp...@googlegroups.com
The close of the staging repo is to make sure no other artefacts are pushed to the staging, making re repository immutable. The close is done automatically by the release build when all the artefacts of the build are pushed to the repo.
Moving forward, I did a change in the Config build to also publish the spec files. Here is a sample:
The specs are still being copied to the Eclipse Downloads section:
If we want to only copy the spec files after the staging repo is approved, then we need to split the release in two steps in the build script. Alternatively, we could leave it as is, because it seems that the download link is not published anywhere. For instance the main downloads area does not reference it in anyway: https://projects.eclipse.org/projects/technology.microprofile/downloads, so the link could be made public if the repo is approved.
Thoughts?
Cheers,
Roberto
You received this message because you are subscribed to the Google Groups "MicroProfile" group.
To unsubscribe from this group and stop receiving emails from it, send an email to microprofile...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/c934103b-70c3-4abd-b2b8-1e7d12313720n%40googlegroups.com.
Roberto Cortez
Nov 5, 2020, 11:18:45 AM11/5/20
to microp...@googlegroups.com
I’ve just created a Dry Run issue for MP Config Spec Review:
This is not the official / final one, but to try out the process and collect feedback.
Cheers,
Roberto
To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/ACD39D93-B360-4CBB-9200-2ED961A3EB83%40yahoo.com.
Emily Jiang
Nov 5, 2020, 12:18:18 PM11/5/20
to MicroProfile
Hi Roberto,
Thanks you so much for getting this implemented! I know you have been working around the clock to get it done! Thanks for your dedication! As for the stopping pushing artificats to download, personally I think it is not a big deal and would like to leave as it is. We are voting against staging repo, which is most important thing.
By the way, I moved the issue you created for Config here so that we can start working on the microprofile-wg and kick off the review process.
Thanks
Emily
Scott Stark
Nov 10, 2020, 2:51:18 PM11/10/20
to MicroProfile
That looks good to me. I'll take a look at starting work on the next rev of a promotion script that just takes the staging repo id and then both releases the next repo to maven central and then copies the staged contents to a microprofile project downloads area. I can work with the JWT project to test this.
Sergey Beryozkin
Nov 10, 2020, 3:36:54 PM11/10/20
to microp...@googlegroups.com
Hi Scott
MP JWT 1.2-RC2 has just made it to Central, so the next one is RC3
Thanks, Sergey
> To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/c3a01111-121a-4741-84ec-e4c5be1da8e5n%40googlegroups.com.
MP JWT 1.2-RC2 has just made it to Central, so the next one is RC3
Thanks, Sergey
Roberto Cortez
Nov 10, 2020, 8:12:42 PM11/10/20
to microp...@googlegroups.com
Maybe we can make it a little easier. I’ve already implemented the change to use the staging prefix in the downloads area:
So maybe just a job that moves the folder is enough?
To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/CAMsYBfX0GaieFF2r6QPCGYB84sCrjkgcbfPPMxe3MMe-50%3DD0A%40mail.gmail.com.
Reply all
Reply to author
Forward
0 new messages