Community standardization effort for library:// support

[Dave Dykstra]

Hello all,

The Apptainer community announcement
https://apptainer.org/news/community-announcement-20211130/
included the following:

As part of this transition, only open community standard interfaces
will be supported. As of right now, this will include removing the
"Library" and "Remote Builder" support. However it would be
preferable if these become open community maintained standards (not
corporate controlled) so we can leave them intact and/or re-add them
at a later date.

In order to keep one or both of those interfaces, we need to have some
community-maintained standard, because the Linux Foundation told us that
they only want their projects to use protocols that have open standards.
The Apptainer Technical Steering Committee (TSC) is especially interested
in keeping "Library" support, which shows up as the library:// URL in
commands, because it is widely used and because there are already open
server implementations. The Apptainer project implements the client
side, but an open standard is needed on the server side. There are two
open implementations of the server side
https://github.com/singularityhub/sregistry
https://github.com/csf-ngs/hinkskalle
but the problem is that there's no community agreement on maintaining
the server API, to make sure that no incompatible changes are made
without wide agreement.

This does not need to take much effort. For example, it would be
sufficient for a group of 3 or more people from different organizations
to simply declare one of the above as a reference implementation and to
write down and make public in some kind of charter the process it would
take to change the server API with group approval. It would require the
cooperation of the owner of the server implementation, of course.

I am willing to join a group and help to draft the charter. Who else
is interested? I am hoping that this can be arranged quickly so
library:// support can be retained in Apptainer 1.0.0. Please reply
here and include any questions you may have.

Dave
TSC member

[v]

I'd be happy to put together a spec based on what I developed for Singularity Registry Server - it's all public in the code to implement the original client so I can probably get most of it right, and then we don't need to remove the library:// endpoint.

Will work on this soon!

--
You received this message because you are subscribed to the Google Groups "singularity" group.
To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.
To view this discussion on the web visit https://groups.google.com/a/lbl.gov/d/msgid/singularity/Ybpt3TTJfRO/lru7%40fnal.gov.

[v]

If anyone else has implementations (client or registry) that they know of (not listed there) please send them my way!

[Dave Dykstra]

Vanessa,

That would be awesome if you were able to make a spec quickly! That
would be an even clearer method of defining the API than declaring one
implementation as a reference implementation, although a bit more work.

Then I assume you would be willing to join me in an organization that
approves future changes to the spec. All we would need is at least one
more person and the charter (which as I said I am willing to draft).
Any other volunteers? Anybody who has an interest in seeing that the
protocol stays compatible, and/or an interest in seeing that Apptainer
continues to support library://.

I suggest we plan on putting the documents (the spec and the charter) in
a new git repo under https://github.com/singularityhub.

Do you have a suggestion for the group name? I was thinking maybe the
"Singularityhub Library Protocol Consortium?"

Dave

> To view this discussion on the web visit https://groups.google.com/a/lbl.gov/d/msgid/singularity/CAM%3Dpu%2B%2BASBdTexLZw0xM2krNVhHTtHLY31Zq_64C_hJPbK9FhQ%40mail.gmail.com .

[v]

I was just going to call it singularityhub/library-api, if that works!

To view this discussion on the web visit https://groups.google.com/a/lbl.gov/d/msgid/singularity/Ybp%2BvzUWEEcSRV47%40fnal.gov.

[Dave Dykstra]

Sure, that works as a repo name.

Dave

> To view this discussion on the web visit https://groups.google.com/a/lbl.gov/d/msgid/singularity/CAM%3Dpu%2BKJ07Dk-AAxidggUT0OvRmKJ2r%2BCRQQkmsZroHbod%2BpLg%40mail.gmail.com .

[v]

Done.

https://twitter.com/vsoch/status/1471384975815548933

To view this discussion on the web visit https://groups.google.com/a/lbl.gov/d/msgid/singularity/YbqhCHTQGLiAUI8X%40fnal.gov.

[Dave Dykstra]

Vanessa, I am in awe. You are absolutely amazing. Thank you!

I will submit a community charter PR today.

Dave

> To view this discussion on the web visit https://groups.google.com/a/lbl.gov/d/msgid/singularity/CAM%3Dpu%2BKORNKaoM%3D0PGpDP3EZ_QC7_eS1cYn%3Dap32QMEUqqKCOA%40mail.gmail.com .

[Gregory M. Kurtzer]

Thank you so much Dave and Vanessa for standing this up and making the Container Library API an open standard for the community!

Good things are coming. Good things!

To view this discussion on the web visit https://groups.google.com/a/lbl.gov/d/msgid/singularity/YbtUK0KXsEXfYxcr%40fnal.gov.

--

Gregory M. Kurtzer

CEO, CIQ: http://ctrliq.com

Director, The Rocky Enterprise Software Foundation: http://www.rockylinux.org

[Dave Dykstra]

The community charter proposal is now at
https://github.com/singularityhub/library-api/pull/1

We still need at least one more volunteer that is willing to review
changes to the specification.

Dave

[v]

It also would be good to get others that have implemented it to review what I wrote in case I missed something.

To view this discussion on the web visit https://groups.google.com/a/lbl.gov/d/msgid/singularity/Ybu6eWoqcyjB5vyI%40fnal.gov.

[David Godlove]

I'm willing to be the 3rd member on this effort just to get the ball rolling and ensure we can keep the library:// directive in the client.  But I don't know how much time I have to review implementation and so forth.  If anyone else can help with this I'd be much obliged.  

To view this discussion on the web visit https://groups.google.com/a/lbl.gov/d/msgid/singularity/CAM%3Dpu%2BKuL54s7W4qEOWXJCAUx%2BAm7rNqoq0q7w-Uhr_%2B6w_PgQ%40mail.gmail.com.

[Dave Dykstra]

Thanks a lot, Dave! I added you to the charter in PR #1.

Dave

> > https://groups.google.com/a/lbl.gov/d/msgid/singularity/CAM%3Dpu%2BKuL54s7W4qEOWXJCAUx%2BAm7rNqoq0q7w-Uhr_%2B6w_PgQ%40mail.gmail.com
> > <https://groups.google.com/a/lbl.gov/d/msgid/singularity/CAM%3Dpu%2BKuL54s7W4qEOWXJCAUx%2BAm7rNqoq0q7w-Uhr_%2B6w_PgQ%40mail.gmail.com?utm_medium=email&utm_source=footer >

> > .
> >
>
> --
> You received this message because you are subscribed to the Google Groups "singularity" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.

> To view this discussion on the web visit https://groups.google.com/a/lbl.gov/d/msgid/singularity/CAN9aCed5if7xgBNw8qm_QTYv5h-cnrtikpD_DAU3Ap2w9tL2_A%40mail.gmail.com .

[v]

Thanks (another Dave!)

This was originally done by Sylabs, the company that Greg founded for the open source software Singularity, and originally created by them, so three maintainers from different orgs is not sufficient - we need at least one representative from Sylabs to be here as well. What ethically would not seem OK to me is to “take over” their spec without including them and then try to control their implementation.

To view this discussion on the web visit https://groups.google.com/a/lbl.gov/d/msgid/singularity/YbzJMlA0Vu/wStgC%40fnal.gov.

[Dave Dykstra]

I totally agree it's best for Sylabs to be involved. The good news for
those not on the SingularityCE mailing list is that Adam Hughes from
Sylabs has also just agreed to be on the list of maintainers, so we are
good to go.

Dave

> To view this discussion on the web visit https://groups.google.com/a/lbl.gov/d/msgid/singularity/CAM%3Dpu%2BJQY3b6c6Pb8w8%3DAQC4w7KOCJZYywPqOHYzrQJHTnu%2BFQ%40mail.gmail.com .

[v]

Yay!

To view this discussion on the web visit https://groups.google.com/a/lbl.gov/d/msgid/singularity/YbzUtaFOt2Ua%2BGqE%40fnal.gov.

[Gregory M. Kurtzer]

This is amazing news and I'd like to personally thank Vanessa and DaveD for spearheading this effort, DaveG for jumping in, and of course Adam who brings his experience as an original architect of the Sylabs Library product.

Many users in the open source community will appreciate the Library API being a collaborative open standard, not only for Apptainer and the Singularity Registry, but now other projects (and products) can be based on this open API.

Much gratitude for this initiative and have an excellent holiday season!

To view this discussion on the web visit https://groups.google.com/a/lbl.gov/d/msgid/singularity/CAM%3Dpu%2BL1jEgr47tRhTnre-GJ0YeDE8ys4EJEV_4ahFj2EYX9Vw%40mail.gmail.com.