Singularity in Debian
66 views
Skip to first unread message
Afif Elghraoui
Jan 4, 2019, 10:26:54 PM1/4/19
to singu...@lbl.gov
Hi, list
I'm one of the co-maintainers of singularity-container in Debian
(through the Debian HPC team [1]). The official Debian package for
Singularity used to be kept up to date in the backports repositories
[2], but it's been deemed unsuitable for Debian Stable because it's
unlikely that we'll have security patches to apply to the version that
ends up there as time goes on [3].
Please be aware that, because of this, singularity-container is going to
get removed from Testing and no newer versions will enter backports. The
official package will only exist in Debian Unstable.
Yaroslav has been maintaining singularity-container separately in
NeuroDebian. None of that is affected by the above.
regards
Afif
1. https://wiki.debian.org/Teams/HPC
2. https://backports.debian.org
3. https://bugs.debian.org/917867
--
Afif Elghraoui | عفيف الغراوي
http://afif.ghraoui.name
I'm one of the co-maintainers of singularity-container in Debian
(through the Debian HPC team [1]). The official Debian package for
Singularity used to be kept up to date in the backports repositories
[2], but it's been deemed unsuitable for Debian Stable because it's
unlikely that we'll have security patches to apply to the version that
ends up there as time goes on [3].
Please be aware that, because of this, singularity-container is going to
get removed from Testing and no newer versions will enter backports. The
official package will only exist in Debian Unstable.
Yaroslav has been maintaining singularity-container separately in
NeuroDebian. None of that is affected by the above.
regards
Afif
1. https://wiki.debian.org/Teams/HPC
2. https://backports.debian.org
3. https://bugs.debian.org/917867
--
Afif Elghraoui | عفيف الغراوي
http://afif.ghraoui.name
David Dykstra
Jan 7, 2019, 2:33:39 PM1/7/19
to singu...@lbl.gov
Hi Arif,
It seems to me that there's only a real issue during this transition
period between 2.x and 3.x. Once 3.x is stable enough to become the
default for Debian, EPEL/Fedora, and OpenSuse, then any new security
problems can be addressed by simply updating to the latest 3.x or by the
distribution package supporter backporting the security fix to the
previous release, their choice. Perhaps we need to get the singularity
core team to agree they will support security updates for the latest
stable older version for a limited period of time during periods of
instability, and maybe that would be enough. What do you think? I
think the singularity team wants to be good member of the open source
community and so maybe they would agree to that modification to their
security support policy.
Dave
> Afif Elghraoui | ???????? ??????????????
> http://afif.ghraoui.name
>
> --
> You received this message because you are subscribed to the Google Groups "singularity" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.
It seems to me that there's only a real issue during this transition
period between 2.x and 3.x. Once 3.x is stable enough to become the
default for Debian, EPEL/Fedora, and OpenSuse, then any new security
problems can be addressed by simply updating to the latest 3.x or by the
distribution package supporter backporting the security fix to the
previous release, their choice. Perhaps we need to get the singularity
core team to agree they will support security updates for the latest
stable older version for a limited period of time during periods of
instability, and maybe that would be enough. What do you think? I
think the singularity team wants to be good member of the open source
community and so maybe they would agree to that modification to their
security support policy.
Dave
> http://afif.ghraoui.name
>
> --
> You received this message because you are subscribed to the Google Groups "singularity" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.
Afif Elghraoui
Jan 8, 2019, 12:08:02 AM1/8/19
to singu...@lbl.gov
Oh, this isn't about the current 2.x to 3.x transition period--there's
technically enough time to get 3.x into the next Stable. We can backport
patches, but the patch needs to exist and be identifiable somehow, and
that's what we can't be sure of. Because of that, we had to block it.
regards
Afif
على ١/٥/١٤٤٠ هـ ٢:٣٣ م، كتب David Dykstra:
technically enough time to get 3.x into the next Stable. We can backport
patches, but the patch needs to exist and be identifiable somehow, and
that's what we can't be sure of. Because of that, we had to block it.
regards
Afif
على ١/٥/١٤٤٠ هـ ٢:٣٣ م، كتب David Dykstra:
David Godlove
Jan 8, 2019, 12:57:12 AM1/8/19
to singularity
As I said in a PM to Afif, there is the small outside chance that we could run into a situation similar to the 2.x to 3.x transition totally within the 3.x series. Let's say we have released 3.1.0 and a security exploit is discovered for 3.0.0. If enough code has changed between 3.1.0 and 3.0.0 then the exploit may not affect 3.1.0. If that's the case, there is no explicit guarantee that a patch would be developed and published for 3.0.0.
Although in practice, the older version probably would be patched (and indeed we did just release a patch the 2.6 series even though it is technically not supported) there is no _explicit guarantee_ that an older version will be supported.
I think that is the reason that Afif and the gang have decided to drop Singularity from Stable. Right Afif?
Afif Elghraoui
Jan 8, 2019, 1:08:28 AM1/8/19
to singu...@lbl.gov
Yep, that's pretty much it.
على ٢/٥/١٤٤٠ هـ ١٢:٥٦ ص، كتب David Godlove:
على ٢/٥/١٤٤٠ هـ ١٢:٥٦ ص، كتب David Godlove:
David Dykstra
Jan 9, 2019, 4:15:39 PM1/9/19
to singu...@lbl.gov
Afif,
How is this different from other open source projects? What change to
the singularity team's security policy would be required to make it be
acceptable?
I think most open source projects would at least notify everyone if an
exploit is found in only older releases and advise everyone to upgrade
to the latest version as the fix. Is this not sufficient for Debian's
Stable? Is the assumption that any security fix has to be a minimum
change to the stable version, and not an upgrade to a version with new
functionality?
Are there other significant open source packages that stay only in
Debian Unstable; is this acceptable to users? I believe that it can be
a major pain for them, since it often requires newer versions of all
dependent libraries.
Is there an alternative widely used repository available? Red Hat
Enterprise Linux has EPEL, and they normally want to keep the first two
release numbers of packages to stay stable per OS version as well, but
exceptions can be granted there if a backport patch is not available.
Dave
On Tue, Jan 08, 2019 at 01:08:20AM -0500, Afif Elghraoui wrote:
> Yep, that's pretty much it.
>
>
How is this different from other open source projects? What change to
the singularity team's security policy would be required to make it be
acceptable?
I think most open source projects would at least notify everyone if an
exploit is found in only older releases and advise everyone to upgrade
to the latest version as the fix. Is this not sufficient for Debian's
Stable? Is the assumption that any security fix has to be a minimum
change to the stable version, and not an upgrade to a version with new
functionality?
Are there other significant open source packages that stay only in
Debian Unstable; is this acceptable to users? I believe that it can be
a major pain for them, since it often requires newer versions of all
dependent libraries.
Is there an alternative widely used repository available? Red Hat
Enterprise Linux has EPEL, and they normally want to keep the first two
release numbers of packages to stay stable per OS version as well, but
exceptions can be granted there if a backport patch is not available.
Dave
On Tue, Jan 08, 2019 at 01:08:20AM -0500, Afif Elghraoui wrote:
> Yep, that's pretty much it.
>
>
> David Godlove:
> > As I said in a PM to Afif, there is the small outside chance that we
> > could run into a situation similar to the 2.x to 3.x transition totally
> > within the 3.x series. Let's say we have released 3.1.0 and a security
> > exploit is discovered for 3.0.0. If enough code has changed between
> > 3.1.0 and 3.0.0 then the exploit may not affect 3.1.0. If that's the
> > case, there is no explicit guarantee that a patch would be developed and
> > published for 3.0.0.
> >
> > Although in practice, the older version probably would be patched (and
> > indeed we did just release a patch the 2.6 series even though it is
> > technically not supported) there is no _explicit guarantee_ that an
> > older version will be supported.
> >
> > I think that is the reason that Afif and the gang have decided to drop
> > Singularity from Stable. Right Afif?
> >
> > On Tue, Jan 8, 2019 at 12:08 AM Afif Elghraoui <af...@debian.org
> > <mailto:af...@debian.org>> wrote:
> >
> > Oh, this isn't about the current 2.x to 3.x transition period--there's
> > technically enough time to get 3.x into the next Stable. We can
> > backport
> > patches, but the patch needs to exist and be identifiable somehow, and
> > that's what we can't be sure of. Because of that, we had to block it.
> >
> > regards
> > Afif
> >
> > As I said in a PM to Afif, there is the small outside chance that we
> > could run into a situation similar to the 2.x to 3.x transition totally
> > within the 3.x series. Let's say we have released 3.1.0 and a security
> > exploit is discovered for 3.0.0. If enough code has changed between
> > 3.1.0 and 3.0.0 then the exploit may not affect 3.1.0. If that's the
> > case, there is no explicit guarantee that a patch would be developed and
> > published for 3.0.0.
> >
> > Although in practice, the older version probably would be patched (and
> > indeed we did just release a patch the 2.6 series even though it is
> > technically not supported) there is no _explicit guarantee_ that an
> > older version will be supported.
> >
> > I think that is the reason that Afif and the gang have decided to drop
> > Singularity from Stable. Right Afif?
> >
> > On Tue, Jan 8, 2019 at 12:08 AM Afif Elghraoui <af...@debian.org
> > <mailto:af...@debian.org>> wrote:
> >
> > Oh, this isn't about the current 2.x to 3.x transition period--there's
> > technically enough time to get 3.x into the next Stable. We can
> > backport
> > patches, but the patch needs to exist and be identifiable somehow, and
> > that's what we can't be sure of. Because of that, we had to block it.
> >
> > regards
> > Afif
> >
Afif Elghraoui
Jan 12, 2019, 5:38:28 PM1/12/19
to singu...@lbl.gov
على ٣/٥/١٤٤٠ هـ ٤:١٥ م، كتب David Dykstra:
> Afif,
>
> How is this different from other open source projects?
Other security-sensitive projects tend to have LTS branches that
>
> How is this different from other open source projects?
continue to be supported with only security udates for a specified
amount of time.
> What change to
> the singularity team's security policy would be required to make it be
> acceptable?
>
in Stable.
> I think most open source projects would at least notify everyone if an
> exploit is found in only older releases and advise everyone to upgrade
> to the latest version as the fix. Is this not sufficient for Debian's
> Stable? Is the assumption that any security fix has to be a minimum
> change to the stable version, and not an upgrade to a version with new
> functionality?
https://www.debian.org/security/faq#oldversion
>
> Are there other significant open source packages that stay only in
> Debian Unstable; is this acceptable to users? I believe that it can be
> a major pain for them, since it often requires newer versions of all
> dependent libraries.
ends up in Stable). But some people do use Unstable and maybe a
combination of Testing and Unstable, but you'd have to be vigilant and
know what you're doing.
>
> Is there an alternative widely used repository available? Red Hat
> Enterprise Linux has EPEL, and they normally want to keep the first two
> release numbers of packages to stay stable per OS version as well, but
> exceptions can be granted there if a backport patch is not available.
destined for the next Stable--to ensure an upgrade path between releases.
I'm not sure how readily an exception is granted for lack of a patch in
Debian, but it can cause disruption even if it is granted. Browsers are
given exceptions-- firefox-esr (which even still has major release
updates in Stable) and chromium, which can break packaged browser
extensions, etc.
I could ask specifically about getting potential exceptions for
Singularity, but it was never mentioned as a possibility when the
security team reached out.
regards
Afif
David Dykstra
Jan 15, 2019, 12:28:25 PM1/15/19
to Afif Elghraoui, singu...@lbl.gov
Hi Afif,
On Sat, Jan 12, 2019 at 05:38:20PM -0500, Afif Elghraoui wrote:
...
exceptions to that. The project I am most familiar with is squid, and
they don't have LTS branches. They support security updates for older
versions for a while after major upgrades, but mostly new releases come
to the latest stable version (and newer unstable versions if there is
one). They always say which versions are impacted in a security
announcement.
> > What change to
> > the singularity team's security policy would be required to make it be
> > acceptable?
>
> We would need to know that we'll have patches to backport to the version in
> Stable.
I don't think it necessarily needs to be that strong, based on the Q&A
you link below (more details on that point below). The singularity
security policy at
https://www.sylabs.io/singularity/security-policy/
already says they will always get a CVE for a vulnerability, and it
doesn't say that doesn't apply to older releases, so assume it does
apply. It also says that they will say in the announcement which
releases are affected. How about if they further say that they will
provide a patch for older versions if it is easy, but not promise to
create a new release based on non-current open source versions? I know
that squid has sometimes provided security patches for older versions
without making a new release.
Singularity team & Sylabs: are you willing to amend your security policy
to say that if the latest version is not vulnerable that you'll at least
provide a source code patch for the newest vulnerable release, if the
patch is small?
> > I think most open source projects would at least notify everyone if an
> > exploit is found in only older releases and advise everyone to upgrade
> > to the latest version as the fix. Is this not sufficient for Debian's
> > Stable? Is the assumption that any security fix has to be a minimum
> > change to the stable version, and not an upgrade to a version with new
> > functionality?
>
> All I know about that is what's written here:
> https://www.debian.org/security/faq#oldversion
That clearly allows for exceptions. It says
In some cases it is not possible to backport a security fix, for
example when large amounts of source code need to be modified or
rewritten. If that happens it might be necessary to move to a new
upstream version, but this has to be coordinated with the security
team beforehand.
That sounds quite a lot like what the singularity security policy is
saying. If they develop a fix for the latest release, and it would be
hard to do it for an older release, they could just recommend that
everyone upgrade to the latest version. If the Debian package provider
also thinks it is too much work to backport, he or she would need to ask
the Debian security team for an exception, and if they agreed it is too
hard they would grant it.
The Debian Q&A also says it is most important to maintain stability for
library APIs, which is not an issue with this package.
> > Are there other significant open source packages that stay only in
> > Debian Unstable; is this acceptable to users? I believe that it can be
> > a major pain for them, since it often requires newer versions of all
> > dependent libraries.
>
> I /think/ firefox is like that (as opposed to firefox-esr, which is what
> ends up in Stable). But some people do use Unstable and maybe a combination
> of Testing and Unstable, but you'd have to be vigilant and know what you're
> doing.
Yes it would be quite a burden on users to ask them to get singularity
only from Unstable. firefox doesn't sound like a good example because
there is firefox-esr.
> > Is there an alternative widely used repository available? Red Hat
> > Enterprise Linux has EPEL, and they normally want to keep the first two
> > release numbers of packages to stay stable per OS version as well, but
> > exceptions can be granted there if a backport patch is not available.
>
> We have stable-backports, but packages are only allowed there if they're
> destined for the next Stable--to ensure an upgrade path between releases.
This is a clear difference between the Debian and Fedora/EPEL/CentOS
communities, and I think it is unfortunate. Some packages that are in
Fedora stay in EPEL forever and never make it to RHEL/CentOS. I think
it would be good to have some mechanism for similar packages in Debian.
> I'm not sure how readily an exception is granted for lack of a patch in
> Debian, but it can cause disruption even if it is granted. Browsers are
> given exceptions-- firefox-esr (which even still has major release updates
> in Stable) and chromium, which can break packaged browser extensions, etc.
>
> I could ask specifically about getting potential exceptions for Singularity,
> but it was never mentioned as a possibility when the security team reached
> out.
Please do. Talk specifically about the situation for 3.x only. I don't
think there should be expectations for continued security support for
2.x for very long because of the difficulty of backporting fixes.
Backports between different 3.x levels when a security fix comes in a
new 3.x.y release should be much easier at that point in general because
of the common language. Of course this means that Debian packaging for
3.x needs to get completed first.
Dave
On Sat, Jan 12, 2019 at 05:38:20PM -0500, Afif Elghraoui wrote:
...
> > How is this different from other open source projects?
>
> Other security-sensitive projects tend to have LTS branches that continue to
> be supported with only security udates for a specified amount of time.
I don't think that's always the case; I would bet that there are
>
> Other security-sensitive projects tend to have LTS branches that continue to
> be supported with only security udates for a specified amount of time.
exceptions to that. The project I am most familiar with is squid, and
they don't have LTS branches. They support security updates for older
versions for a while after major upgrades, but mostly new releases come
to the latest stable version (and newer unstable versions if there is
one). They always say which versions are impacted in a security
announcement.
> > What change to
> > the singularity team's security policy would be required to make it be
> > acceptable?
>
> We would need to know that we'll have patches to backport to the version in
> Stable.
you link below (more details on that point below). The singularity
security policy at
https://www.sylabs.io/singularity/security-policy/
already says they will always get a CVE for a vulnerability, and it
doesn't say that doesn't apply to older releases, so assume it does
apply. It also says that they will say in the announcement which
releases are affected. How about if they further say that they will
provide a patch for older versions if it is easy, but not promise to
create a new release based on non-current open source versions? I know
that squid has sometimes provided security patches for older versions
without making a new release.
Singularity team & Sylabs: are you willing to amend your security policy
to say that if the latest version is not vulnerable that you'll at least
provide a source code patch for the newest vulnerable release, if the
patch is small?
> > I think most open source projects would at least notify everyone if an
> > exploit is found in only older releases and advise everyone to upgrade
> > to the latest version as the fix. Is this not sufficient for Debian's
> > Stable? Is the assumption that any security fix has to be a minimum
> > change to the stable version, and not an upgrade to a version with new
> > functionality?
>
> All I know about that is what's written here:
> https://www.debian.org/security/faq#oldversion
In some cases it is not possible to backport a security fix, for
example when large amounts of source code need to be modified or
rewritten. If that happens it might be necessary to move to a new
upstream version, but this has to be coordinated with the security
team beforehand.
That sounds quite a lot like what the singularity security policy is
saying. If they develop a fix for the latest release, and it would be
hard to do it for an older release, they could just recommend that
everyone upgrade to the latest version. If the Debian package provider
also thinks it is too much work to backport, he or she would need to ask
the Debian security team for an exception, and if they agreed it is too
hard they would grant it.
The Debian Q&A also says it is most important to maintain stability for
library APIs, which is not an issue with this package.
> > Are there other significant open source packages that stay only in
> > Debian Unstable; is this acceptable to users? I believe that it can be
> > a major pain for them, since it often requires newer versions of all
> > dependent libraries.
>
> I /think/ firefox is like that (as opposed to firefox-esr, which is what
> ends up in Stable). But some people do use Unstable and maybe a combination
> of Testing and Unstable, but you'd have to be vigilant and know what you're
> doing.
only from Unstable. firefox doesn't sound like a good example because
there is firefox-esr.
> > Is there an alternative widely used repository available? Red Hat
> > Enterprise Linux has EPEL, and they normally want to keep the first two
> > release numbers of packages to stay stable per OS version as well, but
> > exceptions can be granted there if a backport patch is not available.
>
> We have stable-backports, but packages are only allowed there if they're
> destined for the next Stable--to ensure an upgrade path between releases.
communities, and I think it is unfortunate. Some packages that are in
Fedora stay in EPEL forever and never make it to RHEL/CentOS. I think
it would be good to have some mechanism for similar packages in Debian.
> I'm not sure how readily an exception is granted for lack of a patch in
> Debian, but it can cause disruption even if it is granted. Browsers are
> given exceptions-- firefox-esr (which even still has major release updates
> in Stable) and chromium, which can break packaged browser extensions, etc.
>
> I could ask specifically about getting potential exceptions for Singularity,
> but it was never mentioned as a possibility when the security team reached
> out.
think there should be expectations for continued security support for
2.x for very long because of the difficulty of backporting fixes.
Backports between different 3.x levels when a security fix comes in a
new 3.x.y release should be much easier at that point in general because
of the common language. Of course this means that Debian packaging for
3.x needs to get completed first.
Dave
David Godlove
Jan 17, 2019, 9:26:17 PM1/17/19
to singularity
Hey Guys,
This is a good discussion. I spoke with Greg about this and we agreed that we should make one thing absolutely clear.
We are maintaining several LTS versions in SingularityPRO. (2.4, 2.5, 2.6, and 3.x in the future). This means that we will need to develop security patches for vulnerabilities discovered in any and all of these releases. We want to make clear, that Sylabs will NEVER keep these patches closed source. We will always release security patches that we've developed for LTS PRO versions.
Now, this does not necessarily mean there will be new point releases on deprecated series. It would require extra work for us to cut new releases on old versions and it would defeat the purpose of deprecating the release. And it's not even clear if we will actually apply the patches ourselves to the tips of the release branches in vault. If the PRO and open source series have diverged widely enough it may take extra work to reconcile conflicts and apply the patches. But we will always at least release them in one form or another so that they will be available to interested parties. That is no extra work on our end and it's the right thing to do.
Does that help assuage any fears you have about including Singularity in stable Afif?
Afif Elghraoui
Jan 18, 2019, 3:01:49 AM1/18/19
to singu...@lbl.gov
على ١١/٥/١٤٤٠ هـ ٩:٢٥ م، كتب David Godlove:
>
> Does that help assuage any fears you have about including Singularity in
> stable Afif?
Yes! We can backport patches as long as they exist :)
> Does that help assuage any fears you have about including Singularity in
> stable Afif?
Thanks for clearing this up. I guess I have some Go dependencies to
package now.
Afif
David Dykstra
Jan 18, 2019, 11:12:43 AM1/18/19
to singu...@lbl.gov
Thanks a lot for that clarification, Dave, that sounds great.
I was going to ask that someone update the security policy with this
info if needed, but re-reading point 8 again it already says "the
patches will be merged from the private development space into the
public repository" and does not limit that to only the latest version,
so the policy may already cover this. It also says "a release will
immediately be made", although it doesn't define what release means if a
vulnerability only applies to older versions. Maybe that would be good
to clarify.
Dave
I was going to ask that someone update the security policy with this
info if needed, but re-reading point 8 again it already says "the
patches will be merged from the private development space into the
public repository" and does not limit that to only the latest version,
so the policy may already cover this. It also says "a release will
immediately be made", although it doesn't define what release means if a
vulnerability only applies to older versions. Maybe that would be good
to clarify.
Dave
Dave Godlove
Jan 23, 2019, 5:18:38 PM1/23/19
to singularity
Heya Dave,
Sorry for the delay. We made an edit to the security page. After the numbered list identifying the security procedure we've added a note to clarify that all patches will be made public. Does that make sense?
Dave Godlove
Engineering Coordinator, Sylabs Inc.
David Dykstra
Jan 24, 2019, 4:23:53 PM1/24/19
to singu...@lbl.gov
That works, thanks a lot.
Dave
Dave
Reply all
Reply to author
Forward
0 new messages