RFC: DSSE for Digital Signatures

[Adam Hughes]

Hi everyone!

I'm looking for some feedback on a proposed extension to the SIF format to include support for digital signatures using non-PGP key material within SingularityCE.

As background, digital signatures in SIF currently use the cleartext signed message format defined in RFC4880 §7. This works well for current singularity sign / singularity verify commands since they only support PGP, but does not allow key material from alternative sources to be used (ex. https://github.com/sylabs/singularity/issues/1095).

While looking to add support for embedded attestations (https://github.com/sylabs/sif/issues/230), I came across the Dead Simple Signing Envelope (DSSE) format, which is the underlying format used by projects like Sigstore and Syft to store in-toto attestations. As part of that work, I have added a DSSE encoder/decoder that can be used for signed attestations. For attestations, this seems like a no-brainer, since other projects in the space are already leveraging DSSE, and interoperability is desired.

With all that being said, I'm looking for feedback on using DSSE more generally within SIF... specifically as the on-disk format for digital signatures using non-PGP key material. This could be used to implement sign/verify with X.509 certificates (https://github.com/sylabs/singularity/issues/1095) but is flexible enough to be leveraged with other sources of key material as well. As Dave Dykstra has pointed out, there is an x.509 proposal with a branch of a fork of SIF, which adds support for x.509 using a PEM-based format. This should be weighed as a potential alternative to using DSSE for digital signatures.

My personal thoughts, having taken some time to compare the two approaches:

• The PEM-based approach is (at the moment) specific to x.509. DSSE is able to support a variety of key material sources, and there are mature Go modules that do so today (OpenSSL keys, cloud KMS, etc.).
• The PEM-based approach expands the integrity package public API to include x.509-specific methods to parse and verify keys, certificates, etc. I don't believe any x.509-specific code is required in SIF with the DSSE approach, due to the well thought out interfaces provided by the Sigstore team.
• SIF is likely to support embedded digital attestations in the near future, and DSSE is prevalent in that space. Re-using DSSE for digital signatures should reduce maintenance by supporting two signature formats (PGP cleartext, DSSE) instead of three (PGP cleartext, PEM-based, DSSE).
• The maturity of the PEM-based approach is unclear. Some of the work made it into main branch of the fork, but following discussion in https://github.com/apptainer/sif/pull/149 the remainder is currently housed in a separate x509 branch.

I'd love to hear others' views on this subject, either here or in Slack. It's clear that there is a lot of interest at the user level for this functionality!

Thanks,

Ad

[Fotis Nikolaidis]

Hi Adam,

Author of X509 patch (for Apptainer) here.
I drop some comments to fuel the discussion.

Signing images with X509 practically consists of two discrete steps:
- Use the signer certificate to sign the image.
- Validate that the signer's certificate is trustworthy (signed by a trusted entity, is not expired, and is not revoked).

The former is straightforward.
The latter, as described in RFC https://datatracker.ietf.org/doc/html/rfc5280, requires inspecting the "IssuingCertificateURL" of the signer
and recursively download and validate all the certificates in the signing chain (e.g., Root CA -> Institute CA -> Lab CA -> User Certificate).

For a long time, the certificates were expected to be in the PEM format, described in RFC https://www.rfc-editor.org/rfc/rfc1423.html.
However, some PEM-related interfaces are being deprecated from Golang's crypto libraries lately (e.g., https://pkg.go.dev/crypto/x509).

The only remaining PEM-related interface requires the system certificates (e.g.,/etc/ssl)  to be in PEM format. This is reasonable as it would break existing installations.

If I understand correctly, DSSE is an alternative solution to PEM, and in-toto attestations are an alternative to x509.

However, none of them are RFC-based, so I wonder if the slow-moving HPC community is likely to adopt them.

Additionally, I do not see in-toto attestations to support online revocation checks.

Without these checks, there is no way to protect sif images against malicious (e.g., leaked) certificates.

To conclude, the question is, "do we need online revocation checks or not ?"
If we do *NOT* need it,  DSSE + in-toto should be ok.
if we need it,  the PEM + X509 is the industry standard and we should follow it.

Cheers,
Fotis

[Adam Hughes]

Hello Fotis, thanks for joining the conversation!

I think we've got two issues in play, and it's confusing the discussion:

1. How is key material handled?
2. How is the signed metadata stored within the SIF?

I believe your response is targeting the first question, and I don't disagree with any of what you've said there. Support for X.509 certificates is something that users are requesting in the SingularityCE community, and from the sounds of it, Apptainer as well. I would certainly expect that any implementations of X.509 will be able to load key material stored in a reasonable format (PEM), and verify it according to the specifications you mention.

All of that is orthogonal to the second question: how to store the metadata block (link) and digital signature(s) that accompany it within the SIF? We cannot use a PGP cleartext signed message, so an alternative format needs to be chosen. This is where I'm proposing a DSSE envelope be used.

For completeness, in-toto would be an alternative payload stored within the DSSE envelope, replacing the metadata block from the previous paragraph. It's not relevant for the `singularity sign` / `singularity verify` discussion in particular; I mention it only because supporting it in the future will likely mean the SIF code base will have DSSE in it for that reason. Maintaining two signature envelope formats long term is better than three.

Given the above, I respectfully disagree that this choice has anything to do with whether we need revocation checks or not. Proper handling of key material is imperative regardless of how the signed metadata is stored within a SIF.

Let me know what you think!

Ad

[Fotis Nikolaidis]

Hi Adam,

Happy to join.

I think I 'm now getting to understand your original concern better.

In my current implementation, the signature is certificate.PublicKeyAlgorithm(crypto.sha256(image)), and the signature "metadata" is the "SubjectKeyID" of the certificate.

The concept is similar to what you did with the PGP fingerprint.

https://github.com/apptainer/sif/blob/x509/pkg/integrity/sign.go#L180-L183

Since I do not store the certificate within the image, the user is responsible for circulating the right certificate (signature.SubjectKeyID == certificate.SubjectKeyID).
But this is what usually happens with x509.

An alternative would be to use the "IssuingCertificateURL" field in the image metadata and have sif download the certificate automatically.
This is practically what I do to fetch the RootCA and IntermediateCAs certificates.
https://github.com/apptainer/sif/blob/x509/pkg/integrity/x509.go#L366

That said, I still haven't understood if you are looking for a way to embed the entire certificate (chain) within the signature's metadata, or just a pointer to the certificate (e.g., fingerprint, SubjectKeyID).

Cheers,
Fotis

[Adam Hughes]

Hey Fotis,

I think we're on the same page now. Apologies that my original post wasn't as clear as it could have been.

The raw signature format with DSSE will use the same encryption algorithm and (where applicable) hash. I've actually chosen not to set a key ID directly in the signature "metadata", since a DSSE envelope can have multiple signatures contained within it, and picking one of them seemed arbitrary. Fortunately, the DSSE envelope has a keyID field beside each signature for this purpose, so I've filled that in.

I've also taken the approach of requiring the user to supply one or more relevant sources of public key material when verifying. While there is a proposal to add an optional `certificate` field within the DSSE envelope (https://github.com/secure-systems-lab/dsse/pull/50), this hasn't been accepted (yet), and isn't implemented by any of the Go modules I've looked at. 

The proposed public API for the DSSE work is in a PR here: https://github.com/sylabs/sif/pull/245, and includes a couple of new images with DSSE envelopes in `test/images`. There is more work required to expose and implement X.509 support on the Singularity side, but If you have any feedback on that, I'd be very grateful for it.

Finally, I gave a basic overview of the DSSE work at the start of today's community call. I mention that in case it helps anyone following this thread: https://www.youtube.com/watch?v=NOfTTO16Y3Q. I was a little under the weather today, but hopefully it came through clear enough to follow!

Thanks, and take care,

Ad

[Fotis Nikolaidis]

Hi Adam,

Did you receive my last reply ? I cannot see it on the list.
(Perhaps I accidentally selected "reply to author" instead of all).

Cheers,
Fotis

[Adam Hughes]

Hey Fotis! 

I did receive it, but yeah, looks like you might've replied directly rather than to the group. Have started writing up a response, but perhaps best if you re-send your message so the community can follow?

Thanks,

Ad

[Fotis Nikolaidis]

The problem is that I 've lost the answer :(
That's why I was not sure if you received it.

I 'm guessing if you forward your response to the group, it will also carry my post.

[Adam Hughes]

Ah, right! OK, here's what you sent me:

> Hi Adam,
> 
> Apologies for the long delay -- it's been a hectic week.
> 
> I had a look at the DSSE implementation, and it's quite similar to the X509 implementation for apptainer/sif. 
> 
> More specifically, the X509 implementation consists of three "layers".
> - The X509 signing/verification protocol [1]
> - The storing of signature into sif image [2]
> - The command line stuff [3]
> 
> I guess your proposed changes with DSSE effect (2) and perhaps (3). 
> The (1) can be used as it is for X.509 support on the Singularity side.
> 
> In any case, let me know how I can help.
> 
> [1] https://github.com/fnikolai/sif/blob/7a6b05b9ea48f8d856aa8f8dc3feabf94a82a140/pkg/integrity/x509.go
> [2] https://github.com/apptainer/sif/commit/702c8e2eb4a078ad24bdd1d2d3914ed840a20875#diff-7c2c801fffa1c6105556baa23d8258f713cfc439b0324c23498951248b192ac8
> [3] https://github.com/apptainer/apptainer/pull/747/files
> 
> P.S We must also revise the "validation semantics" when multiple signatures exist. When is the image valid?
> - When all the signatures are ok.
> - When at least one of the signatures is ok.
> 
> X509 RFC does not specify anything about this condition.
> 
> Cheers,
> Fotis

Apologies for the slow reply here as well. Pretty hectic going into SC22 next week!

On your three points, I'd agree that the DSSE work is targeted pretty squarely at the second ("the storing of signature into a SIF image"). There's a draft PR up for that to expose this SIF functionality (https://github.com/sylabs/sif/pull/245).

On the other two points, there's still work to be done. I've got a draft PR up (https://github.com/sylabs/singularity/pull/1120) that is a baby step towards x.509 support. That's essentially adding "command line stuff", but for raw key material. In part, this is aimed at exposing the DSSE work at a user level, to give interested folks something to try from the Singularity CLI.

The rest of the work, full X.509 support and the corresponding CLI extensions is yet to come. I'd love to collaborate on that if you're game. The one big difference in our approaches that I can see is that this functionality will be housed in the Singularity code base, rather than SIF itself. The design goal around the SIF integrity package is really to expose the low level functions to sign and verify, and nothing else. Since it's security-critical, the smaller the code base there, the better. Clearly, validation of X.509 certificates, including revocation checks needs to live somewhere, but I intend to keep that outside of the scope of SIF to the extent possible. I anticipate there will be use cases such as air-gapped networks where online revocation checks may need to be optional, and that makes me think that this is likely left in the "client" side of things (Singularity) versus the image format side of things (SIF).

With regard to validation semantics, yes, this one is sort of sticky. This was raised in the DSSE review (https://github.com/sylabs/sif/pull/228#discussion_r1011486065) and as I pointed out there, I believe this will need to be resolved carefully on the Singularity side of things. Really, we need to understand user expectations, and make sure we can meet them with the CLI.

Thanks, and really hope to collaborate on this going forward. Hope you have a great weekend!

Adam

[Fotis Nikolaidis]

Aham. I see your point. My original approach was to house the x509 on the  Singularity/Apptainer code base, but it didn't go far.

The only "signature storing structure" at the time was opengpg.Entity, which made it difficult to store x509 signatures on Sif.

Also, if I 'm not mistaken, the pgp signing/validation takes place within Sif codebase, so it made more sense to place x509 there.
With DSSE, we can probably have a "signature-agnostic" SIF (and move the signing logic to Singularity), but that means we must also move the tests and golden files to Singularity.

About the air-gapped networks, I wonder if it's better to drop online revocation checks or to enforce a model of "trusted intermediate".
In this model, a trusted entity (e.g., HPC admin) takes the responsibility to perform online revocation checks outside the air-gapped network and then sign and store the respective metadata (e.g., time of check, result, etc.) on Sif. Once copied to the air-gapped network, the "validated" image can be verified using the self-signed certificate of the trusted entity.
This functionality is akin to OCSP protocol (already integrated into x509 implementation)

From the perspective of validation semantics, this means that some "signatures" are stronger than others. For instance, the Admin's signature is stronger than the user's signature.

P.S Another question you could help me with. I 'm still not sure about the relationship between Singularity CE and Apptainer. Will any changes be propagated from one project to another, or do we have to duplicate everything?

Thanks,
Fotis

[Dave Dykstra]

Hi Fotis,

On Sat, Nov 12, 2022 at 07:50:21AM -0800, Fotis Nikolaidis wrote:
...

> P.S Another question you could help me with. I 'm still not sure about the
> relationship between Singularity CE and Apptainer. Will any changes be
> propagated from one project to another, or do we have to duplicate
> everything?

It is indeed confusing.

So far, Apptainer has imported every change from sylabs/sif. We have
also ported most but not all of the changes in sylabs/singularity.

Dave

[Adam Hughes]

Hi Fotis,

> The only "signature storing structure" at the time was opengpg.Entity, which made it difficult to store x509 signatures on Sif.

> Also, if I 'm not mistaken, the pgp signing/validation takes place within Sif codebase, so it made more sense to place x509 there.
> With DSSE, we can probably have a "signature-agnostic" SIF (and move the signing logic to Singularity), but that means we must also move the tests and golden files to Singularity.

Yeah, you're correct that the signing/verification for PGP occurs in the SIF integrity package, although locating and dealing with PGP entities happens on the Singularity side. For non-PGP, I think it makes sense to take a similar approach, where the low level sign/verify operations and storing of the signatures live in SIF, and higher level concepts like loading and validating certificates live in Singularity. I believe the API in https://github.com/sylabs/sif/pull/245 is minimally viable for this purpose, but the proof is always in the pudding!

> About the air-gapped networks, I wonder if it's better to drop online revocation checks or to enforce a model of "trusted intermediate".

> In this model, a trusted entity (e.g., HPC admin) takes the responsibility to perform online revocation checks outside the air-gapped network and then sign and store the respective metadata (e.g., time of check, result, etc.) on Sif. Once copied to the air-gapped network, the "validated" image can be verified using the self-signed certificate of the trusted entity.
> This functionality is akin to OCSP protocol (already integrated into x509 implementation)

Yes, I think there are a few options in this case. In practice, I'm guessing simply disabling the online check could be desirable, or perhaps leveraging a certificate revocation list. I will make some notes in https://github.com/sylabs/singularity/issues/1095, as perhaps Don can provide some insight into his requirements.

> P.S Another question you could help me with. I 'm still not sure about the relationship between Singularity CE and Apptainer. Will any changes be propagated from one project to another, or do we have to duplicate everything?

If you're a developer, you'll find that SingularityCE is effectively the upstream code base for both projects. See https://sylabs.io/2022/05/whats-in-a-singularityce-release/ for the facts on what's happening within both communities. If you're a user, you've probably been bombarded by messaging telling you Singularity is now Apptainer, a campaign apparently spearheaded by one large, VC-backed company. Take that for what you will. Within the SingularityCE community, we remain focussed on the important work of making performance intensive computing easier. I think you'll find it a welcoming place to collaborate and solve problems together.

Adam

[Dave Dykstra]

Since I am a part of this thread, I don't want to get into a debate but
I feel that I can't leave these statements by Adam unchallenged, for the
record, lest people think that I agree with them.

I object to the characterization of SingularityCE as upstream from
Apptainer. I see it rather as not letting improvements in an open source
peer project go to waste.

I also object to the raising of doubt over the fact that Singularity is
now Apptainer. SingularityCE is a fork.

Dave

On Wed, Nov 16, 2022 at 08:50:31AM -0800, 'Adam Hughes' via Singularity Community Edition wrote:
...

[v]

"Upstream" implies the direction of commits, and so if it's the case that Apptainer inherits from SingularityCE, then SingularityCE is indeed upstream. If it's a two way street, then they are both sort of upstreams, but then probably we'd need a new term. They are both roots.

But does this really matter? Let's try to figure out ways to communicate and language that is supportive of one another and not get caught up over terms like "upstream" - there are really better uses of time. :)

--
You received this message because you are subscribed to the Google Groups "Singularity Community Edition" group.
To unsubscribe from this group and stop receiving emails from it, send an email to singularity-c...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/singularity-ce/Y3rbkHLQOPTZDT%2Br%40fnal.gov.

[David Trudgian]

Hi all,

Let's get back on topic to the DSSE work here. If there's a question relevant to SingularityCE vs Apptainer for contributions in this topic, then of course we can address that here. Otherwise, I'd recommend that a new thread, or the Slack #general channel would be the best place to continue a more general debate.

I've left some comments on the GitHub PR. I think it looks like there is relatively broad agreement going forward with Adam's sign proposal, and verify points 1-3 at:

https://github.com/sylabs/singularity/issues/1095#issuecomment-1317335696

... but some more input is needed for online revocation checks / revocation lists. Also, if anyone else has thoughts on the UI i.e. flags, subcommand implementation, then it'd be great to hear those.

Cheers,

DT

To view this discussion on the web visit https://groups.google.com/d/msgid/singularity-ce/CAM%3Dpu%2BKm5MhnHabV-gWRVadJ7O1RhE%2B-4dOTf9nLEd80DMrDrw%40mail.gmail.com.

--

David Trudgian

Sylabs Inc.

[Adam Hughes]

Hey everyone,

We've made significant progress on this. For those not following along on GitHub, here's an update:

• Support for digital signatures in DSSE format has been added to SIF. Full details are available in the release notes here: https://github.com/sylabs/sif/releases/tag/v2.9.0.
  
• Basic support for sign/verify with PEM-encoded key material has been merged into the Singularity main branch. Full details available here: https://github.com/sylabs/singularity/pull/1120.
  
• Support for 'singularity verify' with an X.509 certificate is in a draft PR here: https://github.com/sylabs/singularity/pull/1139.
  
• The above has led to some follow-on work:
  
  • Environment variables for key material: https://github.com/sylabs/singularity/issues/1148
  • Add optional revocation checks: https://github.com/sylabs/singularity/issues/1152 
    
  • Improve organization of sign/verify flags: https://github.com/sylabs/singularity/issues/1153

Finally, there is an open question about what should happen when 'singularity verify' is run against an image that contains multiple signatures. Would love to hear any feedback on that: https://github.com/sylabs/singularity/issues/1150

Thanks everyone for all of the feedback and contributions!

Adam