Policy proposal: Log public keys should be in PEM format, rather than DER

96 views
Skip to first unread message

Rob Percival

unread,
Aug 22, 2017, 12:04:11 PM8/22/17
to Certificate Transparency Policy
When applying for a log to be included in Chromium, the CT log policy requires that the log's public key be provided in DER form. However, in practice, most log operators have supplied their public key in PEM form. The PEM form is both more common and easier to use. OpenSSL, for example, assumes PEM form by default, and the PEM can easily be copied into a JSON log list. Therefore, I propose updating the policy to formally allow, and indeed require, that public keys are provided in PEM form. I've created a PR for the ct-policy GitHub repo that makes this change: https://github.com/GoogleChrome/ct-policy/pull/10. Does anyone object?

Ryan Sleevi

unread,
Aug 22, 2017, 1:09:44 PM8/22/17
to Rob Percival, Certificate Transparency Policy
On Tue, Aug 22, 2017 at 12:04 PM, Rob Percival <robpe...@chromium.org> wrote:
When applying for a log to be included in Chromium, the CT log policy requires that the log's public key be provided in DER form. However, in practice, most log operators have supplied their public key in PEM form.

The following were DER:

The following were PEM:

The following did not include the key in either form:

The PEM form is both more common and easier to use.

See https://bugs.chromium.org/p/chromium/issues/detail?id=483625#c3 as an example where this is not true.
 
OpenSSL, for example, assumes PEM form by default, and the PEM can easily be copied into a JSON log list. 

To be honest, I don't think this is a particularly compelling reason; that is, I'm not sure the inherent "OpenSSL does it" reason is good (after all, there's no need to integrate with OpenSSL here in this area), and the format of the JSON log list doesn't seem a good reason to drive policy, considering that the format is arbitrary.
 
Therefore, I propose updating the policy to formally allow, and indeed require, that public keys are provided in PEM form. I've created a PR for the ct-policy GitHub repo that makes this change: https://github.com/GoogleChrome/ct-policy/pull/10. Does anyone object?

I do not find this a particularly compelling change, certainly not on the requirement side, and I think the example in #c3 shows where it can end up in a bad place. 

In some ways, it highlights logs that have carefully reviewed the inclusion requirements, and lots that haven't been. One could argue Brown M&Ms, but this is hardly the hill I want to make that stand on. I think it also highlights a problematic inconsistency in ingesting requests, and ensuring compliance with the necessary information.

The question, of course, is why pick a format at all - what are we trying to achieve? The goal is consistency, for sure, and to ensure there is an unambiguous representation. For example, the choice of "DER-encoded SubjectPublicKeyInfo ASN.1" was intentionally and explicitly chosen, because OpenSSL can represent RSA public keys multiple ways - as an RSAPublicKey (RFC 3447) or as a SubjectPublicKeyInfo (RFC 5280 & RFC 3279) - and similarly with EC keys (RFC 3279 & RFC 5480). With PEM encoding, you run into issues with the PEM block header and what the underlying format is chosen.

This is why an explicitly specified encoding was requested - DER encoding (avoiding the PEM block header issues), SubjectPublicKeyInfo (avoiding the ambiguity as to the type and representation of the key).

Rob Percival

unread,
Aug 22, 2017, 2:03:18 PM8/22/17
to Certificate Transparency Policy, robpe...@chromium.org, rsl...@chromium.org
Thanks for the breakdown of PEM vs DER vs nothing. It indeed appears that PEM is more common and, in the example you highlighted, it was easy to identify the adjacent garbage in the file due to its textual nature. It would likely have been harder to notice the problem in a binary DER file.
 
 
OpenSSL, for example, assumes PEM form by default, and the PEM can easily be copied into a JSON log list. 

To be honest, I don't think this is a particularly compelling reason; that is, I'm not sure the inherent "OpenSSL does it" reason is good (after all, there's no need to integrate with OpenSSL here in this area), and the format of the JSON log list doesn't seem a good reason to drive policy, considering that the format is arbitrary.

My main point here was that at least some of the people actually using the attached public key pass it to OpenSSL and/or copy it into a log list (I know of a number of people who do this). Providing the key in any format other than PEM is an inconvenience for them.
 
 
Therefore, I propose updating the policy to formally allow, and indeed require, that public keys are provided in PEM form. I've created a PR for the ct-policy GitHub repo that makes this change: https://github.com/GoogleChrome/ct-policy/pull/10. Does anyone object?

I do not find this a particularly compelling change, certainly not on the requirement side, and I think the example in #c3 shows where it can end up in a bad place. 

In some ways, it highlights logs that have carefully reviewed the inclusion requirements, and lots that haven't been. One could argue Brown M&Ms, but this is hardly the hill I want to make that stand on. I think it also highlights a problematic inconsistency in ingesting requests, and ensuring compliance with the necessary information. 

Those of us handling inclusion requests have generally been pragmatic on the issue, given that it's easier for us to handle the PEM files and the DER can be obtained by anyone who needs it. However, assuming this proposal is withdrawn, we can certainly begin enforcing DER format public keys from now on.
 
The question, of course, is why pick a format at all - what are we trying to achieve? The goal is consistency, for sure, and to ensure there is an unambiguous representation. For example, the choice of "DER-encoded SubjectPublicKeyInfo ASN.1" was intentionally and explicitly chosen, because OpenSSL can represent RSA public keys multiple ways - as an RSAPublicKey (RFC 3447) or as a SubjectPublicKeyInfo (RFC 5280 & RFC 3279) - and similarly with EC keys (RFC 3279 & RFC 5480). With PEM encoding, you run into issues with the PEM block header and what the underlying format is chosen.

This is why an explicitly specified encoding was requested - DER encoding (avoiding the PEM block header issues), SubjectPublicKeyInfo (avoiding the ambiguity as to the type and representation of the key).

I would argue that my PR does not undermine the goal of having an explicitly specified encoding. If you take a look, you'll see that it references an RFC that is quite specific about the required PEM format (as well as providing an example and further reading, should someone want more specifics on what a SubjectPublicKeyInfo structure is). To answer your perhaps rhetorical question of what we are trying to achieve, I don't believe it's just consistency, but also convenience. We could push for consistency with either PEM or DER, but PEM is demonstrably more convenient for the people I work with. You make a good point about the fact that sometimes PEM files containing raw RSA/ECDSA DER-encoded keys are mistaken for PEM files containing a SubjectPublicKeyInfo structure (i.e. "BEGIN RSA PUBLIC KEY" vs "BEGIN PUBLIC KEY" PEM block header). However, this may happen with the DER format too and it would be considerably harder to identify the problem. With either format, the file would be rejected during review though, as it simply wouldn't be accepted by Google's compliance monitor.

Given that the primary argument for this change is convenience, it's certainly not worth a lengthy debate. If you feel strongly that the benefits of keeping the policy as it is (and presumably enforcing it from now on) outweigh the convenience of switching to PEM, and no one else cares about what format the files are in, then I'm willing to close my PR and retract this proposal. I had hoped it would be a fairly uncontentious proposal but it seems I was wrong!

Ryan Sleevi

unread,
Aug 22, 2017, 2:18:25 PM8/22/17
to Rob Percival, Certificate Transparency Policy, Ryan Sleevi
On Tue, Aug 22, 2017 at 2:03 PM, Rob Percival <robpe...@chromium.org> wrote:
To answer your perhaps rhetorical question of what we are trying to achieve, I don't believe it's just consistency, but also convenience.

Partially rhetorical - partially part of the documenting each of the requirements and how/why they came about. :)

At the time of the first draft, we didn't have RFC 7468 to clarify "what's the PEM encoding" 

As for debugging why, I'm a big fan of not spending any time debugging ;) It's on the requestor to ensure it works (and debug it) :)
For convenience/log-list, I'm not fully sure I understand that point, since things like the Chromium log list use the base64-encoded, no-newlines, no-pem form ( https://chromium.googlesource.com/chromium/src/+/811dc91fe13ab1939aaab067e429c32d98e65faf/net/data/ssl/certificate_transparency/log_list.json ). So we have to use a transformation for that, and whether command-line (-inform DER) or API (the non-PEM version of the OpenSSL APIs is considerably easier to use than the PEM form, in my experience), doesn't seem that much of a difference.

Yes, to some extent, we're talking about the color of a bikeshed - and of course, proposals to change the color generate far more response :D - but also trying to make sure we know why we're changing and what the risks are.

I'm inclined to take the change, in part because of the reference to 7468 reference, but I'm wanting to make sure that, regardless of this change, we're ensuring that there are no format issues and the requirements are consistently followed.

Andrew Ayer

unread,
Aug 22, 2017, 2:34:19 PM8/22/17
to Rob Percival, Certificate Transparency Policy
On Tue, 22 Aug 2017 09:04:11 -0700 (PDT)
Rob Percival <robpe...@chromium.org> wrote:

> When applying for a log to be included in Chromium, the CT log policy
> <https://www.chromium.org/Home/chromium-security/certificate-transparency/log-policy>
> requires that the log's public key be provided in DER form. However,
> in practice, most log operators have supplied their public key in PEM
> form. The PEM form is both more common and easier to use. OpenSSL,
> for example, assumes PEM form by default, and the PEM can easily be
> copied into a JSON log list. Therefore, I propose updating the policy
> to formally allow, and indeed require, that public keys are provided
> in PEM form. I've created a PR for the ct-policy GitHub repo that
> makes this change: https://github.com/GoogleChrome/ct-policy/pull/10.
> Does anyone object?

I object:

1. RFC7468 doesn't require the PEM to be of the DER. Someone could
submit a PEM of non-DER BER, which would be rather unpleasant.

2. It's easier to copy DER into a base64-encoded JSON string field,
since you can just pipe the DER through `base64 -w 0`. PEM requires
parsing, conversion to DER, and then base64 encoding.

3. It's easier to compute the log ID from the DER, since you can just
hash it with SHA-256.

I find DER easier to work with. All of my monitoring tools currently
use DER (or base64-encoded DER in a JSON or XML document - which is
*not* the same as PEM).

Regards,
Andrew

Eric Mill

unread,
Aug 22, 2017, 11:57:29 PM8/22/17
to Andrew Ayer, Rob Percival, Certificate Transparency Policy
If there are different sets of convenient use cases for each, how about asking log operators to provide both the DER and the PEM [of the DER]?

-- Eric


--
You received this message because you are subscribed to the Google Groups "Certificate Transparency Policy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ct-policy+unsubscribe@chromium.org.
To post to this group, send email to ct-p...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/20170822113345.9796d72b984922fbce252bc0%40andrewayer.name.



--

Rob Percival

unread,
Aug 29, 2017, 11:28:22 AM8/29/17
to Certificate Transparency Policy, robpe...@chromium.org, ag...@andrewayer.name
Thanks for taking the time to send in your objections Andrew. I've replied inline below but, in light of them, I'm happy to rescind this proposal. If requiring PEM would further inconvenience you (and possibly others), it's quite possibly not a net gain.


On Tuesday, August 22, 2017 at 7:34:19 PM UTC+1, Andrew Ayer wrote:
On Tue, 22 Aug 2017 09:04:11 -0700 (PDT)
Rob Percival <robpe...@chromium.org> wrote:

> When applying for a log to be included in Chromium, the CT log policy
> <https://www.chromium.org/Home/chromium-security/certificate-transparency/log-policy>
> requires that the log's public key be provided in DER form. However,
> in practice, most log operators have supplied their public key in PEM
> form. The PEM form is both more common and easier to use. OpenSSL,
> for example, assumes PEM form by default, and the PEM can easily be
> copied into a JSON log list. Therefore, I propose updating the policy
> to formally allow, and indeed require, that public keys are provided
> in PEM form. I've created a PR for the ct-policy GitHub repo that
> makes this change: https://github.com/GoogleChrome/ct-policy/pull/10.
> Does anyone object?

I object:

1. RFC7468 doesn't require the PEM to be of the DER.  Someone could
submit a PEM of non-DER BER, which would be rather unpleasant.

True, we'd have to explicitly exclude that option in the policy. The difference is called out in the RFC, so it would only need to be a short sentence in the policy.

2. It's easier to copy DER into a base64-encoded JSON string field,
since you can just pipe the DER through `base64 -w 0`.  PEM requires
parsing, conversion to DER, and then base64 encoding.

I've found it easier to copy the PEM, as you just copy everything between the header and footer and strip the whitespace. On a Windows machine, where there is no `base64` command, it's much more convenient. You might say that the solution to that problem is to not use Windows though ;) 
 
3. It's easier to compute the log ID from the DER, since you can just
hash it with SHA-256.
 
You have me here! The PEM does add an extra hurdle for this.
 
I find DER easier to work with.  All of my monitoring tools currently
use DER (or base64-encoded DER in a JSON or XML document - which is
*not* the same as PEM).
 
Ours generally use the base64-encoded DER as well. We've found it easy enough copying that straight out of the PEM file. However, the difference in effort is minimal (unless you're unfortunate enough to be using Windows at the time).
 
Regards,
Andrew

Ryan Sleevi

unread,
Aug 29, 2017, 11:37:24 AM8/29/17
to Rob Percival, Certificate Transparency Policy, Andrew Ayer
On Tue, Aug 29, 2017 at 11:28 AM, Rob Percival <robpe...@chromium.org> wrote:
I've found it easier to copy the PEM, as you just copy everything between the header and footer and strip the whitespace. On a Windows machine, where there is no `base64` command, it's much more convenient. You might say that the solution to that problem is to not use Windows though ;) 

Small Windows trick: `certutil -encode` and `certutil -decode` provide equivalent functionality to `base64` :D
 
Ours generally use the base64-encoded DER as well. We've found it easy enough copying that straight out of the PEM file. However, the difference in effort is minimal (unless you're unfortunate enough to be using Windows at the time).

Does that One Small Trick help close the gap? :) 

Rob Percival

unread,
Aug 29, 2017, 11:43:03 AM8/29/17
to Certificate Transparency Policy, robpe...@chromium.org, ag...@andrewayer.name, rsl...@chromium.org
Thanks for the tip! That'll certainly come in useful for this sort of thing. So are we happy to settle this with the policy remaining as-is and more consistent enforcement of public keys being submitted in DER files?
Reply all
Reply to author
Forward
0 new messages