Certificate pinning

[Michael]

It looks like the HTTP transports for the handler don't do certificate pinning. I wanted to understand that better, as -- in many cases -- these reports could contain PII, sensitive user data, or passwords (if they were stack allocated at the time of crash).

Are there other factors or mitigations in place that make minidump interception not a significant vector for attack? If it just hasn't been prioritized, would the list be receptive to a patch adding pinning and how would you want to certificate configuration to be provided -- some options,

 (a) A new startup parameter to the handler; different clients pass in their root certificate list.

 (b) Config file compiled directly into the handler binary.

Thanks,

Michael

[Scott Graham]

Hi Michael,

There's undoubtedly others reading that know more about this than me, but I believe it's just not yet implemented, and that patches to do so would be welcome.

As far as providing data, a configuration mechanism that allows sharing data with Chromium would probably be ideal.

--
You received this message because you are subscribed to the Google Groups "Crashpad-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to crashpad-dev+unsubscribe@chromium.org.
To post to this group, send email to crashp...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/crashpad-dev/234f037b-fcea-461b-aa4b-3ac36694c251%40chromium.org.

[Mark Mentovai]

That’s basically it. I’d like this, we just haven’t done it yet.

I generally try to avoid baking configuration-specific stuff into crashpad_handler. I’d prefer to avoid it for this too, although I imagine that it may make a bit more sense in this case than is typical.

Would something on the crashpad_handler command line like --https-pin-{leaf,intermediate,root}=HASH do the trick? Maybe it doesn’t matter where in the chain the pinned key hashes are found, and we can just go with --https-pin=HASH. The argument would be accepted multiple times, and if present, at least one of hashes would need to have a match in the chain.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/crashpad-dev/CANHK6RYz%2B%2BbqwTu_2Nt_mYiKMw25qW2bQUa114r8wVj6uJvuZA%40mail.gmail.com.

[Mark Mentovai]

I filed https://crashpad.chromium.org/bug/164 with some notes from this discussion, a few more thoughts, and some implementation guidance after a small bit of preliminary research.

[Michael]

I'm not sure how to get Gerrit to send mail to the list or run the trybot if possible, but I put a changelist for the Windows side up at https://chromium-review.googlesource.com/c/471917 based on the suggested approach and notes.

Would appreciate someone taking a look!

On Thursday, March 2, 2017 at 7:08:31 AM UTC-8, Mark Mentovai wrote:

I filed https://crashpad.chromium.org/bug/164 with some notes from this discussion, a few more thoughts, and some implementation guidance after a small bit of preliminary research.

On Thu, Feb 23, 2017 at 9:34 PM, Mark Mentovai <ma...@chromium.org> wrote:

That’s basically it. I’d like this, we just haven’t done it yet.

I generally try to avoid baking configuration-specific stuff into crashpad_handler. I’d prefer to avoid it for this too, although I imagine that it may make a bit more sense in this case than is typical.

Would something on the crashpad_handler command line like --https-pin-{leaf,intermediate,root}=HASH do the trick? Maybe it doesn’t matter where in the chain the pinned key hashes are found, and we can just go with --https-pin=HASH. The argument would be accepted multiple times, and if present, at least one of hashes would need to have a match in the chain.

On Thu, Feb 23, 2017 at 2:58 PM, Scott Graham <sco...@chromium.org> wrote:

Hi Michael,

There's undoubtedly others reading that know more about this than me, but I believe it's just not yet implemented, and that patches to do so would be welcome.

As far as providing data, a configuration mechanism that allows sharing data with Chromium would probably be ideal.

On Wed, Feb 22, 2017 at 3:12 PM, 'Michael' via Crashpad-dev <crashp...@chromium.org> wrote:

It looks like the HTTP transports for the handler don't do certificate pinning. I wanted to understand that better, as -- in many cases -- these reports could contain PII, sensitive user data, or passwords (if they were stack allocated at the time of crash).

Are there other factors or mitigations in place that make minidump interception not a significant vector for attack? If it just hasn't been prioritized, would the list be receptive to a patch adding pinning and how would you want to certificate configuration to be provided -- some options,

 (a) A new startup parameter to the handler; different clients pass in their root certificate list.

 (b) Config file compiled directly into the handler binary.

Thanks,

Michael

--
You received this message because you are subscribed to the Google Groups "Crashpad-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to crashpad-dev...@chromium.org.

To post to this group, send email to crashp...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/crashpad-dev/234f037b-fcea-461b-aa4b-3ac36694c251%40chromium.org.

--
You received this message because you are subscribed to the Google Groups "Crashpad-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to crashpad-dev...@chromium.org.

To post to this group, send email to crashp...@chromium.org.

[Mark Mentovai]

Thanks for getting this conversation started, Michael!

We’re still working on getting Gerrit set up nicely for Chrome’s workflow, and eventually it should send e-mail when expected. Gerrit bug 4390 has more details. Until then, you can “trick” Gerrit into sending e-mail by entering a message: hit “Reply” and type something into the “Say something nice…” box.

The trybots are currently reserved for project members.

Now, on to the meat of this issue!

I put https://chromium-review.googlesource.com/448085 down after recognizing that there was no way to make it work with the current libcurl. libcurl does allow some form of pinning with curl_easy_setopt(…, CURLOPT_PINNEDPUBLICKEY, …), but it only considers the leaf certificate’s key. This is mentioned in curl’s TODO file.

Anyway, after working on this for a while, I developed some pretty strong ideas about how it should be achieved. Mostly, I want to remain as close to RFC 7469-style pinning as possible, so:

• Require at least two pins (§4.3, §2.5). Require that the validated chain contains one match pin, and require that at least one pin be absent from the validated chain. This will encourage people to follow the best practice of having a backup key prepared ahead of time. Of course, it’s always possible to provide a dummy hash instead of a proper backup hash (as is possible with RFC 7469), but this at least raises the bar to the level that irresponsible behavior requires an explicit action.
  
• Pins should come with an expiration date (along the lines of §2.1.2). The date should be baked in to the Crashpad client the same way that the hashes themselves are. After expiration, pinning should be disabled and HTTPS connections should proceed normally. The expiration date should not be more than n days in the future relative to the start time of crashpad_handler. (§4.1, n = 60?)
  
• I feel that this is currently optional for our purposes, but pin replacement/refreshing: by implementing portions of RFC 7469 and not simply allowing for “preloaded” pins, the expiration date can be refreshed, pins can be changed, cleared, etc. If this is done, it’s probably sufficient to deal with everything in memory for the running crashpad_handler instance. I don’t think that there’s currently a need to save any of this to disk for Crashpad’s current purposes.

That’s the policy side of this. On the implementation side, after investigating how the various transport mechanisms handle this (or don’t), I’ve got this:

• We’ll need to design things to push a set of pins into the transport implementation, and then let it decide whether to allow the connection. Some transports (WinHTTP) make hashable SPKIs available, some (NSURLConnection) only provide easy access to SPKIs already hashed, and others (libcurl, judging from existing interfaces) may want to be provided with hashes and make their own assessment.

As for testing:

• As with all things TLS, testing (particularly offline) is difficult.

Suggested roadmap for this feature:

• First, land the support code in the transport implementations. This should ideally be tested, but see above. I can polish off the bits that I wrote and take this part.
  
• Then, work backwards to add a policy layer (requirements for pins, expiration, etc., also tested). You can take this part.
  
• Finally, plug it all in to crashpad_handler.

One final note: I didn’t read all of https://chromium-review.googlesource.com/471917 exhaustively, but I saw naked std::array<uint8_t, 32> being used to ferry SHA-256 hashes around. I think that this is a crutch, and that we should have a proper SHA256Hash type. Can it use std::array<> internally? Sure! (uint8_t[32] is fine too.) I just think that it deserves a proper purpose-specific class of its own, and we should be substituting naked generic types. The class should be able to convert to and from hexadecimal strings and byte buffers, and should also provide for comparisons. This part will also be easy to test independently at the “unit” level.

To unsubscribe from this group and stop receiving emails from it, send an email to crashpad-dev+unsubscribe@chromium.org.

To post to this group, send email to crashp...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/crashpad-dev/882ba18d-dd2c-4359-9171-9069ae3f2e7e%40chromium.org.

[Michael McGee]

Thanks Mark!

I'm interested in learning more about the policy aspects here. In particular, RFC 7469 doesn't strike me as applying particularly well -- it describes mechanisms for pinning trust between independent user agents and hosts, but in Crashpad the host and user agent are strictly controlled.

 * Backup pins are not as useful since deploying a new binary is an effective recovery mechanism.

 * Expiring the pins seems counter productive. Due to the nature of Crashpad, an attacker could simply reject requests for 60 days and then begin intercepting uploads. Deploying a new binary again serves as an effective recovery and rotation mechanism.

 * Preloaded pins achieve the same, if not stricter security goals with less complexity.

Is the goal with implementing parts of RFC 7469 to allow very old versions to continue to operate once their preloaded pins have expired?

[Mark Mentovai]

Michael McGee wrote:

I'm interested in learning more about the policy aspects here. In particular, RFC 7469 doesn't strike me as applying particularly well -- it describes mechanisms for pinning trust between independent user agents and hosts, but in Crashpad the host and user agent are strictly controlled.

 * Backup pins are not as useful since deploying a new binary is an effective recovery mechanism.

It’s “a” mechanism but it’s not the most effective. If you need to change certificates, clients are locked out until they’ve been updated. If the primary key becomes obsolete but the backup key remains viable, then you can swap to the backup on the server side with zero downtime and zero impact to existing clients, which is a much more effective mechanism.

 * Expiring the pins seems counter productive. Due to the nature of Crashpad, an attacker could simply reject requests for 60 days and then begin intercepting uploads. Deploying a new binary again serves as an effective recovery and rotation mechanism.

The rationale is as in RFC 7469 §4.1, and n = 60 (or thereabouts) is a compromise. I don’t want to deny service to older clients that your updater (or mine, or anyone’s) couldn’t reach. On the other hand, if your updater is that good and can reach all of your clients quickly as you seem to be indicating it is, then pin expiration shouldn’t be a problem for you either. Another option is to do periodic pin refresh as I mentioned. In any event, whether or not the pins expire, if you’ve got an emergency that requires an unforeseen certificate change, you’d better push an update with new pins as quickly as possible.

 * Preloaded pins achieve the same, if not stricter security goals with less complexity.

Is the goal with implementing parts of RFC 7469 to allow very old versions to continue to operate once their preloaded pins have expired?

That’s exactly it.

Chrome implements this behavior for its static preload lists (including not just public key pinning but also strict transport security, certificate transparency, and stapling) as well. Have a look at net::TransportSecurityState::IsBuildTimely() and its callers, specifically, net::TransportSecurityState::GetStaticDomainState(). Chrome uses n = 70.

I have no problem making n configurable by the client along with the pins, but I think that there should be some upper limit on what unaltered Crashpad source will accept there, and I don’t think I’d be comfortable setting it beyond 180.

We can’t foresee events that may require us to invalidate a certificate. Both the provision of backup pins and affordance for pin expiration are intended to overcome this uncertainty with some flexibility.