Benchmark purpose: Getting HTTPS resources into the cache even with certificate errors

[Guillaume Abadie]

Hi there,

I'm working on a test-bed to evaluate the performance of NoState-Prefetch called Sandwich.

The key operation of Sandwich, is to "convert" a WPR archive to a chrome HTTP cache directory. How we do it (without going into deep technical details) is by running chrome:

* Clear the HTTP cache,

* Navigate to a few urls we have recorded the WPR archive with.

* Snapshot the HTTP cache directory from the user profile

On android it works just fine with HTTP and HTTPS resources, because we are pushing certificates to the device.

However on desktop, according to telemetry folks, there is currently no programmatic equivalent of what is available for android: Pushing certificates. Therefore HTTPS requests to the running WPR server get certification errors. Therefore we have copied the current solution done in telemetry: use the controversial --ignore-certificate-errors command-line flag since this for test/benchmark purpose. The problem is, even if the SSL errors are ignored generally in net/, they are not ignored on the level of http_cache_transaction. And the latter just removes the cached entries from disk.

Now, the reason why this one line of code (see previous link to http_cache_transaction) is here is very straight forward:

 // Do not cache no-store content.  Do not cache content with cert errors

 // either.  This is to prevent not reporting net errors when loading a

 // resource from the cache.  When we load a page over HTTPS with a cert error

 // we show an SSL blocking page.  If the user clicks proceed we reload the

 // resource ignoring the errors.  The loaded resource is then cached.  If that

 // resource is subsequently loaded from the cache, no net error is reported

 // (even though the cert status contains the actual errors) and no SSL

 // blocking page is shown.  An alternative would be to reverse-map the cert

 // status to a net error and replay the net error.

To us, it sounds like a fair amount of work. So I was wondering if there is currently any interest from other folks in making the "alternative would be to reverse-map the cert status to a net error and replay the net error."?

Since we need to get all resources into the cache for a test/benchmark purpose, we absolutely don't care about certification failures, the reason why we are passing the --ignore-certificate-errors flag to chrome. I am not a security expert and so might be doing a mistake, but I still can't find a reason why cached HTTPS resources having certification failures would be more harmful than cached HTTP resources. Resources holding sensitive information can set the Cache-Control: No-Store response header anyway. Therefore I was thinking of making a very simple change: Would it be possible to still cache the HTTPS resources having a certification failure if we have the --ignore-certificate-errors? I have noticed that there is not a single HasSwitch in the net/’s production code (only in offline tools). Could I have implementation advises if you would be insisting in keeping out any command-line flag checks in net/?

I am also opened to any programmatic alternative ideas or change in chrome production code you may have to be able to cache somehow HTTPS resources fetched from WPR on Linux.

If you need more context, let me give some below in a mini-FAQ, otherwise you can stop ready now:

Q: Why loading web pages from a WPR?

A: This is to repro the performance metrics reliably, tackling potential future web site changes or death.

Q: Why snapshotting cache?

A: This is for benchmarking. NoState-Prefetch is going to prefetch resources and stored them into the HTTP cache until they may actually be used. Sandwich snapshots the cache directory with all resources in it, and then generate new cache directories with sub-sets of resources to use for loading web pages. This give us the ability to see concret performance metrics with different resources fetching policies on different web pages. For instance, how would NoState-Prefetch be performant if we were prefetching only resources discovered by the HTMLPreloadScanner.

Q: Is it possible to avoid using --ignore-certificate-errors?

A: We know that --ignore-certificate-errors is controversial. On Android we are able to push all necessary certificates to rooted devices. However, on Linux there is from my knowledge no alternative. We just copied the approach Telemetry is using, and we are open to programmatic suggestions.

Thanks! =)

Guillaume.

[Chris Bentzel]

Can WPR act as a TLS MITM and you install a new root cert on the testing device?

--
You received this message because you are subscribed to the Google Groups "net-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to net-dev+u...@chromium.org.
To post to this group, send email to net...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/net-dev/CADewTJHSrMEPw0L6XM%2BctvQ2Hmr0npZGn4akQQiWCXB7m45c8A%40mail.gmail.com.

[Matt Mueller]

You can script up cert management on Linux with certutil.  Use the -d sql:$HOME/.pki/nssdb argument to specify the database.

[Ryan Sleevi]

On Thu, Mar 17, 2016 at 9:12 AM, 'Guillaume Abadie' via net-dev <net...@chromium.org> wrote:

Since we need to get all resources into the cache for a test/benchmark purpose, we absolutely don't care about certification failures, the reason why we are passing the --ignore-certificate-errors flag to chrome. I am not a security expert and so might be doing a mistake, but I still can't find a reason why cached HTTPS resources having certification failures would be more harmful than cached HTTP resources.

To respond to this (Why is it this way): HTTP content is inherently insecure. HTTPS pages are (hopefully) secure. Because of this, HTTPS pages are blocked from loading insecure content (e.g. HTTP pages - this is mixed content blocking). With the way the cache and loading are implemented, the assumption is that whatever is in the cache is loaded without network errors - but certificate errors *are* presented as network errors, so there's no implementation presently to replay that logic back and trigger the right security blocking status. Instead, what you would end up with is an insecure resource (cached evil-HTTPS) being played back into a potentially secure context (legit HTTPS), with no signalling to the rest of the network stack or Chrome loader stack that there was an error. It effectively turns a 'one-time' bypass of the HTTPS error into a 'forever-bypass', and worse, without any visible indicator to the user.

Now imagine you have one of these dirty-HTTPS in the cache, and you wanted to set about solving the problem after the user has restarted the browser.

You encounter the item in the cache, and you have to ask:

- Was there an error here? [Yes]

- Should that count as a Must-Revalidate? [Probably]

- If the server's HTTPS cert is *now* good, is it safe to use the cache? [No]

So in the optimal case, you'd end up ignoring the cache entirely. Which is also why it's silly to store in the cache.

Now, you could decide that if a cached resource is dirty-HTTPS, you pop an interstitial to the user in the new session to determine if they still accept the bad-cert. But this is a fairly bad experience for the user - *especially* if the server has now fixed its config, and if you did the network load, there'd be no prompt.

This is some of the logic for why certificate errors, even with --ignore-certificate-errors, are not persisted to the cache. This is also similar to the command-line flags for enabling privileged permissions on insecure origins require the specification of a separate profile directory - to prevent any contamination from the temporary security override into the long-term state of things.

 

Resources holding sensitive information can set the Cache-Control: No-Store response header anyway. Therefore I was thinking of making a very simple change: Would it be possible to still cache the HTTPS resources having a certification failure if we have the --ignore-certificate-errors?

No :)

 

I have noticed that there is not a single HasSwitch in the net/’s production code (only in offline tools). Could I have implementation advises if you would be insisting in keeping out any command-line flag checks in net/?

Preferably :) //net's trying to offer a defined interface contract, and HasSwitch is just a form of hidden-global-flag, and those make everything harder to reason about :)

 

Q: Is it possible to avoid using --ignore-certificate-errors?

A: We know that --ignore-certificate-errors is controversial. On Android we are able to push all necessary certificates to rooted devices. However, on Linux there is from my knowledge no alternative. We just copied the approach Telemetry is using, and we are open to programmatic suggestions.

There are a variety of solutions you can use, depending on your constraints. This range from using certutil (as mattm@ points out) to using custom LD_LIBRARY_DIR to affect how softoken is loaded. If this is only for telemetry machines, this is much easier/safer. If this is also intended for end-user/developer machines, there's a variety of (programatic) options that vary in terms of risk-level and privileges-needed, but certainly are all less invasive than the above.

For other platforms:

- Windows you can do some hijinks with DLL loading that prevents the trust stuff from being in 'production' Chrome, while still has the same results

- OS X I'm not sure of what dynamic interception capabilities exist; if you can override Frameworks, thats one option, and if you can inject into processes (such as Mach-*), that's another, and both avoid the need for 'production' code to know about the trust anchors.

- ChromeOS - depends on if you're talking about images built for dev/test or factory images. Factory you're SOL-by-design, but dev/test has many of the same options as Linux.

[Chris Bentzel]

On Thu, Mar 17, 2016 at 8:30 PM Ryan Sleevi <rsl...@chromium.org> wrote:

On Thu, Mar 17, 2016 at 9:12 AM, 'Guillaume Abadie' via net-dev <net...@chromium.org> wrote:

Since we need to get all resources into the cache for a test/benchmark purpose, we absolutely don't care about certification failures, the reason why we are passing the --ignore-certificate-errors flag to chrome. I am not a security expert and so might be doing a mistake, but I still can't find a reason why cached HTTPS resources having certification failures would be more harmful than cached HTTP resources.

To respond to this (Why is it this way): HTTP content is inherently insecure. HTTPS pages are (hopefully) secure. Because of this, HTTPS pages are blocked from loading insecure content (e.g. HTTP pages - this is mixed content blocking). With the way the cache and loading are implemented, the assumption is that whatever is in the cache is loaded without network errors - but certificate errors *are* presented as network errors, so there's no implementation presently to replay that logic back and trigger the right security blocking status. Instead, what you would end up with is an insecure resource (cached evil-HTTPS) being played back into a potentially secure context (legit HTTPS), with no signalling to the rest of the network stack or Chrome loader stack that there was an error. It effectively turns a 'one-time' bypass of the HTTPS error into a 'forever-bypass', and worse, without any visible indicator to the user.

Now imagine you have one of these dirty-HTTPS in the cache, and you wanted to set about solving the problem after the user has restarted the browser.

You encounter the item in the cache, and you have to ask:

- Was there an error here? [Yes]

- Should that count as a Must-Revalidate? [Probably]

- If the server's HTTPS cert is *now* good, is it safe to use the cache? [No]

So in the optimal case, you'd end up ignoring the cache entirely. Which is also why it's silly to store in the cache.

Now, you could decide that if a cached resource is dirty-HTTPS, you pop an interstitial to the user in the new session to determine if they still accept the bad-cert. But this is a fairly bad experience for the user - *especially* if the server has now fixed its config, and if you did the network load, there'd be no prompt.

This is some of the logic for why certificate errors, even with --ignore-certificate-errors, are not persisted to the cache. This is also similar to the command-line flags for enabling privileged permissions on insecure origins require the specification of a separate profile directory - to prevent any contamination from the temporary security override into the long-term state of things.

 

Resources holding sensitive information can set the Cache-Control: No-Store response header anyway. Therefore I was thinking of making a very simple change: Would it be possible to still cache the HTTPS resources having a certification failure if we have the --ignore-certificate-errors?

No :)

 

I have noticed that there is not a single HasSwitch in the net/’s production code (only in offline tools). Could I have implementation advises if you would be insisting in keeping out any command-line flag checks in net/?

Preferably :) //net's trying to offer a defined interface contract, and HasSwitch is just a form of hidden-global-flag, and those make everything harder to reason about :)

 

You can also take a look at chrome/browser/io_thread.cc for examples where we read command-line flags at a Chrome layer which eventually trickle into the networking stack configuration. 

 

Q: Is it possible to avoid using --ignore-certificate-errors?

A: We know that --ignore-certificate-errors is controversial. On Android we are able to push all necessary certificates to rooted devices. However, on Linux there is from my knowledge no alternative. We just copied the approach Telemetry is using, and we are open to programmatic suggestions.

There are a variety of solutions you can use, depending on your constraints. This range from using certutil (as mattm@ points out) to using custom LD_LIBRARY_DIR to affect how softoken is loaded. If this is only for telemetry machines, this is much easier/safer. If this is also intended for end-user/developer machines, there's a variety of (programatic) options that vary in terms of risk-level and privileges-needed, but certainly are all less invasive than the above.

For other platforms:

- Windows you can do some hijinks with DLL loading that prevents the trust stuff from being in 'production' Chrome, while still has the same results

- OS X I'm not sure of what dynamic interception capabilities exist; if you can override Frameworks, thats one option, and if you can inject into processes (such as Mach-*), that's another, and both avoid the need for 'production' code to know about the trust anchors.

- ChromeOS - depends on if you're talking about images built for dev/test or factory images. Factory you're SOL-by-design, but dev/test has many of the same options as Linux.

--

You received this message because you are subscribed to the Google Groups "net-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to net-dev+u...@chromium.org.
To post to this group, send email to net...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/net-dev/CACvaWvZvoFb0xKVxR0Y_yxHJVduPoKS15-rTCJjNqSTqHUm1CQ%40mail.gmail.com.

[Guillaume Abadie]

Hi there,

sorry for the unforgivable time to respond. Thanks for these details responses.

For now, we are running our code only locally on developer machines. Therefore the solution requires to not compromise at all risks of the machine (under linux). What would be the safest solution for the machine?

Could it be possible to use certutil to store the certificate in a tmp directory and tell in chrome command line arguments to also look into this directory for certification validation?

Thanks

[Matt Mueller]

You can run chrome with the HOME environment variable set to some other dir and it will access the .pki/nssdb/ from there instead (along with all other config files). Some things may be a bit weird since it won't find the normal settings for your user/desktop environment/etc but it seems to work ok.

--
You received this message because you are subscribed to the Google Groups "net-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to net-dev+u...@chromium.org.
To post to this group, send email to net...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/net-dev/CADewTJEi%2BOzCgG_RxZkw_FfhXC-bsQ4P4QCiQgo8kuLe_sRYLA%40mail.gmail.com.

[Guillaume Abadie]

Thanks Matt!

I am trying using certutil with $HOME env, but looks like I'm doing something wrong.

My goal is to load https://en.m.wikipedia.org/wiki/Science on linux from a WPR server with valid certificate fully grammatically. So I have tried setting up ~/.pki/nssdb with my test certifcate with:

> certutil -d sql:/tmp/tmpUOeZRY/.pki/nssdb -f /tmp/tmpUOeZRY/certutil.pwd -N

> certutil -d sql:/tmp/tmpUOeZRY/.pki/nssdb -f /tmp/tmpUOeZRY/certutil.pwd -A -t ,, -n localhost -i /tmp/tmpUOeZRY/testca.pem

And then launch chrome with HOME=/tmp/tmpUOeZRY. After having checked within the code that the path to the db was correct, I am still having a certifcate failure on the web page:

[30651:30671:0411/175051:ERROR:cert_verify_proc_nss.cc(984)] CERT_PKIXVerifyCert for en.m.wikipedia.org failed err=-8179

One first weird thing is that certutil always ask me a password, the reason why I am using -f /tmp/tmpUOeZRY/certutil.pwd that just have a "Hello~world2016" string:

> certutil -d sql:/tmp/tmpUOeZRY/.pki/nssdb -N

Enter a password which will be used to encrypt your keys.

The password should be at least 8 characters long,

and should contain at least one non-alphabetic character.

Enter new password:

But I don't know how to give this pwd to chrome in the first place. So I have also tested with and empty /tmp/tmpUOeZRY/certutil.pwd. Still not working.

Would you guys see mistakes or have concrete example to share on how to set that up properly?

[Ryan Sleevi]

On Mon, Apr 11, 2016 at 9:45 AM, Guillaume Abadie <gab...@google.com> wrote:

Thanks Matt!

I am trying using certutil with $HOME env, but looks like I'm doing something wrong.

My goal is to load https://en.m.wikipedia.org/wiki/Science on linux from a WPR server with valid certificate fully grammatically. So I have tried setting up ~/.pki/nssdb with my test certifcate with:

> certutil -d sql:/tmp/tmpUOeZRY/.pki/nssdb -f /tmp/tmpUOeZRY/certutil.pwd -N

> certutil -d sql:/tmp/tmpUOeZRY/.pki/nssdb -f /tmp/tmpUOeZRY/certutil.pwd -A -t ,, -n localhost -i /tmp/tmpUOeZRY/testca.pem

See certutil -H

You want --empty-password to go with -N

 

[Guillaume Abadie]

Hi there,

Thanks Ryan for the handy --empty-password flag!

A little bit of an update. The issue was that I was generating my self signed root certificate using WPR's same code path generating the certificate for as for android. But looks like it is incomplete for desktop because just generating this certificate with: openssl req -x509 -newkey rsa:2048 -keyout 01key.pem -out 01cert.pem -nodes -days 365 -sha256 worked fine on desktop.

However now the new issue I'm having is that I'm hitting ERR_SSL_VERSION_OR_CIPHER_MISMATCH:

After some drilling, I have found out that WPR's HTTPS server is restricting to SSL v2 and v3. So I switched that to TLSv1_2_METHOD. But still same ERR_SSL_VERSION_OR_CIPHER_MISMATCH error. And the thing is HTTPS requests to WPR (with still SSLv23_METHOD) and without certificate validation works fine on desktop as long as the --ignore-certificate-errors is set. It works also fine on android with SSLv23_METHOD and certificate successfully pushed to the device. Therefore it is not clear to me where the issue is coming from.

Attached below are two network logs exported from chrome://net-internals, one with SSL_METHOD = SSL.SSLv23_METHOD and the other with SSL_METHOD = SSL.TLSv1_1_METHOD.

Thanks for help!

Guillaume.

[Ryan Sleevi]

On Wed, Apr 13, 2016 at 10:16 AM, Guillaume Abadie <gab...@google.com> wrote:

After some drilling, I have found out that WPR's HTTPS server is restricting to SSL v2 and v3.

No, that's not a correct read. It is an unfortunate artifact of poor naming, but SSLv23_METHOD is (effectively) the "Not restricted to any particular protocol" (It's named so because it enables certain compatability logic for the client hellos, rather than explicitly failing; the actual version negotiation for SSLv23_METHOD is handled via the option flags of SSL_OP_NO_(SSLv2/SSLv3/TLSv1/TLSv1_1/TLSv1_2)_METHOD flags. This has all been refactored in BoringSSL)

 

So I switched that to TLSv1_2_METHOD. But still same ERR_SSL_VERSION_OR_CIPHER_MISMATCH error. And the thing is HTTPS requests to WPR (with still SSLv23_METHOD) and without certificate validation works fine on desktop as long as the --ignore-certificate-errors is set. It works also fine on android with SSLv23_METHOD and certificate successfully pushed to the device. Therefore it is not clear to me where the issue is coming from.

The error is being sent by the Python server - it's reporting a fatal alert with error code 40 (aka handshake_failure). You can see this with Wireshark.

You'd want to trace/debug more from the Python side, but --ignore-certificate-errors should have no impact on this, because it's the peer (the Python server) sending the error. I'd encourage you to capture a NetLog with the --ignore-certificate-errors, because that would only squelch certificate-related errors, and ERR_SSL_VERSION_OR_CIPHER_MISMATCH is not such an error.

[Ryan Sleevi]

On Wed, Apr 13, 2016 at 11:45 AM, Ryan Sleevi <rsl...@chromium.org> wrote:

The error is being sent by the Python server - it's reporting a fatal alert with error code 40 (aka handshake_failure). You can see this with Wireshark.

You'd want to trace/debug more from the Python side, but --ignore-certificate-errors should have no impact on this, because it's the peer (the Python server) sending the error. I'd encourage you to capture a NetLog with the --ignore-certificate-errors, because that would only squelch certificate-related errors, and ERR_SSL_VERSION_OR_CIPHER_MISMATCH is not such an error.

Also, I think it would be helpful if you explain more your test setup.

If I were to guess, you've got an apples:oranges comparison somewhere. For example, you might be testing locally with one, but then comparing against "--ignore-certificate-errors" from a different machine (e.g. WPT bots). Can you confirm that --ignore-certificate-errors works with your local instance? Can you confirm that you don't have any cached .pyc files around / zombie instances?

If I were to make a wild stab in the dark, I suspect you're comparing official bots versus local instance. I'd also be willing to bet you're doing the comparison on a system with an older OpenSSL/pyOpenSSL, which has a bad default cipher config, such that there's no cipher overlap.

The SSLContext being initialized by WPR ( https://github.com/chromium/web-page-replay/blob/master/certutils.py#L53 ) doesn't ever explicitly set the ciphersuites, so it'll be influenced by things like the openssl.cnf file and whatever version of OpenSSL you're running. To get deterministic behaviour, you'd want to make sure you're enabling a consistent set of ciphersuites. Given that the server is rejecting the connection immediately, the two likely suspects are:

- You have a cached local change that you've missed somewhere, configuring it to only support TLS1.2, which ends up rejecting client hellos done in the 'compatability format' like Chrome

- You've got an older OpenSSL/crappy OpenSSL cnf that

  - Doesn't have TLS1.2 enabled by default for SSLv23_METHOD

  - Doesn't have a sane set of ciphersuites enabled by default

To rule out the latter, I believe you can do something like

// Really, SSL_CTX_clear_options() could also work here, but being conservative (perhaps incorrectly so)

SSL_CTX_set_options(context, SSL_CTX_get_options(context) & (~SSL_OP_NO_TLSv1_2 & ~SSL_OP_NO_TLSv1_1));

SSL_CTX_set_cipher_list(context, "ALL");

You'd have to find the appropriate Python bindings for those, but that should hopefully rule out the "stupid local configuration issue" possibility.

 

[David Benjamin]

On Wed, Apr 13, 2016 at 3:08 PM Ryan Sleevi <rsl...@chromium.org> wrote:

On Wed, Apr 13, 2016 at 11:45 AM, Ryan Sleevi <rsl...@chromium.org> wrote:

The error is being sent by the Python server - it's reporting a fatal alert with error code 40 (aka handshake_failure). You can see this with Wireshark.

You'd want to trace/debug more from the Python side, but --ignore-certificate-errors should have no impact on this, because it's the peer (the Python server) sending the error. I'd encourage you to capture a NetLog with the --ignore-certificate-errors, because that would only squelch certificate-related errors, and ERR_SSL_VERSION_OR_CIPHER_MISMATCH is not such an error.

Also, I think it would be helpful if you explain more your test setup.

If I were to guess, you've got an apples:oranges comparison somewhere. For example, you might be testing locally with one, but then comparing against "--ignore-certificate-errors" from a different machine (e.g. WPT bots). Can you confirm that --ignore-certificate-errors works with your local instance? Can you confirm that you don't have any cached .pyc files around / zombie instances?

If I were to make a wild stab in the dark, I suspect you're comparing official bots versus local instance. I'd also be willing to bet you're doing the comparison on a system with an older OpenSSL/pyOpenSSL, which has a bad default cipher config, such that there's no cipher overlap.

The SSLContext being initialized by WPR ( https://github.com/chromium/web-page-replay/blob/master/certutils.py#L53 ) doesn't ever explicitly set the ciphersuites, so it'll be influenced by things like the openssl.cnf file and whatever version of OpenSSL you're running. To get deterministic behaviour, you'd want to make sure you're enabling a consistent set of ciphersuites. Given that the server is rejecting the connection immediately, the two likely suspects are:

- You have a cached local change that you've missed somewhere, configuring it to only support TLS1.2, which ends up rejecting client hellos done in the 'compatability format' like Chrome

What compatibility format? We definitely do not send V2ClientHellos if that's what you mean.

If you're referring to the record-layer version number, that thing is made of lies in all TLS versions and OpenSSL doesn't anything about it in the ClientHello for TLS1_2_method() either.