Testing cache in web platform tests

[Martin Šrámek]

Hello folks!

I have run into a peculiar problem when writing tests for the cache deletion in Clear-Site-Data.

I can't test whether the cache is deleted, because I can't write into it in the first place. The reason is that content_shell --run-layout-test runs on https://web-platform.test, which does not have a valid certificate; and in that case, we disable cache completely. My first guess, using the --ignore-certificate-errors flag, does not solve the problem.

This problem was discussed on security-dev in the past, concluding that the flag can't make websites work to full extent, as if there was a valid certificate, because users tend to use it to shoot themselves in the foot. I'm not going to argue against that. However, could we make this possible at least for content_shell --run-layout-test?

In the meantime, I can mark the tests as manual and run them on http://localhost, where they will pass, since localhost is a secure context without the need for certificates. But it would be nice to be able to automate.

Best regards,

Martin

[Ryan Sleevi]

On Tue, Jul 11, 2017 at 5:20 PM, Martin Šrámek <msr...@chromium.org> wrote:
> Hello folks!
>
> I have run into a peculiar problem when writing tests for the cache deletion
> in Clear-Site-Data.
>
> I can't test whether the cache is deleted, because I can't write into it in
> the first place. The reason is that content_shell --run-layout-test runs on
> https://web-platform.test, which does not have a valid certificate; and in
> that case, we disable cache completely. My first guess, using the
> --ignore-certificate-errors flag, does not solve the problem.

Correct. --ignore-certificate-errors properly ignores errors, but the
//net stack is explicitly coded to not cache data for errors (and this
propagates to other things, like service workers). What you want is,
well, a not-errorful connection, which isn't unreasonable to do for a
lot of our test harnesses, and helps make sure we aren't shipping
giant footguns in the production code. But you caught up that thread
:)

>
> This problem was discussed on security-dev in the past, concluding that the
> flag can't make websites work to full extent, as if there was a valid
> certificate, because users tend to use it to shoot themselves in the foot.
> I'm not going to argue against that. However, could we make this possible at
> least for content_shell --run-layout-test?

I'd be nervous about adding it to content_shell, given how many
downstream consumers are now apparently using it to build their
(production) products on, seemingly.

From quickly scanning
https://cs.chromium.org/chromium/src/content/shell/app/shell_main_delegate.cc?rcl=07eb249cdc03c7ca1eceda2c755def72fcbcf5f5&l=161
it doesn't look like --run-layout-test affects profile initialization
(e.g. no good path to inject test dependencies in). I also suspect
it'd be pretty nasty to try to plumb this flag all the way down to the
IO thread initialization - as it's one of those "disable security"
flags that are generally to be avoided.

That said, https://cs.chromium.org/chromium/src/chrome/common/chrome_switches.cc?rcl=07eb249cdc03c7ca1eceda2c755def72fcbcf5f5&l=492
tries to address the gap between the security/usability/risk side,
depending on what you're using for your test server, but that's in
//chrome (along with
https://cs.chromium.org/chromium/src/chrome/browser/ssl/ignore_errors_cert_verifier.h?rcl=07eb249cdc03c7ca1eceda2c755def72fcbcf5f5&l=21
) in part because it's generally unwise to ignore certificate errors
(and few of our tests need to). It wasn't plumbed into //content
because that sort of "giant dangerous footgun" was something we were
intentionally trying to limit spreading, but...

[Matt Falkenhagen]

This net-dev thread is also relevant: https://groups.google.com/a/chromium.org/d/msg/net-dev/wbhbSn9ICfc/XDmCkpboAAAJ

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CACvaWvb5X0Dv15_sULRhQJwXf8UWvwiUxqoA2xhWiHKCNfdSrA%40mail.gmail.com.

[Matthew Menke]

I assume there's a reason for not just using a browser_test?  Embedded test server's SSL mode gets around cert errors somehow (I can't remember how).

[Ryan Sleevi]

On Wed, Jul 12, 2017 at 9:02 AM, Matthew Menke <mme...@google.com> wrote:
> I assume there's a reason for not just using a browser_test? Embedded test
> server's SSL mode gets around cert errors somehow (I can't remember how).

It works because of the single-threaded (ish) initialization phase.
It's not supported in content_shell or chrome (intentionally). For
those, you need to associate the CertVerifier with the profile - which
is why I mentioned the other two paths (either plumbing it down during
the "run-layout-tests" handling in a way to affect the profile
initialization, or moving the 'global flag' down from //chrome into
//content so that the profile initialization reads the command-line
flag)

[Mike West]

On Wed, Jul 12, 2017 at 5:55 PM, Ryan Sleevi <rsl...@chromium.org> wrote:

On Wed, Jul 12, 2017 at 9:02 AM, Matthew Menke <mme...@google.com> wrote:
> I assume there's a reason for not just using a browser_test?  Embedded test
> server's SSL mode gets around cert errors somehow (I can't remember how).

There is! :) We can't easily share `browser_tests` with other vendors. Whenever possible, we should prefer to test web-facing features via layout tests that live in a shared repository that all browsers can pull from.

 

It works because of the single-threaded (ish) initialization phase.
It's not supported in content_shell or chrome (intentionally). For
those, you need to associate the CertVerifier with the profile - which
is why I mentioned the other two paths (either plumbing it down during
the "run-layout-tests" handling in a way to affect the profile
initialization, or moving the 'global flag' down from //chrome into
//content so that the profile initialization reads the command-line
flag)

Which of whose would you prefer? It seems like exposing a flag to content_shell in layout test mode will be the most straightforward thing to do as a first step.

-mike

[Ryan Sleevi]

On Wed, Jul 12, 2017 at 12:06 PM, Mike West <mk...@chromium.org> wrote:
>> It works because of the single-threaded (ish) initialization phase.
>> It's not supported in content_shell or chrome (intentionally). For
>> those, you need to associate the CertVerifier with the profile - which
>> is why I mentioned the other two paths (either plumbing it down during
>> the "run-layout-tests" handling in a way to affect the profile
>> initialization, or moving the 'global flag' down from //chrome into
>> //content so that the profile initialization reads the command-line
>> flag)
>
>
> Which of whose would you prefer? It seems like exposing a flag to
> content_shell in layout test mode will be the most straightforward thing to
> do as a first step.

If there is a specific need for it in content_shell (and it sounds
like there is), and we believe we can mitigate the security footgun
aspect (e.g. the need for user_data_dir), then moving the command-line
flag from //chrome into //content seems reasonable. We didn't at the
time because there wasn't a need, but it sounds like there is now :)

[Jochen Eisinger]

I'd propose to move the IgnoreErrorsCertVerifier to //net and add LayoutTestURLRequestContextGetter::GetCertVerifier to return an instance of it instead of the default verifier for ShellURLRequestContextGetter::GetURLRequestContext() to use.

[Rick Byers]

[+ecosystem-infra for WPT discussion]

I'm curious but lack the context on how https WPT tests are supposed to work at all (in an official Chrome build).  They don't seem to work for me using upstream WPT and 'wptrun'.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CALjhuiesD_TW7Z0bNXojZTaYhdNbFMhiX%2Ba9QmECwkhSGaeUVA%40mail.gmail.com.

[Jochen Eisinger]

On Thu, Jul 13, 2017 at 6:45 PM Rick Byers <rby...@chromium.org> wrote:

[+ecosystem-infra for WPT discussion]

I'm curious but lack the context on how https WPT tests are supposed to work at all (in an official Chrome build).  They don't seem to work for me using upstream WPT and 'wptrun'.

They mostly don't work in an official build - at the very least, you have to disable the popup blocker, and a bunch of other security features :/

[Philip Jägenstedt]

On Fri, Jul 14, 2017 at 10:36 AM Jochen Eisinger <joc...@chromium.org> wrote:

On Thu, Jul 13, 2017 at 6:45 PM Rick Byers <rby...@chromium.org> wrote:

[+ecosystem-infra for WPT discussion]

I'm curious but lack the context on how https WPT tests are supposed to work at all (in an official Chrome build).  They don't seem to work for me using upstream WPT and 'wptrun'.

They mostly don't work in an official build - at the very least, you have to disable the popup blocker, and a bunch of other security features :/

Are these things specific to HTTPS, or just generally broken when running wpt on an official Chrome build?

As for the invalid certs, there is code in the wpt repo to generate certificates for testing, as if it was its own CA it seems. Are there command line flags that could be used to add certificates to be considered valid, that could make https://web-platform.test behave in every way identical to a public site with a real certificate?

[Jochen Eisinger]

On Mon, Jul 31, 2017 at 10:27 AM Philip Jägenstedt <foo...@chromium.org> wrote:

On Fri, Jul 14, 2017 at 10:36 AM Jochen Eisinger <joc...@chromium.org> wrote:

On Thu, Jul 13, 2017 at 6:45 PM Rick Byers <rby...@chromium.org> wrote:

[+ecosystem-infra for WPT discussion]

I'm curious but lack the context on how https WPT tests are supposed to work at all (in an official Chrome build).  They don't seem to work for me using upstream WPT and 'wptrun'.

They mostly don't work in an official build - at the very least, you have to disable the popup blocker, and a bunch of other security features :/

Are these things specific to HTTPS, or just generally broken when running wpt on an official Chrome build?

at least the popup blocker is independent of https 

As for the invalid certs, there is code in the wpt repo to generate certificates for testing, as if it was its own CA it seems. Are there command line flags that could be used to add certificates to be considered valid, that could make https://web-platform.test behave in every way identical to a public site with a real certificate?

we configured wptserve to use a checked in cert here: https://cs.chromium.org/chromium/src/third_party/WebKit/Tools/Scripts/webkitpy/thirdparty/wpt/wpt.config.json?rcl=dc7b9066e7192a15ec7c54bbf0088ec497a91096&l=14