Re: [blink-dev] Simulating offline in LayoutTests

[Josh Karlin]

+net...@chromium.org 

It would be nice if dev-tools and layout tests could have more control over the network. For instance, going offline in dev-tools today doesn't stop background sync events from firing because the net::NetworkChangeNotifier isn't affected.

On Tue, Jun 7, 2016 at 2:33 AM Marijn Kruisselbrink <m...@chromium.org> wrote:

There seems to be an internals.setNetworkConnectionInfo method, but as far as I can tell that only effects what navigator.connection returns; it doesn't actually effect the network stack in any way. It would be nice if there was some way through internals or testRunner to do that though. I wonder if somehow it is possible to (ab)use testRunner.evaluateInWebInspector to call the devtools Network.emulateNetworkConditions command, but probably that wouldn't be a good idea anyway.

On Mon, Jun 6, 2016 at 9:00 PM, Joel Weinberger <j...@chromium.org> wrote:

Well, a unit test would be helpful as well, but I'm definitely looking for a way to do it in a layout test, as part of some integration tests for appcache I'm working on.

On Mon, Jun 6, 2016 at 8:15 PM Daniel Cheng <dch...@chromium.org> wrote:

Does it have to be a layout test? Could it be a unit test instead?

Daniel

On Mon, Jun 6, 2016 at 8:05 PM Joel Weinberger <j...@chromium.org> wrote:

Hi all. I'm interested in making a test where the network is offline from the perspective of the network stack. In particular, the AppCache LayoutTests that use network-simulator.php are not what I need, because while it treats those individual resources as giving network errors, I want the network stack itself to think it's offline.

What I'd particularly like is for navigator.onLine == false. I would have expected there to be some sort of test of this for the throttling in network tools, but I haven't found one. Any ideas? Thanks!

--Joel

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

[Ryan Sleevi]

That seems like it would add a lot of complexity for not a lot of clear value.

Are there concrete use cases for it? In a "real world" scenario, wouldn't someone be better off controlling the state of a VM? Do we expect a billion users to benefit from this functionality - and the code cost? Do we expect it will result in reasonable-to-understand code?

For better or worse, the lower you get in the network stack, the more closely it's integrated with the OS - as a necessary condition. The more layers we start adding between that - especially to thread all the way up to the //chrome layer to control state - the harder it becomes to reason about code, security, side-effects, and reasonable expectations.

I totally understand the desire for it to be available to LayoutTests, but I'm not sure the costs would justify it. Like Daniel, I guess I struggle to understand why a unit test doesn't work better here. What does the integration test buy?

--
You received this message because you are subscribed to the Google Groups "net-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to net-dev+u...@chromium.org.
To post to this group, send email to net...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/net-dev/CAANMuaPiW%2BcYDLwsgDi1AVvXFfstP1YkjsNvyt1GcwOP8hoUmA%40mail.gmail.com.

[Joel Weinberger]

I think Josh's example is a pretty compelling one. It sounds like today, we can't actually have an integration test for background sync because the network state isn't changed, so you can't actually test the Web API layer properly. That's in addition to the developer not being able to test their application properly.

In my case, I need to check the caching behavior of the AppCache when the user agent had no network connectivity and how that compares to the Web API level. Happy to give more specifics later, but on phone now.

Thanks for the help digging into this.

-Joel 

[Josh Karlin]

Background Sync uses browser tests for API testing, partly for this reason. I'm okay with that, but it would be nice if the dev-tools networking options worked as users would expect. The result of which might be new internal APIs that layout tests could also use.

[Joel Weinberger]

Makes sense, and I'm guessing I'll have to do the best that I can with browsers tests as well. I'm just sad that we can't test Web APIs that depend on network state at what we've traditionally used as the Web API testing layer. Put differently, I don't think you could write a Web Platform Test for the W3C for the parts of background sync you're talking about right?

In short, I'd love a new internals API to test behavior like this, but understand that the complexity may not be worth it. As time goes on, though, and especially with the "offline first" PWA ideal, I imagine this will become a bigger hole.

[Yoav Weiss]

On Tue, Jun 7, 2016 at 4:22 PM, Joel Weinberger <j...@chromium.org> wrote:

Makes sense, and I'm guessing I'll have to do the best that I can with browsers tests as well. I'm just sad that we can't test Web APIs that depend on network state at what we've traditionally used as the Web API testing layer. Put differently, I don't think you could write a Web Platform Test for the W3C for the parts of background sync you're talking about right?

In short, I'd love a new internals API to test behavior like this, but understand that the complexity may not be worth it. As time goes on, though, and especially with the "offline first" PWA ideal, I imagine this will become a bigger hole.

FWIW, a non-standard internals API wouldn't really help on the WPT front. Also, this is not the only feature that requires a browser test to properly tests it (e.g. preconnect)

But I agree it would've been great to be able to test Web APIs using Layout tests, rather than browser tests.

[Ryan Sleevi]

On Tue, Jun 7, 2016 at 7:22 AM, Joel Weinberger <j...@chromium.org> wrote:

Makes sense, and I'm guessing I'll have to do the best that I can with browsers tests as well. I'm just sad that we can't test Web APIs that depend on network state at what we've traditionally used as the Web API testing layer. Put differently, I don't think you could write a Web Platform Test for the W3C for the parts of background sync you're talking about right?

I guess I philosophically disagree that it would be a good thing, for the normal reason that I would argue a unit test is superior to an integration tests. The costs, and risks, of integration tests (which LayoutTests are) are necessarily higher, riskier, and ultimately result in shipping 'testing' code to users. There needs to be a relatively high bar when asking all users to bear those costs.

I'm trying to unpack the value assertion you're making, which is that LayoutTests are better than a unit test in this regard, and understand why you feel that way.

In short, I'd love a new internals API to test behavior like this, but understand that the complexity may not be worth it. As time goes on, though, and especially with the "offline first" PWA ideal, I imagine this will become a bigger hole.

That statement doesn't really help me understand your needs, or why they exist. It makes a vague reference, but doesn't get to the core of what a LayoutTest brings here. 

[Joel Weinberger]

Yoav: doesn't WPT define generic internals-like controls via testharness.js that user agents then back appropriately? I imagined something like that for turning off network connectivity; perhaps that's more fine grained of a control than we generally provide, though.

On Tue, Jun 7, 2016 at 8:02 AM Ryan Sleevi <rsl...@chromium.org> wrote:

On Tue, Jun 7, 2016 at 7:22 AM, Joel Weinberger <j...@chromium.org> wrote:

Makes sense, and I'm guessing I'll have to do the best that I can with browsers tests as well. I'm just sad that we can't test Web APIs that depend on network state at what we've traditionally used as the Web API testing layer. Put differently, I don't think you could write a Web Platform Test for the W3C for the parts of background sync you're talking about right?

I guess I philosophically disagree that it would be a good thing, for the normal reason that I would argue a unit test is superior to an integration tests. The costs, and risks, of integration tests (which LayoutTests are) are necessarily higher, riskier, and ultimately result in shipping 'testing' code to users. There needs to be a relatively high bar when asking all users to bear those costs.

I think we're speaking past each other. I'm not saying that unit tests are less important than integration tests. I am saying that both unit tests and integration tests are extremely important.

[Yoav Weiss]

On Tue, Jun 7, 2016 at 7:07 PM, Joel Weinberger <j...@chromium.org> wrote:

Yoav: doesn't WPT define generic internals-like controls via testharness.js that user agents then back appropriately? I imagined something like that for turning off network connectivity; perhaps that's more fine grained of a control than we generally provide, though.

There wasn't anything similar last time I checked. (which was ~ a year ago, admittedly)

[Matthew Menke]

The decision was made to make dev-tools networking options be per-tab, as opposed to global.  NetworkChangeNotifier is a global.  I really don't think we want to try to unwrangle per-tab modifiers to NCN's behavior.

[Ojan Vafai]

Can this be done as a SimTest? That would have many fewer layers than a BrowserTest, run faster and be less flaky.

--
You received this message because you are subscribed to the Google Groups "net-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to net-dev+u...@chromium.org.
To post to this group, send email to net...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/net-dev/9f4df822-015d-4279-8d2d-f66269af1243%40chromium.org.