Will Chen uploaded patch set #4 to this change.
DevTools: deflake devtools tests by resetting storage state
Ideally, layout tests should clear its storage state after each test, but
there's a ~100 ms overhead to doing so and most layout tests aren't affected
by storage state.
This resets storage state at the start of every DevTools test which:
1) deflakes tests where the inspected page uses indexeddb, websql, etc.
2) simplifies test infrastructure code to reset DevTools' local storage (used
to store certain settings)
Bug: 667560
Change-Id: Ia7b70aebabd970957ae3581e47ee31a10992c657
---
M content/shell/browser/layout_test/blink_test_controller.cc
M content/shell/browser/layout_test/layout_test_devtools_bindings.cc
M content/shell/browser/layout_test/layout_test_devtools_bindings.h
M content/shell/browser/shell_devtools_bindings.cc
M content/shell/browser/shell_devtools_bindings.h
M content/shell/common/shell_messages.h
M content/shell/renderer/layout_test/blink_test_runner.cc
M content/shell/renderer/layout_test/blink_test_runner.h
M content/shell/test_runner/test_interfaces.cc
M content/shell/test_runner/test_runner.cc
M content/shell/test_runner/test_runner.h
M content/shell/test_runner/web_test_delegate.h
M third_party/WebKit/LayoutTests/http/tests/devtools/service-workers/service-workers-redundant-expected.txt
M third_party/WebKit/LayoutTests/http/tests/devtools/startup/database-open.html
M third_party/WebKit/LayoutTests/http/tests/inspector/inspector-test.js
15 files changed, 48 insertions(+), 40 deletions(-)
To view, visit change 792534. To unsubscribe, or for help writing mail filters, visit settings.
Will Chen uploaded patch set #5 to this change.
DevTools: deflake devtools tests by resetting storage state
Ideally, layout tests should clear its storage state after each test, but
there's a ~100 ms overhead to do so and most layout tests aren't affected
@dgozman - ptal
@eostroukhov - I redid the CL after being inspired by your suggestion.
Patch set 8:Commit-Queue +1
1 comment:
this should never be in our test expectations - this is an internal string that code mirror uses for text measurements (and isn't actually visible).
To view, visit change 792534. To unsubscribe, or for help writing mail filters, visit settings.
Will Chen would like Dmitry Gozman to review this change.
15 files changed, 41 insertions(+), 38 deletions(-)
The code looks good, but I have some questions.
1 comment:
File content/shell/browser/layout_test/layout_test_devtools_bindings.cc:
Patch Set #8, Line 160: // Reset storage state before each DevTools test to avoid flakiness.
What do we use storage for? If that's LocalStorage for settings, we should pass a different storage backend to settings while under test (we actually support that).
For everything else, should we just do an explicit clear in tests which require it, like indexeddb?
To view, visit change 792534. To unsubscribe, or for help writing mail filters, visit settings.
Will Chen uploaded patch set #9 to this change.
DevTools: deflake application panel tests by resetting storage state
Bug: 667560
Change-Id: Ia7b70aebabd970957ae3581e47ee31a10992c657
---
M content/shell/browser/layout_test/blink_test_controller.cc
M content/shell/browser/layout_test/layout_test_devtools_bindings.cc
M content/shell/browser/layout_test/layout_test_devtools_bindings.h
M content/shell/browser/shell_devtools_bindings.cc
M content/shell/browser/shell_devtools_bindings.h
M content/shell/common/shell_messages.h
M content/shell/renderer/layout_test/blink_test_runner.cc
M content/shell/renderer/layout_test/blink_test_runner.h
M content/shell/test_runner/test_interfaces.cc
M content/shell/test_runner/test_runner.cc
M content/shell/test_runner/test_runner.h
M content/shell/test_runner/web_test_delegate.h
M third_party/WebKit/LayoutTests/http/tests/devtools/service-workers/service-workers-redundant-expected.txt
M third_party/WebKit/LayoutTests/http/tests/devtools/startup/database-open.html
M third_party/WebKit/LayoutTests/http/tests/inspector/inspector-test.js
15 files changed, 41 insertions(+), 38 deletions(-)
To view, visit change 792534. To unsubscribe, or for help writing mail filters, visit settings.
I redid the CL with @dgozman's suggestion and narrowed the scope to just application panel tests and used eugene's suggestion of just calling the agent directly.
Patch set 10:Commit-Queue +1
Non-committer LGTM
Looks like this does not work? Can we use testRunner.clearAllDatabases right now (or expose more if we need)? Then we can follow up with protocol fix and switch.
1 comment:
File third_party/WebKit/Source/devtools/front_end/application_test_runner/ResourcesTestRunner.js:
Patch Set #10, Line 18: // TODO(chenwilliam): This has a race condition because clearDataForOrigin is asynchronous
Let's change the protocol!
To view, visit change 792534. To unsubscribe, or for help writing mail filters, visit settings.
Will Chen uploaded patch set #13 to this change.
DevTools: deflake application panel tests by resetting storage state
This improves ClearDataForOrigin by making it explicit that it's
asynchronous and (mostly) fixing the race condition.
The callback doesn't guarantee the storage has actually been deleted but
at least it has been scheduled (see storage_partition.h), whereas before
it returned with a success response immediately.
Bug: 667560
Change-Id: Ia7b70aebabd970957ae3581e47ee31a10992c657
---
M content/browser/devtools/protocol/storage_handler.cc
M content/browser/devtools/protocol/storage_handler.h
M content/browser/devtools/protocol_config.json
M third_party/WebKit/Source/devtools/front_end/application_test_runner/ResourcesTestRunner.js
M third_party/WebKit/Source/devtools/front_end/test_runner/TestRunner.js
5 files changed, 48 insertions(+), 11 deletions(-)
To view, visit change 792534. To unsubscribe, or for help writing mail filters, visit settings.
Will Chen uploaded patch set #14 to this change.
DevTools: deflake application panel tests by resetting storage state
This improves ClearDataForOrigin by making it explicit that it's
asynchronous and (mostly) fixing the race condition.
The callback doesn't guarantee the storage has actually been deleted but
at least it has been scheduled (see storage_partition.h), whereas before
it returned with a success response immediately.
We could update third_party/WebKit/Source/devtools/front_end/resources/ClearStorageView.js
to actually wait for this callback to be called to update the UI when the user
clicks on the clear storage button, but let's defer this for a follow-up patch.
Bug: 667560
Change-Id: Ia7b70aebabd970957ae3581e47ee31a10992c657
---
M content/browser/devtools/protocol/storage_handler.cc
M content/browser/devtools/protocol/storage_handler.h
M content/browser/devtools/protocol_config.json
M third_party/WebKit/Source/devtools/front_end/application_test_runner/ResourcesTestRunner.js
M third_party/WebKit/Source/devtools/front_end/test_runner/TestRunner.js
5 files changed, 48 insertions(+), 11 deletions(-)
To view, visit change 792534. To unsubscribe, or for help writing mail filters, visit settings.
Will Chen uploaded patch set #15 to this change.
DevTools: deflake application panel tests by resetting storage state
This improves ClearDataForOrigin by making it explicit that it's
asynchronous and (mostly) fixing the race condition.
The callback doesn't guarantee the storage has actually been deleted but
at least it has been scheduled (see storage_partition.h), whereas before
it returned with a success response immediately.
We could update third_party/WebKit/Source/devtools/front_end/resources/ClearStorageView.js
to actually wait for this callback to be called to update the UI when the user
clicks on the clear storage button rather than rely on a 500ms timeout,
but let's defer this for a follow-up patch.
Bug: 667560
Change-Id: Ia7b70aebabd970957ae3581e47ee31a10992c657
---
M content/browser/devtools/protocol/storage_handler.cc
M content/browser/devtools/protocol/storage_handler.h
M content/browser/devtools/protocol_config.json
M third_party/WebKit/Source/devtools/front_end/application_test_runner/ResourcesTestRunner.js
M third_party/WebKit/Source/devtools/front_end/test_runner/TestRunner.js
5 files changed, 48 insertions(+), 11 deletions(-)
To view, visit change 792534. To unsubscribe, or for help writing mail filters, visit settings.
Will Chen uploaded patch set #16 to this change.
DevTools: deflake application panel tests by resetting storage state
This relies on a (deprecated) mechanism for waiting for an async init task and calls
ClearDataForOrigin.
This makes ClearDataForOrigin more robust by making it explicit that it's
asynchronous and (mostly) fixing the race condition.
The callback doesn't guarantee the storage has actually been deleted but
at least it has been scheduled (see storage_partition.h), whereas before
it returned with a success response immediately.
We could update third_party/WebKit/Source/devtools/front_end/resources/ClearStorageView.js
to actually wait for this callback to be called to update the UI when the user
clicks on the clear storage button rather than rely on a 500ms timeout,
but let's defer this for a follow-up patch.
Bug: 667560
Change-Id: Ia7b70aebabd970957ae3581e47ee31a10992c657
---
M content/browser/devtools/protocol/storage_handler.cc
M content/browser/devtools/protocol/storage_handler.h
M content/browser/devtools/protocol_config.json
M third_party/WebKit/Source/devtools/front_end/application_test_runner/ResourcesTestRunner.js
M third_party/WebKit/Source/devtools/front_end/test_runner/TestRunner.js
5 files changed, 48 insertions(+), 11 deletions(-)
To view, visit change 792534. To unsubscribe, or for help writing mail filters, visit settings.
ptal. I couldn't use testRunner.clearAllDatabases since it's just WebSQL and this covers a wide range of storages.
Patch set 19:Commit-Queue +1
Patch set 19:Code-Review +1
2 comments:
File content/browser/devtools/protocol/storage_handler.cc:
Patch Set #19, Line 320: base::BindOnce(&ClearedDataForOrigin, base::Passed(std::move(callback))));
base::BindOnce(&ClearDataForOriginCallback::sendSuccess, std::move(callback))
File third_party/WebKit/Source/devtools/front_end/test_runner/TestRunner.js:
Patch Set #19, Line 667: TestRunner.deprecatedWaitForInitAsyncTask = function() {
Instead of bringing more deprecated stuff, should we just make an explicit helper function ResourcesTestRunner.init() and await it in all tests?
To view, visit change 792534. To unsubscribe, or for help writing mail filters, visit settings.
Will Chen uploaded patch set #21 to this change.
DevTools: deflake application panel tests by resetting storage state
This creates an ApplicationTestRunner.resetState() method to reset storage state
and is added to the beginning of every test that relies on a storage API.
This makes ClearDataForOrigin more robust by making it explicit that it's
asynchronous and (mostly) fixing the race condition.
The callback doesn't guarantee the storage has actually been deleted but
at least it has been scheduled (see storage_partition.h), whereas before
it returned with a success response immediately.
We could update third_party/WebKit/Source/devtools/front_end/resources/ClearStorageView.js
to actually wait for this callback to be called to update the UI when the user
clicks on the clear storage button rather than rely on a 500ms timeout,
but let's defer this for a follow-up patch.
Bug: 667560
Change-Id: Ia7b70aebabd970957ae3581e47ee31a10992c657
---
M content/browser/devtools/protocol/storage_handler.cc
M content/browser/devtools/protocol/storage_handler.h
M content/browser/devtools/protocol_config.json
M third_party/WebKit/LayoutTests/http/tests/devtools/appcache/appcache-iframe-manifests.js
M third_party/WebKit/LayoutTests/http/tests/devtools/appcache/appcache-manifest-with-non-existing-file.js
M third_party/WebKit/LayoutTests/http/tests/devtools/appcache/appcache-swap.js
M third_party/WebKit/LayoutTests/http/tests/devtools/application-panel/resources-panel-on-navigation.js
M third_party/WebKit/LayoutTests/http/tests/devtools/application-panel/resources-panel-resource-preview.js
M third_party/WebKit/LayoutTests/http/tests/devtools/application-panel/resources-panel-selection-on-reload.js
M third_party/WebKit/LayoutTests/http/tests/devtools/application-panel/resources-panel-websql.js
M third_party/WebKit/LayoutTests/http/tests/devtools/application-panel/storage-view-reports-quota.js
M third_party/WebKit/LayoutTests/http/tests/devtools/cache-storage/cache-data.js
M third_party/WebKit/LayoutTests/http/tests/devtools/cache-storage/cache-deletion.js
M third_party/WebKit/LayoutTests/http/tests/devtools/cache-storage/cache-entry-deletion.js
M third_party/WebKit/LayoutTests/http/tests/devtools/cache-storage/cache-live-update-cache-content.js
M third_party/WebKit/LayoutTests/http/tests/devtools/cache-storage/cache-live-update-list.js
M third_party/WebKit/LayoutTests/http/tests/devtools/cache-storage/cache-names.js
M third_party/WebKit/LayoutTests/http/tests/devtools/cache-storage/cache-track-valid-origin.js
M third_party/WebKit/LayoutTests/http/tests/devtools/indexeddb/database-data.js
M third_party/WebKit/LayoutTests/http/tests/devtools/indexeddb/database-names.js
M third_party/WebKit/LayoutTests/http/tests/devtools/indexeddb/database-refresh-view.js
M third_party/WebKit/LayoutTests/http/tests/devtools/indexeddb/database-structure.js
M third_party/WebKit/LayoutTests/http/tests/devtools/indexeddb/delete-entry.js
M third_party/WebKit/LayoutTests/http/tests/devtools/indexeddb/live-update-indexeddb-content.js
M third_party/WebKit/LayoutTests/http/tests/devtools/indexeddb/live-update-indexeddb-list.js
M third_party/WebKit/LayoutTests/http/tests/devtools/indexeddb/resources-panel.js
M third_party/WebKit/LayoutTests/http/tests/devtools/indexeddb/track-valid-origin.js
M third_party/WebKit/LayoutTests/http/tests/devtools/indexeddb/transaction-promise-console.js
M third_party/WebKit/LayoutTests/http/tests/devtools/indexeddb/upgrade-events.js
M third_party/WebKit/LayoutTests/http/tests/devtools/service-workers/lazy-addeventlisteners.js
M third_party/WebKit/LayoutTests/http/tests/devtools/service-workers/service-worker-agents.js
M third_party/WebKit/LayoutTests/http/tests/devtools/service-workers/service-worker-manager.js
M third_party/WebKit/LayoutTests/http/tests/devtools/service-workers/service-worker-network-fetch-blocked.js
M third_party/WebKit/LayoutTests/http/tests/devtools/service-workers/service-worker-network-fetch.js
M third_party/WebKit/LayoutTests/http/tests/devtools/service-workers/service-worker-pause.js
M third_party/WebKit/LayoutTests/http/tests/devtools/service-workers/service-worker-v8-cache.js
M third_party/WebKit/LayoutTests/http/tests/devtools/service-workers/service-workers-bypass-for-network-cors.js
M third_party/WebKit/LayoutTests/http/tests/devtools/service-workers/service-workers-bypass-for-network-navigation.js
M third_party/WebKit/LayoutTests/http/tests/devtools/service-workers/service-workers-bypass-for-network-redirect.js
M third_party/WebKit/LayoutTests/http/tests/devtools/service-workers/service-workers-force-update-on-page-load.js
M third_party/WebKit/LayoutTests/http/tests/devtools/service-workers/service-workers-navigation-preload.js
M third_party/WebKit/LayoutTests/http/tests/devtools/service-workers/service-workers-redundant.js
M third_party/WebKit/LayoutTests/http/tests/devtools/service-workers/service-workers-view.js
M third_party/WebKit/LayoutTests/http/tests/devtools/service-workers/user-agent-override.js
M third_party/WebKit/LayoutTests/http/tests/devtools/storage-panel-dom-storage-update.js
M third_party/WebKit/LayoutTests/http/tests/devtools/storage-panel-dom-storage.js
M third_party/WebKit/Source/devtools/front_end/application_test_runner/ResourcesTestRunner.js
47 files changed, 152 insertions(+), 12 deletions(-)
To view, visit change 792534. To unsubscribe, or for help writing mail filters, visit settings.
thanks.
Patch set 22:Commit-Queue +2
2 comments:
File content/browser/devtools/protocol/storage_handler.cc:
Patch Set #19, Line 320: const String& origin,
base::BindOnce(&ClearDataForOriginCallback::sendSuccess, std::move(callback))
Done
File third_party/WebKit/Source/devtools/front_end/test_runner/TestRunner.js:
Instead of bringing more deprecated stuff, should we just make an explicit helper function Resources […]
OK.
To view, visit change 792534. To unsubscribe, or for help writing mail filters, visit settings.
CQ is trying the patch.
Note: The patchset sent to CQ was uploaded after this CL was approved.
"period" https://chromium-review.googlesource.com/c/792534/22
Follow status at: https://chromium-cq-status.appspot.com/v2/patch-status/chromium-review.googlesource.com/792534/22
Bot data: {"action": "start", "triggered_at": "2017-12-15T23:59:08.0Z", "cq_cfg_revision": "b547f51ef97353cccb06eebcda71133909f61295", "revision": "27b791c041c6b9ac4ac918c43a80122c14b171d5"}
Commit Bot merged this change.
DevTools: deflake application panel tests by resetting storage state
This creates an ApplicationTestRunner.resetState() method to reset storage state
and is added to the beginning of every test that relies on a storage API.
This makes ClearDataForOrigin more robust by making it explicit that it's
asynchronous and (mostly) fixing the race condition.
The callback doesn't guarantee the storage has actually been deleted but
at least it has been scheduled (see storage_partition.h), whereas before
it returned with a success response immediately.
We could update third_party/WebKit/Source/devtools/front_end/resources/ClearStorageView.js
to actually wait for this callback to be called to update the UI when the user
clicks on the clear storage button rather than rely on a 500ms timeout,
but let's defer this for a follow-up patch.
Bug: 667560
Change-Id: Ia7b70aebabd970957ae3581e47ee31a10992c657
Reviewed-on: https://chromium-review.googlesource.com/792534
Commit-Queue: Will Chen <chenw...@chromium.org>
Reviewed-by: Dmitry Gozman <dgo...@chromium.org>
Cr-Commit-Position: refs/heads/master@{#524555}
47 files changed, 152 insertions(+), 12 deletions(-)