webnn: fix an issue of conformance tests ran TIMEOUT under Page Heap [chromium/src : main]
Dai, Feng (Gerrit)
Dai, Feng added 1 comment![]( "Open in Gerrit")
Related details
- Hu, Ningxin
- Reilly Grant
Code-Coverage
Code-Owners
Code-Review
Review-Enforcement
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
Reilly Grant (Gerrit)
Reilly Grant added 1 comment![]( "Open in Gerrit")
}, {explicit_timeout: true});I've never used `explicit_timeout` before but my understanding is that it expects that the test will call `timeout()` explicitly to signal that the test has taken longer than expected. Adding it here but not adding any code to call `timeout()` simply means that the test will never time out.
I don't think that's your intention here but I suspect I'm missing something. Typically when running tests under a tool like MSan an option is passed to the test runner to increase the timeout with the understanding that the tool generally makes execution slower. Is that an option here?
Related details
- Dai, Feng
- Hu, Ningxin
Code-Coverage
Code-Owners
Code-Review
No-Unresolved-Comments
Review-Enforcement
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
Dai, Feng (Gerrit)
Dai, Feng added 1 comment![]( "Open in Gerrit")
}, {explicit_timeout: true});I've never used `explicit_timeout` before but my understanding is that it expects that the test will call `timeout()` explicitly to signal that the test has taken longer than expected. Adding it here but not adding any code to call `timeout()` simply means that the test will never time out.
I don't think that's your intention here but I suspect I'm missing something. Typically when running tests under a tool like MSan an option is passed to the test runner to increase the timeout with the understanding that the tool generally makes execution slower. Is that an option here?
Is that an option here?
According to https://web-platform-tests.org/writing-tests/testharness-api.html#setup, there's such a `timeout_multiplier` setting (a Number - Multiplier to apply to timeouts.).
Current we add the meta element "// META:
timeout=long", the timeout is `60000` ms, with this `timeout_multiplier` setting, the timeout will increase to `60000 * timeout_multiplier` ms .
Since these conformance tests could run on different devices, and execution time aren't the same, so there would be some tests running TIMEOUT` even if we use fixed timeout timing with `timeout_multiplier` setting.
If we configure `explicit_timeout` setting to be "true", no matter what kind of devices, the test will never time out [by test running canceling the timeout with this setting](https://chromium.googlesource.com/chromium/src/+/refs/heads/main/third_party/blink/web_tests/external/wpt/resources/testharness.js#3748) as our expected.
Here're the relevant `explicit_timeout` codes in [`setup()` of testharness.js](https://chromium.googlesource.com/chromium/src/+/refs/heads/main/third_party/blink/web_tests/external/wpt/resources/testharness.js#3729) . Please take a look, thanks.
```js
Tests.prototype.setup = function(func, properties)
{
......
for (var p in properties) {
if (properties.hasOwnProperty(p)) {
var value = properties[p];
if (p === "allow_uncaught_exception") {
this.allow_uncaught_exception = value;
} else if (p === "explicit_done" && value) {
this.wait_for_finish = true;
} else if (p === "explicit_timeout" && value) {
this.timeout_length = null;
if (this.timeout_id)
{
clearTimeout(this.timeout_id);
}
} else if (p === "single_test" && value) {
this.set_file_is_test();
} else if (p === "timeout_multiplier") {
this.timeout_multiplier = value;
if (this.timeout_length) {
this.timeout_length *= this.timeout_multiplier;
}
} else if (p === "hide_test_state") {
this.hide_test_state = value;
} else if (p === "output") {
this.output = value;
} else if (p === "debug") {
settings.debug = value;
}
}
}
......
};
```
Related details
- Hu, Ningxin
- Reilly Grant
Code-Coverage
Code-Owners
Code-Review
Review-Enforcement
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
Dai, Feng (Gerrit)
Sorry, there's a typos in above comment, the `running` should be `runner` as below updated.
If we configure `explicit_timeout` setting to be "true", no matter what kind of devices, the test will never time out [by test runner canceling the timeout with this setting](https://chromium.googlesource.com/chromium/src/+/refs/heads/main/third_party/blink/web_tests/external/wpt/resources/testharness.js#3748) as our expected.
Dwayne Robinson (Gerrit)
Dwayne Robinson added 1 comment![]( "Open in Gerrit")
Dwayne has no expertise to offer on this timeout attribute - deferring to Reilly (but I am curious to know how long they actually take with and without page heap enabled).
Dai, Feng (Gerrit)
Dai, Feng added 1 comment![]( "Open in Gerrit")
Dwayne has no expertise to offer on this timeout attribute - deferring to Reilly (but I am curious to know how long they actually take with and without page heap enabled).
For example of running subgraph tests (conformance_tests/subgraph.https.any.js) by ORT OV CPU EP on my local MTL laptop, without PageHeap the execution test time costs about 1.4s. With PageHeap and configured `explicit_timeout` setting, it would cost about 11.9s to 12.2s.
For this case, if we configured `timeout_multiplier` as `2`, the subgraph tests would sometimes run with flaky results. Similarly, it's not easy to give a specific `timeout_multiplier` number for other tests.
|subgraph tests \ execution time(ms)| Run 1 | Run 2 | Run 3|
|--|--|--|--|
|w/o PageHeap | 1540.368896484375 | 1400.59716796875 | 1429.67822265625|
|w/ PageHeap + explicit_timeout(true) | 119788.70092773438 | 122319.15307617188 | 121820.81176757812|
Related details
- Dwayne Robinson
- Hu, Ningxin
- Reilly Grant
Code-Coverage
Code-Owners
Code-Review
Review-Enforcement
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
Dai, Feng (Gerrit)
Dai, Feng added 1 comment![]( "Open in Gerrit")
Dai, FengDwayne has no expertise to offer on this timeout attribute - deferring to Reilly (but I am curious to know how long they actually take with and without page heap enabled).
For example of running subgraph tests (conformance_tests/subgraph.https.any.js) by ORT OV CPU EP on my local MTL laptop, without PageHeap the execution test time costs about 1.4s. With PageHeap and configured `explicit_timeout` setting, it would cost about 11.9s to 12.2s.
For this case, if we configured `timeout_multiplier` as `2`, the subgraph tests would sometimes run with flaky results. Similarly, it's not easy to give a specific `timeout_multiplier` number for other tests.
|subgraph tests \ execution time(ms)| Run 1 | Run 2 | Run 3|
|--|--|--|--|
|w/o PageHeap | 1540.368896484375 | 1400.59716796875 | 1429.67822265625|
|w/ PageHeap + explicit_timeout(true) | 119788.70092773438 | 122319.15307617188 | 121820.81176757812|
Updated.
With PageHeap and configured explicit_timeout setting, it would cost about `119`s to `122`s.
Dwayne Robinson (Gerrit)
Dwayne Robinson added 2 comments![]( "Open in Gerrit")
🫡
Dai, FengDwayne has no expertise to offer on this timeout attribute - deferring to Reilly (but I am curious to know how long they actually take with and without page heap enabled).
Dai, FengFor example of running subgraph tests (conformance_tests/subgraph.https.any.js) by ORT OV CPU EP on my local MTL laptop, without PageHeap the execution test time costs about 1.4s. With PageHeap and configured `explicit_timeout` setting, it would cost about 11.9s to 12.2s.
For this case, if we configured `timeout_multiplier` as `2`, the subgraph tests would sometimes run with flaky results. Similarly, it's not easy to give a specific `timeout_multiplier` number for other tests.
|subgraph tests \ execution time(ms)| Run 1 | Run 2 | Run 3|
|--|--|--|--|
|w/o PageHeap | 1540.368896484375 | 1400.59716796875 | 1429.67822265625|
|w/ PageHeap + explicit_timeout(true) | 119788.70092773438 | 122319.15307617188 | 121820.81176757812|
Updated.
With PageHeap and configured explicit_timeout setting, it would cost about `119`s to `122`s.
Oof, so 77x to 87x longer. 🐢
(ideally, increased timeouts would only apply *when* pageheap is enabled, rather than permanently bumping up the normal case too, but JS has no visibility into that low level detail, and there's not even an official C API to test it.)
Related details
- Dai, Feng
- Hu, Ningxin
- Reilly Grant
Code-Coverage
Code-Owners
Code-Review
Review-Enforcement
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |