WebNN: Reject batched MatMul with large K dimension size on OV NPU [chromium/src : main]
Xu, Mingming1 (Gerrit)
Xu, Mingming1 added 3 comments![]( "Open in Gerrit")
OV NPU can't support models with MatMul whose number of
columns in the first matrix is too large, resulting in hange issue
during model compilation.Xu, Mingming1When you say "OV NPU can't support ....", is this a limitation of the hardware or a bug in the EP itself? Does the "hang" really mean compilation will never complete or just that it takes a long time? If the latter, how long is "a long time"? What is the ETA for a fix?
In general, I think we should avoid doing workarounds like this and, instead, insist that vendors fix their software. Otherwise, workarounds become permanent and linger forever.
Xu, Mingming1>When you say "OV NPU can't support ....", is this a limitation of the hardware or a bug in the EP itself?
It's a NPU driver bug, not hardware limitation.
>Does the "hang" really mean compilation will never complete or just that it takes a long time? If the latter, how long is "a long time"?
As I know, it's the latter case, it takes too long time (even one day). /cc @qiuji...@intel.com
>What is the ETA for a fix?
A NPU compiler fix is WIP. Maybe the fix can be in the next NPU driver.
Xu, Mingming1The NPU compiler fix PR has been landed yet.
Done
// Workaround: Split MatMul operations with excessively large K dimension size
// into smaller chunks to prevent hang issues during model compilation on some
// NPU devices.Xu, Mingming1As discussed with @rafael....@microsoft.com offline, in this CL, we agreed to fail the graph build for this issue. We'll explore the workaround in EP. Once new EP release has the fix, we'll increase the minimum required version for that EP and remove the handling from Chromium. This would be a TODO.
Updated, PTAL, thanks!
k_dimension_size > matmul_k_dimension_limit_.value()) {Xu, Mingming1If this issue only exists for grouped / batched matmul, we should only reject that.
Done
Related details
- Hu, Ningxin
- Qiujiao Wu
- Rafael Cintron
Code-Coverage
Code-Owners
Code-Review
Review-Enforcement
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
Hu, Ningxin (Gerrit)
Hu, Ningxin voted and added 2 comments![]( "Open in Gerrit")
Votes added by Hu, Ningxin
| Code-Review | +1 |
2 comments
This CL reject such MatMul operations to prevent hang issues.nit: rejects
Related details
- Qiujiao Wu
- Rafael Cintron
- Xu, Mingming1
Code-Coverage
Code-Owners
Code-Review
No-Unresolved-Comments
Review-Enforcement
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
Xu, Mingming1 (Gerrit)
Xu, Mingming1 voted and added 1 comment![]( "Open in Gerrit")
Votes added by Xu, Mingming1
| Commit-Queue | +1 |
1 comment
This CL reject such MatMul operations to prevent hang issues.Xu, Mingming1nit: rejects
Done
Related details
- Qiujiao Wu
- Rafael Cintron
Code-Coverage
Code-Owners
Code-Review
Review-Enforcement
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
Xu, Mingming1 (Gerrit)
Xu, Mingming1 added 1 comment![]( "Open in Gerrit")
I found that the hang issue can cause the entire GPU process to hang.
I did a test: I opened more than 10 web pages running the same model (which has a hang issue caused by batched MatMul). Each page should compile the model, which blocks a background thread and never completes. The Chrome UI hung for a while, and eventually the GPU process crashed with the error:
`The GPU process crashed! Exit code: RESULT_CODE_HUNG.`
I'm concerned that this could be a serious issue. What's your opinion?
Related details
- Qiujiao Wu
- Rafael Cintron
Code-Coverage
Code-Owners
Code-Review
No-Unresolved-Comments
Review-Enforcement
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
Hu, Ningxin (Gerrit)
Hu, Ningxin voted and added 1 comment![]( "Open in Gerrit")
Votes added by Hu, Ningxin
| Code-Review | +1 |
1 comment
I found that the hang issue can cause the entire GPU process to hang.
I did a test: I opened more than 10 web pages running the same model (which has a hang issue caused by batched MatMul). Each page should compile the model, which blocks a background thread and never completes. The Chrome UI hung for a while, and eventually the GPU process crashed with the error:
`The GPU process crashed! Exit code: RESULT_CODE_HUNG.`
I'm concerned that this could be a serious issue. What's your opinion?
Thanks @mingmi...@intel.com for the analysis, I agreed this is a critical issue. You may want to document the reproduce steps at the issue https://issues.chromium.org/issues/467442135. Could a web page cause the GPU process hung issue by initiating 10 async model compilations without waiting for the completion?
Related details
- Qiujiao Wu
- Rafael Cintron
- Xu, Mingming1
Xu, Mingming1 (Gerrit)
Xu, Mingming1 added 1 comment![]( "Open in Gerrit")
Hu, NingxinI found that the hang issue can cause the entire GPU process to hang.
I did a test: I opened more than 10 web pages running the same model (which has a hang issue caused by batched MatMul). Each page should compile the model, which blocks a background thread and never completes. The Chrome UI hung for a while, and eventually the GPU process crashed with the error:
`The GPU process crashed! Exit code: RESULT_CODE_HUNG.`
I'm concerned that this could be a serious issue. What's your opinion?
Thanks @mingmi...@intel.com for the analysis, I agreed this is a critical issue. You may want to document the reproduce steps at the issue https://issues.chromium.org/issues/467442135. Could a web page cause the GPU process hung issue by initiating 10 async model compilations without waiting for the completion?
Sure, please see steps in https://issues.chromium.org/issues/467442135
Related details
- Hu, Ningxin
- Qiujiao Wu
- Rafael Cintron
Xu, Mingming1 (Gerrit)
Xu, Mingming1 added 2 comments![]( "Open in Gerrit")
Hu, NingxinI found that the hang issue can cause the entire GPU process to hang.
I did a test: I opened more than 10 web pages running the same model (which has a hang issue caused by batched MatMul). Each page should compile the model, which blocks a background thread and never completes. The Chrome UI hung for a while, and eventually the GPU process crashed with the error:
`The GPU process crashed! Exit code: RESULT_CODE_HUNG.`
I'm concerned that this could be a serious issue. What's your opinion?
Xu, Mingming1Thanks @mingmi...@intel.com for the analysis, I agreed this is a critical issue. You may want to document the reproduce steps at the issue https://issues.chromium.org/issues/467442135. Could a web page cause the GPU process hung issue by initiating 10 async model compilations without waiting for the completion?
Sure, please see steps in https://issues.chromium.org/issues/467442135
Done
Related details
- Hu, Ningxin
- Qiujiao Wu
- Rafael Cintron
Code-Coverage
Code-Owners
Code-Review
Review-Enforcement
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
Hu, Ningxin (Gerrit)
Hu, Ningxin voted and added 1 comment![]( "Open in Gerrit")
Votes added by Hu, Ningxin
| Code-Review | +1 |
- Rafael Cintron
- Wu, Qiujiao
- Xu, Mingming1
Code-Coverage
Code-Owners
Code-Review
Review-Enforcement
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
Blink W3C Test Autoroller (Gerrit)
Message from Blink W3C Test Autoroller![]( "Open in Gerrit")
Exportable changes to web-platform-tests were detected in this CL and a pull request in the upstream repo has been made: https://github.com/web-platform-tests/wpt/pull/57011.
When this CL lands, the bot will automatically merge the PR on GitHub if the required GitHub checks pass; otherwise, ecosystem-infra@ team will triage the failures and may contact you.
WPT Export docs:
https://chromium.googlesource.com/chromium/src/+/main/docs/testing/web_platform_tests.md#Automatic-export-process
Reilly Grant (Gerrit)
Reilly Grant added 5 comments![]( "Open in Gerrit")
OV NPU can't support models with batched MatMul whose K
dimension size is too large, resulting in hange issue during
model compilation.
This CL rejects such MatMul operations to prevent hang issues.```suggestion
The OpenVINO NPU EP will take a very long time to compile models
with batched MatMul operations whose K dimension size is too large.
This CL rejects such MatMul operations to prevent WebNN from
becoming unresponsive.
```
// dimension size to prevent hang issues during model compilation on some NPU
// devices.```suggestion
// dimension size to prevent the EP from becoming unresponsive during model
// compilation on some NPU devices.
```
// TODO(crbug.com/467468912): When the hang issue is fixed, remove the
// Limitation and increase the minimum required OV EP version.```suggestion
// TODO(crbug.com/467468912): When the OpenVINO issue is fixed, remove
// the limitation and increase the minimum required EP version.
```
if (!model_info_result.has_value()) {Use the ASSIGN_OR_RETURN macro to simplify this.
// It's safe to keep `first_selected_device_` as `SessionOptions` also keeps a
// reference of `Environment` which owns all EP devices.```suggestion
// It's safe to keep `first_selected_device_` as `env_` owns all EP devices.
```
Related details
- Rafael Cintron
- Wu, Qiujiao
- Xu, Mingming1
Code-Coverage
Code-Owners
Code-Review
No-Unresolved-Comments
Review-Enforcement
Xu, Mingming1 (Gerrit)
Xu, Mingming1 added 5 comments![]( "Open in Gerrit")
OV NPU can't support models with batched MatMul whose K
dimension size is too large, resulting in hange issue during
model compilation.
This CL rejects such MatMul operations to prevent hang issues.```suggestion
The OpenVINO NPU EP will take a very long time to compile models
with batched MatMul operations whose K dimension size is too large.This CL rejects such MatMul operations to prevent WebNN from
becoming unresponsive.
```
👍Done
// dimension size to prevent hang issues during model compilation on some NPU
// devices.```suggestion
// dimension size to prevent the EP from becoming unresponsive during model
// compilation on some NPU devices.
```
Done
// TODO(crbug.com/467468912): When the hang issue is fixed, remove the
// Limitation and increase the minimum required OV EP version.```suggestion
// TODO(crbug.com/467468912): When the OpenVINO issue is fixed, remove
// the limitation and increase the minimum required EP version.
```
Done
Use the ASSIGN_OR_RETURN macro to simplify this.
Good catch! Thanks!
// It's safe to keep `first_selected_device_` as `SessionOptions` also keeps a
// reference of `Environment` which owns all EP devices.```suggestion
// It's safe to keep `first_selected_device_` as `env_` owns all EP devices.
```
- Rafael Cintron
- Reilly Grant
- Wu, Qiujiao
Code-Coverage
Code-Owners
Code-Review
Review-Enforcement
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
Reilly Grant (Gerrit)
Reilly Grant voted and added 1 comment![]( "Open in Gerrit")
Votes added by Reilly Grant
| Code-Review | +1 |
1 comment
Related details
- Rafael Cintron
- Wu, Qiujiao
- Xu, Mingming1
Code-Coverage
Code-Owners
Code-Review
Review-Enforcement
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
Zainudin Muji (ZF-DK) (Gerrit)
Zainudin Muji (ZF-DK) added 1 comment![]( "Open in Gerrit")
Rafael Cintron (Gerrit)
Rafael Cintron added 5 comments![]( "Open in Gerrit")
const char* ep_name = ort_api->EpDevice_EpName(first_selected_device);
const auto iter = kKnownEPs.find(UNSAFE_BUFFERS(base::cstring_view(ep_name)));
if (iter == kKnownEPs.end()) {
return std::nullopt;
}By the time we get to this code, shouldn't the EP always be in the known EP list?
const std::vector<uint32_t>& input_a_shape =
GetOperand(matmul.a_operand_id).descriptor.shape();
uint32_t k_dimension_size = input_a_shape.back();
const OperandDescriptor& output_descriptor =
GetOperand(matmul.output_operand_id).descriptor;
bool is_batched_matmul = output_descriptor.Rank() > 2;To avoid having correctly implemented EPs pay the price of all of this lookup and checking, please structure the code such that the `batched_matmul_k_dimension_limit_` has_value() and value() calls are made first.
// Limitation: Reject batched MatMul operations with excessively large KPlease also link to the OV bug and provide a rough ETA for when we can expect a fix.
// The OpenVINO NPU limits the batched MatMul K dimension size to
// 8192.I see that you provided a detailed analysis of the situation in `GraphBuilderOrt::AddMatMulOperation`, including links to crbugs. Please add a comment here referring to that code, or vice versa, so people can easily find more details.
std::optional<uint32_t> npu_batched_matmul_k_dimension_limit;Should `npu_batched_matmul_k_dimension_limit` go in the `EpWorkarounds` structure? I know that it is already passed around in various place which might make accessing the value easier.
Related details
- Wu, Qiujiao
- Xu, Mingming1
Code-Coverage
Code-Owners
Code-Review
No-Unresolved-Comments
Review-Enforcement
Hu, Ningxin (Gerrit)
Hu, Ningxin voted and added 1 comment![]( "Open in Gerrit")
Votes added by Hu, Ningxin
| Code-Review | +1 |
1 comment
// Limitation: Reject batched MatMul operations with excessively large KPlease also link to the OV bug and provide a rough ETA for when we can expect a fix.
https://github.com/microsoft/onnxruntime/issues/26643, the fix is expected to be available in NPU driver Feb release.
Xu, Mingming1 (Gerrit)
Xu, Mingming1 added 5 comments![]( "Open in Gerrit")
const char* ep_name = ort_api->EpDevice_EpName(first_selected_device);
const auto iter = kKnownEPs.find(UNSAFE_BUFFERS(base::cstring_view(ep_name)));
if (iter == kKnownEPs.end()) {
return std::nullopt;
}By the time we get to this code, shouldn't the EP always be in the known EP list?
It's possible. The `kKnownEPs` doesn't contain default EPs like DML EP.
const std::vector<uint32_t>& input_a_shape =
GetOperand(matmul.a_operand_id).descriptor.shape();
uint32_t k_dimension_size = input_a_shape.back();
const OperandDescriptor& output_descriptor =
GetOperand(matmul.output_operand_id).descriptor;
bool is_batched_matmul = output_descriptor.Rank() > 2;To avoid having correctly implemented EPs pay the price of all of this lookup and checking, please structure the code such that the `batched_matmul_k_dimension_limit_` has_value() and value() calls are made first.
Done
// Limitation: Reject batched MatMul operations with excessively large KHu, NingxinPlease also link to the OV bug and provide a rough ETA for when we can expect a fix.
https://github.com/microsoft/onnxruntime/issues/26643, the fix is expected to be available in NPU driver Feb release.
Done
// The OpenVINO NPU limits the batched MatMul K dimension size to
// 8192.I see that you provided a detailed analysis of the situation in `GraphBuilderOrt::AddMatMulOperation`, including links to crbugs. Please add a comment here referring to that code, or vice versa, so people can easily find more details.
Done
std::optional<uint32_t> npu_batched_matmul_k_dimension_limit;Should `npu_batched_matmul_k_dimension_limit` go in the `EpWorkarounds` structure? I know that it is already passed around in various place which might make accessing the value easier.
- Hu, Ningxin
- Rafael Cintron
- Reilly Grant
- Wu, Qiujiao
Code-Coverage
Code-Owners
Code-Review
Review-Enforcement
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
Hu, Ningxin (Gerrit)
Hu, Ningxin added 4 comments![]( "Open in Gerrit")
// The fix is expected to be available in NPU driver Feb release.```suggestion
// The fix is expected to be available in NPU driver Feb '26 release.
```
std::optional<uint32_t> GetBatchedMatMulKDimensionLimit(
const OrtEpDevice* first_selected_device) {
const OrtApi* ort_api = PlatformFunctions::GetInstance()->ort_api();
const char* ep_name = ort_api->EpDevice_EpName(first_selected_device);
const auto iter = kKnownEPs.find(UNSAFE_BUFFERS(base::cstring_view(ep_name)));
if (iter == kKnownEPs.end()) {
return std::nullopt;
}
OrtHardwareDeviceType hardware_device_type = ort_api->HardwareDevice_Type(
ort_api->EpDevice_Device(first_selected_device));
if (hardware_device_type != OrtHardwareDeviceType_NPU) {
return std::nullopt;
}
return iter->second.workarounds.npu_batched_matmul_k_dimension_limit;
}Place this method into the unnamed namespace.
std::optional<uint32_t> npu_batched_matmul_k_dimension_limit;Nit: for struct, data member definition should be declared before functions.
// The maximum K dimension size of batched MatMul on certain NPU devices.You may want to explain it is unnecessary to compute `|=` operation result across EP devices, because there will be only one NPU device EP.
Related details
- Rafael Cintron
- Reilly Grant
- Wu, Qiujiao
- Xu, Mingming1
Code-Coverage
Code-Owners
Code-Review
No-Unresolved-Comments
Review-Enforcement
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
Xu, Mingming1 (Gerrit)
Xu, Mingming1 added 4 comments![]( "Open in Gerrit")
// The fix is expected to be available in NPU driver Feb release.```suggestion
// The fix is expected to be available in NPU driver Feb '26 release.
```
Done
std::optional<uint32_t> GetBatchedMatMulKDimensionLimit(
const OrtEpDevice* first_selected_device) {
const OrtApi* ort_api = PlatformFunctions::GetInstance()->ort_api();
const char* ep_name = ort_api->EpDevice_EpName(first_selected_device);
const auto iter = kKnownEPs.find(UNSAFE_BUFFERS(base::cstring_view(ep_name)));
if (iter == kKnownEPs.end()) {
return std::nullopt;
}
OrtHardwareDeviceType hardware_device_type = ort_api->HardwareDevice_Type(
ort_api->EpDevice_Device(first_selected_device));
if (hardware_device_type != OrtHardwareDeviceType_NPU) {
return std::nullopt;
}
return iter->second.workarounds.npu_batched_matmul_k_dimension_limit;
}Place this method into the unnamed namespace.
Done
std::optional<uint32_t> npu_batched_matmul_k_dimension_limit;Nit: for struct, data member definition should be declared before functions.
Done
// The maximum K dimension size of batched MatMul on certain NPU devices.You may want to explain it is unnecessary to compute `|=` operation result across EP devices, because there will be only one NPU device EP.
- Hu, Ningxin
- Rafael Cintron
- Reilly Grant
- Wu, Qiujiao
Code-Coverage
Code-Owners
Code-Review
Review-Enforcement
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
Hu, Ningxin (Gerrit)
Hu, Ningxin voted and added 1 comment![]( "Open in Gerrit")
Votes added by Hu, Ningxin
| Code-Review | +1 |
Related details
- Rafael Cintron
- Reilly Grant
- Wu, Qiujiao
- Xu, Mingming1
Code-Coverage
Code-Owners
Code-Review
Review-Enforcement
Reilly Grant (Gerrit)
Reilly Grant voted and added 1 comment![]( "Open in Gerrit")
Votes added by Reilly Grant
| Code-Review | +1 |
1 comment
- Rafael Cintron
- Wu, Qiujiao
- Xu, Mingming1
Rafael Cintron (Gerrit)
Rafael Cintron voted Code-Review+1![]( "Open in Gerrit")
- Wu, Qiujiao
- Xu, Mingming1
Hu, Ningxin (Gerrit)
Hu, Ningxin voted Commit-Queue+2![]( "Open in Gerrit")
| Commit-Queue | +2 |
Chromium LUCI CQ (Gerrit)
Chromium LUCI CQ submitted the change![]( "Open in Gerrit")
Change information
WebNN: Reject batched MatMul with large K dimension size on OV NPU
The OpenVINO NPU EP will take a very long time to compile models
with batched MatMul operations whose K dimension size is too large.
This CL rejects such MatMul operations to prevent WebNN from
becoming unresponsive.
- M services/webnn/ort/device_allocator.cc
- M services/webnn/ort/graph_builder_ort.cc
- M services/webnn/ort/graph_builder_ort.h
- M services/webnn/ort/graph_impl_ort.cc
- M services/webnn/ort/model_editor.h
- M services/webnn/ort/ort_session_options.cc
- M services/webnn/ort/ort_session_options.h
- M services/webnn/public/cpp/execution_providers_info.h
- M third_party/blink/web_tests/external/wpt/webnn/conformance_tests/matmul.https.any.js
Code-Review: +1 by Rafael Cintron, +1 by Reilly Grant, +1 by Hu, Ningxin
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
Blink W3C Test Autoroller (Gerrit)
Message from Blink W3C Test Autoroller![]( "Open in Gerrit")
The WPT PR for this CL has been merged upstream! https://github.com/web-platform-tests/wpt/pull/57011
Related details
Code-Coverage
Code-Owners
Code-Review
Review-Enforcement
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |