matmul CUBLAS_STATUS_ARCH

unread,

Oct 6, 2020, 9:42:36 PM10/6/20

to CuPy User Group

matmul results in error cupy_backends.cuda.libs.cublas.CUBLASError: CUBLAS_STATUS_ARCH_MISMATCH

Similar error with cupy.dot.

The GPUs are Tesla K40c on one machine and Tesla K80 on the other.

CuPy was installed as cupy-cuda110.

Thanks, Andras

unread,

Oct 10, 2020, 6:47:44 PM10/10/20

to CuPy User Group

More specifically it works for 1D arrays as inner product, like (1,10)x(10,1), but does not work for matrices.

unread,

Oct 11, 2020, 1:34:18 AM10/11/20

to CuPy User Group

So I found in the CUDA Toolkit documentation that

CUBLAS_STATUS_ARCH_MISMATCH: cublasGemmEx is only supported for GPU with architecture capabilities equal or greater than 5.0

My K40 and K80 GPUs are 3.5

Would downgrading CUDA and CuPy to a lower version help, or should I just kill myself with a 5-year old $30K machine that is useless now?

unread,

Oct 11, 2020, 11:23:10 PM10/11/20

to CuPy User Group

Hi Andras,

Thank you for the feedback!

This issue seems specific to CUDA 11.0, which uses cublasGemmEx regardless of the compute capability.

We'll discuss and try fixing the issue here: https://github.com/cupy/cupy/issues/4111

Meanwhile using CUDA 10.2 or earlier should workaround the issue.

Thanks,

Kenichi

2020年10月11日日曜日 14:34:18 UTC+9 Andras Balogh:

unread,

Oct 12, 2020, 12:28:07 PM10/12/20

to CuPy User Group

Fantastic! Thanks!

Reply all

Reply to author

Forward

matmul CUBLAS_STATUS_ARCH_MISMATCH