Testing CuPy Functionality in CI for Public Projects

[Ognyan Moore]

Hello CuPy team and friends,

My name is Ogi, I'm a pyqtgraph maintainer.  We have recently introduced optional CuPy support for part of our library, which was feasible due to CuPy's support for __array_function__, so thank you for that!  Performance gains for us when dealing with large images is quite substantial!

We do want to test these CuPy paths as part of our CI process though, as we periodically do make changes that result in CuPy functionality breaking (see: https://github.com/cupy/cupy/issues/5021 ).  As a result, we would like to test CuPy functionality in our CI pipelines, which brings us to an issue I have no idea about; what CI service should I even look at?  I could run a self-hosted container, but GitHub has explicit warnings against doing this for public projects.

The solution doesn't have to be free (would be great if it was though!) but I'm curious what the CuPy did.  I took a glance n the pipelines, and saw that likely Jenkins was the platform used, but I couldn't identify any kind of config within the repo (could have easily missed it, as I've never used Jenkins).

So with that, I figured I would ask here, what do you all do for CI pipelines testing CuPy code?  What would you recommend for projects that use CuPy as a dependency?

Thanks!

Ogi

[Kenichi Maehashi]

Hello Ogi,

I'm Kenichi from the CuPy team. Thank you for using CuPy and congratulations on the v0.12 release including GPU support!

As you see our primary CI pipeline is on Jenkins. The master node (Jenkins Web UI) is on AWS EC2, and the worker (a GPU node to runs test) is on-premise.

Test scripts are placed in the separate repository: https://github.com/chainer/chainer-test

I don't think this repo is useful for downstream projects (it's a bit complicated as we have to test against various CUDA versions and library versions), but in summary, `./run_test.py --test cupy-py3` builds a docker image containing all dependencies (including CUDA Toolkit) and runs CuPy build and unit tests inside the container to isolate the test environment.

Speaking of Jenkins,

Pros are:

- Flexible; you can do almost everything you want

- Plenty of resources and plug-ins; GitHub integration to trigger tests via test phrase ("Jenkins, test this please"), launching worker node in the cloud AWS/GCP/Azure only when needed (this can reduce cost if the test run is not very frequent)

Cons are:

- Not a SaaS solution; you need to manage whole master/worker servers.

- No per-test isolation support; you need to do it by yourself (e.g., Docker as described above)

Other choices are (although we don't have much experience):

- CircleCI Cloud provides the GPU instance: https://circleci.com/pricing/

- GitHub Actions with self-hosted runners with GPU, as you mentioned. GitHub actions are often targeted by attackers, but I think the risk can be reduced by setting the workflow to only trigger by a test phrase.

Finally, unfortunately, there's no service offering test infrastructure with CUDA for free, AFAIK. You may need to change the development flow (e.g., use a test phrase to trigger GPU test instead of running tests automatically for all pull-requests) to reduce the number of test runs.

This will also reduce the security risk of malicious pull-request attacks like:

https://therecord.media/github-investigating-crypto-mining-campaign-abusing-its-server-infrastructure/

Hope this helps!

Thanks,

Kenichi

2021年4月6日火曜日 0:58:08 UTC+9 ognyan...@gmail.com:

[Leo Fang]

Hello Ogi,

Someone recently told and demoed a pyqtgraph-based application to me and it was very impressive. I didn't know it also supports CuPy! I'm curious how is CuPy being utilized in your project. Perhaps we can have an offline exchange? (My email can be found via my GitHub handle @leofang.)

Regarding GPU CI, I recently learned another option https://cirun.io/, see https://github.com/pystatgen/sgkit/issues/512. But I have never used it.

Best,

Leo

Kenichi Maehashi 在 2021年4月7日 星期三上午12:16:40 [UTC-4] 的信中寫道：

[Amit Kumar]

Hi everyone!

"GPU CI" caught my eye, I wanted to ask if this is still a problem, i.e. running GPU tests in the CI. I am the creator of Cirun.io and also a committer to sgkit project. We recently started using Cirun for GPU testing in GitHub Actions. If people are interested I can help setting up Cirun for CuPy. Please note that Cirun is free for Open Source projects.

-

Best,

Amit