GPU issues and OpenFX announcement (meeting tuesday)

[pierre...@gmail.com]

Sorry for x-posting but I figure there might be someone interested in this issue on this list.
We are having a discussion at OpenFX about GPU APIs tuesday. OpenFX is an API for VFX plugins and video processing libraries.

Details:
http://openeffects.org/news_items/ofx-association-2020-annual-meeting-virtual
https://zoom.us/j/853083971
(do not repost please)

Also I read that Windows and MacOS might become part of VFX reference platform specs and as a group (maybe 20 commercial software vendors have products supporting/depending on the OFX API) we probably form a relevant assemblage of all possible cases (or at least quorum). As you know there is OpenGL, OpenCL, Cuda, DirectX/Compute, Metal, Vulkan and different derivatives (moltenVK - vulkan on metal), SyCL etc. This is likely an issue with renderers using GPU and even FileIO.

One unresolved issue is multi-GPU on one system. First, it's impossible except perhaps for game engines to support it all. This is likely an issue with anything that compiles against a third-party library somehow. Second there is no low-level API agnostic method to specify that. I asked nVidia whose driver supports OpenGL, OpenCL, Cuda, DirectX/Compute and Vulkan and they said it does not exist. (sub-example if you don't get it, if you use OpenCL there are more GPU reported than under CUDA if you have an embedded graphics chip, maybe more if you have Intel chips with FPGA?). It looks like there is a need to put some support code somewhere on the internet that addresses the matrix OS X GPU api so the host of a library can tell which GPU to use in the language that the client library understands (e.g. Host currently runs on this API, library supports this/these API). This is not unique to us, google returns similar issues with some crypto-mining libs as well as some ML frameworks.

So there appears to be a need for abstraction probably per OS and GPU vendor, here's a pile of examples (the hosts named here are for illustration purposes, the list is not complete but a good start, and the issue is not particular to OpenFX - it's just more acute in that API context as there is no master app acting as reference platform - OpenFX is not a stateless machine or a library like OpenGL that everyone must sort of implement the same, but a set of properties to allow each host to best represent what it can support).

Baselight X - client server model (e.g. 4 clients setup all using a server with N GPU, one per thinner client in a client session room) - all fine for host, but for plugins how to tell which GPU if we don't even support the same API.
Flame: background rendering - they have a background rendering mode - Reactor - so if you have a second GPU, you can launch a render while continuing to render. Same for all examples, right now it works there via OpenGL context passing. For example if they were to reimplement Action with Vulkan on the Mac (moltenVK) as that's one case where there could be a speedup (an incentive for Apple OS driver to legitimate deprecated OpenGL/OpenCL), then what happens at plugin level, there is not even moltenVK - Metal interop for textures...
Resolve: They decimate the render timeline (used to be I think in 10 frames chunk - unclear now) - spawning one render thread per GPU in your machine.
Nuke/Fusion type applications: They are internally RAM based but Fusion supports simultaneous frame rendering even in interactive mode (i.e. part of the graph on a different render thread) and Nuke internally supports dual GPU officially now (not at level we discuss, perhaps for their own tools). This is a clear case of no VRAM texture/buffer passed and the only thing relevant is which GPU and perhaps VRAM Memory management hints.
Furthermore, more and more video in and out libraries use GPU a well in a wild west keyboard cowboys sort of way. The GPU driver also has to deal with situations like a user watching a movie on second screen while working on first screen... (multiple apps using same GPU). No one has a QA pipeline with 3 parties using GPU. What happened to us with Adobe video apps last year is they added on Windows support for Intel MP4 decoding using Intel SDK for embedded graphics and we suddenly for plugins that internally get a second image got all kinds of artifacts *we do support OpenGL/OpenCL on HD graphics for laptops without a discrete GPU. Turns out (took a while to figure out) we then needed to add a flush after each kernel basically which makes this a bit slower. Actually doing so fixed another Premiere type issue as they now have render entry points for Metal/OpenCL/Cuda ... so as we are still only OpenCL we run as CPU effect. When a CPU effect, they spawn a frame render per core - so if you have 8 cores (16 hyper-treads) they launch 8 frame renders at once. This totally breaks nVidia driver (the way I explain to myself is the way I understand the nVidia warp engine works looping until there is a slot opened) so suddenly we are much much slower using GPU as the warp engine is over-threaded...
Piling on, although OpenGL is not going away on Windows, nVidia still has marketing based implementation of not supporting multi-GPU with OpenGL on Windows with GTX cards (only with Quadro cards) - so OpenGL is useless for multi-GPU at API level for host-plugin ecosystem if you have four RTX 280 for example.

Pierre Jasmin
RE:Vision Effects