How to Create GPU SkImage in a Background Thread and Draw it on a Main SkSurface with OpenGL and Skia?

[stephane]

I'm integrating Skia with OpenGL and facing a multi-threading challenge. I need to create a SkImage on a GPU in a background thread and render it on a SkSurface in the main thread. How can I safely share the OpenGL context and synchronize these operations between the threads? Any examples or best practices for this scenario in Skia and OpenGL would be very helpful.

[Michael Katz]

I don't think you can share the OpenGL context across threads. But if you capture the image as an SkImage in the background thread, I think you can access that in the main thread because the SkImage itself is not tied to the OpenGL context. If I'm remembering correctly. (You'd still need thread locking around access to the SkImage, of course.)

On Friday, February 23, 2024 at 01:40:13 PM PST, stephane <zeus9...@gmail.com> wrote:

I'm integrating Skia with OpenGL and facing a multi-threading challenge. I need to create a SkImage on a GPU in a background thread and render it on a SkSurface in the main thread. How can I safely share the OpenGL context and synchronize these operations between the threads? Any examples or best practices for this scenario in Skia and OpenGL would be very helpful.

--
You received this message because you are subscribed to the Google Groups "skia-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to skia-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/skia-discuss/13b52d59-156a-4cd2-9cd4-a42928bcad16n%40googlegroups.com.

[stephane]

Thanks but  according to EGL 1.4 specification it should be possible by using
EGLContext from the main thread as "share_context" parameter to
eglCreateContext during creation of the second EGLContext on the
second thread.

eglCreateContext's third parameter is defined as:
share_context Specifies another EGL rendering context with which to share data, as defined
by the client API corresponding to the contexts. Data is also shared with all
other contexts with which share_context shares data. EGL_NO_CONTEXT 
indicates that no sharing is to take place.

[K. Moon]

OpenGL is a single-threaded API. I don't have experience with sharing data across contexts like this, but it sounds like using two separate contexts that share data, not one. Support for this sort of thing is generally not great; you're generally expected to use Vulkan for this use case.

As I said, though, I don't have experience with this. It seems more like an EGL question than a Skia question, though.

To view this discussion on the web visit https://groups.google.com/d/msgid/skia-discuss/e3433443-06b1-49a6-8d3f-9fc8a952349fn%40googlegroups.com.

[stephane]

Yes, I use two separate contexts to share data. It works like a charm in my OpenGL app, but for some reason, it's not working with Skia.

Here's what I'm doing:

Main UI Thread:
MainContext = eglCreateContext(display, config, EGL_NO_CONTEXT (share_context), attrib_list);

Background Thread:
BackgroundContext = eglCreateContext(display, config, MainContext (share_context), attrib_list);

According to the EGL documentation, share_context specifies another EGL rendering context with which to share data, as defined by the client API corresponding to the contexts. Data is also shared with all other contexts that share data with share_context.

Therefore, textures created in the background thread should be available in the MainContext running in the main UI thread. However, I can't successfully draw them on the main SkSurface running in the main UI thread. What am I doing wrong? As I mentioned, this approach works in my OpenGL app without any problems, but I don't understand why it's not working with Skia.
 

[K. Moon]

Do you have an example of the Skia calls you're using? How are you creating the SkImage?

To view this discussion on the web visit https://groups.google.com/d/msgid/skia-discuss/4bcd2cbd-2057-4bd4-aa3e-d55baf6601e7n%40googlegroups.com.

[stephane]

Yer sure : 


MAINTHREAD
LMainContext = eglCreateContext(display, config, EGL_NO_CONTEXT (share_context), attrib_list); 
LMainEglSurface = eglCreateWindowSurface(...,ANativeWindow, ....);

eglMakeCurrent(LDisplay,  LMainEglSurface, LMainEglSurface  LMainContext  );

LMainGrDirectContext := GrDirectContext::MakeGL
LMainGrBackendRenderTarget = GrBackendRenderTarget::CreateGl(...);
FMainBackBufferSurface = SkSurface::MakeFromRenderTarget(LMainGrDirectContext, LMainGrBackendRenderTarget, ....);
LMainCanvas = FMainBackBufferSurface::getcanvas

Background Thread
LBackgroundContext = eglCreateContext(display, config,  LMainContext (share_context), attrib_list);
LSurface = eglCreatePbufferSurface(...);

eglMakeCurrent(LDisplay,  LSurface, LSurface   LBackgroundContext   );
LGrDirectContext := GrDirectContext::MakeGL
LGrBackEndTexture = grdirectcontext::createtexture( LGrDirectContext , ...);
Lsksurface = sksurface::makefromtexture(LGrDirectContext , LGrBackEndTexture, ..)
...draw on the sksurface ...
LSkImage = Lsksurface::makeimagesnapshot;

MAINTHREAD
LMainCanvas::drawimagerect(LSkImage )

and nothing is draw :( What did i do wrong ?

[John Wiseman]

You can upload the image data in your background thread with your shared context using SkImages::CrossContextTextureFromPixmap. Then, you can use the returned SkImage in your main context.

[stephane]

Thanks, I understand that I can do this. However, isn't it somewhat too expensive in terms of performance? Because I need to create a GrBackendTexture and then bind a SkSurface to it, ensuring that all operations are executed on the GPU side for optimal performance. Then, after finishing, I need to extract the pixmap from my GPU surface and from this pixmap create a cross-context SkImage. Doesn't this design incur a performance penalty?

[Greg Daniel]

So a quick rundown of what is possible here with Ganesh

1. Ganesh is not thread safe. Thus it can only be used on one thread at a time and it is the clients responsibility to ensure this happens. This includes drawing to GPU surfaces as well as things like creating and freeing resources owning GPU objects (e.g. gpu texture backed SkImages/Surfaces).
   
2. You cannot share objects that wrap GPU objects between different GrDirectContexts. So an SkSurface made on one GrDirectContext cannot be used on a different one*.
3. A GL context is not thread safe. A client is responsible for not using a GL context on multiple threads at the same time and calling makeCurrent to start using a GL Context on a new thread.
   
4. GL does allow you to make share groups which allow you to use GL objects between different GL Contexts. In general Skia does not test this so we cannot recommend using this system ourselves. Now saying that, we do know in the past we've seen driver bugs when trying to play around with share groups especially when rendering to the objects on different contexts (with appropriate synchronization of course). It does seem like things like texture creation and uploads do tend to work with less bugs across share groups.

Okay so saying that what is our recommendations here.

1. If you don't need your threads to run in parallel you could use one GL Context and on GrDirectContext that you are manually synchronizing to make sure only one thread is active at a time. This approach only really is beneficial if you have other CPU work to do on your main thread while doing some Skia work on a helper thread. Otherwise the lack of parallelization won't be worth the overhead of managing the threads.
2. Create N GL Contexts in a share group and N GrDirectContexts. Manually create GL Textures yourself and wrap them in GrBackendTextures on the various GrDirectContexts (remember you need a different SkImage/SkSurface on each GrDirectContext). It is your responsibility to make sure the synchronization between contexts are correct and you don't have two trying to use the same Resource at the same time (i.e. make sure to give Skia gpu semaphores/fences to wait on). We make no promises that the use of a share group will work without driver bugs. Your best bet with the model is to use the helper threads mostly for image creation and uploading data. Once you start rendering on the helper threads, that's where you may or may not start hitting driver bugs.
3. As mentioned earlier, we do have SkImages::CrossContextTextureFromPixmap. This sort of wraps the idea of 2. in a helper utility where we are managing the BackendTexture objects for you behind the scenes. I generally don't recommend this approach as it has a lot of caveats and was designed to work for a specific use case in Flutter. However, if that use case works for you then this is an option. Though it's worth mentioning, we don't plan on adding a similar utility like this to our new gpu backend, Graphite, so if you're trying to multithread GL now and in the future you might want to start with something like 2. FWIW all other backend GPU APIs (Vulkan, Metal, Dawn, etc.) will natively support multithreading in Graphite.

--

You received this message because you are subscribed to the Google Groups "skia-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to skia-discuss...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/skia-discuss/b1eb1486-4676-4a16-9f7d-fd26a03017d6n%40googlegroups.com.

[stephane]

Thank you so much, Greg, for your comprehensive response! I'm confident that many others will appreciate this information as well, especially considering the scarcity of resources on this topic online.

I must admit, I'm somewhat surprised to learn that creating images in parallel with Skia/GPU – specifically in background threads for later use in the main UI thread – isn't as straightforward as I hoped. It seems a bit complex to generate images in the background and then seamlessly integrate them into the main UI SkSurface.

From what I understand, the viable approach is to establish N OpenGL contexts within a share group, coupled with N GrDirectContexts. In each thread, I would need to create a unique OpenGL Texture ID, encapsulate it within a GrBackendTexture, and then construct an SkSurface on top of this GrBackendTexture for rendering. Upon completion, this OpenGL texture ID must be transferred to the main UI thread. There, it would require recreation into a new GrBackendTexture, followed by the generation of a new SkSurface, and ultimately converting this into an SkImage for rendering on the main UI's GPU-based SkSurface. So, there's no straightforward method to pass an SkImage, created on a background thread (on the top of an OpenGL Texture ID), directly to the main UI thread?

Additionally, I'm curious if this same complexity exists with Vulkan?

I'm beginning to consider sticking with raster images due to their simpler multithreading capabilities. Are there any resources that compare the performance trade-offs between raster SkImages and GPU-based SkImages?

Again, thank you immensely for your insights and assistance!

Stephane

[Brian Salomon]

On Mon, Feb 26, 2024 at 11:21 AM stephane <zeus9...@gmail.com> wrote:

Thank you so much, Greg, for your comprehensive response! I'm confident that many others will appreciate this information as well, especially considering the scarcity of resources on this topic online.

I must admit, I'm somewhat surprised to learn that creating images in parallel with Skia/GPU – specifically in background threads for later use in the main UI thread – isn't as straightforward as I hoped. It seems a bit complex to generate images in the background and then seamlessly integrate them into the main UI SkSurface.

From what I understand, the viable approach is to establish N OpenGL contexts within a share group, coupled with N GrDirectContexts. In each thread, I would need to create a unique OpenGL Texture ID, encapsulate it within a GrBackendTexture, and then construct an SkSurface on top of this GrBackendTexture for rendering. Upon completion, this OpenGL texture ID must be transferred to the main UI thread. There, it would require recreation into a new GrBackendTexture, followed by the generation of a new SkSurface, and ultimately converting this into an SkImage for rendering on the main UI's GPU-based SkSurface. So, there's no straightforward method to pass an SkImage, created on a background thread (on the top of an OpenGL Texture ID), directly to the main UI thread?

You don't need to go back into SkSurface on the main thread, just wrap the texture directly in SkImage using Skimages::AdoptTextureFrom().

Also on the producing thread could could make an SkSurface using SkSurfaces::RenderTarget (allowing Skia to make the texture), draw to the surface, snap an SkImage, and then call SkImages::MakeBackendTextureFromImage to take ownership of the SkImage's texture before passing it to the main thread. You'd want to make sure you destroy the SkSurface and then std::move the last ownership of the SkImage into MakeBackendTextureFromImage so you don't wind up with an extra copy.

Something like this:

auto surface = SkSurface::RenderTarget(direct_context, ....);

draw_stuff(surface->getCanvas());

auto image = surface->makeImageSnapshot();

surface = {};

GrBackendTexture texture;

BackendTextureReleaseProc dont_care_unused_in_open_gl;

assert(SkImages::MakeBackendTextureFromImage(context, std::move(image), &texture, &dont_care_unused_in_open_gl));

// pass texture over to the other thread and call SkImages::AdoptTextureFrom(context, texture, ...);

Also the same GrBackendTexture should be valid on both threads.

To view this discussion on the web visit https://groups.google.com/d/msgid/skia-discuss/abf1ad9f-7f3d-4045-86d9-ed42a1136ed6n%40googlegroups.com.

[stephane]

Thank you, Brian. Unfortunately, I haven't been able to get it working successfully. 😕 SkSurface::RenderTarget requires a GrBackendRenderTarget, which in turn needs an FBO and all the associated overhead. Perhaps I'm doing something wrong. Additionally, I'm uncertain about the performance gains considering all this overhead. My tests indicate that drawing a raster image and a GPU image results in similar performance. So, perhaps I should consider switching to Vulkan, which, I believe, supports multithreading?

[Brian Salomon]

You shouldn't need a GrBackendTarget if you start with this API:

https://source.chromium.org/chromium/chromium/src/+/main:third_party/skia/include/gpu/ganesh/SkSurfaceGanesh.h;l=64?q=SkSurfaces::RenderTar&sq=&ss=chromium%2Fchromium%2Fsrc:third_party%2Fskia%2F

However, thinking about this some more I realize you would need a GL fence inserted by the producing thread and waited on in the consumer thread. IIRC Skia used to have some support for fences but that was removed. So you'd have to add fencing yourself.

I'm not sure any of this actually gets any better if you use Vulkan on Ganesh. Maybe with Graphite?

To view this discussion on the web visit https://groups.google.com/d/msgid/skia-discuss/572de1e1-74f8-4905-929b-988fc28eaef5n%40googlegroups.com.

[Greg Daniel]

IIRC Skia used to have some support for fences but that was removed

I'm pretty sure all backends still support gpu to gpu synchronization with semaphores (this would be fences in GL under the hood). We did recently remove the generic GrFence object, but that is a different use case and even for that GLs underlying code for it is still active.

Using the Vulkan backend of Ganesh could allow you to use the same Vulkan Instance/Device across multiple threads. However, again since Ganesh itself isn't thread safe, you would still need to make N GrDirectContexts that all share the same VkDevice. But as before objects from one GrDirectContext aren't really shareable on another and thus the need to pass things around by pulling out the raw VkImages and using GrBackendTextures. One other caveat with Vulkan is that vkQueueSubmit is not a thread safe call. Thus if you're using the same VkQueue on multiple GrDirectContexts you need to find a way to make sure calls to vkQueueSubmit are not done simultaneously.  In Ganesh the vast majority would come from calling GrDirectContext::submit, however there are random other APIs that could cause us to submit work (e.g. read pixels). So it may be hard to guard the public Skia calls. Thus I would suggest that you give us a ptr to a local vkQueueSubmit function via the getProc passed into Vulkan creation. This function would manage a mutex and then forward the vkQueueSubmit call to the driver.

To view this discussion on the web visit https://groups.google.com/d/msgid/skia-discuss/CALsh4BfM0uke3WaQqhHmpEcQLbG-HdYCaY0aiVBK2yD9Rtm91w%40mail.gmail.com.

[Brian Salomon]

On Wed, Feb 28, 2024 at 9:36 AM 'Greg Daniel' via skia-discuss <skia-d...@googlegroups.com> wrote:

IIRC Skia used to have some support for fences but that was removed

I'm pretty sure all backends still support gpu to gpu synchronization with semaphores (this would be fences in GL under the hood). We did recently remove the generic GrFence object, but that is a different use case and even for that GLs underlying code for it is still active.

For my own edification, wouldn't this use case need GrBackendSemaphore support so that the client can create a semaphore to pass to GrDirectContext::flush() on the producing thread and then GrDirectContext::wait() on the consuming? I see support in GL is disabled here.

I saw this commit that removed GL backend semaphore support. Though, from the original change's description, maybe it never was actually usable with GL because insertion and creation are linked in GL.

To view this discussion on the web visit https://groups.google.com/d/msgid/skia-discuss/CAL_WtP5BEWX9wTmb7vRQVwyDKHXtn6Wvw8Kzc%2BVckvxYPzTLFg%40mail.gmail.com.

[Greg Daniel]

Hmm you're right I forgot we did that. I think the main reason it was removed was that it was an API that wasn't ever used so we wanted to stop maintaining it. Though I'm a little confused about the description that it wasn't usable with GL. Looking through the old code it does seem as if it should have worked with GL. In general, when Skia was asked to signal a GL semaphore it is true we created the sync object, but then we would set it on the GrBackendSemaphore for the client to then read and do what they wanted with (i.e. they would pass in uninitialized GrBackendSemaphores to the signal semaphores on GrFlushInfo). For waiting there shouldn't have been any issue since the client would have the sync object already.

I recall Kevin giving his reasoning why he thought things were broken originally (and with no users was worth just deleting rather than fixing), but I'm just not see what the issue actually was back then.

To view this discussion on the web visit https://groups.google.com/d/msgid/skia-discuss/CALsh4BfPEHsFauFJTzc0FyphLX2XCO%2B21O-h5E-3rCunPWe8ew%40mail.gmail.com.