Convert YUV CVPixelBuffer to SkImage without RGB conversion

[mrousavy]

Hey all!

I'm working in an iOS Camera app where I want to build a Preview View (for the Camera Frames) in Skia (w/ Metal).

The Frame type I get is a CMSampleBuffer, which I then convert to a CVPixelBuffer.

It's source format is always Y'CbCr (aka YUV), with various variations:

• 4:2:0 or 4:2:2 or 4:4:4
• Full-Range (420f), Video-Range (420v) and apparently also HDR (420x)
• 8-Bit or 10-Bit
• Bi-Planar and not Bi-Planar

Let's assume for now that we are working with the kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange format - aka Y'CbCr (aka YUV), 4:2:0, video-range (420v), 8 Bit, and Bi-Planar.

I want to avoid a YUV -> RGB conversion, as that turns out to be quite slow.

Now here's what I came up with: https://gist.github.com/mrousavy/03658f52ad4209a8cb348683f36beeb2

This code does not work.

1. I am able to correctly create the two Metal Textures, one for Y and one for CbCr (UV).

2. I am not able to create a GrYUVABackendTextures instance, as the buffers (Y, U, V, and A) are then all marked isValid:false.

I guess this leads me to two questions:

1. Do you guys know what I'm doing wrong? Is there a different, easier way to do that?
2. Is Skia even able to work directly on the YUV SkImage, or does it internally convert to RGB anyways? I'm asking because I can also have the Camera Frame delivery pipeline send me RGB directly instead of YUV - I assume there is an internal conversion happening so this introduces an additional overhead, but if Skia does that conversion anyways I might as well do it at that Camera pipeline level instead.

[Brian Osman]

Hmm, what you're doing there looks right to me (although I'm not the expert -- other team members might notice something subtle with how the YUV info is being configured). As for your second question: Yes, if you construct an SkImage from YUV planes like this, Skia will work directly from the planes for most things (eg, sampling the image will actually sample each plane and do the math to compute RGB in the shader).

If you can debug into the Skia code, you may be able to see what's happening with the dimensions -- it probably happens in `int SkYUVAInfo::PlaneDimensions`, which is called from the YUVAInfo constructor.

--
You received this message because you are subscribed to the Google Groups "skia-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to skia-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/skia-discuss/80c9be56-9c04-4f44-9379-96e0a66c961dn%40googlegroups.com.

[Greg Daniel]

Shoud you need to use half the width and height in the CbCr textures?

To view this discussion on the web visit https://groups.google.com/d/msgid/skia-discuss/CAMcBjo5sSAmnoPfJhv-Q84CFwgEqzWehyDC1cCgJeSRERMGSAw%40mail.gmail.com.

[mrousavy]

"sampling the image will actually sample each plane and do the math to compute RGB in the shader"

Does this mean if I have a SkImage backed by a YUV Texture, the Shader will not run as efficiently as if the SkImage was in the RGB space already? Is there an overhead here?

If that's the case, then I can forget all the tricky YUV parts and just do the RGB conversion myself at the Camera pipeline level....

"Shoud you need to use half the width and height in the CbCr textures?"

You're saying _width / 2 and _height / 2? Tried that, didn't work either.

I mean, if it's 4:2:0, the width should be _width / 4 and _height / 2, no? (See this visual explanation of YUV) But that didn't work either.

Btw - the full code is here if you're wondering: https://github.com/mrousavy/react-native-vision-camera/pull/1345

I am trying to integrate the react-native-skia abstraction into my react-native-vision-camera library so users can efficiently draw on Camera frames.

I am currently struggling with performance, as converting to RGB colorspace takes a ton of time and we are not able to run at anything above 60 FPS on a quite modern iPhone 11 Pro (Cameras can do 120 FPS or even 240 FPS).

That's why I want to try and stay in the YUV colorspace.

Christian Falch also discovered that most of the time is spent on the CPU (for one Frame, we have 3ms on the GPU and 16.7ms on the CPU, but I think this might just be Metal's [_layer nextDrawable] call which is blocking until the next Frame is ready, so exactly 16.7ms.).

We are also not entirely sure if we are working 100% on the GPU, or if the CVPixelBuffer goes over to the CPU...

[mrousavy]

Also I can see that the SkSurface constructor takes a SkColorType:

    static sk_sp<SkSurface> MakeFromBackendRenderTarget(GrRecordingContext* context,

                                                const GrBackendRenderTarget& backendRenderTarget,

                                                GrSurfaceOrigin origin,

                                                SkColorType colorType,

                                                sk_sp<SkColorSpace> colorSpace,

                                                const SkSurfaceProps* surfaceProps,

                                                RenderTargetReleaseProc releaseProc = nullptr,

                                                ReleaseContext releaseContext = nullptr);

And it seems like this can only be RGB based, so can the SkSurface itself not work in YUV?

[mrousavy]

Okay update: I managed to create the SkImage from the YUV CVPixelBuffer. Woohoo 🎉

Here's what I did wrong:

1. I made a stupid mistake of using the width/height of the Canvas, not of the CVPixelBuffer. I was wondering why the input image was just 300 pixels wide lmao.

2. For the CbCr texture, I need to use half the width/height of the Y plane - you were right. This is only the case for 4:2:0 though.

3. I was using the same Metal Texture Cache for both Y and CbCr Textures, I am now using two separate ones. Not sure if that's required.

Overall, this is very hard-coded. It only works for YUV 4:2:0 8-Bit video-range images. I guess I need to add a few conditionals to also make it work for 4:4:4, 4:2:2, then 10-Bit, then also video-range vs full-range.

The problem I have now is probably memory related - The Camera is only able to draw like ~20 frames on the screen, then it freezes. I think something here is leaking, but I am not sure what.

Do you guys maybe spot anything here? SkiaMetalCanvasProvider.mm

[Jim Van Verth]

I'm not seeing any obvious leaks. Instruments could probably tell you.

To view this discussion on the web visit https://groups.google.com/d/msgid/skia-discuss/98fa3939-159d-4d07-8bd0-72a105553cd7n%40googlegroups.com.

--

Jim Van Verth | Software Engineer | jvan...@google.com | 919-210-7664

[Jim Van Verth]

SkSurface is meant as a render target, and you can't render to a YUV texture.

That 4:2:0 example has a Y w,h resolution of 4x2 and a UV resolution of 2x1, so the UV dims are w/2 and h/2.

And yes, it's likely that the nextDrawable call is blocking. You could try to use SkSurface::MakeFromCAMetalLayer which will delay the call somewhat.

To view this discussion on the web visit https://groups.google.com/d/msgid/skia-discuss/ffbe7dac-68c2-4cf3-8e3c-2e6bde98165an%40googlegroups.com.

[mrousavy]

Aha - okay - Does this mean that Skia does internal conversions from YUV -> RGB anyways?

Since this obviously comes with a performance overhead, I think it makes more sense to just configure the Camera pipeline to output me RGB buffers instead of YUV, so Apple does the conversion internally and I don't have to fiddle with constructing an SkImage from YUV buffers, right?

If I understood Apple's code correctly, they were able to draw the YUV directly to a Metal canvas for previewing and didn't have to convert to RGB viewspace at all, so it sucks that I can't do the same with Skia as this is probably an inevitable performance hit (not sure how noticeable it will be)

[Greg Daniel]

When you say Apple is able to "draw the YUV directly to a Metal canvas", are you saying the Metal canvas has a YUV format or just that you're sampling a YUV texture when drawing to a "normal" format canvas (e.g. RGBA8). If it is the latter, Skia should be able to do that without having to create a separate RGBA texture to copy the YUV into. You can create an SkImage that directly can reference the YUV planes using SkImage::MakeFromYUVATextures, https://source.chromium.org/chromium/chromium/src/+/main:third_party/skia/include/core/SkImage.h;l=439. 

To view this discussion on the web visit https://groups.google.com/d/msgid/skia-discuss/ade3e585-80dd-4ba8-802a-9a65ad605796n%40googlegroups.com.

[Jim Van Verth]

Apple has two YUV formats: GBGR422 and BGRG422. Neither can be used as a render target format -- they can only be read from (in https://developer.apple.com/metal/Metal-Feature-Set-Tables.pdf they are marked as Filter) so there's some conversion in there somewhere. Skia doesn't currently support them, just storing the Y channel as an R texture and the UV (or CbCr) channels as a smaller RG texture. We treat the SkImage as a single unit, will read/sample from both textures, and can write the result to a renderable format.

To view this discussion on the web visit https://groups.google.com/d/msgid/skia-discuss/CAL_WtP4wQhGeQrQtkHn4gSkGmz_P1oTDjfdALvi7OWspRN93XQ%40mail.gmail.com.

[mrousavy]

Okay - then I'll just configure the video output to directly give me RGB CMSampleBuffers. That way I always work with the RGB format and don't need to worry about YUV at all, since that introduces a conversion at some point anyways.

Here's the code I am now using, which seems to be really fast, not sure if there's a faster way; https://gist.github.com/mrousavy/05a02b29bedf2d1e30dcafc2168f60ab

Thanks!