NV12 support on Linux platforms

[Kieran Bingham]

Hi all,

There are now many devices running Linux which utilise the Intel IPU3 and IMGU image processing system for cameras. This provides a way to connect MIPI raw sensors to the Intel CPU and process and capture a video feed. This is quite drastically different from UVC cameras and requires additional software support.

To support this in Linux, there is an IPU3 driver, along with specific drivers for each sensor, and while the V4L2 kernel APIs are still used in the kernel, to manage this in userspace - an extra camera stack - libcamera (https://libcamera.org) is used.

The IPU3 (or rather the IMGU) is however only capable of producing NV12 (*1)

Is there any further restriction or otherwise that prevents NV12 support in webRTC?

(*1) - While I expect the hardware could probably do other formats we don't have any visibility of the firmware sources so we are limited to the NV12 format it provides.

Looking at https://webrtc.googlesource.com/src/+/refs/heads/main/modules/video_capture/linux/video_capture_linux.cc#133  I suspect that extending that table of supported formats may be all that is required. Is there anything else? I see the image is already converted to I420 at https://webrtc.googlesource.com/src/+/refs/heads/main/modules/video_capture/linux/video_capture_linux.cc#412 so can the input formats simply be extended to list all formats supported by libYUV?

If we can resolve this or otherwise support NV12 - then devices such as the Microsoft Surface will be able to use the camera support through the V4L2 layer of libcamera. Currently they are working around this limitation by capturing video using gstreamer, converting the pixel format in software, and feeding frames back through V4L2-loopback to allow Firefox/Chrome to capture (https://github.com/linux-surface/linux-surface/wiki/Camera-Support#gstreamer-loopback-device-for-other-applications).

--

Regards

Kieran