Codec selection in getUserMedia

[Jialun Hu]

Hello,

I am trying to discover some workaround to have getUserMedia select MJPEG over NV12 with identical size and FPS, with minimal or ideally no code change, and have been walking through the media code.

As I understand it, Blink gets all device info from MediaDevicesDispatcherHost::GetVideoInputCapabilities, does constraint processing, and opens a particular stream for consumption by means of GenerateStreams. I am having trouble connecting the dots though,

- UserMediaProcessor::SelectVideoDeviceSettings only receives NV12 formats; how and why are the other pixel formats filtered out?
- VideoCaptureDeviceMFWin has GetBestMatchedCapability that seems to perform format selection logic. It is called after constraint processing, but does not seem to reflect the selected settings or affect anything. How is it used, where does it get its requested format from, and how does it work with selection on renderer side?

Any tips on general code navigation across task and mojo boundaries would also be much appreciated, as I find them rather hard and slow to navigate both statically on Code Search and dynamically inside a debugger.

Best regards,
~jialun.hu

[Dale Curtis]

Are you looking to have MJPEG frames delivered to the JS/HTML side? Or just don't want NV12 selected during capturing? In cases where we select MJPEG we always convert to a YUV format since the rest of the pipeline can't handle MJPEG frames:

https://source.chromium.org/chromium/chromium/src/+/main:media/capture/video/video_capture_device_client.cc;l=645;drc=7440118889a85f0e016efd3b4fd351dd5a23c3a2

For the MediaFoundation one I think you could modify the listing here to drop formats you don't want.

https://source.chromium.org/chromium/chromium/src/+/main:media/capture/video/win/video_capture_device_mf_win.cc;l=398;drc=7440118889a85f0e016efd3b4fd351dd5a23c3a2

+Ilya Nikolaevskiy for more details. 

- dale

--
You received this message because you are subscribed to the Google Groups "media-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to media-dev+...@chromium.org.
To view this discussion visit https://groups.google.com/a/chromium.org/d/msgid/media-dev/f37b04c3-8257-4017-9a6c-20224e3a7afan%40chromium.org.

[Jialun Hu]

Thank you very much Dale and Ilya for the really detailed explanation.

> Are you looking to have MJPEG frames delivered to the JS/HTML side? Or just don't want NV12 selected during capturing?

> If you are OK with the frames being decoded to NV12 in chrome and exposed like that to JS, but using Mjpeg on the camera

My use case is for a camera device which exposes a "cooked" MJPEG stream (with configurable quality, FX, etc. builtin) and a raw unpacked NV12 stream on the same source. I would like to select the former, while Chromium prefers the latter for performance reasons. 

As long as the result originates from the MJPEG one from the device, it does not matter if it is decoded by MediaFoundation, Chromium or someone else.

> Usually a capturer gets a list of possible formats from the camera, then chooses the "best" one.

Is my understanding correct that the Blink side, after processing constraints, passes down a VideoCaptureParams::requested_format to the media utility process, which then selects one suitable stream on the device?

> We will oppose changing this default in chrome for performance reasons. If some specific camera doesn't work nicely unless mjpeg is selected, it would be fine to make a fix for that camera specifically, but not changing the default preference.

This is indeed a very specific use case and not likely something that could be upstreamed. Unfortunately, it is a general issue that is not device-specific -- the fact that streams under a single source are sometimes not mutually substitutable in content despite sharing the same heuristics (FPS, size, etc).

And on a related note, what is the blessed/expected flow here? Should camera vendors expose such streams through separate UVC devices (but users would see multiple entries for one single physical device)? Should browsers offer an "advanced" dropdown for power users to manually select the underlying stream?

~jialun.hu

On 20 Aug 2025, at 4:19 PM, Ilya Nikolaevskiy <il...@google.com> wrote:

As Dale pointed out, even if mjpeg is used by the camera, the pipeline itself can't handle MJPEG, it's always converted to YUV early in the capture pipeline. 

There are a lot of places which would break otherwise. Another one is VideoCaptureImpl. But to find them all some PoC will be needed.

I also think MJPEG media::VideoFrames wouldn't be renderable, you wouldn't be able to pass them to webrtc, media recorder, etc. There are half a dozen consumers which you will also have to fix.

Also, the pixel format isn't exposed anywhere to the JS/HTML, there's just no API to control it. So there will be a very huge hassle to add it to GetUserMedia constraints, where the resolution is already exposed. It will involve bureaucracy: standardization talks. After that, the capturers already can expose pixel format,  they only expose just one usually (NV12 or I420 based on gpu memory buffers being allowed or not).

Then you'll have to change each individual capturer in media/capture/video, add a new capability, which is the same resolution but MJPEG.

Windows capturer for example forces to expose everything as NV12/I420 here.

Overall, this is quite a huge project.

If you are OK with the frames being decoded to NV12 in chrome and exposed like that to JS, but using Mjpeg on the camera, it's doable.

Again, you'll have to update each platform-specific capturer individually. Usually a capturer gets a list of possible formats from the camera, then chooses the "best" one. On windows it's done here.

However, we generally prefer to avoid mjpeg, because decoding it is expensive. If the camera can provide an uncompressed format for the resolution we prefer it.

Also, note, on Windows and MacOS the system usually exposes "fake" nv12, which is actually mjpeg, decoded by the system. We try to avoid this one too.

We will oppose changing this default in chrome for performance reasons. If some specific camera doesn't work nicely unless mjpeg is selected, it would be fine to make a fix for that camera specifically, but not changing the default preference. 

--

Best regards,

Ilya