Web video capture (getUserMedia) pixel format with no chroma subsampling (YUV444 or RGB24) on chromium

[Baptiste Parsy]

Hi,

I am writing to this address in hopes of getting some advice regarding an issue I am unable to solve.

Despite conducting a good number of experiments and thorough research, I could not find a way to obtain a full color video/webcam stream with no chroma subsampling in chromium. The goal is to be able to display a video stream from a capture card and displaying it directly onto a canvas and drawing some overlay elements on top, so network throughput is not an issue, but the stream quality is. My company is making analyses and overlays over video streams of endoscopes, thus the need to output a good quality stream back to the clinicians.

I have been testing by feeding a still image into v4l2 with ffmpeg with RGB24 pixel format (YUV444 doesn't seem to be supported by v4l2). However, whatever I do, I cannot change the resulting stream to be something other than vp8 I420 but I cannot find how to change the vp8 codec used with another one supporting a full color profile.

I attempted using the experimental MediaStreamTrackProcessor since it said it would allow me to access raw stream bytes, however these are already I420 unless I missed a way to change it. But this makes me wonder if there could be some other experimental APIs or ongoing w3c proposals I could have missed and which could achieve what I need. I came to realize that I could have used MediaRecorder too which might have been simpler.

Here is the test code I have been using:

navigator.mediaDevices.enumerateDevices().then(async devices => {
  console.log(devices);
  const videoDevice = devices.find(device => device.kind === "videoinput" && device.label === "Virtual Webcam");

  const stream = await navigator.mediaDevices.getUserMedia({
    video: {
      deviceId: videoDevice.deviceId,
      width: {ideal: 1920},
      height: {ideal: 1080},
      frameRate: {ideal: 25},
    }
  });

  const [track] = stream.getVideoTracks();
  const {readable} = new MediaStreamTrackProcessor({track});
  const reader = readable.getReader();
  const result = await reader.read();
  console.log(result.value.format); // Outputs I420 no matter which ffmpeg pix_fmt
});

If there is no simpler solution, my company is willing to spend some people and some time to contribute to chromium to make it happen if it must come to it, but it is quite difficult to know if this is realistic as I fear we would need to write a serious and backed proposal first if I understand correctly. I hope to obtain some advice on the feasibility of this; and it would be incredibly helpful if I could at least get a very approximate number of developers and amount of time that such an endeavour would require.

Thanks a lot for your time!

Best Regards,
Baptiste Parsy,
Software Engineer at Odin Vision (Odin Medical Ltd.)
+33 6 17 20 87 69

**CONFIDENTIAL NOTICE** You have received this message from Odin Medical Limited. The information contained in this email is confidential, may be privileged and is intended solely for the use of the named addressee. If you are not the intended recipient, please contact the sender immediately and delete this email including any attachments, from your computer system or applicable device. Unauthorised disclosure, copying or other use of this email and/or attachment is prohibited. If you are the intended recipient and do not wish to receive further communication from us or to exercise your rights under the General Data Protection Regulation, please contact us using the following email address: c...@odin-vision.com and we will endeavour to comply with your request.

[Dale Curtis]

+top...@chromium.org 

We only support capture in NV12/I420 formats at this time -- I think we'll even convert to I420/NV12 if the camera serves us something else. I don't think there's even any way to specify the desired capture pixel format.

I think all our conversion code is in pipeline these days, so we could probably stop converting at time of capture and only convert when the frame reaches the ingest pipeline. It might even be a performance win if we end up dropping some captured frames prior to conversion.

- dale

--
You received this message because you are subscribed to the Google Groups "media-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to media-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/media-dev/CADTcc0tKHj6n%2BT5sqWR450AJNMNzZji16cB%2BUq_necfHMh2KSA%40mail.gmail.com.