Efficient Webrtc rendering of a large and synchronized video stream

[aric lasry]

Hi everybody. I will first ask my question and then provide details on why I am doing that.

Question:

Is there an efficient / fast way to get the video frame data from WebRTC to a web worker (at 20hz, without blocking too much the main thread)?

More details:

I use Webrtc native to stitch 7 cameras feed together and send them as one video track through Webrtc. Then on the browser, the video track gets rendered into a hidden <video> element and is then un-stitched into 7 canvas using <canvas>.getContext('2d').drawImage(<video>, ....).

The stitching is for video synchronization.

The problem is that the 7 call to canvas.drawImage(width=~400, width=~320) takes too much time to keep a 60fps browser rendering and slows down my javascript main event loop. Which delays some data channel operations.

So instead of drawing into a regular canvas, I tried drawing into an offscreenCanvas, then call imageData = offscreenCanvas.getImageData() and send imageData to a worker so that the worker can put back the image data into a transferred "onscreen" canvas.

mainThread.js

const transferedOnscreenCanvas = document.getElementById('onscreen-canvas').transferControlToOffscreen();
const offscreenCanvas =new OffscreenCanvas(width, height);
offscreenCanvas.drawImage(...);
const imageData = offscreenCanvas.getContext('2d').getImageData()
worker.postMessage({imageData:imageData, canvas: transferedOnscreenCanvas}, [transferedOnscreenCanvas]);

worker.js

onMessage(event) {
 const onscreenCanvas = event.data.canvas;
 const imageData = event.data.imageData;

 onscreenCanvas.putImageData(imageData, 0, 0);
}

The problem is that getImageData() takes as much time as drawing on an onscreen canvas and is done on the main thread.

Another idea would be to use 7 video track and 7 video elements, would that be more efficient and synchronized?

All ideas are welcomed.

Thanks.

[Alexandre GOUAILLARD]

any reason why you try to do that in the browser instead of doing it server side?

--

---
You received this message because you are subscribed to the Google Groups "discuss-webrtc" group.
To unsubscribe from this group and stop receiving emails from it, send an email to discuss-webrt...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/bdfd2955-484a-4da4-b6b9-6ad20282cabd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Alex. Gouaillard, PhD, PhD, MBA

------------------------------------------------------------------------------------

President - CoSMo Software Consulting, Singapore

------------------------------------------------------------------------------------

sg.linkedin.com/agouaillard

• 
  

[aric lasry]

I don’t see any step that could be done on the server here.. what do you mean?

Also, we want a peer to peer connection to minimize latency.

You received this message because you are subscribed to a topic in the Google Groups "discuss-webrtc" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/discuss-webrtc/UiEV4EtVfw0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to discuss-webrt...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/CAHgZEq4inNZ9myHa2jsR1_DuGCa09K82Y-8kpu_q8XWdx5R1uA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Aric Lasry

646-705-1274

[Nils Ohlmeier]

Why do you stitch the 7 camera inputs together on the sender side?

Wouldn’t be a lot easier to simply send all 7 video tracks through the same PeerConnection and present/render them on the receiving end in any way you want.

Best regards

  Nils Ohlmeier

[aric lasry]

I do that for synchronization.

Will 7 video track on the same peerconnection would ensure synchronization?

Also, by using canvas, we can apply some filtering on the frontend like contrast and brightness.

That could be done on the sending side if not on the rendering side.

You received this message because you are subscribed to a topic in the Google Groups "discuss-webrtc" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/discuss-webrtc/UiEV4EtVfw0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to discuss-webrt...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/D680B3B8-FCED-44EC-8587-B370D25077ED%40mozilla.com.

For more options, visit https://groups.google.com/d/optout.

--

Aric Lasry

646-705-1274