Performance issue/support with WebRTC in Chrome browser

[Jet Stream Test channel]

Hello,

this is my first message, so my apologies if my issue is at the wrong list. But we have a WebRTC performance issue regarding the frames per second. And this is only happening in the Google Chrome browser. We do not see it in FireFox. I do not want to start a browser ware. I think that Google Chrome is just as good as Firefox, so we would like to support both browsers with WebRTC.

So we have created a web application where you are able to mix video's, webcams, desktop screen sharing etc. For this we use the general WebRTC javascript code and functions. All the mixing is happening on a canvas which is NOT visible, so that should reduce GPU load when it does not have to draw the canvas on screen.

So we pick up the webcam in the browser, and let it play in a HTML5 video tag which is also NOT visible. Again, we do this, as this would/should reduce the GPU/CPU load as there is nothing to draw. The same thing we do with a MP4 video file. That is loaded locally and played also in a HTML5 video player. Then we have at least two video players running outside the visible window. Then we have a function running that is grabbing a frame from the MP4 movie and draw it on the canvas. Then it will grab a frame from the webcam HTML5 video and add that as a picture in picture on the canvas. We are mixing the video sources together on a canvas. This code is running inside a 'requestAnimationFrame' function for optimal browser usage. We are getting 30 fps out of this.

Then we create a single HTML5 video for preview which has a source which is the hidden canvas. And we setup a WebRTC stream/connection with the same canvas as source. We use the command: 'canvas.captureStream(30)' of which we would expect that it will grab at a speed of 30fps from the canvas. This is then send to a WebRTC server to be processed further.

An audio track is added to the video track that is coming from the canvas. So we stream 1 video track and 1 audio track to the WebRTC server.

The audio track is in Opus, the video track is in H264.

Now we see that Chrome is fluctuating a lot with the FPS. We try to get 30fps, but 25fps is also fine. But it looks like that we cannot get Chrome at even 25fps. While with FireFox we do not have this issue. So is there a way that we can enforce the FPS a bit more stable. We have used the following custom Chrome SDP parameters for bandwidth and CPU load: 'googCpuOveruseDetection' = false and x-google-min-bitrate, x-google-max-bitrate and x-google-start-bitrate which works well. The bitrate is pretty stable and is working fine. We are having issues that sometimes the FPS is dropping, and having hard time to get back.

Could it be that the website is causing some extra rendering as the design is in flexbox css. And there is also some animations going on. Could it be that when we update some CSS, like an audio volume indicator, can hurt the WebRTC performance? And we update also some small parts as the video is playing and the progress bar of that video is growing.

I have added a screenshot, with orange area's where animations / DOM updates are happening. Could this have much impact on the WebRTC part?

For the canvas mixing,we calculate our own fps, which is the value of 27.48, which should be 30. But this FPS is not moving much up or down, so it stays around 28-27 fps. Then the WebRTC fps is only 15 fps. This is calculated based on amount of frames sent to the server. As Chrome does not provide a fps stats yet. So we calculate the difference between each second based on total send frames. Not sure how reliable that is, but else we do not have a way to get the WebRTC fps on screen.

We see the following message on the console: [OpenH264] this = 0x0x559d5a957580, Warning:Actual input framerate 0.000000 is different from framerate in setting 13.000000, suggest to use other rate control modes

As we are trying to get 25 fps or 30 fps, we do not understand why Chrome is saying that he want 13.0000 fps and not 25fps.

We are using the video codec H264, as this will/should make it easier for us to transform to HLS streaming, As the video codec is the same, we do not need video transcoding which save a lot of CPU power. Audio transcoding is not that heavy so we transcode the Opus audio codec to AAC audio on the WebRTC server. So, for testing VP8 or VP9 could be used, but then we see the same issues with the FPS. It looks like Chrome can't get them stable somehow.

If someone wants to test it, or just play with it, we have the application live in beta at: https://my.streaminar.com It is free and open for everybody, so if somebody has some time and willing to test, that would be very awesome.

Also, it somebody can point me to how to debug this better, or some pointers to look at that would also be very help full.

Kind regards

Joshua

Jet-Stream BV

[Mirko Bonadei]

Opening a bug on https://bugs.chromium.org/p/chromium with component Blink>WebRTC>Video is probably the best thing to do in this case.