Reducing latency for 60fps video streaming

[Van Catha]

Hello,
    What are the options for in browser non-extension NaCL to receive a stream.  Right now I am testing using websockets to stream video to my NaCL plugin but there is poor performance in the chrome client. The receive callbacks tend to bunch up and proc all together. Meaning the 60fps stream turns into clumps of frames at a time, which lead to constant dropped frames as we cannot paint them that fast.  Note that I did confirm with wireshark and with timers on the server that the websocket frames leave the server 15-19ms apart.  Also the test was on localhost so network problems fall out, the only theory left is a slow consumer/buncher in the client websocket implementation.

I opened an issue here https://bugs.chromium.org/p/chromium/issues/detail?id=692257 about it but this post is not about that.

This post is about the other options available and I am particularly looking for something that is implemented well in chrome and maybe is UDP. UDP is not as important as having closer to realtime properties.

Some options I considered are Streams API (a long running GET).  Would QUIC work here, I believe it has support for FEC if my QUIC server implements/forces FEC?  I am not sure if WebRTC is a candidate here.  Also I do not think there is any RTP support to get a binary stream that I can pipe to the nacl video decoder frame by frame.

I really want to stay in browser and outside of 3rd party extension.

Also if anyone is familiar with NaCL is it possible to run the WebSocket connection on another thread? Outside the mainthreads eventloop. When I tried I could not get it right.

And with NaCL if your doing callback based SwapBuffers on the mainthread would this block websocket sends/recvs?

[Ben Smith]

Hello,

On Saturday, February 18, 2017 at 8:09:01 AM UTC-8, Van Catha wrote:

Hello,
    What are the options for in browser non-extension NaCL to receive a stream.  Right now I am testing using websockets to stream video to my NaCL plugin but there is poor performance in the chrome client. The receive callbacks tend to bunch up and proc all together. Meaning the 60fps stream turns into clumps of frames at a time, which lead to constant dropped frames as we cannot paint them that fast.  Note that I did confirm with wireshark and with timers on the server that the websocket frames leave the server 15-19ms apart.  Also the test was on localhost so network problems fall out, the only theory left is a slow consumer/buncher in the client websocket implementation.

I opened an issue here https://bugs.chromium.org/p/chromium/issues/detail?id=692257 about it but this post is not about that.

This post is about the other options available and I am particularly looking for something that is implemented well in chrome and maybe is UDP. UDP is not as important as having closer to realtime properties.

You'll only be able to use UDP for a Chrome App.

 

Some options I considered are Streams API (a long running GET).  Would QUIC work here, I believe it has support for FEC if my QUIC server implements/forces FEC?  I am not sure if WebRTC is a candidate here.  Also I do not think there is any RTP support to get a binary stream that I can pipe to the nacl video decoder frame by frame.

I believe WebRTC can do this, though I don't know the specifics. I assume you can get a raw ArrayBuffer; if so, you can postMessage that to your NaCl module, and it should be pretty fast.

 

I really want to stay in browser and outside of 3rd party extension.

Also if anyone is familiar with NaCL is it possible to run the WebSocket connection on another thread? Outside the mainthreads eventloop. When I tried I could not get it right.

It should be possible to use all the Pepper API functions off of the main thread, though if they are non-blocking, you'll have to set up a MessageLoop on that thread first. That said, it probably won't help any if you're blocking the main thread, since AIUI most browser APIs still use that for communication.

 

And with NaCL if your doing callback based SwapBuffers on the mainthread would this block websocket sends/recvs?

I'm not certain about this case, but in general you should avoid blocking the main thread. Often it's best to run all of the code for your NaCl module on a new thread so the main thread isn't ever blocked.