Game RTP stream displayed on browser via WebRTC and Kurento

[Gabriel Pimenta]

I am working on a project that involves real time game streaming. We chose RTP as the streaming protocol for this task via FFmpeg. We are also using NVENC and CUDA hardware acceleration. FFmpeg command:

ffmpeg.exe -y -vsync 0 -hwaccel cuda -hwaccel_output_format cuda -f gdigrab -framerate 60 -i title="Some game" -c:v h264_nvenc -preset p2 -b:v 5M -bufsize 5M -maxrate 10M -sdp_file D:\video-streamer.sdp -f rtp rtp://127.0.0.1:26958

From my understanding the only way to stream RTP to a HTML5 video tag is via WebRTC, and we thought about doing it with Kurento's media pipeline along with WebRTC and RTP endpoints, similarly to the RTP receiver tutorial:

https://doc-kurento.readthedocs.io/en/stable/tutorials/java/tutorial-rtp-receiver.html

I was able to achive real time streaming to the browser this way, but the stream displayed on the browser freezes a lot.

• Does this media pipeline do any transcoding that we can disable? I am wondering if the stream is being transcoded unnecessarily at some point.
• Why does the stream freeze on the browser? Is there any other parameter that I should set that I am missing?

I would appreciate any help. Thanks in advance.

Gabriel Robaina

[Neil Young]

As long as your KMS instance is not explicitly forced to use H.264 only there is most likely transcoding involved. You should be able to see that on various ways:

1) KMS traces that

2) You could examine chrome://webrtc-internals. If you see VP8 then yes

In order to stick with plain H.264 you could edit SdpEndpoint.conf.json and remove the VP8 codec from KMS

Other than that there are for sure more efficient ways to omit ffmpeg and RTPEndPoint. Why don't you go with a pure WebRTC solution, e.g. via GStreamer? You just need to figure out a capture pipeline in GStreamer (would that work? https://stackoverflow.com/questions/33747500/using-gstreamer-to-capture-screen-and-show-it-in-a-window), then you could do that in either Java, Python, Rust by help of GStreamer "webrtcbin", some simple signaling with KMS and good.

[Gabriel Pimenta]

Hello mr. Young, thanks for your reply.

I will certainly take a look into removing the VP8 codec from KMS like you suggested.

About the GStreamer alternative you mentioned, would that grant me more performance/quality on the video stream, or would that just be a "cleaner" solution? 

--
You received this message because you are subscribed to the Google Groups "kurento" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kurento+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kurento/d448a9af-e26e-4089-8139-e26037f86712n%40googlegroups.com.

--

Pimenta

[Neil Young]

I would expect a performance boost, since RtpEndpoint is eliminated. Also, you would have to publish your RTP stream to the Internet (given the KMS resides in the Internet), here WebRTC has advantages, since the connectivity is evaluated by ICE.

But I never used a screen capture device as input for GStreamer. I always just used USB or CSI (Raspberry PI) cameras. There the end to end latency is in the sub-0.5-secs area

[Gabriel Pimenta]

Let me see if I understood this alternative correctly...

With RTP endpoint I need to publish the RTP stream to the address KMS is expecting the video to be, and then KMS sends the video to the browser client via WebRTCEndpoint, and this may have some transcoding involved.

With your alternative, and GStreamer, KMS is responsible only for the signaling, and GStreamer is able to stream straight to the browser client, in a peer-to-peer fashion.

Is that correct?

You received this message because you are subscribed to a topic in the Google Groups "kurento" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/kurento/KJjJi7e_tNg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to kurento+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kurento/6e11bc16-14a7-47ba-939c-c7821cbe195an%40googlegroups.com.

--

Pimenta

[Neil Young]

Not entirely. Correct for the first sentence: Your RTP stream needs to have an RTSP URL which resolves on the internet, if the KMS is on the internet. So you are responsible for having it publicly available, with all the disadvantages and problems arising from that requirement.

The second thing is: Yes, it is possible to have a P2P connection with GStreamer and a browser, but in this case you don't need KMS. You would just need some code which organizes that SDP/ICE exchange. This is referred to be a "WebRTC signaling server", some are for free available on the Internet. You could use mine for tests (https://webrtcserver.ddns.net) or you could roll your own. This solution is the fastest thinkable, but has the drawback, that (w/o further means) there cannot be more than one P2P connection between a publisher  and a subscriber and 1toMany can become difficult fast.There are of course use cases, in which this is perfectly OK.

A Media Server, like KMS, _always_ just negotiates SDP and ICE with a remote peer and its own instance. So in any event, your media will end up in the KMS and would have to be retrieved from that by another, but separated WebRTC connection. So a former P2P connection boils down to two P2P connections: One for publisher -> KMS, one for KMS -> subscriber.

Let me know if you have further questions. Also in a private exchange. I can support you with anything :)

[Gabriel Pimenta]

Thank you! You made it so much clearer to me.

In my situation a one to one pub/sub would be enough. I will certainly take a look at using a WebRTC Signaling Server such as yours.

I will give it a try tonight. I hope GStreamer is not as difficult to learn as FFmpeg is (IMO)

To view this discussion on the web visit https://groups.google.com/d/msgid/kurento/7ccab060-47c2-4029-96ca-58367e7aaa9an%40googlegroups.com.

--

Pimenta

[Gabriel Pimenta]

I gave it a try tonight and got great results!

I have been following the sendrecv tutorial from gstreamer while using a free signaling server: https://gitlab.freedesktop.org/gstreamer/gst-examples/-/tree/master/webrtc

I managed to come up with a RTP stream with NVENC H264 hardware encoding to the browser quite easily:

"gst-launch-1.0 dx9screencapsrc cursor=true ! videoconvert ! queue ! nvh264enc zerolatency=true ! rtph264pay ! queue ! capsfilter caps=application/x-rtp,media=video,encoding-name=H264,framerate=60/1,width=1920,height=1080,payload=97"

Good progress so far. Thank you again for leading me to this alternative.

My next step is figuring out how to gather some quality metrics from the network and the stream.

--

Pimenta

[Neil Young]

This is exactly what I meant. Congrats. Out of curiosity: On what hardware/OS are you running that and what language for the GST client are you using? Looks like Windows as OS? 

If one side is a browser then maybe chrome://webrtc-internals or Firefox's about://webrtc maybe of help.

[Gabriel Pimenta]

I am working on Windows, since I want to stream games and those are available mostly on Windows.

I am using Java for the GST client since it is the language I am the most familiar with.

I will take a look into that tool. Thank you.

To view this discussion on the web visit https://groups.google.com/d/msgid/kurento/aad801a3-1ad4-41f1-a605-3213d44ffd3dn%40googlegroups.com.

--

Pimenta

[Neil Young]

Thanks for sharing 

Sent from my iPhone

Am 09.11.2021 um 13:17 schrieb Gabriel Pimenta <piment...@gmail.com>:



To view this discussion on the web visit https://groups.google.com/d/msgid/kurento/CAKUYLW%2Bqa3%3DT_uS5PvO%3DCwjwp-60%2BAgCcTn_6J0Om7Lm5EAsNw%40mail.gmail.com.

[Gabriel Pimenta]

Hello mr. Young, how are you?

Indeed, chrome://webrtc-internals had everything I needed regarding metrics.

I came across a problem now: Chrome doesnt allow getUserMedia on untrusted (non HTTPS, non localhost) origins. This means that I am not able to send a MediaStream when I open my website from a smartphone. Also, when I try to send an empty MediaStream, or one made from a canvas.captureStream, the server doesnt send anything back. It seems that I am required to send some user media stream (audio or video) in order for the server to send the RTP stream correctly to the client browser.

Any insights? How should I substitute getUserMedia when it is not available on the client?

Thanks once again

To view this discussion on the web visit https://groups.google.com/d/msgid/kurento/23C64C16-32CB-4A21-9610-1203EBC59C94%40googlemail.com.

--

Pimenta

[Neil Young]

Didn't you only want to send something form your device towards the browser? Then you don't have to call getUserMedia() in the browser.

Chrome can be convinced to trust insecure domains: 

chrome://flags/#unsafely-treat-insecure-origin-as-secure

It should be possible to send something to the browser w/o getting anything back (sendonly)

[Gabriel Pimenta]

I thought so as well...

In practice, it all breaks when I stop sending media from the browser to the server. I am using the same code example as a base: https://gitlab.freedesktop.org/gstreamer/gst-examples/-/tree/master/webrtc

I will figure out how to do it tonight.

To view this discussion on the web visit https://groups.google.com/d/msgid/kurento/5081e119-7ff0-4d6e-9f05-d36b587ab9f0n%40googlegroups.com.

--

Pimenta

[Gabriel Pimenta]

Hello mr. Young,

I managed to alter the code in order for the server to use sendonly SDPs, as you suggested. It all works now, and I am able to stream without any data coming from the browser.

Now I am looking into improving the quality of the real time stream. I have been using the following Gstreamer pipeline (pp 1):

dx9screencapsrc cursor=true ! videoconvert ! queue ! nvh264enc zerolatency=true ! rtph264pay ! queue ! capsfilter caps=application/x-rtp,media=video,encoding-name=H264,framerate=60/1,width=1920,height=1080,payload=97

As suggested in this Google article there are more optimizations to be done on the NVENC H264 Gstreamer pipeline. They did provide some python code, but not the full command for the pipeline, but it should look like this (pp 2):

dx9screencapsrc cursor=true ! capsfilter caps=video/x-raw,framerate=60/1 ! cudaupload ! cudaconvert ! capsfilter caps=video/x-raw(memory:CUDAMemory),format=I420 ! nvh264enc bitrate=2000 rc-mode=cbr gop-size=-1 qos=true preset=low-latency-hq ! capsfilter caps=video/x-h264,profile=high ! rtph264pay ! capsfilter caps=application/x-rtp,media=video,encoding-name=H264,payload=123

pp1 works just fine. The pipeline starts, and then WebRTCBin is able to receive and generate SDP offers.

pp2 doesnt work. With the same code, the pipeline starts with a SUCCESS status, but the WebRTCBin callback for ON_NEGOTIATION_NEEDED is never called, and the whole signaling process is not finished, so the live stream doesnt start on the browser. No errors or exceptions are thrown.

Do you have any idea on why this might be happening? Debugging this is quite hard without any exceptions or errors... I dont know where to start looking

--

Pimenta

[Neil Young]

No, sorry. You are asking too much, I of course can't debug your pipeline remotely. I would suggest to raise GST_DEBUG level in order to get some insight

[Gabriel Pimenta]

Hello Mr Young, how are you?

Over the last weeks I was able to advance in the project to a state where I can already collect results and metrics. I thank you for helping me get to this point.

I have one last problem though: When analyzing the results, using WebRTC internals, I noticed the client browser's bitrate fluctuates at around 250 kbits/sec. But, on my media pipeline I configured nvh264enc for a bitrate of 2250 kbit/sec:

dxgiscreencapsrc cursor=true ! capsfilter caps="video/x-raw,framerate=60/1" ! queue ! nvh264enc bitrate=2250 rc-mode=cbr gop-size=-1 qos=true preset=low-latency-hq ! capsfilter caps="video/x-h264,profile=high" ! queue ! rtph264pay ! capsfilter caps="application/x-rtp,media=video,encoding-name=H264,width=1280,height=720,payload=123"

I know WebRTC limits the bitrate based on bandwidth constraints, but I am doing all my experiments in LAN. I expected bitrate to be much closer to 2250 kbit/sec from the GStreamer pipeline encoder.

How can I make sure where this limitation is coming from (pipeline or client browser)? I tried reading the bitrate from the GStreamer pipeline but couldn't find anything using DEBUG logs because it's too verbose. I have also tried reading the "bitrate" property on some Bin elements (found some SO answer related to it) but couldn't find anything useful either.

Also, is there any possibility this limitation is coming from the client browser?

Like you said, of course you can't debug my pipeline remotely. Still, any directions you can give me are really useful. 

Again, thanks for the help so far.

To view this discussion on the web visit https://groups.google.com/d/msgid/kurento/098a7894-7318-4d06-978f-470f75fd301dn%40googlegroups.com.

--

Pimenta

[Neil Young]

No, I don't think it is related to the browser. But it is really hard to do a remote debugging

[Gabriel Pimenta]

Hello Mr. Young, how have you been?

The project finished up successfully, and I managed to run all the experiments I needed thanks to the help you gave me.

Now, for the next steps, I would like to study more about internet media streaming. I am interested specifically in delivering media content in an edge environment. Can you recommend any books, blogs, etc., about video streaming in general?

I would like to know more about different audio and video codecs and how to design a video streaming architecture considering different network conditions, for example.

Thank you again,

To view this discussion on the web visit https://groups.google.com/d/msgid/kurento/b343f2f2-c21a-41fb-8b5c-5bef02f00174n%40googlegroups.com.

--

Pimenta

[Neil Young]

No recommendation just a hint: For edge devices webrtc together with GStreamer webrtcbin plugin works good 

I have solutions for that if you are interested

[Gabriel Pimenta]

I have a question though: Why is webrtc any better than a pure P2P RTP stream for a LAN environment? GStreamer can output an RTP stream, and I would like to run that sometime for comparison. While implementing, it seems like WebRTC was built for web conferencing on the browser, and goes nice with capturing video from cameras. 

I would be interested in one of the solutions you have for that. Thanks again

To view this discussion on the web visit https://groups.google.com/d/msgid/kurento/e0d96bbc-3520-41ab-8d4b-b8103225d2bbn%40googlegroups.com.

--

Pimenta

[decades software]

If your LAN conditions never change (no errors, no bandwidth troubles), you don't have to overcome NAT/Firewalls, you don't need video encryption, you don't need data - well, then RTP might be the better choice.

Write me a PM regarding the solution if possible.

To view this discussion on the web visit https://groups.google.com/d/msgid/kurento/CAKUYLW%2BmnFzkWQjg7fj5J-O7U-NU9p_aAyisjiRHcTv-JR%2BOmQ%40mail.gmail.com.