Indefinite buffering when KMS falls behind due to high CPU load

[David Robison]

For testing purposes, I have a VM configured with one CPU. When I attempt to run three high-definition videos through KMS, the CPU becomes over taxed and KMS begins to buffer the incoming video. This leads to huge delays in displaying the video. I've seen as much as 2 minutes of video being buffered inside KMS and the player running at almost half speed and 2 minutes behind the live video. This also causes huge memory accumulation within KMS, often in the tens of gigabytes range. There seem to be two issues leading to the buffering of video and the delayed delivery of video to the browser.

1) I've set a 500ms network buffer on the rtspsrc gstreamer filter. However, as latency grows, the delay increases and packets seem to accumulate in the rtpjitterbuffer. The first "fix" is to allow the rtspsrc to discard packets at the latency level and not to increase it beyond the set network buffer size. I made a change in the kmsplayerendpoint.c file:

element_added (GstBin * bin, GstElement * element, gpointer data)
{
  KmsPlayerEndpoint *self = KMS_PLAYER_ENDPOINT (data);


  if (g_strcmp0 (gst_plugin_feature_get_name (GST_PLUGIN_FEATURE
              (gst_element_get_factory (element))), RTSPSRC) == 0) {
    g_object_set (G_OBJECT (element), "latency", self->priv->network_cache,
        "drop-on-latency", TRUE, NULL);
  }
}

The change was to add "drop-on-latency" to the g_object_set command.

2) There is a queue element just before the vp8 encoder which can buffer up to 1 second of video. Other places where the queue element is used, KMS restricts the amount of buffering either to a smaller number or smaller time frame. When the vp8 encoder falls behind, packets can stack up in the queue and new frames will block until space is available in the queue. I made a change to lower the amount of buffering the queue can hold and to allow it to discard old frames when the buffer fills up. The change is made in the kms_enc_tree_bin_configure function in the kmsenctreebin.c file:

  queue = gst_element_factory_make ("queue", NULL);
  g_object_set (queue, "leaky", 2, "max-size-buffers", 1, NULL);

These two fixes allow KMS to better handle high CPU load when decoding, transcoding, and delivering video. It also keeps KMS from consuming an increasing amount of memory due to the indefinite buffering of video. 

Do these changes seem reasonable? Are there reasons for not making these changes? If not, then I can/will add them to a pull request.

Thanks, David

[Micael Gallego]

Hi David, 

Thank you for your analysis. We will take a look to your proposed changes and come back to you in some days.

Best regards

Micael Gallego

Kurento / OpenVidu Project Lead

--
You received this message because you are subscribed to the Google Groups "kurento" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kurento+unsubscribe@googlegroups.com.
To post to this group, send email to kur...@googlegroups.com.
Visit this group at https://groups.google.com/group/kurento.
To view this discussion on the web visit https://groups.google.com/d/msgid/kurento/65f75c90-9523-4dd9-bb6c-0386fba5c72e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Andrew Moss]

Hi,

I am glad someone else has been encountering this problem. We are trying to build a product for providing RTSP streams from CCTV cameras over WebRTC. This indefinite buffering (followed by a crash when the memory is exhausted) is a major issue.

I had spotted the drop-on-latency option but hadn't found time to try it out. We would be very keen to see fixed!

Thanks for your hard work!

Andrew Moss
Indigo vision Ltd

[Andrew Moss]

Hi David,

Any update on this?

Many thanks,

Andrew

[David Robison]

I have a pull request on GitHub, I believe it has been accepted. Not sure when it will appear in a formal release.

[Micael Gallego]

We will release a new version of KMS in the following days/weeks.

Thank you for your patience.

Best regards

Micael Gallego

Kurento / OpenVidu Project Lead

--

You received this message because you are subscribed to the Google Groups "kurento" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kurento+unsubscribe@googlegroups.com.
To post to this group, send email to kur...@googlegroups.com.
Visit this group at https://groups.google.com/group/kurento.

To view this discussion on the web visit https://groups.google.com/d/msgid/kurento/3dfc98bb-f978-4a2d-b313-db8be6ec39dc%40googlegroups.com.

[Andrew Moss]

Hi David,

Maybe I am blind, but I couldn't locate your pull request or any associated commit in github. Could you please provide a link?

Thanks for your hard work on this, by the way!

Andrew

[David Robison]

Sorry, my memory is not what it used to be. I just sent the pull request for kms-core and kms-elements for this fix.