KMS scalability and other issues

[Johan]

Hi all!

I've evaluated KMS for a few days, but I'm having troubles with scalability.
First of all, let's say we have a decent server hosting the KMS, how many concurrent streams should it be able to handle (Assuming a resolution 100px x 100px with around 10 fps)?

I'm currently trying to build a conferencing system that should support these two scenarios:
1) A teacher (always shown) + one student (max one visible) + 20 receiver students
2) Small groups of students (5 - 6 people), everyone sending to everyone
Of 1) and 2) there may be several ongoing conferences, which means we can get up to several hundreds, maybe thousands of streams via the KMS.

An Example: Ten groups of six persons, all streaming to each and everyone within a group (room). Furthermore, there may be ten other groups simolutaneously (two senders and twenty receivers). Is this an utopia with the KMS? What kind of numbers should I be able to expect, assuming that I run the server on decent HW. If not possible, could it be possible using several servers and load balancing somehow?

My current experience:

Setup: required Ubuntu with 2GiB ram and four HT processors avail.

First I tried the group call demo with the java app and KMS 5.0.4. I could get a group of maximum 5 users streaming to each other. If a sixth user joined, the KMS broke down and had to be started again. Then I tried KMS 5.1.0-something and it scaled up to 6 concurrent streamers within the same group, but this feels quite small.

So, I modified the client- and application code for the application server (Group Call) to support enabling/disabling of the users' web cams. So, when ten people join a chat, clicking on your icon will start streaming your video to everyone. For a group of twenty persons, this works ok as long as the amount of senders is max two. If I start a third one to stream, the KMS shuts down. For me, it is very important that the KMS would not shut down, that is a bit critical.

Here are parts of the error log just before the KMS shuts down (also attached in a file. The error occurrences are separated with '------'):


------------------------------------------------------------------------------------------------------------------------------------------------------------------------
2015-01-19 15:40:15.477781 9449 [3039681344]   debug KurentoMediaSet           MediaSet.cpp:82 doGarbageCollection() Running garbage collector
2015-01-19 15:40:15.479799 9449 [2943310656]    info KurentoWebSocketTransport WebSocketTransport.cpp:247 keepAliveSessions() Keep alive ba4be2a1-2b09-444e-a368-f81825a6168c
(kurento-media-server:9449): GLib-ERROR **: /build/buildd/g
lib2.0-2.40.0/./glib/gmem.c:103: failed to allocate 65619 bytes

------------------------------------------------------------------------------------------------------------------------------------------------------------------------
(kurento-media-server:9955): GStreamer-WARNING **: failed to create thread: Error creating thread: Resource temporarily unavailable
(kurento-media-server:9955): GStreamer-WARNING **: failed to create thread: Error creating thread: Resource temporarily unavailable
2015-01-19 15:52:07.662451 9955 [2968488768]   error basesrc                   gstbasesrc.c:3441 gst_base_src_start_complete() <nicesrc46> Failed to perform initial seek
2015-01-20 08:45:31.440373 9955 [2943310656]    info KurentoWebSocketTransport WebSocketTransport.cpp:247 keepAliveSessions() Keep alive ba4be2a1-2b09-444e-a368-f81825a6168c
2015-01-20 08:45:31.440390 9955 [3039419200]   debug KurentoMediaSet           MediaSet.cpp:82 doGarbageCollection() Running garbage collector
GLib: Cannot convert message: Could not open converter from 'UTF-8' to 'ANSI_X3.4-1968'
(kurento-media-server:9955): GLib-ERROR **: /build/buildd/glib2.0-2.40.0/./glib/gmem.c:103: failed to allocate 2089 bytes

------------------------------------------------------------------------------------------------------------------------------------------------------------------------

(kurento-media-server:10907): GLib-ERROR **: creating thread 'KmsLoop': Error creating thread: Resource temporarily unavailable
------------------------------------------------------------------------------------------------------------------------------------------------------------------------
<
(kurento-media-server:11441): GStreamer-WARNING **: failed to create thread: Error creating thread: Resource temporarily unavailable
(kurento-media-server:11441): GStreamer-WARNING **: failed to create thread: Error creating thread: Resource temporarily unavailable
2015-01-20 10:36:06.535142 11441 [2968488768]   error basesrc                   gstbasesrc.c:3441 gst_base_src_start_complete() <nicesrc40> Failed to perform initial seek
2015-01-20 10:36:12.981481 11441 [3039767360]   debug KurentoMediaSet           MediaSet.cpp:82 doGarbageCollection() Running garbage collector
2015-01-20 10:36:12.990988 11441 [2943310656]    info KurentoWebSocketTransport WebSocketTransport.cpp:247 keepAliveSessions() Keep alive ba4be2a1-2b09-444e-a368-f81825a6168c
2015-01-20 10:37:12.991220 11441 [2943310656]    info KurentoWebSocketTransport WebSocketTransport.cpp:247 keepAliveSessions() Keep alive ba4be2a1-2b09-444e-a368-f81825a6168c
GLib: Cannot convert message: Could not open converter from 'UTF-8' to 'ANSI_X3.4-1968'
(kurento-media-server:11441): GLib-ERROR **: /build/buildd/glib2.0-2.40.0/./glib/gmem.c:103: failed to allocate 1447 bytes
(kurento-media-server:11441): GLib-ERROR **: /build/buildd/glib2.0-2.40.0/./glib/gmem.c:103: failed to allocate 1469 bytes

------------------------------------------------------------------------------------------------------------------------------------------------------------------------
If I check the Kurento Media Server Status, I get:
* checking Kurento Media Server...                                              * /usr/bin/kurento-media-server is not running but the pid file exists...     

[Johan]

Is there anyone that could check why the KMS fails and shuts down? Running the 32 bit version of ubuntu, does that matter much?

[Johan]

I can answer this much, that I tried to install the 64bit version of Ubuntu 14.1 and that also gave me the later version of KMS (~53 instead of ~50) and now it works better! Thanks for the ~53-fix! Still, however, it would be nice to get some info on the upper questions :)

[Ivan Gracia]

Well, we can't provide exact figures right now, but in some recent load tests we were able to get 150 one2one connections of webrtc on an i5/16GB RAM. You can expect that these numbers will be a bit better in the future, but don't expect miracles: there is a los of enctyption going on for SRTP, and that is demanding. We are looking into hardware-accelerated encryption/decryption, and the numbers will go higher, but can't tell you how much will you be able to get until we haven't tested it.

Ivan Gracia

On Wed, Jan 21, 2015 at 10:48 AM, Johan <johan.a...@gmail.com> wrote:

I can answer this much, that I tried to install the 64bit version of Ubuntu 14.1 and that also gave me the later version of KMS (~53 instead of ~50) and now it works better! Thanks for the ~53-fix! Still, however, it would be nice to get some info on the upper questions :)

--
You received this message because you are subscribed to the Google Groups "kurento" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kurento+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Johan]

Hi Ivan and thanks for your reply!
I understood that one shall not expect any miracles, so I made my own kind of load balancing. I set up a few KMSes and via the Web Application, I divided different rooms to different servers. For the user it will be transparent, so joining another room may get the streams from a different KMS server. Anyways, is there a good way to check if the server is up and running? (I guess there is, have not just checked out the API between KMS and the java backend). It quite often hangs and stopping the service does not help, so I have to kill the process.

Thanks,
Johan

[Ivan Gracia]

What version of KMS are you using? v5.4.0 is known to become unresponsive in certain circumstances.

Nice set up you've made! Load balancing is definitely the way to go ;-)

Ivan Gracia