Kurento crash

Mikhail Novikov

unread,

Feb 6, 2018, 1:11:50 PM2/6/18

to kurento

from time to time i am getting a crash with this stack trace:

^[[31;1mSegmentation fault^[[0m (thread ^[[33;1m139796485371648^[[0m, pid ^[[33;1m7829^[[0m)
Stack trace:
^[[34;1m[g_socket_send_message]^[[0m
/usr/lib/x86_64-linux-gnu/libgio-2.0.so.0^[[32;1m:0x7B044^[[0m
^[[34;1m[nice_output_stream_new]^[[0m
/usr/lib/x86_64-linux-gnu/libnice.so.10^[[32;1m:0x2769E^[[0m
^[[34;1m[nice_output_stream_new]^[[0m
/usr/lib/x86_64-linux-gnu/libnice.so.10^[[32;1m:0x27813^[[0m
^[[34;1m[nice_agent_recv_nonblocking]^[[0m
/usr/lib/x86_64-linux-gnu/libnice.so.10^[[32;1m:0x11879^[[0m
^[[34;1m[gst_nice_src_get_type]^[[0m
/usr/lib/x86_64-linux-gnu/gstreamer-1.5/libgstnice15.so^[[32;1m:0x3902^[[0m
^[[34;1m[gst_nice_sink_get_type]^[[0m
/usr/lib/x86_64-linux-gnu/gstreamer-1.5/libgstnice15.so^[[32;1m:0x4203^[[0m
^[[34;1m[gst_base_sink_do_preroll]^[[0m
/usr/lib/x86_64-linux-gnu/libgstbase-1.5.so.0^[[32;1m:0x2A1B2^[[0m
^[[34;1m[gst_base_sink_do_preroll]^[[0m
/usr/lib/x86_64-linux-gnu/libgstbase-1.5.so.0^[[32;1m:0x2B620^[[0m
^[[34;1m[gst_flow_get_name]^[[0m
/usr/lib/x86_64-linux-gnu/libgstreamer-1.5.so.0^[[32;1m:0x6E5CF^[[0m
^[[34;1m[gst_pad_push]^[[0m
/usr/lib/x86_64-linux-gnu/libgstreamer-1.5.so.0^[[32;1m:0x76533^[[0m
^[[34;1m[gst_proxy_pad_chain_default]^[[0m
/usr/lib/x86_64-linux-gnu/libgstreamer-1.5.so.0^[[32;1m:0x5F5E3^[[0m
^[[34;1m[gst_flow_get_name]^[[0m
/usr/lib/x86_64-linux-gnu/libgstreamer-1.5.so.0^[[32;1m:0x6E5CF^[[0m
^[[34;1m[gst_pad_push]^[[0m
/usr/lib/x86_64-linux-gnu/libgstreamer-1.5.so.0^[[32;1m:0x76533^[[0m
^[[32;1m0x1B48D^[[0m at /usr/lib/x86_64-linux-gnu/gstreamer-1.5/libgstcoreelements.so
^[[34;1m[gst_flow_get_name]^[[0m
/usr/lib/x86_64-linux-gnu/libgstreamer-1.5.so.0^[[32;1m:0x6E5CF^[[0m
^[[34;1m[gst_pad_push]^[[0m
/usr/lib/x86_64-linux-gnu/libgstreamer-1.5.so.0^[[32;1m:0x76533^[[0m
^@

Where to look?

Thanks!

Jorge Maiquez

unread,

Mar 12, 2018, 9:56:01 AM3/12/18

to kurento

Hi all,

Has anyone figured out what is causing this the segmentation fault in g_socket_send_message? There are various reports of this in this news group, but not a single positive response from the community.

Is the kurento team aware of this? Is there a workaround?

We didn't see this error a single time in our development and staging environments. But now that we deployed it to production, our client has seen it twice in less than a week.

Any tips are greatly appreciated. The client has a large webinar planned for tomorrow.

Thanks!

Jorge Maiquez, Digital Samba

Jon Ruddell

unread,

Mar 12, 2018, 12:19:24 PM3/12/18

to kurento

Do you have any more information such as your KMS version, OS info, and how to reproduce the crash?

Kapa6ac79

unread,

Mar 12, 2018, 4:26:22 PM3/12/18

to kurento

I confirm the problem, when you switch to the latest version, the same bug appears. The error occurs when there is a load on the server (from about 50+ clients), but there is no clear algorithm for reproducing the error. It arises suddenly. The WebRTC one-to-many broadcast mode was tested. OS version of Ubuntu 16.04, KMS 6.7.0

среда, 7 февраля 2018 г., 2:11:50 UTC+8 пользователь Mikhail Novikov написал:

Alex Kandrashkin

unread,

Mar 13, 2018, 3:20:51 AM3/13/18

to kurento

Also have this issue (not the latest KMS)

KMS version: 6.7.0~1.g6ebaa27

Found modules:

Module: 'core' version '6.6.3'

Module: 'elements' version '6.6.3'

Module: 'filters' version '6.7.0~1.g0314843'

Ubuntu 16.04.3 LTS (GNU/Linux 4.4.0-112-generic x86_64)

Do you have it on the latest KMS ?

dawn_...@outlook.com

unread,

Mar 13, 2018, 3:24:14 AM3/13/18

to kurento

I also met this problem！

this is my stack trace:

[31;1mSegmentation fault [0m (thread [33;1m139918407882496 [0m, pid [33;1m26118 [0m)

Stack trace:

[34;1m[g_socket_send_message] [0m

/usr/lib/x86_64-linux-gnu/libgio-2.0.so.0 [32;1m:0x7B044 [0m

[34;1m[nice_output_stream_new] [0m

/usr/lib/x86_64-linux-gnu/libnice.so.10 [32;1m:0x2ACBF [0m

[34;1m[nice_output_stream_new] [0m

/usr/lib/x86_64-linux-gnu/libnice.so.10 [32;1m:0x2AF3B [0m

[34;1m[nice_agent_recv_nonblocking] [0m

/usr/lib/x86_64-linux-gnu/libnice.so.10 [32;1m:0x12AE9 [0m

[34;1m[gst_nice_src_get_type] [0m

/usr/lib/x86_64-linux-gnu/gstreamer-1.5/libgstnice15.so [32;1m:0x36B2 [0m

[34;1m[gst_nice_sink_get_type] [0m

/usr/lib/x86_64-linux-gnu/gstreamer-1.5/libgstnice15.so [32;1m:0x3FB3 [0m

[34;1m[gst_base_sink_do_preroll] [0m

/usr/lib/x86_64-linux-gnu/libgstbase-1.5.so.0 [32;1m:0x2A1B2 [0m

[34;1m[gst_base_sink_do_preroll] [0m

/usr/lib/x86_64-linux-gnu/libgstbase-1.5.so.0 [32;1m:0x2B620 [0m

[34;1m[gst_flow_get_name] [0m

/usr/lib/x86_64-linux-gnu/libgstreamer-1.5.so.0 [32;1m:0x6E5CF [0m

[34;1m[gst_pad_push] [0m

/usr/lib/x86_64-linux-gnu/libgstreamer-1.5.so.0 [32;1m:0x76533 [0m

[34;1m[gst_proxy_pad_chain_default] [0m

/usr/lib/x86_64-linux-gnu/libgstreamer-1.5.so.0 [32;1m:0x5F5E3 [0m

[34;1m[gst_flow_get_name] [0m

/usr/lib/x86_64-linux-gnu/libgstreamer-1.5.so.0 [32;1m:0x6E5CF [0m

[34;1m[gst_pad_push] [0m

/usr/lib/x86_64-linux-gnu/libgstreamer-1.5.so.0 [32;1m:0x76533 [0m

[32;1m0x1B48D [0m at /usr/lib/x86_64-linux-gnu/gstreamer-1.5/libgstcoreelements.so

[34;1m[gst_flow_get_name] [0m

/usr/lib/x86_64-linux-gnu/libgstreamer-1.5.so.0 [32;1m:0x6E5CF [0m

[34;1m[gst_pad_push] [0m

/usr/lib/x86_64-linux-gnu/libgstreamer-1.5.so.0 [32;1m:0x76533 [0m

**

libnice:ERROR:agent.c:2342:agent_signal_component_state_change: assertion failed: (TRANSITION (DISCONNECTED, FAILED) || TRANSITION (GATHERING, FAILED) || TRANSITION (CONNECTING, FAILED) || TRANSITION (CONNECTED, FAILED) || TRANSITION (READY, FAILED) || TRANSITION (DISCONNECTED, GATHERING) || TRANSITION (GATHERING, CONNECTING) || TRANSITION (CONNECTING, CONNECTED) || TRANSITION (CONNECTED, READY) || TRANSITION (READY, CONNECTED) || TRANSITION (FAILED, CONNECTING) || TRANSITION (FAILED, GATHERING) || TRANSITION (DISCONNECTED, CONNECTING))

在 2018年2月7日星期三 UTC+8上午2:11:50，Mikhail Novikov写道：

Jorge Maiquez

unread,

Mar 13, 2018, 3:42:03 AM3/13/18

to kurento

Thanks for the responses.

We don’t know how to reproduce this. If we did, then we could at least manage our client in some way.

The KMS setup was unchanged in the switch from staging to production (same box, same KMS version, same OS, etc). You can see the respective version information in Alex’s response above.

Kapa6ac79, can you please confirm that you see the same error on the very latest 2018-01-18 version of KMS?

Jon, are you aware if this is something that was specifically worked on in the 2018-01-18 version? I can’t see anything in the change logs that would indicate that is the case.

Regardless, it seems we are not the only ones experiencing this problem, and bug like is not something you want kicking around in production.

Any hints/experiences much appreciated!

On Tuesday, February 6, 2018 at 7:11:50 PM UTC+1, Mikhail Novikov wrote:

Jorge Maiquez

unread,

Mar 13, 2018, 5:35:57 AM3/13/18

to kurento

Are there any gstreamer guru's out there that could suggest which direction we should explore to trigger segmentation errors specifically in

g_socket_send_message

We don't have enough experience with gstreamer or that particular method to compile targeted test cases for reproducing this error. It would be good to have an educated-guess starting point. Is this more likely to be load related (didn't seem to be the case for us), or some dodgy camera/mic device driver on the client, or something else entirely?

Thanks,

Jorge

Jorge Maiquez

unread,

Mar 14, 2018, 8:45:41 AM3/14/18

to kurento

Quick update. The client's session went well (2 broadcasters, 140 viewers), so we got lucky this time.

Can someone please suggest what things we can (stress) test to try to trigger the g_socket_send_message segmentation fault?

Micael Gallego

unread,

Mar 16, 2018, 6:38:47 PM3/16/18

to kur...@googlegroups.com

We are working on that right now.

--
You received this message because you are subscribed to the Google Groups "kurento" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kurento+u...@googlegroups.com.
To post to this group, send email to kur...@googlegroups.com.
Visit this group at https://groups.google.com/group/kurento.
To view this discussion on the web visit https://groups.google.com/d/msgid/kurento/61baf5e2-14ed-456f-815a-8537910e4f23%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Message has been deleted

Jorge Maiquez

unread,

Mar 17, 2018, 6:35:17 AM3/17/18

to kurento

Hi Micael,

Does this mean that you know how to reproduce it? If so, can you give us some more info about the conditions that trigger the error?

This client will be ramping up soon, and I'd like to take active steps to minimize the risk of this occurring, even if that means we have to cripple our application slightly in the short term.

Basically, is there a workaround we can use until you guys have fixed this?

If you need help with any specific testing, let me know.

Thanks & all the best,

Jorge Maiquez

Digital Samba

On Friday, March 16, 2018 at 11:38:47 PM UTC+1, Micael Gallego Carrillo wrote:

We are working on that right now.

Jorge Maiquez

unread,

Mar 21, 2018, 3:27:59 AM3/21/18

to kurento

Any more details you can share with us Micael?

Micael Gallego

unread,

Mar 21, 2018, 4:18:57 AM3/21/18

to kur...@googlegroups.com

We are digging into the problem right now... When we have somo advances we will publish more information about it

--

You received this message because you are subscribed to the Google Groups "kurento" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kurento+u...@googlegroups.com.
To post to this group, send email to kur...@googlegroups.com.
Visit this group at https://groups.google.com/group/kurento.

To view this discussion on the web visit https://groups.google.com/d/msgid/kurento/1ebc0389-2e71-4f82-adea-9d699bc9095b%40googlegroups.com.

Message has been deleted

Jorge Maiquez

unread,

Mar 21, 2018, 9:12:47 AM3/21/18

to kurento

I'm not sure why my messages keep getting deleted in this thread. Here is my last message again:

Thanks for the update. We are flying blind in production right now, so if there is anything at all that we can do right now to mitigate the error from occurring, please share- even it it's not the final solution.

And if you need assistance with testing, just let me know.

On Wednesday, March 21, 2018 at 9:18:57 AM UTC+1, Micael Gallego Carrillo wrote:

We are digging into the problem right now... When we have somo advances we will publish more information about it

Juan Navarro

unread,

Mar 23, 2018, 8:43:30 AM3/23/18

to kurento

Hi,

we are having some issues with the overzealous spam filter that Google has in their Google Groups. It even happens to us some times, that messages get deleted! Currently looking for solutions, but it seems that other communities have had similar problems in the past. Sort of totally disabling spam filtering, it seems there is not much we can do because the messages don't appear in the "awaiting for moderation" list in the administration view... they instead get outright deleted.

Jorge Maiquez

unread,

Mar 23, 2018, 10:10:06 AM3/23/18

to kurento

Hi Juan,

After one of my messages gets deleted, I see additional links in the Reply section of the UI, and one of those is (paraphrased) "click here to post". This then leads to a captcha, and after I verify I'm not a robot, I'm able to post a reply successfully.

It's not ideal, but maybe this helps someone who has the same problem to at least be able to post a reply without ripping their hair out :-)

Have a great weekend,

Jorge

Jorge Maiquez

unread,

Mar 30, 2018, 1:55:05 AM3/30/18

to kurento

Any update on this?

On Wednesday, March 21, 2018 at 2:12:47 PM UTC+1, Jorge Maiquez wrote:

ankit...@gmail.com

unread,

Mar 30, 2018, 10:22:29 PM3/30/18

to kurento

Jorge,

First of all, I am just a user like you. While the work is being done on this. Use monit to monitor KMS and restart it. There must be a way in you client API to get reconnected event and then use this event to reconnect your app. I do this in my NodeJS app

Regards

Message has been deleted

Jorge Maiquez

unread,

Mar 31, 2018, 12:36:26 AM3/31/18

to kur...@googlegroups.com

Hi Ankit,

Monit looks like a great tool, thanks for sharing. But unfortunately, monitoring isn’t the problem in this case.

The problem is that KMS/gstreamer will seem to be working fine one moment, and then it will throw this error in the middle of a 400 user session, for example. Even if we can detect that it is about to go down (CPU pegged at 100%, etc), it doesn’t help much if a large customer session is already in progress. The result will be an interruption of service and an unhappy customer.

Really, what we’re looking for is some kind of “best practice” guideline that will allow us to minimize the risk of the error occurring, until the Kurento team has solved the root cause of the error.

In the blog post from March 22nd, the roadmap lists the following item:
- “Update GStreamer and several other underlying support libraries to their latest versions.”

Will that solve this problem? And is that already included in 6.7.1?

Any additional info would be greatly appreciated.

Thanks!
Jorge

--
You received this message because you are subscribed to a topic in the Google Groups "kurento" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/kurento/_rf1ANq5Cm8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to kurento+u...@googlegroups.com.

To post to this group, send email to kur...@googlegroups.com.
Visit this group at https://groups.google.com/group/kurento.

To view this discussion on the web visit https://groups.google.com/d/msgid/kurento/6c88bf15-99f4-4f0d-952a-77f01c8e9ddf%40googlegroups.com.

Micael Gallego

unread,

Mar 31, 2018, 6:06:52 AM3/31/18

to kur...@googlegroups.com

Hi Jorge,

We have detected some problems in the libraries in Trusty version of KMS that leads to KMS crash. If you are using Trusty, please update to Xenial and report us if your problems are gone.

By the way, KMS 6.7.1 is not still updated to recent library versions, as this update requires a lot of work.

Best regards

Micael Gallego

Kurento / OpenVidu Project Lead

To unsubscribe from this group and all its topics, send an email to kurento+unsubscribe@googlegroups.com.

To post to this group, send email to kur...@googlegroups.com.
Visit this group at https://groups.google.com/group/kurento.
To view this discussion on the web visit https://groups.google.com/d/msgid/kurento/6c88bf15-99f4-4f0d-952a-77f01c8e9ddf%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "kurento" group.

To unsubscribe from this group and stop receiving emails from it, send an email to kurento+unsubscribe@googlegroups.com.

To post to this group, send email to kur...@googlegroups.com.
Visit this group at https://groups.google.com/group/kurento.

To view this discussion on the web visit https://groups.google.com/d/msgid/kurento/80BCC103-55C1-44E0-B5D6-918DA428ED1D%40gmail.com.

Jorge Maiquez

unread,

Mar 31, 2018, 6:42:00 AM3/31/18

to kurento

Hi Micael,

Thanks for getting back. We are on Xenial and the problems we reported were on that version.

Are you able to reproduce the gstreamer_send_message error on Trusty? If so, can you share details?

Is there anything else you can suggest for us to try?

Thanks!

Jorge

On Saturday, March 31, 2018 at 12:06:52 PM UTC+2, OpenVidu wrote:

Hi Jorge,

We have detected some problems in the libraries in Trusty version of KMS that leads to KMS crash. If you are using Trusty, please update to Xenial and report us if your problems are gone.

By the way, KMS 6.7.1 is not still updated to recent library versions, as this update requires a lot of work.

Best regards

Micael Gallego
Kurento / OpenVidu Project Lead

Juan Navarro

unread,

Apr 12, 2018, 11:49:21 AM4/12/18

to kurento

Tracking issue: https://github.com/Kurento/bugtracker/issues/247

Thanks!

TheV

unread,

May 25, 2018, 7:34:58 AM5/25/18

to kurento

We can reliably (9/10) reproduce this crash in our performance testing. At this point crashing is the expected behavior rather than the exception. Completely unusable in production.

Juan Navarro

unread,

May 25, 2018, 10:38:28 AM5/25/18

to kur...@googlegroups.com

The 3rd-party libnice library is the weak link here. The crash happens in that library, not in Kurento code. See https://github.com/Kurento/bugtracker/issues/247

Simply updating the library to latest versions (which would probably fix this issue) is not easy and straightforward for us because it makes some of our integration tests fail for some use cases. So this task has been registered in the issue tracker but delayed multiple times due to more pressing issues.

You may have success in updating the library for your use case without extra adaptation work, so I would suggest that you look into that. If you are not able to do so, or would need some help with this, you can also ask for our commercial support (which will also raise this issue's priority). Check https://doc-kurento.readthedocs.io/en/stable/business/

Regards,
Juan

> Sent: Friday, May 25, 2018 at 1:34 PM
> From: TheV <ry...@solutera.lt>
> To: kurento <kur...@googlegroups.com>
> Subject: [kurento-public] Re: Kurento crash

> --
> You received this message because you are subscribed to a topic in the Google Groups "kurento" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/kurento/_rf1ANq5Cm8/unsubscribe.

> To unsubscribe from this group and all its topics, send an email to kurento+u...@googlegroups.com.

> To post to this group, send email to kur...@googlegroups.com.
> Visit this group at https://groups.google.com/group/kurento.

> To view this discussion on the web visit https://groups.google.com/d/msgid/kurento/8ed3c791-ce28-4883-9cfc-6fda4f818c09%40googlegroups.com.

Mikhail Novikov

unread,

May 26, 2018, 1:31:00 AM5/26/18

to kurento

I will quote that to any new clients asking to implement Kurento in
their projects...

Jorge Maiquez

unread,

May 28, 2018, 3:54:14 AM5/28/18

to kurento

Hi Juan,

Can you give examples of which use cases fail for you after updating the library? It doesn't make sense for us to invest time into this if we are talking about common use cases.

Thanks,

Jorge