Kurento crash

1347 views
Skip to first unread message

Mikhail Novikov

unread,
Feb 6, 2018, 1:11:50 PM2/6/18
to kurento
from time to time i am getting a crash with this stack trace:

^[[31;1mSegmentation fault^[[0m (thread ^[[33;1m139796485371648^[[0m, pid ^[[33;1m7829^[[0m)
Stack trace:
^[[34;1m[g_socket_send_message]^[[0m
/usr/lib/x86_64-linux-gnu/libgio-2.0.so.0^[[32;1m:0x7B044^[[0m
^[[34;1m[nice_output_stream_new]^[[0m
/usr/lib/x86_64-linux-gnu/libnice.so.10^[[32;1m:0x2769E^[[0m
^[[34;1m[nice_output_stream_new]^[[0m
/usr/lib/x86_64-linux-gnu/libnice.so.10^[[32;1m:0x27813^[[0m
^[[34;1m[nice_agent_recv_nonblocking]^[[0m
/usr/lib/x86_64-linux-gnu/libnice.so.10^[[32;1m:0x11879^[[0m
^[[34;1m[gst_nice_src_get_type]^[[0m
/usr/lib/x86_64-linux-gnu/gstreamer-1.5/libgstnice15.so^[[32;1m:0x3902^[[0m
^[[34;1m[gst_nice_sink_get_type]^[[0m
/usr/lib/x86_64-linux-gnu/gstreamer-1.5/libgstnice15.so^[[32;1m:0x4203^[[0m
^[[34;1m[gst_base_sink_do_preroll]^[[0m
/usr/lib/x86_64-linux-gnu/libgstbase-1.5.so.0^[[32;1m:0x2A1B2^[[0m
^[[34;1m[gst_base_sink_do_preroll]^[[0m
/usr/lib/x86_64-linux-gnu/libgstbase-1.5.so.0^[[32;1m:0x2B620^[[0m
^[[34;1m[gst_flow_get_name]^[[0m
/usr/lib/x86_64-linux-gnu/libgstreamer-1.5.so.0^[[32;1m:0x6E5CF^[[0m
^[[34;1m[gst_pad_push]^[[0m
/usr/lib/x86_64-linux-gnu/libgstreamer-1.5.so.0^[[32;1m:0x76533^[[0m
^[[34;1m[gst_proxy_pad_chain_default]^[[0m
/usr/lib/x86_64-linux-gnu/libgstreamer-1.5.so.0^[[32;1m:0x5F5E3^[[0m
^[[34;1m[gst_flow_get_name]^[[0m
/usr/lib/x86_64-linux-gnu/libgstreamer-1.5.so.0^[[32;1m:0x6E5CF^[[0m
^[[34;1m[gst_pad_push]^[[0m
/usr/lib/x86_64-linux-gnu/libgstreamer-1.5.so.0^[[32;1m:0x76533^[[0m
^[[32;1m0x1B48D^[[0m at /usr/lib/x86_64-linux-gnu/gstreamer-1.5/libgstcoreelements.so
^[[34;1m[gst_flow_get_name]^[[0m
/usr/lib/x86_64-linux-gnu/libgstreamer-1.5.so.0^[[32;1m:0x6E5CF^[[0m
^[[34;1m[gst_pad_push]^[[0m
/usr/lib/x86_64-linux-gnu/libgstreamer-1.5.so.0^[[32;1m:0x76533^[[0m
^@

Where to look?

Thanks!

Jorge Maiquez

unread,
Mar 12, 2018, 9:56:01 AM3/12/18
to kurento
Hi all,

Has anyone figured out what is causing this the segmentation fault in g_socket_send_message? There are various reports of this in this news group, but not a single positive response from the community.

Is the kurento team aware of this? Is there a workaround?

We didn't see this error a single time in our development and staging environments. But now that we deployed it to production, our client has seen it twice in less than a week.

Any tips are greatly appreciated. The client has a large webinar planned for tomorrow.

Thanks!
Jorge Maiquez, Digital Samba

Jon Ruddell

unread,
Mar 12, 2018, 12:19:24 PM3/12/18
to kurento
Do you have any more information such as your KMS version, OS info, and how to reproduce the crash?

Kapa6ac79

unread,
Mar 12, 2018, 4:26:22 PM3/12/18
to kurento
I confirm the problem, when you switch to the latest version, the same bug appears. The error occurs when there is a load on the server (from about 50+ clients), but there is no clear algorithm for reproducing the error. It arises suddenly. The WebRTC one-to-many broadcast mode was tested. OS version of Ubuntu 16.04, KMS 6.7.0

среда, 7 февраля 2018 г., 2:11:50 UTC+8 пользователь Mikhail Novikov написал:

Alex Kandrashkin

unread,
Mar 13, 2018, 3:20:51 AM3/13/18
to kurento
Also have this issue (not the latest KMS)
KMS version:  6.7.0~1.g6ebaa27
Found modules:
   Module: 'core' version '6.6.3'
   Module: 'elements' version '6.6.3'
   Module: 'filters' version '6.7.0~1.g0314843'

Ubuntu 16.04.3 LTS (GNU/Linux 4.4.0-112-generic x86_64)

Do you have it on the latest KMS ? 

dawn_...@outlook.com

unread,
Mar 13, 2018, 3:24:14 AM3/13/18
to kurento
I also met this problem!


this is my stack trace:

[31;1mSegmentation fault [0m (thread [33;1m139918407882496 [0m, pid [33;1m26118 [0m)
Stack trace:
[34;1m[g_socket_send_message] [0m
/usr/lib/x86_64-linux-gnu/libgio-2.0.so.0 [32;1m:0x7B044 [0m
[34;1m[nice_output_stream_new] [0m
/usr/lib/x86_64-linux-gnu/libnice.so.10 [32;1m:0x2ACBF [0m
[34;1m[nice_output_stream_new] [0m
/usr/lib/x86_64-linux-gnu/libnice.so.10 [32;1m:0x2AF3B [0m
[34;1m[nice_agent_recv_nonblocking] [0m
/usr/lib/x86_64-linux-gnu/libnice.so.10 [32;1m:0x12AE9 [0m
[34;1m[gst_nice_src_get_type] [0m
/usr/lib/x86_64-linux-gnu/gstreamer-1.5/libgstnice15.so [32;1m:0x36B2 [0m
[34;1m[gst_nice_sink_get_type] [0m
/usr/lib/x86_64-linux-gnu/gstreamer-1.5/libgstnice15.so [32;1m:0x3FB3 [0m
[34;1m[gst_base_sink_do_preroll] [0m
/usr/lib/x86_64-linux-gnu/libgstbase-1.5.so.0 [32;1m:0x2A1B2 [0m
[34;1m[gst_base_sink_do_preroll] [0m
/usr/lib/x86_64-linux-gnu/libgstbase-1.5.so.0 [32;1m:0x2B620 [0m
[34;1m[gst_flow_get_name] [0m
/usr/lib/x86_64-linux-gnu/libgstreamer-1.5.so.0 [32;1m:0x6E5CF [0m
[34;1m[gst_pad_push] [0m
/usr/lib/x86_64-linux-gnu/libgstreamer-1.5.so.0 [32;1m:0x76533 [0m
[34;1m[gst_proxy_pad_chain_default] [0m
/usr/lib/x86_64-linux-gnu/libgstreamer-1.5.so.0 [32;1m:0x5F5E3 [0m
[34;1m[gst_flow_get_name] [0m
/usr/lib/x86_64-linux-gnu/libgstreamer-1.5.so.0 [32;1m:0x6E5CF [0m
[34;1m[gst_pad_push] [0m
/usr/lib/x86_64-linux-gnu/libgstreamer-1.5.so.0 [32;1m:0x76533 [0m
[32;1m0x1B48D [0m at /usr/lib/x86_64-linux-gnu/gstreamer-1.5/libgstcoreelements.so
[34;1m[gst_flow_get_name] [0m
/usr/lib/x86_64-linux-gnu/libgstreamer-1.5.so.0 [32;1m:0x6E5CF [0m
[34;1m[gst_pad_push] [0m
/usr/lib/x86_64-linux-gnu/libgstreamer-1.5.so.0 [32;1m:0x76533 [0m
 **
libnice:ERROR:agent.c:2342:agent_signal_component_state_change: assertion failed: (TRANSITION (DISCONNECTED, FAILED) || TRANSITION (GATHERING, FAILED) || TRANSITION (CONNECTING, FAILED) || TRANSITION (CONNECTED, FAILED) || TRANSITION (READY, FAILED) || TRANSITION (DISCONNECTED, GATHERING) || TRANSITION (GATHERING, CONNECTING) || TRANSITION (CONNECTING, CONNECTED) || TRANSITION (CONNECTED, READY) || TRANSITION (READY, CONNECTED) || TRANSITION (FAILED, CONNECTING) || TRANSITION (FAILED, GATHERING) || TRANSITION (DISCONNECTED, CONNECTING))

在 2018年2月7日星期三 UTC+8上午2:11:50,Mikhail Novikov写道:

Jorge Maiquez

unread,
Mar 13, 2018, 3:42:03 AM3/13/18
to kurento

Thanks for the responses.


We don’t know how to reproduce this. If we did, then we could at least manage our client in some way. 


The KMS setup was unchanged in the switch from staging to production (same box, same KMS version, same OS, etc). You can see the respective version information in Alex’s response above.


Kapa6ac79, can you please confirm that you see the same error on the very latest 2018-01-18 version of KMS?


Jon, are you aware if this is something that was specifically worked on in the 2018-01-18 version? I can’t see anything in the change logs that would indicate that is the case.


Regardless, it seems we are not the only ones experiencing this problem, and bug like is not something you want kicking around in production.


Any hints/experiences much appreciated!


On Tuesday, February 6, 2018 at 7:11:50 PM UTC+1, Mikhail Novikov wrote:

Jorge Maiquez

unread,
Mar 13, 2018, 5:35:57 AM3/13/18
to kurento
Are there any gstreamer guru's out there that could suggest which direction we should explore to trigger segmentation errors specifically in
g_socket_send_message

We don't have enough experience with gstreamer or that particular method to compile targeted test cases for reproducing this error. It would be good to have an educated-guess starting point. Is this more likely to be load related (didn't seem to be the case for us), or some dodgy camera/mic device driver on the client, or something else entirely?

Thanks,
Jorge

Jorge Maiquez

unread,
Mar 14, 2018, 8:45:41 AM3/14/18
to kurento
Quick update. The client's session went well (2 broadcasters, 140 viewers), so we got lucky this time.

Can someone please suggest what things we can (stress) test to try to trigger the g_socket_send_message segmentation fault?

Micael Gallego

unread,
Mar 16, 2018, 6:38:47 PM3/16/18
to kur...@googlegroups.com
We are working on that right now.

--
You received this message because you are subscribed to the Google Groups "kurento" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kurento+u...@googlegroups.com.
To post to this group, send email to kur...@googlegroups.com.
Visit this group at https://groups.google.com/group/kurento.
To view this discussion on the web visit https://groups.google.com/d/msgid/kurento/61baf5e2-14ed-456f-815a-8537910e4f23%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Message has been deleted
Message has been deleted
Message has been deleted

Jorge Maiquez

unread,
Mar 17, 2018, 6:35:17 AM3/17/18
to kurento

Hi Micael,


Does this mean that you know how to reproduce it? If so, can you give us some more info about the conditions that trigger the error?


This client will be ramping up soon, and I'd like to take active steps to minimize the risk of this occurring, even if that means we have to cripple our application slightly in the short term.


Basically, is there a workaround we can use until you guys have fixed this?


If you need help with any specific testing, let me know.


Thanks & all the best,


Jorge Maiquez

Digital Samba


On Friday, March 16, 2018 at 11:38:47 PM UTC+1, Micael Gallego Carrillo wrote:
We are working on that right now.

Jorge Maiquez

unread,
Mar 21, 2018, 3:27:59 AM3/21/18
to kurento
Any more details you can share with us Micael?

Micael Gallego

unread,
Mar 21, 2018, 4:18:57 AM3/21/18
to kur...@googlegroups.com
We are digging into the problem right now... When we have somo advances we will publish more information about it

--
You received this message because you are subscribed to the Google Groups "kurento" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kurento+u...@googlegroups.com.
To post to this group, send email to kur...@googlegroups.com.
Visit this group at https://groups.google.com/group/kurento.
Message has been deleted

Jorge Maiquez

unread,
Mar 21, 2018, 9:12:47 AM3/21/18
to kurento
I'm not sure why my messages keep getting deleted in this thread. Here is my last message again:

Thanks for the update. We are flying blind in production right now, so if there is anything at all that we can do right now to mitigate the error from occurring, please share- even it it's not the final solution.

And if you need assistance with testing, just let me know.


On Wednesday, March 21, 2018 at 9:18:57 AM UTC+1, Micael Gallego Carrillo wrote:
We are digging into the problem right now... When we have somo advances we will publish more information about it

Juan Navarro

unread,
Mar 23, 2018, 8:43:30 AM3/23/18
to kurento
Hi,

we are having some issues with the overzealous spam filter that Google has in their Google Groups. It even happens to us some times, that messages get deleted! Currently looking for solutions, but it seems that other communities have had similar problems in the past. Sort of totally disabling spam filtering, it seems there is not much we can do because the messages don't appear in the "awaiting for moderation" list in the administration view... they instead get outright deleted.

Jorge Maiquez

unread,
Mar 23, 2018, 10:10:06 AM3/23/18
to kurento
Hi Juan,

After one of my messages gets deleted, I see additional links in the Reply section of the UI, and one of those is (paraphrased) "click here to post". This then leads to a captcha, and after I verify I'm not a robot, I'm able to post a reply successfully.

It's not ideal, but maybe this helps someone who has the same problem to at least be able to post a reply without ripping their hair out :-)

Have a great weekend,
Jorge

Jorge Maiquez

unread,
Mar 30, 2018, 1:55:05 AM3/30/18
to kurento
Any update on this?


On Wednesday, March 21, 2018 at 2:12:47 PM UTC+1, Jorge Maiquez wrote:

ankit...@gmail.com

unread,
Mar 30, 2018, 10:22:29 PM3/30/18
to kurento
Jorge,

First of all, I am just a user like you. While the work is being done on this. Use monit to monitor KMS and restart it. There must be a way in you client API to get reconnected event and then use this event to reconnect your app. I do this in my NodeJS app

Regards
Message has been deleted
Message has been deleted
Message has been deleted
Message has been deleted
Message has been deleted
Message has been deleted

Jorge Maiquez

unread,
Mar 31, 2018, 12:36:26 AM3/31/18
to kur...@googlegroups.com
Hi Ankit,

Monit looks like a great tool, thanks for sharing. But unfortunately, monitoring isn’t the problem in this case. 

The problem is that KMS/gstreamer will seem to be working fine one moment, and then it will throw this error in the middle of a 400 user session, for example. Even if we can detect that it is about to go down (CPU pegged at 100%, etc), it doesn’t help much if a large customer session is already in progress. The result will be an interruption of service and an unhappy customer.

Really, what we’re looking for is some kind of “best practice” guideline that will allow us to minimize the risk of the error occurring, until the Kurento team has solved the root cause of the error.

In the blog post from March 22nd, the roadmap lists the following item:
- “Update GStreamer and several other underlying support libraries to their latest versions.”

Will that solve this problem? And is that already included in 6.7.1?

Any additional info would be greatly appreciated.

Thanks!
Jorge

-- 
You received this message because you are subscribed to a topic in the Google Groups "kurento" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/kurento/_rf1ANq5Cm8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to kurento+u...@googlegroups.com.

To post to this group, send email to kur...@googlegroups.com.
Visit this group at https://groups.google.com/group/kurento.

Micael Gallego

unread,
Mar 31, 2018, 6:06:52 AM3/31/18
to kur...@googlegroups.com
Hi Jorge, 

We have detected some problems in the libraries in Trusty version of KMS that leads to KMS crash. If you are using Trusty, please update to Xenial and report us if your problems are gone.

By the way, KMS 6.7.1 is not still updated to recent library versions, as this update requires a lot of work.

Best regards

Micael Gallego
Kurento / OpenVidu Project Lead

To unsubscribe from this group and all its topics, send an email to kurento+unsubscribe@googlegroups.com.

To post to this group, send email to kur...@googlegroups.com.
Visit this group at https://groups.google.com/group/kurento.
To view this discussion on the web visit https://groups.google.com/d/msgid/kurento/6c88bf15-99f4-4f0d-952a-77f01c8e9ddf%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "kurento" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kurento+unsubscribe@googlegroups.com.

To post to this group, send email to kur...@googlegroups.com.
Visit this group at https://groups.google.com/group/kurento.

Jorge Maiquez

unread,
Mar 31, 2018, 6:42:00 AM3/31/18
to kurento
Hi Micael,

Thanks for getting back. We are on Xenial and the problems we reported were on that version.

Are you able to reproduce the gstreamer_send_message error on Trusty? If so, can you share details? 

Is there anything else you can suggest for us to try?

Thanks!
Jorge

On Saturday, March 31, 2018 at 12:06:52 PM UTC+2, OpenVidu wrote:
Hi Jorge, 

We have detected some problems in the libraries in Trusty version of KMS that leads to KMS crash. If you are using Trusty, please update to Xenial and report us if your problems are gone.

By the way, KMS 6.7.1 is not still updated to recent library versions, as this update requires a lot of work.

Best regards

Micael Gallego
Kurento / OpenVidu Project Lead

Juan Navarro

unread,
Apr 12, 2018, 11:49:21 AM4/12/18
to kurento

TheV

unread,
May 25, 2018, 7:34:58 AM5/25/18
to kurento
We can reliably (9/10) reproduce this crash in our performance testing. At this point crashing is the expected behavior rather than the exception. Completely unusable in production.

Juan Navarro

unread,
May 25, 2018, 10:38:28 AM5/25/18
to kur...@googlegroups.com
The 3rd-party libnice library is the weak link here. The crash happens in that library, not in Kurento code. See https://github.com/Kurento/bugtracker/issues/247

Simply updating the library to latest versions (which would probably fix this issue) is not easy and straightforward for us because it makes some of our integration tests fail for some use cases. So this task has been registered in the issue tracker but delayed multiple times due to more pressing issues.

You may have success in updating the library for your use case without extra adaptation work, so I would suggest that you look into that. If you are not able to do so, or would need some help with this, you can also ask for our commercial support (which will also raise this issue's priority). Check https://doc-kurento.readthedocs.io/en/stable/business/

Regards,
Juan


> Sent: Friday, May 25, 2018 at 1:34 PM
> From: TheV <ry...@solutera.lt>
> To: kurento <kur...@googlegroups.com>
> Subject: [kurento-public] Re: Kurento crash
> --
> You received this message because you are subscribed to a topic in the Google Groups "kurento" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/kurento/_rf1ANq5Cm8/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to kurento+u...@googlegroups.com.
> To post to this group, send email to kur...@googlegroups.com.
> Visit this group at https://groups.google.com/group/kurento.
> To view this discussion on the web visit https://groups.google.com/d/msgid/kurento/8ed3c791-ce28-4883-9cfc-6fda4f818c09%40googlegroups.com.

Mikhail Novikov

unread,
May 26, 2018, 1:31:00 AM5/26/18
to kurento
I will quote that to any new clients asking to implement Kurento in
their projects...

Jorge Maiquez

unread,
May 28, 2018, 3:54:14 AM5/28/18
to kurento
Hi Juan,

Can you give examples of which use cases fail for you after updating the library? It doesn't make sense for us to invest time into this if we are talking about common use cases.

Thanks,
Jorge 

Jorge Maiquez

unread,
Jun 4, 2018, 8:56:06 AM6/4/18
to kurento
Any chance of a getting an answer to my previous question?

We are now just weeks away from our production deployment deadline, and I need to know what our options are for working around this problem. In the current state, our biggest concern is the stability of KMS, particularly this bug.

Thanks,
Jorge

TheV

unread,
Jun 4, 2018, 11:13:08 AM6/4/18
to kurento
Did some testing with different KMS versions down to 6.6.1 and while they all are affected in some form, 6.7.1 is the only one to crash with every load test (particularly during disconnects).

Paulo R. Lanzarin

unread,
Jun 4, 2018, 12:41:00 PM6/4/18
to kur...@googlegroups.com
Reposting from another thread:

If you aren't afraid of ignoring integration tests and such, and also ignore the ugliness of the workaround 
(desperate times, desperate measures):
https://github.com/prlanzarin/libnice/tree/crash-fix-upstream.

This is merged with libnice upstream. I added checks for NULL gsocket occurrences and commented out an assertion regarding
ICE state transition that was aborting Kurento. I reckon the assert is there for a reason, and there's probably some underlying condition
making it fail; however, I lack the time to go deeper into that. Assertion abortions also shouldn't be used in production hehe.
If anyone digs what's the underlying condition for that btw, I'd appreciate news regarding the problem.

It's been working nice for me with heavy load sessions (~300 streams or more, sometimes). However, use at your own risk :).

s, 

Paulo. 

--
You received this message because you are subscribed to the Google Groups "kurento" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kurento+unsubscribe@googlegroups.com.
To post to this group, send email to kur...@googlegroups.com.
Visit this group at https://groups.google.com/group/kurento.

Jorge Maiquez

unread,
Jun 5, 2018, 5:38:24 AM6/5/18
to kurento
Thanks TheV. Unfortunately, we need the latest version for other reasons.

And thanks for pointing to that thread Paulo. I think we will have to give this a shot, or at least deploy it and have it ready for a hot-swap when things go hairy in production.

Juan, I would still appreciate an answer on the use cases.
To unsubscribe from this group and stop receiving emails from it, send an email to kurento+u...@googlegroups.com.

To post to this group, send email to kur...@googlegroups.com.
Visit this group at https://groups.google.com/group/kurento.

Juan Navarro

unread,
Jun 5, 2018, 7:05:58 AM6/5/18
to kur...@googlegroups.com
Hi,

We are in the process of resurrecting our integration tests infrastructure and all browser-level tests will be reimplemented in a new product developed by the kurento team, ElasTest, which expands on the idea of the old ElasticRTC (bought to oblivion by Twilio). Sadly, this means that the tests we conducted while trying to upgrade libnice were manually done and are not reproducible.

The use cases that failed were related to basic browser-to-browser calls via Kurento Media Server; we found out that our own fork version of libnice contained several undocumented changes from the upstream version, and given that the older version was working mostly fine, we decided to lower this task's priority due to some other issues having more importance right then.

But now, complaints about libnice crashes have rised, so I've raised this concern with the team and we've decided to dedicate some time again and try upgrading again to the latest available libnice version, hoping that it will already contain fixes for all or most of its memory access errors. We're currently in the process of providing commercial support for one client, but in the second to third week of June we should be able to start working in this problem again.

Thanks for your patience; meanwhile, any help pinpointing the actual cause of these crashes could help in the integration of the new version of libnice; the comment about it being much more common in version 6.7.1 than it was in older ones is very interesting.

I'll be updating in this list about any news in this regard;
Juan
You received this message because you are subscribed to a topic in the Google Groups "kurento" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/kurento/_rf1ANq5Cm8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to kurento+u...@googlegroups.com.

To post to this group, send email to kur...@googlegroups.com.
Visit this group at https://groups.google.com/group/kurento.

Jorge Maiquez

unread,
Jun 7, 2018, 1:27:09 AM6/7/18
to kurento
Hi Juan,

Thanks for the more detailed response- much appreciated. We have not been able to pinpoint the trigger for the crashes, and this is why I asked about the use cases.

I understand that you give priority to commercial projects. Unfortunately, the time frames you have given for *starting* work on libnice integration will push us past our deadline. And we can't simply continue with the status quo and hope for the best. That will simply result in us losing customers. 

It looks like Paulo's reposted suggestion is currently the only option we have, and as you can imagine, it doesn't exactly fill me with confidence for a production deployment.

These kinds of bugs- especially when they are not taken care of quickly- erode confidence in the production capabilities of any platform. I suppose complaints about libnice crashes have increased because more and more folks out there are considering using KMS in a serious capacity. All the more reason to put this bug to bed quickly- before confidence is lost.

I'm sure none of this is news to you, and I look forward to updates on this problem.

All the best,
Jorge

Paulo R. Lanzarin

unread,
Jun 7, 2018, 10:04:16 AM6/7/18
to kur...@googlegroups.com

Given the commotion in this topic I'll try to clean up the fix in the solution I posted before. I'll do that in my free time though, so no promises, starting this weekend. If anyone wanna join the effort  please contact me or maintain the discussion in this thread. However, I'll just find the underlying cause and turn it into a real fix instead of a workaround. I really can't give a damn about integration tests because I don't really know what those are about, nor do I know where tests those are.

Juan, if you can link me the tests procedures I'd be happy to have a look. Also, feel free to give a look at the branch because the breaking points inside libnice were pinpointed and that's a hell of a start.

Jorge, if you're desperate I'd recommend to try out that branch. Despite the ugliness and my bad marketing skills, it works perfectly. Haven't seen any leaks nor crashes around libnice, and we've been running it for some time now and with some high stress involved.

To unsubscribe from this group and stop receiving emails from it, send an email to kurento+unsubscribe@googlegroups.com.

To post to this group, send email to kur...@googlegroups.com.
Visit this group at https://groups.google.com/group/kurento.

Jorge Maiquez

unread,
Jun 8, 2018, 8:48:26 AM6/8/18
to kurento
That's awesome Paulo. I wish my coding skills were up to scratch, but as it stands, I'd just be a total liability on that front. I will however contribute by reporting our experiences back to this thread. We'll get started next week. Thanks for the initiative!
<d

Alexandru Duzsardi

unread,
Jun 9, 2018, 6:04:58 AM6/9/18
to kurento
@Jorge
Here are the build instructions on ubuntu 16.04 , if you want to give it a spin with Paulo's fix

# as root or user with sudo privileges
export DISTRO="xenial"
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 5AFA7A83
sudo tee "/etc/apt/sources.list.d/kurento.list" >/dev/null <<EOF
# Kurento Media Server - Release packages
deb [arch=amd64] http://ubuntu.openvidu.io/6.7.1 $DISTRO kms6
EOF
sudo apt-get update
sudo apt-get build-dep libnice
sudo apt-get install fakeroot libgstreamer1.5-dev libgstreamer-plugins-base1.5-dev gobject-introspection libgirepository1.0-dev libgnutls28-dev

# as a regular user
mkdir BUILD
cd BUILD
apt-get source libnice
mv libnice-0.1.13/ libnice-0.1.13_ubu
git clone https://github.com/prlanzarin/libnice libnice-0.1.13
cd libnice-0.1.13
git checkout crash-fix-upstream
./autogen.sh
dpkg-buildpackage -rfakeroot -uc -b

you will end up with a few deb packages in ../BUILD dir
gir1.2-nice-0.1_0.1.13-3_amd64.deb    gstreamer1.5-nice_0.1.13-3_amd64.deb  libnice-dbg_0.1.13-3_amd64.deb  libnice-doc_0.1.13-3_all.deb
gstreamer1.0-nice_0.1.13-3_amd64.deb  libnice10_0.1.13-3_amd64.deb          libnice-dev_0.1.13-3_amd64.deb

of which i guess you only need to install libnice10_0.1.13-3_amd64.deb ( maybe uninstall the version from apt repositories first )

Jorge Maiquez

unread,
Jun 11, 2018, 7:11:00 AM6/11/18
to kurento
Thank you Alexandru- much appreciated. 

I'll ask someone from my team- with capabilities greater than me- to confirm here whether or not this worked for us.

Alexandru Duzsardi

unread,
Jun 11, 2018, 10:08:30 AM6/11/18
to kurento
Do you guys know if these messages are related to this issue ?

Jun 11 14:00:24 dev1-kurento2 kurento[11369]: (kurento-media-server:11369): GStreamer-CRITICAL **: gst_mini_object_unlock: assertion '(state & access_mode) == access_mode' failed
Jun 11 14:00:24 dev1-kurento2 kurento[11369]: (kurento-media-server:11369): GStreamer-CRITICAL **: gst_mini_object_unlock: assertion '(state & access_mode) == access_mode' failed
Jun 11 14:00:24 dev1-kurento2 kurento[11369]: (kurento-media-server:11369): GStreamer-CRITICAL **: gst_mini_object_unlock: assertion '(state & access_mode) == access_mode' failed
Jun 11 14:00:24 dev1-kurento2 kurento[11369]: (kurento-media-server:11369): GStreamer-CRITICAL **: gst_mini_object_unlock: assertion '(state & access_mode) == access_mode' failed
Jun 11 14:00:24 dev1-kurento2 kurento[11369]: (kurento-media-server:11369): GStreamer-CRITICAL **: gst_mini_object_unlock: assertion '(state & access_mode) == access_mode' failed
Jun 11 14:00:24 dev1-kurento2 kurento[11369]: (kurento-media-server:11369): GStreamer-CRITICAL **: gst_mini_object_unlock: assertion '(state & access_mode) == access_mode' failed
Jun 11 14:00:24 dev1-kurento2 kurento[11369]: (kurento-media-server:11369): GStreamer-CRITICAL **: gst_mini_object_unlock: assertion '(state & access_mode) == access_mode' failed
Jun 11 14:00:24 dev1-kurento2 kurento[11369]: (kurento-media-server:11369): GStreamer-CRITICAL **: gst_mini_object_unlock: assertion '(state & access_mode) == access_mode' failed
Jun 11 14:00:24 dev1-kurento2 kurento[11369]: (kurento-media-server:11369): GStreamer-CRITICAL **: gst_mini_object_unlock: assertion '(state & access_mode) == access_mode' failed
Jun 11 14:00:24 dev1-kurento2 kurento[11369]: (kurento-media-server:11369): GStreamer-CRITICAL **: gst_mini_object_unlock: assertion '(state & access_mode) == access_mode' failed
Jun 11 14:00:24 dev1-kurento2 kurento[11369]: (kurento-media-server:11369): GStreamer-CRITICAL **: gst_mini_object_unlock: assertion '(state & access_mode) == access_mode' failed
Jun 11 14:00:24 dev1-kurento2 kurento[11369]: (kurento-media-server:11369): GStreamer-CRITICAL **: gst_mini_object_unlock: assertion '(state & access_mode) == access_mode' failed
Jun 11 14:00:24 dev1-kurento2 kurento[11369]: (kurento-media-server:11369): GStreamer-CRITICAL **: gst_mini_object_unlock: assertion '(state & access_mode) == access_mode' failed


Also do you guys have a way to reproduce the crash with some load tests or something ?
We tried but it doesn't seem to react based on load or anything else ... it just crashes whenever it wants randomly.
Thaks!

Paulo R. Lanzarin

unread,
Jun 11, 2018, 11:11:51 AM6/11/18
to kur...@googlegroups.com
Hey Alexandru,

Thanks for posting the build instructions. I forgot about that completely.
Those assertions are certainly something to be looked at. I can't actually reproduce them yet,
but I'll skim through the code for the glib lock routine that might be throwing that. You're seeing
that with the stock libnice packages?

Regarding reproducing the crash: yeah, it's not deterministic. But we've been able to make it crash
based on load in a WebRTC only environment. With the stock packages, everytime it gets near
~140 streams it crashes, and we've assembled a test framework to reproduce it. It's a videoconferencing
app though, and the stress scripts are tightly coupled with our app, so I think it's no use that I share it here
(and I don't think I can hehe).

Have you tried the workaround? Did it work for you in any way?

s, 

Paulo.

--
You received this message because you are subscribed to the Google Groups "kurento" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kurento+u...@googlegroups.com.
To post to this group, send email to kur...@googlegroups.com.
Visit this group at https://groups.google.com/group/kurento.

Alexandru Duzsardi

unread,
Jun 11, 2018, 11:55:18 AM6/11/18
to kurento

Yes, those messages are with the stock libnice package.

Oh , thanks for the info , we actually tried the other way around ... one streamer and ~100 viewers.
We installed the patched version on some servers but we didn't really tested it since we couldn't make it crash on the other version either or it crashed very rarely and randomly, at least how we tested the load :)
We'll try again tomorrow with more streamers and viewers and maybe than we have some actual data to compare.
Thank you anyway for the work you've done.

Juan Navarro

unread,
Jun 12, 2018, 6:57:02 AM6/12/18
to kur...@googlegroups.com
Hi,
thank you very much for the work done here; I'm following closely this thread and, if the results are satisfactory, I agree to apply these changes as a contingency measurement in our copy of libnice while we work in upgrading it from upstream.
You received this message because you are subscribed to a topic in the Google Groups "kurento" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/kurento/_rf1ANq5Cm8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to kurento+u...@googlegroups.com.

To post to this group, send email to kur...@googlegroups.com.
Visit this group at https://groups.google.com/group/kurento.

Jorge Maiquez

unread,
Jun 12, 2018, 7:10:52 AM6/12/18
to kurento
FYI, we managed to build without any issues (with some slight mods to the instructions). 

The next step is to deploy it in the only environment where we have seen the crash to date0- hopefully tomorrow.

>> everytime it gets near ~140 streams it crashes

That's awesome info. We'll try to run this simulation on both the stock and the branch. I'll report all news/findings here.

Thanks guys!
Message has been deleted

Alexandru Duzsardi

unread,
Jun 14, 2018, 4:41:31 AM6/14/18
to kurento
applied the patched libnice yesterday on a test environment , didn't have kurento crashes until now with the tests we run
we will do more test and see what happens

# lsb_release -a
No LSB modules are available.
Distributor ID:    Ubuntu
Description:    Ubuntu 16.04.4 LTS
Release:    16.04
Codename:    xenial

# kurento-media-server -v
Kurento Media Server version: 6.7.1
Found modules:
   
'core' version 6.7.1
   
'elements' version 6.7.1
   
'filters' version 6.7.1

# dpkg -L libnice10 | grep so
/usr/lib/x86_64-linux-gnu/libnice.so.10.7.0
/usr/lib/x86_64-linux-gnu/libnice.so.10

Alexandru Duzsardi

unread,
Jun 20, 2018, 2:25:18 AM6/20/18
to kurento
The libnice error seems like it went away with the patched build but from time to time i see another crash happening
Any ideas what's causing it ?

kurento[31894]: Segmentation fault (thread 139745719990016, pid 31894)
kurento[31894]: Stack trace:
kurento[31894]: [sigc::internal::signal_emit1<void, kurento::ElementDisconnected, sigc::nil>::emit(sigc::internal::signal_impl*, kurento::ElementDisconnected const&)]
kurento[31894]: /usr/lib/x86_64-linux-gnu/libkmscoreimpl.so.6:0x10AD69
kurento[31894]: [kurento::MediaElementImpl::mediaFlowInStateChange(int, char*, KmsElementPadType)]
kurento[31894]: /usr/lib/x86_64-linux-gnu/libkmscoreimpl.so.6:0x10276F
kurento[31894]: [virtual thunk to kurento::MediaElementImpl::getGstreamerDot()]
kurento[31894]: /usr/lib/x86_64-linux-gnu/libkmscoreimpl.so.6:0xF9989
kurento[31894]: [g_closure_invoke]
kurento[31894]: /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0:0xFFA5
kurento[31894]: [g_signal_handler_disconnect]
kurento[31894]: /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0:0x21FC1
kurento[31894]: [g_signal_emit_valist]
kurento[31894]: /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0:0x2AD5C
kurento[31894]: [g_signal_emit]
kurento[31894]: /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0:0x2B08F
kurento[31894]: [check_if_flow_media]
kurento[31894]: /usr/lib/x86_64-linux-gnu/libkmsgstcommons.so.6:0x1F554
kurento[31894]: [gst_mini_object_steal_qdata]
kurento[31894]: /usr/lib/x86_64-linux-gnu/libgstreamer-1.5.so.0:0x6C29B
kurento[31894]: [g_hook_list_marshal]
kurento[31894]: /lib/x86_64-linux-gnu/libglib-2.0.so.0:0x3A904
kurento[31894]: [gst_mini_object_steal_qdata]
kurento[31894]: /usr/lib/x86_64-linux-gnu/libgstreamer-1.5.so.0:0x6AAFB
kurento[31894]: [gst_flow_get_name]
kurento[31894]: /usr/lib/x86_64-linux-gnu/libgstreamer-1.5.so.0:0x6E98B
kurento[31894]: [gst_pad_push]
kurento[31894]: /usr/lib/x86_64-linux-gnu/libgstreamer-1.5.so.0:0x76533
kurento[31894]: [gst_proxy_pad_chain_default]
kurento[31894]: /usr/lib/x86_64-linux-gnu/libgstreamer-1.5.so.0:0x5F5E3
kurento[31894]: [gst_flow_get_name]
kurento[31894]: /usr/lib/x86_64-linux-gnu/libgstreamer-1.5.so.0:0x6E5CF
systemd[1]: kurento.service: Main process exited, code=killed, status=6/ABRT
systemd[1]: kurento.service: Unit entered failed state.
systemd[1]: kurento.service: Failed with result 'signal'.
systemd[1]: kurento.service: Service hold-off time over, scheduling restart.
systemd[1]: Stopped Kurento Media Server daemon.
systemd[1]: Started Kurento Media Server daemon.

Paulo R. Lanzarin

unread,
Jun 20, 2018, 5:03:47 PM6/20/18
to kur...@googlegroups.com
Hey Alexandru,

Thanks for the report. That's odd, haven't seen that. I'd like to carry some tests to identify if
that's something caused by the fix or a bug that already exists on upstream.
Could you clarify a few things: what's the setup that caused that error (nature of endpoints,
number of users)? Are you able to reproduce it reliably? If so, can you share steps so I can
investigate it?
I also see that you're trying to generate a gstreamer dot diagram, is that right? Could you try
disabling that and see if the error persists?

Also, would you care to reproduce it and send me a full media server log besides the error log?

Thanks for trying it out.

s,

Paulo.

--
You received this message because you are subscribed to the Google Groups "kurento" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kurento+u...@googlegroups.com.
To post to this group, send email to kur...@googlegroups.com.
Visit this group at https://groups.google.com/group/kurento.

Juan Navarro

unread,
Jun 20, 2018, 5:38:37 PM6/20/18
to kur...@googlegroups.com
Hi Alexandru,
please also install these packages in your machine, so next time the segmentation fault happens, we'll have file name and line number to look for in the code:

sudo apt-get install libglib2.0-0-dbg libgstreamer1.5-0-dbg kms-core-dbg
You received this message because you are subscribed to a topic in the Google Groups "kurento" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/kurento/_rf1ANq5Cm8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to kurento+u...@googlegroups.com.

To post to this group, send email to kur...@googlegroups.com.
Visit this group at https://groups.google.com/group/kurento.

Alexandru Duzsardi

unread,
Jun 25, 2018, 2:38:06 AM6/25/18
to kurento
@Paulo i have no idea what getGstreamerDot is or how to disable it and we are not able to reproduce it , it just happens randomly.
One to many video streaming , it doesn't seem to matter how many users are connected.

Another thing is that it seems like i've missed some step in building the libnice library because apt complains that it's broken
The following packages have unmet dependencies:
 kms-elements : Depends: libnice10 (>= 0.1.13.1.xenial~20170725160546.81.eebfdab) but 0.1.13-3 is installed
E: Unmet dependencies. Try using -f.


Any idea how to solve it ? I can't install the libraries with the debug symbols left in.

Alexandru Duzsardi

unread,
Jun 25, 2018, 5:35:57 AM6/25/18
to kurento
solved the dependency issue by incrementing the build version to 0.1.13.2

Abhishek Mishra

unread,
Jun 25, 2018, 5:41:23 AM6/25/18
to kur...@googlegroups.com
Install dependency manually may resolve your problem

sudo apt-get install libnice10

--
You received this message because you are subscribed to the Google Groups "kurento" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kurento+u...@googlegroups.com.
To post to this group, send email to kur...@googlegroups.com.
Visit this group at https://groups.google.com/group/kurento.

Sergiy Kukunin

unread,
Jul 26, 2018, 3:46:58 AM7/26/18
to kurento
Any progress on this? A half of year has passed (since Feb) since this was reported, and no solution is available. 

I tried the patched libnice lib, but get another crash: Segmentation fault.

We can't go live because of this bug. It would be very disappointing, if we need to switch out from Kurento.

Juan Navarro

unread,
Aug 13, 2018, 4:10:49 PM8/13/18
to kurento

libnice is a third-party library that implements ICE connectivity, but the project seems to be understaffed. If you benefit in any form from it (directly or indirectly by using Kurento), then consider contributing in any way you can, as they have a good number of issues pending to be solved. One crash that has affected users of Kurento before seems to be well known and tracked here: https://gitlab.freedesktop.org/libnice/libnice/issues/20


We've been working at Kurento on the issue of this thread, and found out that the latest development branch of libnice seems to work better. In our tests, either versions 0.1.13 and 0.1.14 crashed, but the development branch for 0.1.15 (or as libnice creators like to put it, "0.1.14.1") is currently working p