OpenH264 crash

[Maxim Solodovnik]

Our Mac users observing disconnections

While checking the logs I found below errors and crash

Is this known issue?

Can someone help?

OS: Ubuntu 18.04.5 LTS

KMS: 6.14.0

grep -a "Error:bParseOnly" /var/log/kurento-media-server/errors.log|wc -l

17836

tail -n 50 /var/log/kurento-media-server/errors.log

[OpenH264] this = 0x0x7fee00040f00, Error:bParseOnly should be false for this API calling! 

[OpenH264] this = 0x0x7fee00040f00, Error:bParseOnly should be false for this API calling! 

[OpenH264] this = 0x0x7fee00040f00, Error:bParseOnly should be false for this API calling! 

[OpenH264] this = 0x0x7fee00040f00, Error:bParseOnly should be false for this API calling! 

[OpenH264] this = 0x0x7fee00040f00, Error:bParseOnly should be false for this API calling! 

[OpenH264] this = 0x0x7fee00040f00, Error:bParseOnly should be false for this API calling! 

[OpenH264] this = 0x0x7fee00040f00, Error:bParseOnly should be false for this API calling! 

[OpenH264] this = 0x0x7fee00040f00, Error:bParseOnly should be false for this API calling! 

[OpenH264] this = 0x0x7fee00040f00, Error:bParseOnly should be false for this API calling! 

[OpenH264] this = 0x0x7fed08028b30, Warning:Change QP Range from(0,51) to (12,42)

Segmentation fault (thread 140658400540416, pid 29936)

Stack trace:

[??]

??:0

0x4528 at /usr/lib/x86_64-linux-gnu/gstreamer-1.5/libgstopenh264.so

[gst_video_decoder_set_use_default_pad_acceptcaps]

/usr/lib/x86_64-linux-gnu/libgstvideo-1.5.so.0:0x41226

[gst_flow_get_name]

/usr/lib/x86_64-linux-gnu/libgstreamer-1.5.so.0:0x6D2E7

[gst_flow_get_name]

/usr/lib/x86_64-linux-gnu/libgstreamer-1.5.so.0:0x6D74D

[gst_flow_get_name]

/usr/lib/x86_64-linux-gnu/libgstreamer-1.5.so.0:0x6DAB9

[gst_mini_object_steal_qdata]

/usr/lib/x86_64-linux-gnu/libgstreamer-1.5.so.0:0x6B85F

[gst_pad_push_event]

/usr/lib/x86_64-linux-gnu/libgstreamer-1.5.so.0:0x775BF

[??]

??:0

0x2D4B4 at /usr/lib/x86_64-linux-gnu/gstreamer-1.5/libgstcoreelements.so

[gst_tag_setter_get_tag_merge_mode]

/usr/lib/x86_64-linux-gnu/libgstreamer-1.5.so.0:0xA0A00

[g_thread_pool_new]

/usr/lib/x86_64-linux-gnu/libglib-2.0.so.0:0x74B40

[g_test_get_filename]

/usr/lib/x86_64-linux-gnu/libglib-2.0.so.0:0x74175

[start_thread]

/lib/x86_64-linux-gnu/libpthread.so.0:0x76DB

[clone]

/lib/x86_64-linux-gnu/libc.so.6:0x121A3F

ls -la /usr/lib/x86_64-linux-gnu/libopenh*

lrwxrwxrwx 1 root root      42 Jul 30  2019 /usr/lib/x86_64-linux-gnu/libopenh264.so -> /usr/lib/x86_64-linux-gnu/libopenh264.so.4

lrwxrwxrwx 1 root root      42 Aug 30 04:48 /usr/lib/x86_64-linux-gnu/libopenh264.so.0 -> /usr/lib/x86_64-linux-gnu/libopenh264.so.4

-rw-r--r-- 1 root root 1418566 Jul 30  2019 /usr/lib/x86_64-linux-gnu/libopenh264.so.4

I can try to modify current scripts to generate core dump if it will help ....

[Maxim Solodovnik]

Shall I provide any additional details?

Can someone give me a hint what can be done here?

[Juan Navarro]

Install debug symbols so we can know the source file and line where the crash happens:

https://doc-kurento.readthedocs.io/en/6.14.0/dev/dev_guide.html?install-debug-symbols#install-debug-symbols

I'm currently on vacation so I won't be able to look into this until next week

Regards,
Juan


> Sent: Monday, September 07, 2020 at 4:46 AM
> From: "Maxim Solodovnik" <solom...@gmail.com>
> To: "kurento" <kur...@googlegroups.com>
> Subject: [kurento-public] Re: OpenH264 crash

>
> Shall I provide any additional details?
> Can someone give me a hint what can be done here?
>
> On Thursday, September 3, 2020 at 9:13:19 AM UTC+7 Maxim Solodovnik wrote:
>
> > Our Mac users observing disconnections
> > While checking the logs I found below errors and crash
> >
> > Is this known issue?
> > Can someone help?
> >
> > OS: Ubuntu 18.04.5 LTS
> > KMS: 6.14.0
> >

> > *grep -a "Error:bParseOnly" /var/log/kurento-media-server/errors.log|wc -l*
> >
> > 17836
> >
> > *tail -n 50 /var/log/kurento-media-server/errors.log*

> > *ls -la /usr/lib/x86_64-linux-gnu/libopenh**

> > lrwxrwxrwx 1 root root 42 Jul 30 2019
> > /usr/lib/x86_64-linux-gnu/libopenh264.so ->
> > /usr/lib/x86_64-linux-gnu/libopenh264.so.4
> > lrwxrwxrwx 1 root root 42 Aug 30 04:48
> > /usr/lib/x86_64-linux-gnu/libopenh264.so.0 ->
> > /usr/lib/x86_64-linux-gnu/libopenh264.so.4
> > -rw-r--r-- 1 root root 1418566 Jul 30 2019
> > /usr/lib/x86_64-linux-gnu/libopenh264.so.4
> >
> >
> > I can try to modify current scripts to generate core dump if it will help
> > ....
> >
> >
>

> --
> You received this message because you are subscribed to the Google Groups "kurento" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to kurento+u...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/kurento/ec2de746-6a83-41ce-8e43-420ed490c5f3n%40googlegroups.com.
>

[Maxim Solodovnik]

Thanks a lot Juan,

here is the stack from today's crash

2020-09-08T09:54:47+02:00 -- New execution

[OpenH264] this = 0x0x7f492c052710, Warning:SliceArgumentValidationFixedSliceMode(), unsupported setting with Resolution and uiSliceNum combination under RC on! So uiSliceNum is changed to 4!

[OpenH264] this = 0x0x7f492c052710, Warning:Change QP Range from(0,51) to (12,42)

Segmentation fault (thread 139951266047744, pid 26003)

Stack trace:

[gst_openh264enc_set_format]

./ext/openh264/gstopenh264enc.cpp:758

[gst_video_decoder_set_use_default_pad_acceptcaps]

/usr/lib/x86_64-linux-gnu/libgstvideo-1.5.so.0:0x41226

[gst_flow_get_name]

/usr/lib/x86_64-linux-gnu/libgstreamer-1.5.so.0:0x6D2E7

[gst_flow_get_name]

/usr/lib/x86_64-linux-gnu/libgstreamer-1.5.so.0:0x6D74D

[gst_flow_get_name]

/usr/lib/x86_64-linux-gnu/libgstreamer-1.5.so.0:0x6DAB9

[gst_mini_object_steal_qdata]

/usr/lib/x86_64-linux-gnu/libgstreamer-1.5.so.0:0x6B85F

[gst_pad_push_event]

/usr/lib/x86_64-linux-gnu/libgstreamer-1.5.so.0:0x775BF

[??]

??:0

0x2D4B4 at /usr/lib/x86_64-linux-gnu/gstreamer-1.5/libgstcoreelements.so

[gst_tag_setter_get_tag_merge_mode]

/usr/lib/x86_64-linux-gnu/libgstreamer-1.5.so.0:0xA0A00

[g_thread_pool_thread_proxy]

glib/gthreadpool.c:308

[g_thread_proxy]

glib/gthread.c:784

[start_thread]

/build/glibc-2ORdQG/glibc-2.27/nptl/pthread_create.c:463

[clone]

sysdeps/unix/sysv/linux/x86_64/clone.S:97

[Maxim Solodovnik]

weird enough: debug symbols were installed, KMS was restarted, but I don't see much difference

[Maxim Solodovnik]

trace from tonight:

[OpenH264] this = 0x0x7f09b4021f70, Warning:Change QP Range from(0,51) to (12,42)

Segmentation fault (thread 139676463609600, pid 9361)

[Paulo Lanzarin]

Hey Maxim,

I suspect the dbg symbols changed nothing because the crash itself seems to be happening in openh264, which, AFAIK,

has no dbg symbols from KMS's packages (but I might be wrong).

What I find odd about the log is saying that the QP range changed from (0,51). If my memory is not failing me, openh264

doesn't even support qp=0. Have you trying fiddling with agnosticbin and/or enctreebin in KMS to try and force a default qp

or constrain a qp range in the openh264 encoder setup?

s[],

prlanzarin

To view this discussion on the web visit https://groups.google.com/d/msgid/kurento/bcee51e9-ea45-4630-8829-75c28d301e75n%40googlegroups.com.

[Maxim Solodovnik]

Hello Paulo,

thanks foe your answer

unfortunately your advice is sort of magical spell to me :(

Could you please provide detailed steps of what should I do or provide me with a link so I can self-educate?

TL;DR;

I can't even reproduce this crash :(

I do have client, who have client with Mac

And somehow KMS crashes ~once in 2 days

I check the server in the morning and see the crash :( 

[Paulo Lanzarin]

Is it an option for you to actually disable H.264 usage? You can do that directly via KMS's SdpEndpoint.conf.ini (remove the H.264 entry and restart KMS).

Safari used to be H.264 only, but for more than a year now they also have VP8. So unless you explicitly need H.264, a good bet would be to just removed it

(at least that's what I did on my end).

About parameterizing the openh264 encoder, I was referring to this:
https://github.com/Kurento/kms-core/blob/master/src/gst-plugins/commons/kmsenctreebin.c#L170-L178

See: https://gstreamer.freedesktop.org/documentation/openh264/openh264enc.html

Try adding qp-max/qp-min (1-51).

To view this discussion on the web visit https://groups.google.com/d/msgid/kurento/93ce7f1e-5bc6-440c-89c0-6bef719d1dean%40googlegroups.com.

[Juan Navarro]

On 10/9/20 3:53, Maxim Solodovnik wrote:

Stack trace:

[gst_openh264enc_set_format]

./ext/openh264/gstopenh264enc.cpp:758

Well at least this line has repeated consistently in two of the stack traces. This might after all be an issue in GStreamer, and not OpenH264; the crash seems to happen at gstopenh264enc.cpp:758. Reviewing the history of that file in GStreamer, it seems that they rewrote part of it and removed the offending line in this commit, which got shipped with GStreamer 1.9.1.

I have backported that change into our fork branch fix-openh264. CI is working on it, so in a couple hours it should be ready for testing on this apt repository:

deb [arch=amd64] http://ubuntu.openvidu.io/fix-openh264 xenial kms6

(use that line in your "/etc/apt/sources.list.d/kurento.list", but using "xenial" or "bionic" according to your Ubuntu version)

As usual this will also be available in the form of an experimental Docker image, kurento/kurento-media-server-exp:fix-openh264
(https://hub.docker.com/r/kurento/kurento-media-server-exp).

Please use this build to test and confirm here if the issue is corrected, so we can merge the branch to master.

Regards

--
Juan Navarro
Kurento developer
Connect: @j1elo at GitHub & Twitter

[Juan Navarro]

Meanwhile, could you provide some technical details about the clients that are causing these crashes? You mentioned they are using Mac, so everything related: computer model, Mac OS version, Safari version (if using Safari) or the Chrome / Firefox version if using those.

Thanks

--

You received this message because you are subscribed to the Google Groups "kurento" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kurento+u...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kurento/94e969e7-c305-869f-c79a-bf41db2c486a%40gmx.es.

[Maxim Solodovnik]

Thanks a lot Juan,

I'll update the server ASAP and will keep you updated

The latest info from my customer is:

   Browser: Chrome 85.0.4183.83 (64 bit)

   macOS Catalina version 10.15.5

I'll ask for additional details

The problem is: the server is public with free registration, so some random user can crash everything

I can log navigator.appVersion, navigator.platform and navigator.userAgent on the server

Would it help?

[Maxim Solodovnik]

Thanks Paulo,

I can try to disable H264, will ask our users to test (I don't have Mac so can't test myself :(( )

[Maxim Solodovnik]

New crash after update:

[OpenH264] this = 0x0x7f0bb80053c0, Warning:SliceArgumentValidationFixedSliceMode(), unsupported setting with Resolution and uiSliceNum combination under RC on! So uiSliceNum is changed to 4!

[OpenH264] this = 0x0x7f0bb80053c0, Warning:Change QP Range from(0,51) to (12,42)

Segmentation fault (thread 139688073475840, pid 4763)

[Maxim Solodovnik]

Here are the last lines from KMS logs:

2020-09-11T09:30:14,698203 4763 0x00007f0cc3ccf700    info KurentoWebRtcEndpointImpl WebRtcEndpointImpl.cpp:104 remove_not_supported_codecs_from_array() <kmswebrtcendpoint14>  Removing not supported codec 'AMR/8000'

2020-09-11T09:30:14,698380 4763 0x00007f0cc3ccf700    info KurentoWebRtcEndpointImpl WebRtcEndpointImpl.cpp:540 WebRtcEndpointImpl()  STUN port not found in config; using default value: 3478

2020-09-11T09:30:14,698456 4763 0x00007f0cc3ccf700    info KurentoWebRtcEndpointImpl WebRtcEndpointImpl.cpp:548 WebRtcEndpointImpl()  STUN server not found in config; remember that NAT traversal requires STUN or TURN

2020-09-11T09:30:14,698522 4763 0x00007f0cc3ccf700    info KurentoWebRtcEndpointImpl WebRtcEndpointImpl.cpp:571 WebRtcEndpointImpl()  TURN server not found in config; remember that NAT traversal requires STUN or TURN

2020-09-11T09:30:14,706442 4763 0x00007f0c4f191700   fixme basesink                  gstbasesink.c:3125 gst_base_sink_default_event() <nicesink14>  stream-start event without group-id. Consider implementing group-id handling in the upstream elements

2020-09-11T09:30:14,706927 4763 0x00007f0bc57fa700   fixme default                   gstutils.c:3766 gst_pad_create_stream_id_internal() <nicesrc14:src>  Creating random stream-id, consider implementing a deterministic way of creating a stream-id

2020-09-11T09:30:14,707755 4763 0x00007f0cc6cd5700    info basertpendpoint           kmsbasertpendpoint.c:1118 kms_base_rtp_endpoint_start_transport_send() <kmswebrtcendpoint14>  Media 'video' has REMB

2020-09-11T09:30:14,707806 4763 0x00007f0cc6cd5700    info basertpendpoint           kmsbasertpendpoint.c:1064 kms_base_rtp_endpoint_create_remb_manager() <kmswebrtcendpoint14>  Creating REMB for session ID 0 (kmswebrtcendpoint14-sess0) and remote video SSRC 719777387

2020-09-11T09:30:14,707854 4763 0x00007f0cc6cd5700    info basertpendpoint           kmsbasertpendpoint.c:1075 kms_base_rtp_endpoint_create_remb_manager() <kmswebrtcendpoint14>  REMB: Set RTCP min interval to 500 ms

2020-09-11T09:30:14,775863 4763 0x00007f0cc64d4700    info KurentoWebRtcEndpointImpl WebRtcEndpointImpl.cpp:104 remove_not_supported_codecs_from_array() <kmswebrtcendpoint15>  Removing not supported codec 'AMR/8000'

2020-09-11T09:30:14,776028 4763 0x00007f0cc64d4700    info KurentoWebRtcEndpointImpl WebRtcEndpointImpl.cpp:540 WebRtcEndpointImpl()  STUN port not found in config; using default value: 3478

2020-09-11T09:30:14,776099 4763 0x00007f0cc64d4700    info KurentoWebRtcEndpointImpl WebRtcEndpointImpl.cpp:548 WebRtcEndpointImpl()  STUN server not found in config; remember that NAT traversal requires STUN or TURN

2020-09-11T09:30:14,776161 4763 0x00007f0cc64d4700    info KurentoWebRtcEndpointImpl WebRtcEndpointImpl.cpp:571 WebRtcEndpointImpl()  TURN server not found in config; remember that NAT traversal requires STUN or TURN

2020-09-11T09:30:14,829222 4763 0x00007f0baaffd700   fixme basesink                  gstbasesink.c:3125 gst_base_sink_default_event() <nicesink15>  stream-start event without group-id. Consider implementing group-id handling in the upstream elements

2020-09-11T09:30:14,830330 4763 0x00007f0baa7fc700   fixme default                   gstutils.c:3766 gst_pad_create_stream_id_internal() <nicesrc15:src>  Creating random stream-id, consider implementing a deterministic way of creating a stream-id

2020-09-11T09:30:14,830733 4763 0x00007f0cc54d2700    info basertpendpoint           kmsbasertpendpoint.c:1118 kms_base_rtp_endpoint_start_transport_send() <kmswebrtcendpoint15>  Media 'video' has REMB

2020-09-11T09:30:14,830793 4763 0x00007f0cc54d2700    info basertpendpoint           kmsbasertpendpoint.c:1064 kms_base_rtp_endpoint_create_remb_manager() <kmswebrtcendpoint15>  Creating REMB for session ID 0 (kmswebrtcendpoint15-sess0) and remote video SSRC 961729270

2020-09-11T09:30:14,830856 4763 0x00007f0cc54d2700    info basertpendpoint           kmsbasertpendpoint.c:1075 kms_base_rtp_endpoint_create_remb_manager() <kmswebrtcendpoint15>  REMB: Set RTCP min interval to 500 ms

2020-09-11T09:30:16,141291 4763 0x00007f0bc57fa700    info kmsutils                  kmsutils.c:494 kms_utils_pad_monitor_gaps() <'':sink_video_default>  Add probe: Detect stream gaps

2020-09-11T09:30:16,231076 4763 0x00007f0baa7fc700    info kmsutils                  kmsutils.c:494 kms_utils_pad_monitor_gaps() <'':sink_video_default>  Add probe: Detect stream gaps

2020-09-11T09:30:16,239633 4763 0x00007f0c4cb3d700   fixme videodecoder              gstvideodecoder.c:1057 gst_video_decoder_drain_out() <vp8dec0>  Sub-class should implement drain()

[Maxim Solodovnik]

Is it possible the issue is caused by the fact I'm using /usr/lib/x86_64-linux-gnu/libopenh264.so.4 (NOT /usr/lib/x86_64-linux-gnu/libopenh264.so.0)

I'm not sure how can I change it due to `libopenh264.so.4` is installed to the latest Ubuntu 18.04

[Juan Navarro]

On 11/9/20 13:57, Maxim Solodovnik wrote:

Is it possible the issue is caused by the fact I'm using /usr/lib/x86_64-linux-gnu/libopenh264.so.4 (NOT /usr/lib/x86_64-linux-gnu/libopenh264.so.0)

I'm not sure how can I change it due to `libopenh264.so.4` is installed to the latest Ubuntu 18.04

I agree. It looks like it is still using the previous installation, and not the new one, because the stack trace still points to the same file and line number:

Stack trace:

[gst_openh264enc_set_format]

./ext/openh264/gstopenh264enc.cpp:758

On Wed, Sep 9, 2020 at 10:53 PM Maxim Solodovnik <solom...@gmail.com> wrote:

trace from tonight:

Stack trace:

[gst_openh264enc_set_format]

./ext/openh264/gstopenh264enc.cpp:758

On Wednesday, September 9, 2020 at 8:46:14 AM UTC+7 Maxim Solodovnik wrote:

here is the stack from today's crash

Stack trace:

[gst_openh264enc_set_format]

./ext/openh264/gstopenh264enc.cpp:758

So as long as it says "gstopenh264enc.cpp:758" we can be sure it is still using the old files, not the new patched ones from the branch "fix-openh264". Even more, we can be sure because in the new code, line 758 doesn't contain code! (here).

If you can check with the Docker image, that simplifies testing different versions because it's just a matter of switching from one image name to the other (always remember to do a "docker pull imagename" before "docker run").

But I assume you're using the apt installation. In this case, please perform a complete uninstall (make a backup of your /etc/kurento config files if you have some important parameters in there), as described in Method B, here: https://doc-kurento.readthedocs.io/en/latest/user/installation.html#local-upgrade

After uninstalling all packages, make sure there is no remaining installations of openh264. Use `sudo updatedb && locate openh264` to search for any other .so file that you might have around the system, to delete it.

Doing so will make sure all of Kurento and related packages are first completely uninstalled, including support libraries like gst-plugins-bad and openh264. Of course, before reinstalling again, make sure you are using the correct line in your .list file:

deb [arch=amd64] http://ubuntu.openvidu.io/fix-openh264 xenial kms6

(use that line in your "/etc/apt/sources.list.d/kurento.list", but using "xenial" or "bionic" according to your Ubuntu version)

Run the --version command like shown here, and you should see these exact build versions and dates:

$ GST_DEBUG=KurentoModuleManager:4 /usr/bin/kurento-media-server --version

Loaded module: elements, version: 6.14.1~4.gf37243c, date: Sep 10 2020 20:03:16
Loaded module: core, version: 6.14.1~6.g6e9423a, date: Sep 10 2020 19:51:42
Loaded module: filters, version: 6.14.1~1.g3a8d8a7, date: Sep 10 2020 20:15:57
Kurento Media Server version: 6.14.1~6.g019a7fc
Found modules:
    'core' version 6.14.1~6.g6e9423a
    'elements' version 6.14.1~4.gf37243c
    'filters' version 6.14.1~1.g3a8d8a7

Also when installing the new KMS, remember to install the "kurento-dbg" package.

[Maxim Solodovnik]

Hello All,

Latest KMS 6.14.1 from `fix-openh264` repo is up and running for ~3.5 days

No crashes so far

keep testing :)

[Juan Navarro]

Nice! let's leave it a couple more days, just in case, and then I think it will pretty much be safe to merge the changes into main branch for the next release.

--
You received this message because you are subscribed to the Google Groups "kurento" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kurento+u...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kurento/85012fda-8895-471a-b6f3-f6b72cc46cb9n%40googlegroups.com.

[Maxim Solodovnik]

The server is up and running for 1 week

No crashes :)

Is it time for new release? :))