Audio and Video out of sync in Kurento 6.7.1

[Ayaan]

Audio and Video is out of sync in the new kurento version 6.7.1. It works okay with older versions. Seeing strange errors as follows in kurento logs:

2018-04-07 16:24:47,529020 14631 [0x00007f1cefed6700]   fixme basesink                  gstbasesink.c:3125 gst_base_sink_default_event() <nicesink562>  stream-start event without group-id. Consider implementing group-id handling in the upstream elements

2018-04-07 16:24:47,529270 14631 [0x00007f1cf0caa700]   fixme default                   gstutils.c:3766 gst_pad_create_stream_id_internal() <nicesrc562:src>  Creating random stream-id, consider implementing a deterministic way of creating a stream-id

2018-04-07 17:43:58,444437 3471 [0x00007f5819ffb700] warning dtlssrtpdemux             gstdtlssrtpdemux.c:137 sink_chain() <dtls-srtp-demux>  received invalid buffer: 1

2018-04-07 17:44:28,593477 3471 [0x00007f5819ffb700] warning rtpsource                 rtpsource.c:1147 update_receiver_stats()  duplicate or reordered packet (seqnr 17019, expected 17021)

2018-04-07 17:44:28,870287 3471 [0x00007f5819ffb700] warning srtpdec                   gstsrtpdec.c:1149 gst_srtp_dec_decode_buffer() <srtp-decoder:rtp_sink>  Unable to unprotect buffer (unprotect failed code 9)

[Ayaan]

Anyone else noticed audio and video out of sync issue with Kurento 6.7.1 version?

[cuong chung]

Hi Ayaan,

  I had same issue like you, but it mission audio. After I was test with another pc, it work fine. Maybe, it dependent on physical device.

Best regard

Cuong Chung.

Vào 04:49:08 UTC+7 Chủ Nhật, ngày 08 tháng 4 năm 2018, Ayaan đã viết:

[Ayaan]

Not really. I had issues with different computers as well. The issues seems buffer related with 6.7.1. I rolled back to older kurento version and it is working like charm now.

[Kyle Foley]

Ayaan,

I am having the same issue with 6.7.1, except my issue seems to only happen with the recording. Are you using the compositor or SFU configuration?

Kyle

[Ayaan]

I am using SFU configuration. The audio and video sync issue in 6.7.1 is noticeably very bad. Reverting back to 6.6.3 or earlier works just great. 

[Juan Navarro]

Hi,

only thing I can think of is that prior to release 6.7.0, a Pull Request was merged and it changed the way buffers are managed in kms-core and kms-elements:
https://github.com/Kurento/kms-core/pull/7
https://github.com/Kurento/kms-elements/pull/11

These changes were problematic and later some contributors suggested to fix the issues they were having:
https://github.com/Kurento/kms-core/pull/9

I'm not sure if these changes are the reason for this synchronization problem. Could you run a modified version of KMS without those changes?

It would be a matter of following the steps outlined in the Developing KMS section of the documentation:
http://doc-kurento.readthedocs.io/en/stable/dev/dev_guide.html#developing-kms

- Set the package repos
- Install development packages
- Clone the KMS sources
- Build
- Launch KMS

But, _before_ building and launching, you would undo the changes introduced by those PRs:

cd kms-omni-build
cd kms-core
git revert --no-edit 42c657c d25a8cf
cd ..
cd kms-elements
git revert --no-edit 097cd13
cd ..

Now, you would build and launch KMS as per the instructions in the documentation.

If you can test this custom build of KMS and it clearly works better than the current version, we'll have to re-evaluate the actual impact of those PRs that were accepted for Kurento 6.7.0.

Kind regards,
Juan

[Ayaan]

Thanks Juan. I will try this out. Kurento used to work better until 6.6.2 version and it started giving lot of performance issues especially around audio cut off, choppy audio, audio/video not in sync etc., after 6.6.3 version and it is getting worse with newer kurento versions. 

[Juan Navarro]

"srtpdec unprotect failed code 9" has been appearing in logs everywhere for a while already, so I've created an issue to track progress when we get some time to work on it:
https://github.com/Kurento/bugtracker/issues/246

On Saturday, April 7, 2018 at 11:49:08 PM UTC+2, Ayaan wrote:

[Noah Sun]

I also get this problem in kurento version 6.7.1.  Some time audio is missing.

Logs are as below:

2018-05-24 19:52:41,924647 45284 [0x00007f36a0dfa700] warning rtpsource                 rtpsource.c:1147 update_receiver_stats()  duplicate or reordered packet (seqnr 22937, expected 22939)

2018-05-24 19:52:43,923763 45284 [0x00007f365f5fe700] warning rtpsource                 rtpsource.c:1147 update_receiver_stats()  duplicate or reordered packet (seqnr 10152, expected 10154)

2018-05-24 19:52:45,366418 45284 [0x00007f365f5fe700] warning rtpsource                 rtpsource.c:1147 update_receiver_stats()  duplicate or reordered packet (seqnr 10225, expected 10227)

[Noah Sun]

Hi, Juan. I found the choppy audio problem related to the leaky queue in kms-core/src/gst-plugins/commons/kmsenctreebin.c, maybe 600ms is not enough for some bad network case.

[Juan Navarro]

Could you please:

- Describe your network setup, to have some context about what kind of bad network you are experiencing (is it fibre, 3G, LAN over Wifi?)
- Test with higher values for the buffer?

I'm not very happy with the idea of increasing this buffer value, as the 600ms figure that we're using here was taken from an already-raised value (from 200ms)
https://github.com/Kurento/kms-core/commit/bbd29c051b28b86eda04a37486be905987b70c43

Also we'd be getting close to 1s buffer, and RTP packets coming with such huge delay are already too late anyway and will probably get discarded by the receiver...

This needs a bit more study but I'm starting to feel like this whole change should be reverted and the original use case (https://groups.google.com/d/topic/kurento/_IsI7tcdLVw/discussion) should be left unresolved for now, as it was concerned only about the edge case of CPU exhaustion.

--
You received this message because you are subscribed to a topic in the Google Groups "kurento" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/kurento/Urpu0ZlK6wY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to kurento+u...@googlegroups.com.
To post to this group, send email to kur...@googlegroups.com.
Visit this group at https://groups.google.com/group/kurento.
To view this discussion on the web visit https://groups.google.com/d/msgid/kurento/6a66da83-a007-4734-b78c-e88700c0e849%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Noah Sun]

Hi, Juan,

1. the network of my user maybe 3G, 4G or wifi.

2. Sorry, I diden't make a clear description of my problem. Here is it:

    WebRTC           WebRTC

A -------------> KMS ------------> B

   <-------------  | |     <-------------

    WebRTC    | |     WebRTC

                       | |

                       | | (composite)

                       | |

         RTP live stream / Recorder

It seems that there is no problem between A and B, but there is choppy audio in RTP live stream/Recorder sometime.

I don't know the difference between kmsenctreebin and kmsagnosticbin, I increase the leaky queue as 6s in kmsenctreebin, but the delay between A and B is not increased.

[Noah Sun]

Here is the log, but I don't know what is the queue.

2018-06-11 21:09:53,000020 47715 [0x00007efb20dfa700]   debug queue_dataflow            gstqueue.c:1200 gst_queue_leak_downs

tream() <queue150>  queue is full, leaking item 0x7efa14001a30 on downstream end

2018-06-11 21:09:53,000047 47715 [0x00007efb2a1fc700]   debug queue_dataflow            gstqueue.c:1627 gst_queue_loop() <qu

eue141>  queue is not empty

2018-06-11 21:09:53,000127 47715 [0x00007efb20dfa700]   debug queue_dataflow            gstqueue.c:1203 gst_queue_leak_downs

tream() <queue150>  Storing sticky event segment on srcpad

2018-06-11 21:09:53,000213 47715 [0x00007efb20dfa700]   debug queue_dataflow            gstqueue.c:1200 gst_queue_leak_downs

tream() <queue150>  queue is full, leaking item 0x7efa14001870 on downstream end

2018-06-11 21:09:53,000295 47715 [0x00007efb20dfa700]   debug queue_dataflow            gstqueue.c:1203 gst_queue_leak_downs

tream() <queue150>  Storing sticky event tag on srcpad

2018-06-11 21:09:53,000373 47715 [0x00007efb20dfa700]   debug queue_dataflow            gstqueue.c:1200 gst_queue_leak_downs

tream() <queue150>  queue is full, leaking item 0x7efc70004430 on downstream end

[Noah Sun]

I increase the leaky queue as 6s in kmsenctreebin, but this try not works. sometimes no audio, sometimes audio choppy.

On Tuesday, June 12, 2018 at 2:52:13 PM UTC+8, Noah Sun wrote:

[Artoon Solutions]

    Hello Juan,

    We are developing one-to-many broadcast nodejs app and for that, we have used 6.7.1 version of kurento-media-server on AWS instance with 72 cores CPU and 144 GB of RAM. 

    Case: 1

    We became successful to handle 550 real-time users with the fixed bandwidth. The only issue we have faced was the poor video clarity and packet drops.

    The socket server was started with 20 Kurento threads and 8 nodejs cluster with pm2. 

    Case: 2 

    When we set the below bandwidth limit to get good quality video, we are getting delay after 10 -15 mins.

    webRtcEndpoint.setMinVideoSendBandwidth(300); //delay starts after 15 min with 400 concurrent connections

    webRtcEndpoint.setMinVideoSendBandwidth(400); //delay starts after 10 min with 400 concurrent connections

    The very first 10-15 mins stream works perfectly without any delay, but exact after 10-15 mins it starts lagging and we are getting delayed in it and there is also sort of video delay for

    the publisher while audio works perfectly in this case.

    Can you Provide this info?

    1. Reason to get delayed video after 15 mins. Does it affect by TURN/STUN server?

    2. Reason for lagging and Freezing the Video and Audio after 15 mins.

    3. The reason behind transcoding gets disable.

    FYI: 

    - We are using a One-to-Many nodejs module.

    - We have started the project as nodejs cluster using pm2.

    - KMS server and nodejs cluster are running in the same instance.

    Regards,

    Nitin Bhanderi.

[Manzoor Ahmed Bajwa]

@Juan Navarro did you find any solution. I am having same problem. When I record from web site it works fine. But unfortunately, same code doesn't work for mobile devices (Android + iOS apps). Please let me know.

[Juan Navarro]

Hi,
a new experimental branch contains a tentative fix that probably solves this issue; please use it either by installing with this apt-get repository:

For Ubuntu 16.04 Xenial:

    deb [arch=amd64] http://ubuntu.openvidu.io/fix-srtp-unprotect xenial kms6

For Ubuntu 18.04 Bionic:

    deb [arch=amd64] http://ubuntu.openvidu.io/fix-srtp-unprotect bionic kms6

or with this Docker image (https://hub.docker.com/r/kurento/kurento-media-server-exp/tags):

    kurento/kurento-media-server-exp:fix-srtp-unprotect

For more information and discussion about this bugfix, check the announcement here: https://groups.google.com/d/topic/kurento/cKzgDVrAch4/discussion

--
Juan Navarro
Kurento maintainer & developer
j1elo @ Twitter / GitHub

On Saturday, April 7, 2018 at 11:49:08 PM UTC+2, Ayaan wrote: