PlayerEndPoint bug: "client at host localhost, port 5004 not found"

[israel.r...@gmail.com]

Hello,

I have a setup where I stream cameras to chrome clients.

(RTP from camera to ffmpeg in host to RTSP server to KMS docker instant PlayerEndPoint to KMS WeBRTCEndPoint to chrome). Kurento version is 6.14.

I have many cameras so I load Many KMS dockers to lower the CPU usage in each of them (in this case 30). I always connect to the next "free" kms.

25 of them work perfectly, takes about 6-8 seconds to connect which makes sense (3 seconds GOP).

The other 5, ALWAYS take about 45 seconds to connect.

Always the same 5 kms docker instances take 45 seconds to connect.

If I restart the problematic docker, it usually fixes it - but that's not a reasonable solution.

I attached extensive logs of that docker, the important lines from it are:

0:02:06.414192600     1 0x7f3800067a80 DEBUG             kmselement kmselement.c:959:kms_element_set_sink_input_stats:<kmswebrtcendpoint0> Generating average stats for pad <kmswebrtcendpoint0:sink_video_default>

20 seconds later:

0:02:24.551680800     1 0x7f37e8010400 WARN            multiudpsink gstmultiudpsink.c:1766:gst_multiudpsink_remove:<udpsink0> client at host localhost, port 5004 not found

0:02:24.551728700     1 0x7f37e8010400 DEBUG                rtspsrc gstrtspsrc.c:3578:gst_rtspsrc_stream_configure_udp_sinks:<source> RTP UDP src has sock 0x7f37bc015280

0:02:24.551845000     1 0x7f37e8010400 DEBUG                rtspsrc gstrtspsrc.c:3612:gst_rtspsrc_stream_configure_udp_sinks:<source> configure RTCP UDP sink for 192.168.1.12:10011

20 seconds later:

0:02:44.575613300     1 0x7f37e8010400 WARN            multiudpsink gstmultiudpsink.c:1766:gst_multiudpsink_remove:<udpsink1> client at host localhost, port 5004 not found

0:02:44.575650700     1 0x7f37e8010400 DEBUG                rtspsrc gstrtspsrc.c:3636:gst_rtspsrc_stream_configure_udp_sinks:<source> RTCP UDP src has sock 0x7f37bc0153d0

And from there it connects.

Meaning there are 40 seconds gap because of this "client at host localhost, port 5004 not found"

This line doesn't exist on the working well KMS instances.

In addition, there is nothing, anywhere, that requests port 5004.

Nothing in KMS configuration, nothing in the camera\ffmpeg.

I have no idea why it even try to go there!

And the other line about stream gaps, I saw you had a bug with it that was fixed on kurento 6.11, so maybe it's not really fixed?

Additional information:

A. It's probably not about network issues, since the original stream comes from the host, and client is at the host. Meaning all the sides (server+client) are on the same machine.

Also, it doesn't happen on the different KMS instances, running the same pipeline.

B. It's not CPU or memory issues. The problematic KMS have the same memory\CPU usage as the working others, nothing interesting there.

Any idea what can cause such a strange but 100% repeatable thing?

Thanks,

Robotnick Israel

[israel.r...@gmail.com]

Something to add, cause it might be related.

The Linux which holding the dockers is a virtual linux on hyperV, that runs on a windows. The windows has the chrome client that connects to that linux.

Maybe this causes the port 5004  log?

I just don't understand why does kurento even try to go to 5004 when it never got any request related to that port.

All of the KMS dockers are running on network host, maybe they some how get messages between themselves that they arent supposed to? 

(each running on a different WS port). Is there something like this in KMS to communicate between it's parts?

Though none of them configured to 5004. (in my case anyway)

Thanks,

Israel

[israel.r...@gmail.com]

No one knows what can cause KMS act so strangely?