Facing intermittent Timeout issue due to WebSocket connection closed by Kurento

[Hemrajsinh Gharia]

• Operating system (Ubuntu 16.04, 18.04, etc.) --> Ubuntu 18.04
  
• Installation method (apt-get, Docker, AWS, build from sources) --> Deployed on EC2 through terraform cloud. 
• Kurento version --> 6.13

Hi,

We have been facing an intermittent timeout issue in production. So it can occur at any point in time. For e.g., for 2-3 days there is no issue and all of a sudden we face it. And we need to restart our application server and/or Kurento to fix it. On observing the Kurento logs when it occurred last time, I found that the Kurento server erased active connection and the Kurento client (from application server) was not aware of it. So after that (after Kurento erased the active connection), when someone joins the room and the Kurento client tries to establish a WebRTC connection for that new joiner, it ends up in Timeout exception. Following are the Kurento server logs when this occurred:

2020-03-05T12:22:25,862468 828 0x00007febc1ab9700    info KurentoWebSocketTransport WebSocketTransport.cpp:296 keepAliveSessions()  Keep alive 48bb923b-5a34-4a32-a786-662599b27c0f

2020-03-05T12:22:37,997492 828 0x00007febc4abf700    info KurentoServerMethods      ServerMethods.cpp:814 ping()  WebSocket Ping/Pong with sessionId 48bb923b-5a34-4a32-a786-662599b27c0f

2020-03-05T12:23:25,862611 828 0x00007febc1ab9700    info KurentoWebSocketTransport WebSocketTransport.cpp:296 keepAliveSessions()  Keep alive 48bb923b-5a34-4a32-a786-662599b27c0f

2020-03-05T12:24:25,862751 828 0x00007febc1ab9700    info KurentoWebSocketTransport WebSocketTransport.cpp:296 keepAliveSessions()  Keep alive 48bb923b-5a34-4a32-a786-662599b27c0f

2020-03-05T12:25:25,862890 828 0x00007febc1ab9700    info KurentoWebSocketTransport WebSocketTransport.cpp:296 keepAliveSessions()  Keep alive 48bb923b-5a34-4a32-a786-662599b27c0f

2020-03-05T12:26:25,863035 828 0x00007febc1ab9700    info KurentoWebSocketTransport WebSocketTransport.cpp:296 keepAliveSessions()  Keep alive 48bb923b-5a34-4a32-a786-662599b27c0f

2020-03-05T12:26:37,997508 828 0x00007febc22ba700    info KurentoServerMethods      ServerMethods.cpp:814 ping()  WebSocket Ping/Pong with sessionId 48bb923b-5a34-4a32-a786-662599b27c0f

2020-03-05T12:27:25,863174 828 0x00007febc1ab9700    info KurentoWebSocketTransport WebSocketTransport.cpp:296 keepAliveSessions()  Keep alive 48bb923b-5a34-4a32-a786-662599b27c0f

2020-03-05T12:28:25,863314 828 0x00007febc1ab9700    info KurentoWebSocketTransport WebSocketTransport.cpp:296 keepAliveSessions()  Keep alive 48bb923b-5a34-4a32-a786-662599b27c0f

2020-03-05T12:29:25,863455 828 0x00007febc1ab9700    info KurentoWebSocketTransport WebSocketTransport.cpp:296 keepAliveSessions()  Keep alive 48bb923b-5a34-4a32-a786-662599b27c0f

2020-03-05T12:30:25,863591 828 0x00007febc1ab9700    info KurentoWebSocketTransport WebSocketTransport.cpp:296 keepAliveSessions()  Keep alive 48bb923b-5a34-4a32-a786-662599b27c0f

2020-03-05T12:30:37,997471 828 0x00007febc32bc700    info KurentoServerMethods      ServerMethods.cpp:814 ping()  WebSocket Ping/Pong with sessionId 48bb923b-5a34-4a32-a786-662599b27c0f

2020-03-05T12:31:25,863730 828 0x00007febc1ab9700    info KurentoWebSocketTransport WebSocketTransport.cpp:296 keepAliveSessions()  Keep alive 48bb923b-5a34-4a32-a786-662599b27c0f

2020-03-05T12:32:25,863875 828 0x00007febc1ab9700    info KurentoWebSocketTransport WebSocketTransport.cpp:296 keepAliveSessions()  Keep alive 48bb923b-5a34-4a32-a786-662599b27c0f

2020-03-05T12:33:25,864022 828 0x00007febc1ab9700    info KurentoWebSocketTransport WebSocketTransport.cpp:296 keepAliveSessions()  Keep alive 48bb923b-5a34-4a32-a786-662599b27c0f

2020-03-05T12:34:25,864176 828 0x00007febc1ab9700    info KurentoWebSocketTransport WebSocketTransport.cpp:296 keepAliveSessions()  Keep alive 48bb923b-5a34-4a32-a786-662599b27c0f

2020-03-05T12:34:48,035810 828 0x00007febc32bc700 warning KurentoWebSocketTransport WebSocketTransport.cpp:382 storeConnection()  Erasing old connection associated with: 48bb923b-5a34-4a32-a786-662599b27c0f

2020-03-05T12:34:56,878793 828 0x00007febc6ac3700    info rtpsynchronizer           kmsrtpsynchronizer.c:141 kms_rtp_synchronizer_init_stats_file() <KmsRtpSynchronizer@0x7feb7c06b640>  No path for stats; enable with env variable: 'KMS_RTP_SYNC_STATS_PATH'

2020-03-05T12:34:56,878854 828 0x00007febc6ac3700    info rtpsynchronizer           kmsrtpsynchronizer.c:141 kms_rtp_synchronizer_init_stats_file() <KmsRtpSynchronizer@0x7feb7c06b590>  No path for stats; enable with env variable: 'KMS_RTP_SYNC_STATS_PATH'

2020-03-05T12:35:25,864319 828 0x00007febc1ab9700    info KurentoWebSocketTransport WebSocketTransport.cpp:296 keepAliveSessions()  Keep alive 48bb923b-5a34-4a32-a786-662599b27c0f

2020-03-05T12:36:25,864471 828 0x00007febc1ab9700    info KurentoWebSocketTransport WebSocketTransport.cpp:296 keepAliveSessions()  Keep alive 48bb923b-5a34-4a32-a786-662599b27c0f

2020-03-05T12:37:25,864623 828 0x00007febc1ab9700    info KurentoWebSocketTransport WebSocketTransport.cpp:296 keepAliveSessions()  Keep alive 48bb923b-5a34-4a32-a786-662599b27c0f

2020-03-05T12:38:25,864773 828 0x00007febc1ab9700    info KurentoWebSocketTransport WebSocketTransport.cpp:296 keepAliveSessions()  Keep alive 48bb923b-5a34-4a32-a786-662599b27c0f

2020-03-05T12:39:25,864932 828 0x00007febc1ab9700    info KurentoWebSocketTransport WebSocketTransport.cpp:296 keepAliveSessions()  Keep alive 48bb923b-5a34-4a32-a786-662599b27c0f

2020-03-05T12:40:25,865090 828 0x00007febc1ab9700    info KurentoWebSocketTransport WebSocketTransport.cpp:296 keepAliveSessions()  Keep alive 48bb923b-5a34-4a32-a786-662599b27c0f

2020-03-05T12:41:25,865243 828 0x00007febc1ab9700    info KurentoWebSocketTransport WebSocketTransport.cpp:296 keepAliveSessions()  Keep alive 48bb923b-5a34-4a32-a786-662599b27c0f

2020-03-05T12:42:25,865400 828 0x00007febc1ab9700    info KurentoWebSocketTransport WebSocketTransport.cpp:296 keepAliveSessions()  Keep alive 48bb923b-5a34-4a32-a786-662599b27c0f

2020-03-05T12:43:25,865556 828 0x00007febc1ab9700    info KurentoWebSocketTransport WebSocketTransport.cpp:296 keepAliveSessions()  Keep alive 48bb923b-5a34-4a32-a786-662599b27c0f

2020-03-05T12:44:25,865708 828 0x00007febc1ab9700    info KurentoWebSocketTransport WebSocketTransport.cpp:296 keepAliveSessions()  Keep alive 48bb923b-5a34-4a32-a786-662599b27c0f

2020-03-05T12:45:25,865860 828 0x00007febc1ab9700    info KurentoWebSocketTransport WebSocketTransport.cpp:296 keepAliveSessions()  Keep alive 48bb923b-5a34-4a32-a786-662599b27c0f

2020-03-05T12:46:25,866018 828 0x00007febc1ab9700    info KurentoWebSocketTransport WebSocketTransport.cpp:296 keepAliveSessions()  Keep alive 48bb923b-5a34-4a32-a786-662599b27c0f

As you can see in above Kurento logs, at 12:34:48 the Kurento closed the active connection. From logs, it looks like exactly at every 4 minutes Kurento sends the ping message to connected client. Last `ping()` was at 12:30:37. After that, it was expected to send next at 12:34:37. But I don't see it in logs. Rather at 12:34:48 closed the active connection. After which in the application server, I started getting following Timeout logs for new joining users:

12:40:45.398 [http-nio-8080-exec-14] ERROR com.broadcaster.CallHandler  some...@email.com d61cc7b9-bf17-929f-e934-256bd30b9352 - Failed to handle message

org.kurento.jsonrpc.JsonRpcException: [KurentoClient]  Timeout of 10000 milliseconds waiting from response to request {"id":885,"method":"create","params":{"type":"WebRtcEndpoint","constructorParams":{"mediaPipeline":"c348d8a9-b160-4587-bf42-671b8795bd4f_kurento.MediaPipeline"},"properties":{},"sessionId":"48bb923b-5a34-4a32-a786-662599b27c0f"},"jsonrpc":"2.0"}

at org.kurento.jsonrpc.client.AbstractJsonRpcClientWebSocket.internalSendRequestWebSocket(AbstractJsonRpcClientWebSocket.java:399)

at org.kurento.jsonrpc.client.AbstractJsonRpcClientWebSocket$1.internalSendRequest(AbstractJsonRpcClientWebSocket.java:141)

at org.kurento.jsonrpc.internal.JsonRpcRequestSenderHelper.sendRequest(JsonRpcRequestSenderHelper.java:75)

at org.kurento.jsonrpc.internal.JsonRpcRequestSenderHelper.sendRequest(JsonRpcRequestSenderHelper.java:69)

at org.kurento.jsonrpc.client.JsonRpcClient.sendRequest(JsonRpcClient.java:112)

at org.kurento.client.internal.transport.jsonrpc.RomClientJsonRpcClient.sendRequest(RomClientJsonRpcClient.java:228)

at org.kurento.client.internal.transport.jsonrpc.RomClientJsonRpcClient.create(RomClientJsonRpcClient.java:157)

at org.kurento.client.internal.transport.jsonrpc.RomClientJsonRpcClient.create(RomClientJsonRpcClient.java:147)

at org.kurento.client.internal.client.RomManager.create(RomManager.java:59)

at org.kurento.client.internal.client.RomManager.createWithKurentoObject(RomManager.java:257)

at org.kurento.client.AbstractBuilder.build(AbstractBuilder.java:65)

at com.broadcaster.CallHandler.handleMessage(CallHandler.java:303)

at com.broadcaster.CallHandler.handleTextMessage(CallHandler.java:349)

at org.springframework.web.socket.handler.AbstractWebSocketHandler.handleMessage(AbstractWebSocketHandler.java:43)

at org.springframework.web.socket.handler.WebSocketHandlerDecorator.handleMessage(WebSocketHandlerDecorator.java:75)

at org.springframework.web.socket.handler.LoggingWebSocketHandlerDecorator.handleMessage(LoggingWebSocketHandlerDecorator.java:56)

at org.springframework.web.socket.handler.ExceptionWebSocketHandlerDecorator.handleMessage(ExceptionWebSocketHandlerDecorator.java:58)

at org.springframework.web.socket.adapter.standard.StandardWebSocketHandlerAdapter.handleTextMessage(StandardWebSocketHandlerAdapter.java:114)

at org.springframework.web.socket.adapter.standard.StandardWebSocketHandlerAdapter.access$000(StandardWebSocketHandlerAdapter.java:43)

at org.springframework.web.socket.adapter.standard.StandardWebSocketHandlerAdapter$3.onMessage(StandardWebSocketHandlerAdapter.java:85)

at org.springframework.web.socket.adapter.standard.StandardWebSocketHandlerAdapter$3.onMessage(StandardWebSocketHandlerAdapter.java:82)

at org.apache.tomcat.websocket.WsFrameBase.sendMessageText(WsFrameBase.java:395)

at org.apache.tomcat.websocket.server.WsFrameServer.sendMessageText(WsFrameServer.java:119)

at org.apache.tomcat.websocket.WsFrameBase.processDataText(WsFrameBase.java:495)

at org.apache.tomcat.websocket.WsFrameBase.processData(WsFrameBase.java:294)

at org.apache.tomcat.websocket.WsFrameBase.processInputBuffer(WsFrameBase.java:133)

at org.apache.tomcat.websocket.server.WsFrameServer.onDataAvailable(WsFrameServer.java:82)

at org.apache.tomcat.websocket.server.WsFrameServer.doOnDataAvailable(WsFrameServer.java:171)

at org.apache.tomcat.websocket.server.WsFrameServer.notifyDataAvailable(WsFrameServer.java:151)

at org.apache.tomcat.websocket.server.WsHttpUpgradeHandler.upgradeDispatch(WsHttpUpgradeHandler.java:148)

at org.apache.coyote.http11.upgrade.UpgradeProcessorInternal.dispatch(UpgradeProcessorInternal.java:54)

at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:59)

at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:860)

at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1591)

at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)

at java.lang.Thread.run(Thread.java:748)

Caused by: java.util.concurrent.TimeoutException: Waited 10000 milliseconds (plus 69280 nanoseconds delay) for com.google.common.util.concurrent.SettableFuture@35de0fd4[status=PENDING]

at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:494)

at com.google.common.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:97)

at org.kurento.jsonrpc.client.AbstractJsonRpcClientWebSocket.internalSendRequestWebSocket(AbstractJsonRpcClientWebSocket.java:377)

... 41 common frames omitted

Our application server is running in Kubernetes and Kurento is running on EC2 within the same cluster.

So is there a way we can prevent the connection close? 

And for Kurento client, is there a way so that the client is aware of the closed connection and it reconnects automatically?

- Hemraj

[Hemrajsinh Gharia]

I found that Kurento client automatically reconnects. 

• https://doc-kurento.readthedocs.io/en/6.13.0/user/troubleshooting.html#kms-became-unresponsive-due-to-network-error-or-crash
• https://doc-kurento.readthedocs.io/en/6.13.0/user/troubleshooting.html#application-server
  

But I don't see any Reconnected logs in client. So how we can configure in the client? So that in case of temporary network issue (where ping/pong doesn't happen in time - 4minutes) or any other problem, it can retry connection and eventually reconnects?

Thanks in advance.

- Hemraj

[Hemrajsinh Gharia]

Hello All,

Anyone could share suggestions/feedback? Have anyone came across a similar situation? I appreciate the help.

Thanks

[Hemrajsinh Gharia]

Okay, I found some Kurento client logs in the client application, looking at which it seems like the Kurento client is reconnecting

05 Mar 2020 12:34:48.08212:34:47.998 [JsonRpcClient-hearbeatExec-e1-t0] WARN o.k.jsonrpc.client.JsonRpcClient - [KurentoClient] Error sending heartbeat to server. Exception: [KurentoClient] Timeout of 10000 milliseconds waiting from response to request {"id":359,"method":"ping","jsonrpc":"2.0","params":{"sessionId":"48bb923b-5a34-4a32-a786-662599b27c0f"}}

05 Mar 2020 12:34:48.08212:34:47.998 [JsonRpcClient-hearbeatExec-e1-t0] WARN o.k.jsonrpc.client.JsonRpcClient - [KurentoClient] Stopping heartbeat and closing client: failure during heartbeat mechanism

05 Mar 2020 12:34:48.08212:34:48.002 [nioEventLoopGroup-2-1] INFO o.k.j.c.JsonRpcClientNettyWebSocket - [KurentoClient] channel closed

05 Mar 2020 12:34:48.08212:34:48.016 [AbstractJsonRpcClientWebSocket-disconnectExec-e4-t0] INFO o.k.j.c.JsonRpcClientNettyWebSocket - [KurentoClient] Connecting native client

05 Mar 2020 12:34:48.08312:34:48.016 [AbstractJsonRpcClientWebSocket-disconnectExec-e4-t0] INFO o.k.j.c.JsonRpcClientNettyWebSocket - [KurentoClient] Creating new NioEventLoopGroup

05 Mar 2020 12:34:48.08312:34:48.022 [nioEventLoopGroup-4-1] INFO o.k.j.c.JsonRpcClientNettyWebSocket - [KurentoClient] Initiating new Netty channel. Will create new handler too!

But still, after this whenever new user joins and Kurento client tries to send WebRTC connection request (SDP offer) to Kurento server, I get Timeout exception (Logs provided in the first post above). So how this reconnection is handled? I was looking into  `closeHeartbeatOnFailure()` method at https://github.com/Kurento/kurento-java/blob/master/kurento-jsonrpc/kurento-jsonrpc-client/src/main/java/org/kurento/jsonrpc/client/JsonRpcClient.java#L294 and it looks that the client reconnects as per above logs. But after that, it looks like it is not able to send any request/message to Kurento server over that reconnected WebSocket connection. Also, I don't see `ping/pong` logs on Kurento server (logs in the first post above) after this reconnection logs of the client. 

So there seems like some issue after reconnection. @Ivan Gracia and @Micael Gallego, if you guys are reading this, please suggest how can I fix this. I am happy to contribute if this is a bug.

Thanks,

Hemraj

[Hemrajsinh Gharia]

Still looking for a solution/suggestion. Anyone could help?

[Micael Gallego Carrillo]

This is a bug in the reconnection system.

Please help us to fix the bug providing a way to reproduce it reliably. 

Regards

[Hemrajsinh Gharia]

Thanks for the feedback @Micael Gallego. Unfortunately, we are not clear what causes a disconnection, so recreating it is a challenge. However, I have an idea to try out. I would like to disable the network of the application server. So the application server won't be able to send a ping. And after 4 minutes, Kurento will assume disconnection and discard the WebSocket connection. After that, re-enable the network of the application server and see if it tries to reconnect and whether this bug gets reproduced. We are running an application server in k8s, so disabling network is kind of difficult, but we have found a way and we are planning to try that on the staging server. I will get back to you with the findings.

- Hemraj

[Micael Gallego]

Thank you!

Best regards

Micael Gallego

Kurento / OpenVidu Project Lead

--
You received this message because you are subscribed to the Google Groups "kurento" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kurento+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kurento/bad1a08d-5abc-44db-b0af-4a2e579246f5%40googlegroups.com.

[Hemrajsinh Gharia]

Hey Micael,

Hope you are safe and healthy.

I have been debugging this to find an issue and I guess I found why it is not enabling the heartbeat (start ping/pong) after reconnection. The following are my findings/understanding

•  On ping/pong failing following method invokes
  https://github.com/Kurento/kurento-java/blob/master/kurento-jsonrpc/kurento-jsonrpc-client/src/main/java/org/kurento/jsonrpc/client/JsonRpcClient.java#L286
  
• Which invokes closeWithReconnection() from https://github.com/Kurento/kurento-java/blob/ca361a1ba760c1f48f5489690120820299b258b8/kurento-jsonrpc/kurento-jsonrpc-client/src/main/java/org/kurento/jsonrpc/client/AbstractJsonRpcClientWebSocket.java#L669
• From that it invokes handleReconnectDisconnection and as isClosedByUser() is false, it will invoke reconnect method at  https://github.com/Kurento/kurento-java/blob/ca361a1ba760c1f48f5489690120820299b258b8/kurento-jsonrpc/kurento-jsonrpc-client/src/main/java/org/kurento/jsonrpc/client/AbstractJsonRpcClientWebSocket.java#L613
  Here we are disabling the heartbeat at line no 622, which is required I think.
  
  
   if (heartbeating) {

        disableHeartbeat();

   }
  
  
• Now when it gets reconnected through internalConnectIfNecessary method, we invoke updateSession() at https://github.com/Kurento/kurento-java/blob/ca361a1ba760c1f48f5489690120820299b258b8/kurento-jsonrpc/kurento-jsonrpc-client/src/main/java/org/kurento/jsonrpc/client/AbstractJsonRpcClientWebSocket.java#L739
  (I have a query, does update session aims to preserve the old pipeline and active WebRTC/RTP endpoint?, I guess yes)
• Now in updateSession() there seems a problem at https://github.com/Kurento/kurento-java/blob/ca361a1ba760c1f48f5489690120820299b258b8/kurento-jsonrpc/kurento-jsonrpc-client/src/main/java/org/kurento/jsonrpc/client/AbstractJsonRpcClientWebSocket.java#L781. It should be
  
  
  if (!heartbeating) {

  enableHeartbeat();

  }
  
  Because now we are reconnected and want to start heartbeat in case it was disabled. And we actually disabled it before reconnecting. So this seems a reason why after reconnection it does not start ping/pong.
  
  Please share your thoughts on this. I haven't tested it. But have created a PR https://github.com/Kurento/kurento-java/pull/21
  
  Thanks,
  Hemraj

To unsubscribe from this group and stop receiving emails from it, send an email to kur...@googlegroups.com.

[Hemrajsinh Gharia]

Hey @Micael,

Whenever you have a few, please have a look at the above findings.

And regarding recreating the issue, I think I found why it is happening in our setup. It is due to a high CPU for 2-3 minutes when a new presenter joins the Composite unit. Refer https://groups.google.com/forum/#!topic/kurento/17j2bejoZQk for more details. I think you can also recreate by putting some debug point on ping/pond response code on kurento server and hold it for 10000 milliseconds timeout. This way, on timeout, Kurento Client will assume that there is some issue and will disconnect. After that, you can run from the debug point on Kurento Server and see what happens on the reconnection attempt by the client.

Thanks,

Hemraj

[Micael Gallego]

Thank you for the PR.

We will review it and accept it it we found no issues with it.

Best regards

Micael Gallego

Kurento / OpenVidu Project Lead

To unsubscribe from this group and stop receiving emails from it, send an email to kurento+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kurento/b28dbbfe-a3c3-4432-b3c7-d2d61443c74b%40googlegroups.com.

[Maxim Solodovnik]

Hello,

It seems internal re-connection doesn't work for me :(  (or maybe I'm doing something wrong)

kurento-java 6.13.1

Client is created with simplest code:

`client = KurentoClient.create(kurentoWsUrl);`

What I have tried so far:

Scenario 1

1) stop KMS docker

2) start application

Fail to create connection to KMS 

3) start KMS docker

Result: KMS will never be connected

Scenario 2

0) Scheduled task checking if KMS was initially connected is added

1) stop KMS docker

2) start application

Fail to create connection to KMS 

3) start KMS docker

4) KMS is connected

5) stop KMS docker, wait 30 seconds

6) start KMS docker

Result: no re-connection

What am I doing wrong? :(((

Later on I have implement my own re-connection mechanism

It doesn't work due to disconnectExec seems to lock connection forever in AbstractJsonRpcClientWebSocket.reconnect

Is it possible to fix internal re-connection or make it possible to completely disable it?

To view this discussion on the web visit https://groups.google.com/d/msgid/kurento/b28dbbfe-a3c3-4432-b3c7-d2d61443c74b%40googlegroups.com.

[Hemrajsinh Gharia]

Reconnection is by default enabled to keep retrying forever. If you want to disable it, afaik, I am afraid there is is no env or java property defined. The only way is to fork the repo and set it to false at https://github.com/Kurento/kurento-java/blob/master/kurento-client/src/main/java/org/kurento/client/KurentoClient.java#L134 ad use the customized client.

But the reconnection should work. I have observed that after reconnection, the client does not start sending ping to the server. And in the event of no ping/pong in the default interval time of 4 minutes (I think), the server assumes some issue with the client and clear the resources associated with that client (pipelines). You can add a connection listener to be sure if it reconnects or not. 

[Maxim Solodovnik]

"retrying forever" can be turned OFF if KurentoClient is created using `KurentoClient.createFromJsonRpcClient`

Unfortunately this also doesn't disables internal re-connection

The worst part: client.destroy() seems to also lock forever

So I see no way to do clean-up and create new client ...

[Hemrajsinh Gharia]

How do you initiate JsonRpcClient ?? I am not able to find a way to do it.

[Maxim Solodovnik]

JsonRpcClient can be created as: `new JsonRpcClientNettyWebSocket(kmsUrl)`

[Maxim Solodovnik]

My current logic is:

The I got "disconnected" event:

1) I call "release()" on all WebRtcEndpoint and drop all refences

2) call "destroy()" on the client

Unfortunately this doesn't work

I got timeout exceptions on above operations

How can I clean everything?

I hope to get answer since this seems to be critical issue for us :((((

[Hemrajsinh Gharia]

Hi Maxim,

Added a post regarding reconnection fix https://groups.google.com/forum/#!topic/kurento/zED9jugKUqk. If in case it helps in your scenario.

Thanks