"Ghost" TCP sessions on port 1544 (MRCP)

[Mickael Hubert]

Hi all,

I have an issue when I use unimrcpserver with MRCPv2.

I can use unimrcp to do STT, it works like a charm.

But the TCP sessions (port 1544 - MRCP) stay establish despite the end of communication (SIP + RTP).

There is no traffic on these ghosts sessions.

I can see:

1.1.1.1 = unimrcp server

2.2.2.2 = mrcp client

tcp        0      0 1.1.1.1:1544     2.2.2.2:8733      ESTABLISHED 0          208914     616/unimrcpserver  
tcp        0      0 1.1.1.1:1544     2.2.2.2:8649      ESTABLISHED 0          206168     616/unimrcpserver  
tcp        0      0 1.1.1.1:1544     2.2.2.2:8625      ESTABLISHED 0          203509     616/unimrcpserver  
tcp        0      0 1.1.1.1:1544     2.2.2.2:8565      ESTABLISHED 0          202881     616/unimrcpserver  
tcp        0      0 1.1.1.1:1544     2.2.2.2:8721      ESTABLISHED 0          207531     616/unimrcpserver  
tcp        0      0 1.1.1.1:1544     2.2.2.2:8689      ESTABLISHED 0          207222     616/unimrcpserver  
tcp        0      0 1.1.1.1:1544     2.2.2.2:8769      ESTABLISHED 0          209053     616/unimrcpserver  
tcp        0      0 1.1.1.1:1544     2.2.2.2:8681      ESTABLISHED 0          206469     616/unimrcpserver  
tcp        0      0 1.1.1.1:1544     2.2.2.2:8637      ESTABLISHED 0          203626     616/unimrcpserver  

the only way to kill these sessions is to restart unimrcp server, after this restart I can see:

tcp        0      1 1.1.1.1:1544     2.2.2.2:8737      FIN_WAIT1   0          0          -                  
tcp        0      1 1.1.1.1:1544     2.2.2.2:8617      FIN_WAIT1   0          0          -                  
tcp        0      1 1.1.1.1:1544     2.2.2.2:8725      FIN_WAIT1   0          0          -                  
tcp        0      1 1.1.1.1:1544     2.2.2.2:8581      FIN_WAIT1   0          0          -                  
tcp        0      1 1.1.1.1:1544     2.2.2.2:8745      FIN_WAIT1   0          0          -                  
tcp        0      1 1.1.1.1:1544     2.2.2.2:8657      FIN_WAIT1   0          0          -                  
tcp        0      1 1.1.1.1:1544     2.2.2.2:8733      FIN_WAIT1   0          0          -                  
tcp        0      1 1.1.1.1:1544     2.2.2.2:8649      FIN_WAIT1   0          0          -      

After 1 or 2 minutes, these sessions are killed

Do  you have some explanations about that please (bug maybe) ?

FYI:

- I use DYI STT module (into unimrcp server)

- I use unimrcpserver version 1.7.0

- I can reproduce this issue on debian or ubuntu server

Thanks in advance

[Mickael Hubert]

Hi all,

we can see an issue on client side, it doesn't send TCP/FIN. and the session still opened all the time.

We added TCP KEEPALIVE process (https://www.tldp.org/HOWTO/html_single/TCP-Keepalive-HOWTO/) in libs/mrcpv2-transport/src/mrcp_server_connection.c

apr_socket_opt_set(connection->sock, APR_SO_KEEPALIVE, 1);

it works like a charm.

After specifics timers (tcp_keepalive_time, tcp_keepalive_intvl, tcp_keepalive_probes) server sends TCP/RST to client to break TCP session cleanly.

in the unimrcp logs:

2020-06-25 11:43:33:453450 src/apt_poller_task.c:267 140591585289984 [DEBUG]  Process Signalled Descriptor [MRCPv2-Agent-1]
2020-06-25 11:43:33:453594 src/mrcp_server_connection.c:780 140591585289984 [INFO]   TCP/MRCPv2 Peer Disconnected 1.1.1.1:1544 <-> 2.2.2.2:8493
2020-06-25 11:43:33:453642 src/mrcp_server_connection.c:565 140591585289984 [NOTICE] Destroy TCP/MRCPv2 Connection 1.1.1.1:1544 <-> 2.2.2.2:8493

Do you want a PR on github ?

Have a good day

[Arsen Chaloyan]

Hi Mickael,

Sorry, I may not be able to follow all your analisis but there must be a reason for MRCPv2 TCP connections to remain stuck.

First, there are several configuration parameters that can be set for MRCPv2 connections. The most relevant ones are

      <max-connection-count>100</max-connection-count>
      <max-shared-use-count>100</max-shared-use-count>
      <force-new-connection>false</force-new-connection>
      <inactivity-timeout>600</inactivity-timeout>
      <termination-timeout>3</termination-timeout>

But I would try to find the root cause of the problem. If the client does not close a SIP/MRCPv2 session, then the session will be closed by the server upon expiration of other timeout such as

      <sip-session-expires>600</sip-session-expires>

and all the resources will be properly released in this case.

However, if there is a problem in the communication between the core and the plugin, then yes, the session and associated resources may not be released, as there is no indication that the underlying resources are no longer referenced. Needless to say, any action taken in these circumstances may easily cause a segfault.

--
You received this message because you are subscribed to the Google Groups "UniMRCP" group.
To unsubscribe from this group and stop receiving emails from it, send an email to unimrcp+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/unimrcp/036d29c4-1658-4416-96ab-646c73576e8bo%40googlegroups.com.

--

Arsen Chaloyan
Author of UniMRCP
http://www.unimrcp.org