Packet Loss During Media Traffic Burst - TURN Server 4.5.0.6

[Kannan Murali]

Hi:

I am using the coturn 4.5.0.6 version in CentOS 7.2. I am using this TURN server along with the WebRTC Gateway for our conferencing solution.

I am experiencing UDP packet loss when I do AppShare (the packet loss could be with audio/video also, but they are not human noticeable), But with 

Appshare (for both AppShare and Video we are using H.264 video codec) it's very well noticeable.

I had only one WebRTC client (having 8 RTP sessions - 4 audio and 4 video) connected through the TURN server - not that much of a traffic going on 

in the TURN server. But there is a burst in the media traffic expected with the AppShare.

I identified the packet loss using the following tools:

1. Using the "netstat -s" command:

    .......

    Udp:

    2369115 packets received

    111 packets to unknown port received.

    6197 packet receive errors        <<<<<<<<<<<<<<<<<<<<<

    2368202 packets sent

    0 receive buffer errors

    0 send buffer errors

    ......

2. using the "dropwatch" tool:

4 drops at skb_queue_purge+18 (0xffffffff8151a838)

100 drops at udp_queue_rcv_skb+3b4 (0xffffffff815973b4)

1 drops at __udp4_lib_mcast_deliver+290 (0xffffffff81597890)

2 drops at unix_stream_connect+2ca (0xffffffff815d439a)

1 drops at unix_release_sock+188 (0xffffffff815d2a18)

1 drops at unix_dgram_sendmsg+459 (0xffffffff815d3c09)

2 drops at unix_stream_connect+2ca (0xffffffff815d439a)

6 drops at unix_dgram_sendmsg+459 (0xffffffff815d3c09)

1 drops at unix_release_sock+188 (0xffffffff815d2a18)

1 drops at __udp4_lib_mcast_deliver+290 (0xffffffff81597890)

53 drops at udp_queue_rcv_skb+3b4 (0xffffffff815973b4)

1 drops at ip_error+68 (0xffffffff81560fe8)

1 drops at ip_error+68 (0xffffffff81560fe8)

1 drops at ip_error+68 (0xffffffff81560fe8)

2 drops at __udp4_lib_mcast_deliver+290 (0xffffffff81597890)

1 drops at __udp4_lib_mcast_deliver+290 (0xffffffff81597890)

1 drops at ip_error+68 (0xffffffff81560fe8)

77 drops at udp_queue_rcv_skb+3b4 (0xffffffff815973b4)

1 drops at ip_error+68 (0xffffffff81560fe8)

1 drops at ip_error+68 (0xffffffff81560fe8)

19 drops at udp_queue_rcv_skb+3b4 (0xffffffff815973b4)

1 drops at ip_error+68 (0xffffffff81560fe8)

1 drops at __udp4_lib_mcast_deliver+290 (0xffffffff81597890)

1 drops at ip_error+68 (0xffffffff81560fe8)

I see a AppShare issue only when there is a packet loss reported in the highlighted location (in Bold). Though the number of packet loss 

reported is huge, but from human perception the Appshare quality is not that bad, but definitely we see a freeze and catch up.

Not sure whether the single RTP packet (we limit one single RTP packet size to 1200 bytes or so) is fragmented into many packets and an 

issue in one or more fragmentation reported into so many packet loss - not clear.

Our System Kernel Settings for UDP:

net.core.rmem_default = 212992

net.core.rmem_max = 16777216

net.core.wmem_default = 212992

net.core.wmem_max = 16777216

net.ipv4.udp_mem = 187299       249735  374598

net.ipv4.udp_rmem_min = 4096

net.ipv4.udp_wmem_min = 4096

sunrpc.transports = udp 32768

sunrpc.udp_slot_table_entries = 16

Our RX ring is set to 512.

Are these kernel values are OK for real time media traffic burst like Video or AppShare? If not, could you please suggest the recommended values.

Also, if I change these values do I need to reboot the system?

TURN Server Configuration Clarification:

From the TURN server configuration file, the description is not complete for the following two configuration parameters. It's not clear 

what value is used when this parameters is commented.

# Max bytes-per-second bandwidth a TURN session is allowed to handle

# (input and output network streams are treated separately). Anything above

# that limit will be dropped or temporary suppressed (within

# the available buffer limits).

# This option can also be set through the database, for a particular realm.

#

#max-bps=0

#

# Maximum server capacity.

# Total bytes-per-second bandwidth the TURN server is allowed to allocate

# for the sessions, combined (input and output network streams are treated separately).

#

# bps-capacity=0

I am not suspecting the TURN server as such, but I may need to fine tune the kernel settings or so. Since I am not Linux kernel expert, 

it's becoming difficult to trouble shoot this issue, any hint/clue will will be very helpful.

-KMurali