Mid-call - too many packets in UDP receive queue
Alex Balashov
Revisiting an old issue: the UDP packet queue overrun message, on mr11.5.1.3-1 --
[core] Too many packets in UDP receive queue (more than 50), aborting loop. Dropped packets possible
I am rather confounded about why this is happening in calls such as the one whose call log is attached. This seems to happen infrequently, but when it does, it leads to an audio drop (callee->caller in this case) in one stream direction. The fact that it's only an occasional occurrence also seems to rule out a forwarding loop; there are no indications of that, not in Kamailio config logic, nor in packet forwarding counters or interface bandwidth stats, etc.
Unfortunately, I don't have the NG control messages for this interaction, though perhaps they should be collected for the future.
My main points of perplexion are:
1) If there is a UDP receive queue overrun, this suggests that the stream is forwarded in userspace, right? There is no UDP receive queue as such inside the netfilter UDP forwarding environment, surely? Or do I fail to misunderstand some aspect of the interaction between the daemon and netfilter?
2) Yet, both stream directions are clearly kernelised above:
Mar 27 16:13:14 mobile-rtc-gw rtpengine[3946133]: INFO: [260bd9483ba8fdc0...@45.32.129.87/as30caf8ca/1 port 58124]: [core] Kernelizing media stream: 45.32.129.82:16834 -> 45.32.129.870:58124 | 45.32.129.870:10822 -> 108.45.60.245:62422
Mar 27 16:13:14 mobile-rtc-gw rtpengine[3946133]: INFO: [260bd9483ba8fdc0...@45.32.129.87/8Stek3W/1 port 10822]: [core] Kernelizing media stream: 108.45.60.245:62422 -> 45.32.129.870:10822 | 45.32.129.870:58124 -> 45.32.129.82:16834
3) This doesn't happen following a renegotiation or any other disruptive event, but rather, 38 minutes after kernelisation, so it cannot be attributed to a packet overrun into the userspace daemon while it is still in its few-seconds' learning phase.
It also precedes any subsequent renegotiation by 7 minutes, so it is truly an event "out of nowhere", at least as far as the scope of this call is concerned.
Where, therefore, might one start with trying to hunt down the cause of this menace? :-)
Many thanks in advance!
-- Alex
Richard Fuchs
1) If there is a UDP receive queue overrun, this suggests that the stream is forwarded in userspace, right? There is no UDP receive queue as such inside the netfilter UDP forwarding environment, surely? Or do I fail to misunderstand some aspect of the interaction between the daemon and netfilter?
You're quite right, this would only happen for user space forwarding. So with a stream that is supposed to be handled by the kernel module, this can either indicate that kernel forwarding doesn't work at all, or that the module encountered some packets or traffic that it refused to handle and so it passed them on to user space.
With 11.5 I believe the counters in /proc/rtpengine/X/list are still indicative of whether the module does any work.
Packets that the kernel module refuses to handle may include STUN or ICE packets, other packets that are not RTP if RTP is expected, SRTP packets that failed decryption, RTP packets with unknown SSRCs, and possibly others.
Assuming you're unable to do large scale tracing (i.e. tcpdump on everything, or running rtpengine under strace) to catch the offending traffic, you can try adding a simple log message that dumps the contents of the packet that triggered the overflow, and then go from there.
Cheers