For those who are interested, the process that receives data handles both UDP and TCP connections. When I looked this morning, the UDP listening socket was no longer open in the process -- which implies that something must have closed it (it is opened during process startup as fd 4, and TCP listening socket is fd 5). The code is written in perl and uses IO::Async::Socket and IO::Async::Stream to handle the actual connections so the app doesn't even know about the underlying details. It is hard to see how it could close the wrong socket. I'm wondering if the number of TCP sockets is overflowing some fixed size internal table (I hope not). It has gone past 1k active TCP connections...
I've added code to try and detect if this happens again and restart the process.....