Diagnosing a problem like this is just about impossible because there are so many variables. There's the speed of your computer (mostly data transfer rate of your target hard drive), the speed of your Internet connection, the speed of the server or servers, traffic on the servers, so many things. And conditions are not necessarily the same from one minute, one hour, one day to the next. If you happen to be doing multiple concurrent downloads from a given server, it might impose a ceiling on the cumulative bandwidth it grants you across all your downloads from that site. Your ISP might impose such a ceiling as well. Then there's anything connected with Google. They mercilessly throttle downloads, specifically but not only from YouTube.
There is also an issue with VDH. But this does not occur during downloads but rather when they complete. However, that occurs on Windows, where you say you have not observed the problem.
Zombie downloads in VDH are not unheard of. Not frequent, just occasional. You have to keep an eye on the Network tab in Resource Monitor to make sure a download is still running or has stopped. Or whatever tool is appropriate on your platform. Don't rely exclusively on VDH's blue dot status menu. Use the blue dot but don't take it as gospel.
You are probably not going to get much meaningful help here, I'm afraid.