What is the USB Drive? Is it USB 1? Or is the device attached as a USB 1 device?
Or have you got limited resources on the Guest that holds all USB ports and that is limiting the passthrough from one Guest to another?
I don't have a USB container, because they all only get attached to Dom0. After that, there is the transferr from there to other guests, but that's only images on USB devices.
I have never had any issues with speed doing it this way.
Having said that though, the USB VM option is not very good, and is FULL OF BUGS. It has been that way since they tried to implement it in 3.2.
I have not ever seen the bugs fixed.
Nor have many bugs I have ever reported that are CRITICAL bugs ever been fixed.
Qubes 4 + may improve though. We can only wait and see, since it's having massive updates and rewrites.
I look forward to not being able to use Qubes 4+.