The only thing I can think of is so much trouble, I wouldn't do it. You'd have to define multiple Firefox profiles, but install VDH in only one of them. Avoid that profile most of the time. For most of your web surfing, use only profiles in which you have not installed VDH. When you find a page where you want to download something, load it using your profile that has VDH.
Instead of the profile trick, there is also the VDH Reload extension function. You can find out about that over here:
Do a string search on "reload" in that thread. You'll find a link that explains, complete with screenshots, how to reload VDH. Reloading VDH clears out all the resources it is using. It empties the VDH menu in every active tab. It also closes & relaunches the CoApp. But beware. Don't just use it any old time. If you have any inflight downloads going on in VDH, reloading the extension will abruptly terminate them all & you'll have to start them over from scratch. But if you aren't downloading anything, doing a Reload extension will clear out all those resources you're concerned about. After you do the reload, VDH will of course start incrementally using up resources again. So you'll have to do a reload every every once in a while. But at least it will keep a lid on things.
Best I got.