I wish I could tell you what we used at my last job, but IT set that up. All I know is that the one server had four VMWare VMs on it that we used (2 XP, 1 Vista, and 1 Win 7 64-bit). It might have had other VMs on it too, but for Selenium it seemed to run quite well. I don't believe it was really any slower than if I was running it on my desktop.
I don't use Selenium (yet) on my home system, but with Virtual Box I often have four VMs along with my native OS and it seems to me as if they all run fairly full speed. I had 2 dual core AMD chips running @ 2.8Ghz with 32GB of RAM. I typically only give a VM a single core and anywhere from 2 to 8GB of RAM, depending on how I use it.
I think you will find that as long as your VMs don't use up all available RAM and you don't have any more VM cores than you have physical cores you will get very good performance out of a system.
Mike