Why it is slower:
1) to create individual PDFs, each element that is needed in the document (eg a logo that is the same for every recipient) has to be embedded into each PDF, but when creating a single PDF for multiple recipients, the logo needs to be added only once, and then referenced in the other locations.
2) XMPie also employs some proprietary tricks when processing large recipient lists. When processing a PDF for each recipient, the system will (if processing an InDesign document rather than XLIM) will use InDesign's built-in PDF export process.
You are right to consider server specs as one method of improving performance, but unfortunatly, I have no experience to help you with recommendations. Since they are virtual servers, perhaps it would be easy to increase one parameter, test, revert it back and increase another parameter, test, etc, and report back your findings for posterity? If you decide to do this, after making your configuration changes and rebooting, I would recommend processing a couple jobs first before recording your test times. This will ensure that the production instances are up and running and you don't get times skewed by having to launch some applications.
Other suggestions:
1) try XLIM document production.
2) try other output formats. Perhaps using PostScript output and an Acrobat Distiller hot folder (or some other utility) may reduce total times. (Personally, I doubt this one, but it is worth testing.)
3) there are other PDF utilities that you may be able to test - eg process the full output in uProduce, and pass the PDF to another tool that splits it off into 1 page (or 2 page if double-sided) files.
4) when performance is an issue, always look at the document design and look at ways to optimize the design for speed.