I don't see why not. I think there are some patches for X to be applied - so specific a specific version would be required. You can specify all of this through ebuilds/overlays or the various configuration files for portage.
The issue with Gentoo is constant compiling. If you're going to build a template might I suggest that you consider including distcc -although this may not be ideal from a security standpoint.
I was considering building a hardened template, and with HVM being default in v4 then a fully hardened kernel is a real possibility. There is a precompiled distro called lilblue that's build from the hardened project. It's maintained by one of hardened project devs, and it's done most of the legwork getting X to work with grsecurity/pax. It could be trimmed for the hardware requirements with a standard VM.