Hello, there is no such thing like wrong approach, and exists many ways to install a new server, so go crazy and share what you do.
If you use virtualbox, you should be able to not install vbox additions, and then generate a .img with a dump of the filesystem.. like dd or dumpfs
then on the physical server, you could dd back or importfs, it should work.
Other way is look into mondo/mindi that is a backup solution, one of the cool features is that can backup to a nfs folder, generate an ISO, then you boot from that ISO and you do the restore. this also should work.
other way would be: If you are happy with a build from packer, you may look into building a custom iso installer..
packer take a boot iso, preseed, a scripts (shell, puppet, etc)
you can just have the logic into a custom build installer cd without packer at all.
if you udpate the scripts, just rebuild the iso and that ist.
if your physical hardware doesn't have remote console/remote cd/dvd, you can have a dvd-rw
if you server is hp/dell/ibm and allow remote media, you can automate the whole process by scripts, and build the the server remotely
if your scripts are in a source control, point the iso to consume those scripts over the network, like raw in github, and your iso never would be updated.
apache webserver can serve content based on the source mac-address, you can use this to have an script and provide a generic script, or a custom script for a specific server, so you can have the same url, but different builds, this make the whole build way more cooler.
If you are open to use puppet or any other Configuration Manager, your initial build can be just small, and the take it from there bases on server name/ hw mac, etc.
Alvaro.