hello
packer can create a VB in a Virtualbox and in VMWARE ESXI directly
the idea behind packer is to buikd VM/AMI/Droplet/docker/etc that has been provisioned in the same way
so they look/behabe/have the same packages, etc.
You can have 1 packer template with an specific provisioning step for virtualbox (to install vbox tools) and for the vmware other for the vmware tools.
that is the best packer way.
You could create a virtualbox image, use then some tools like ovftool to deploy the OVA/OVF into ESXI/vcenter but not sure if the best for the long run in case there are issues.
You can use this repo as example/starting point.
Alvaro.