I have one theory. The first time you run virtualbox the first it loads kernel modules (vbox*). Perhaps they’re trying to allocate large contiguous or wired memory regions for doing virtualbox stuff and triggering a bit of a fragmentation clear-up (and allocation errors). If you quit virtualbox the modules will remain resident in kernel memory so the second time around all is present and it doesn't have to shuffle stuff around. I think it also creates stuff in /dev that might lead to a some kind of virtual race condition. Also, if you're using the ZFS ARC, the first time around stuff won't be cached but it might be on the second attempt.
You could test this crazy theory by using kldload to bring in the modules one at a time (vboxdrv and vboxnet, and possibly vboxnetflt vboxnetadp), wait for things to settle and then run virtualbox.
Beware - I'm not a virtualbox expert but I've seen stuff like this with virtualisation all over the place - which is why I stick to jails :-) If you're luck an actual expert may be along later.
Regards, Frank.