Another update, and found the cause of the problem.
After digging into the code, packer sends keystrokes to virtualbox guests by using the "VBoxManage controlvm ... keyboardputscancode" CLI tool, which on my system here takes about 400ms to run (but i've seen it run much faster, but unsure under what conditions). Each keypress sends two scancodes, a down, and an up. (plus two more if shift has to be pressed). Packer only passes a single scan-code to VBoxManage, even though the tool can take multiples, and thus each character takes about 800ms (or longer). If the Mac suddenly gets busy, it can take longer and thus accidentally activate the guest's keyboard repeat.
Its unfortunate that the CLI tool is the only way to send keystrokes into the image. However, packer exacerbates the issue by sending one scan-code at a time. Ideally, it should send them in batches where possible. A <wait> in the boot_command is handled by packer itself, so obviously that'd be a place where a batch is broken up. However, this would remove the ability for packer to insert small delays, such as a 0.1 second delay as I've seen in the vmware builder. But I think this problem, here, is causing more issues. (At least it is for me! :) )
Anyone have comments before I raise this as a ticket/issue?