you can spin a VM manually, ssh into as you would do with packer and run either the script or commands.
as long there is internet connectivity and the access is in place the clone should work.
Having a zip or any other intermediate step feels like a workaround more than a fix and you are creating state, ie a zip file that may not be always up to date.
Learn what are the requirements to clone by an script and make that part of your build pipeline
if you access your repo over http, then check the format and user/pass needed
can be something lile https://user:password@server/path
if you clone as ssh:// or git:// then a private key may be needed.
Spin a VM manually get the clone working by command line then work in to have this scripted.
Packer in this case is just creating the VM and running your script.
Alvaro.