To specify the port when using gcloud compute, you will have to add the --ssh-flag to the command line. With this you can specify all the arguments that can be used with SSH. For example to connect on port 4561, the command would be:
gcloud compute ssh my_instance --zone us-central1-a --ssh-flag="-p 4561"
Make sure you have restarted the ssh service with: sudo service ssh restart
Also keep in mind you will have to add a firewall rule on your network in the Developers Console to allow port 4561.
I hope this helps.