AMI copy to diff regions takes a lot of time, is this expected? is there a efficient/faster way?

[raje...@qubole.com]

we bake an ami in one region and we copy the ami to different regions as below.

For example 

1. Baking an ami in us-east-1 

2. Regions to copy images for amzn-hvm: ap-northeast-1,ap-northeast-2,ap-south-1,ap-southeast-1,eu-central-1,eu-west-1,sa-east-1,us-east-2,us-west-1,us-west-2,ca-central-1,eu-west-2

packer command to copy ami to different regions

['packer build --only=copier-hvm', '-machine-readable', '-var region=us-east-1', '-var hvm_src_ami=ami-XXXXX', 

'-var hvm_deeplearning_src_ami=None', '-var copy_regions=ap-northeast-1,ap-northeast-2,ap-south-1,ap-southeast-1,eu-central-1,eu-west-1,sa-east-1,us-east-2,us-west-1,us-west-2,ca-central-1,eu-west-2', 

'/media/ebs1/workspace/cluster-release-ami-copy/venv/lib/python2.7/site-packages/qweez/packer/cluster/aws/release/copy_config.json']

we are seeing that 

1. Baking ami in us-east-1 takes ~40 to 60 mins

2. Copy ami to diff regions as mentioned above takes ~1hr 15min to 2hr (the copy time keeps varying from 1hr to 2hr)

want to know if packer copies ami to different regions sequentially or it copies in parallel ?

Would like to know what are factors impacting which is causing the copy ami to take too long.

Want to know is there a efficient/faster way to nake and copy ami, i am basically looking to bring the overall time down.

Thanks!

Rajendra

[Rickard von Essen]

Packer copies AMI in parallel 1). This is actually just issuing one API call per region and then waiting for AWS to report that the AMI is available in all regions requested. There is nothing Packer can do to influence the speed.

I expect that this AMI is based on fairly large snapshots? Some thing you can investigate is if there are certain regions that take longer time to complete the copy. For example I would expect copying between the US regions are faster cross the oceans. I expect that the only drastic reduction of times requires that you build a smaller AMI. Experiment this the aws cli copy-image 2) and wait for the AMI's state to be "available" 3).

1) https://github.com/hashicorp/packer/blob/master/builder/amazon/common/step_ami_region_copy.go

2) https://docs.aws.amazon.com/cli/latest/reference/ec2/copy-image.html

3) https://docs.aws.amazon.com/cli/latest/reference/ec2/describe-images.html

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/mitchellh/packer/issues
IRC: #packer-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Packer" group.
To unsubscribe from this group and stop receiving emails from it, send an email to packer-tool...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/packer-tool/95002f27-d4b0-48ac-a77b-ece67bb248db%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.