Error waiting for AMI Copy: ResourceNotReady: exceeded wait attempts on packer v1.3 and v1.2.3

[John Roh]

This is so weird that it doesn't make any sense. Recently, I have added a new jenkins-slave machine to build AMIs on a different accounts that we just aws IAM role permissions so that it much easy to create and to copy AMIs on different regions.

Currently I have three other jenkins-slave running/creating/copying AMIs with custom KMS encrypted images to any regions I want to successfully. However, this new slave machine builds the AMI but it fails to copy so that there is image gets created at all. With the closed tickets below, I am having the same issues. 

https://github.com/hashicorp/packer/issues/6177

https://github.com/hashicorp/packer/issues/6569

Even though I have increased AWS_MAX_ATTEMPTS  values or not, it still doesn't work at all (either 150 or 3000, there is no differences). I have used packer v1.3.0-dev from the given at the ticket, there is still error coming out.

"provisioners": [

{

"type": "shell",

"inline": [

"#!/bin/sh -x",

"export AWS_POLL_DELAY_SECONDS=30",

"export AWS_MAX_ATTEMPTS=3000",

"export AWS_TIMEOUT_SECONDS=3000"

]

},

I have ran pip list that I have confirmed all four jenkin-slave machines got configured as below.

ansible (2.4.1.0)

aws-cfn-bootstrap (1.4)

awscli (1.16.3)

Babel (0.9.6)

backports.ssl-match-hostname (3.5.0.1)

boto3 (1.7.71)

botocore (1.10.84)

certifi (2018.4.16)

cffi (1.6.0)

cfn-tools (0.1.6)

chardet (3.0.4)

Cheetah (2.4.4)

click (6.7)

cloud-init (0.7.9)

colorama (0.3.9)

configobj (4.7.2)

configshell-fb (1.1.23)

cryptography (1.7.2)

decorator (3.4.0)

docutils (0.14)

ecdsa (0.13)

enum34 (1.0.4)

ethtool (0.8)

futures (3.2.0)

httplib2 (0.9.2)

idna (2.7)

iniparse (0.4)

iotop (0.6)

ipaddress (1.0.16)

IPy (0.75)

javapackages (1.0.0)

Jinja2 (2.7.2)

jmespath (0.9.3)

jsonpatch (1.2)

jsonpointer (1.9)

kitchen (1.1.1)

kmod (0.1)

langtable (0.0.31)

lockfile (0.12.2)

lxml (3.2.1)

Markdown (2.4.1)

MarkupSafe (0.11)

paramiko (1.16.1)

passlib (1.6.5)

pcp (1.2)

perf (0.1)

Pillow (2.0.0)

pip (9.0.1)

ply (3.4)

policycoreutils-default-encoding (0.1)

prettytable (0.7.2)

pyasn1 (0.4.4)

pycparser (2.14)

pycrypto (2.6.1)

pycurl (7.19.0)

Pygments (1.4)

pygobject (3.22.0)

pygpgme (0.3)

pyliblzma (0.5.3)

pyparsing (1.5.6)

pyserial (2.6)

pystache (0.5.4)

python-augeas (0.5.0)

python-daemon (2.1.2)

python-dateutil (2.7.3)

python-dmidecode (3.10.13)

python-keyczar (0.71rc0)

python-linux-procfs (0.4.9)

pyudev (0.21.0)

pyxattr (0.5.1)

PyYAML (3.13)

requests (2.19.1)

rsa (3.4.2)

rtslib-fb (2.1.63)

s3transfer (0.1.13)

schedutils (0.4)

seobject (0.1)

sepolicy (1.1)

setuptools (40.0.0)

six (1.11.0)

slip (0.4.0)

slip.dbus (0.4.0)

system-storage-manager (0.4)

targetcli-fb (2.1.fb46)

urlgrabber (3.10)

urllib3 (1.23)

urwid (1.1.1)

wheel (0.31.1)

yum-langpacks (0.4.2)

yum-metadata-parser (1.1.4)

Have you ran into this error and know how to make it working?

Thanks,

John.

[Rickard von Essen]

It would be very helpful if you could try with an older version of Packer, say 1.2.0. And report if it works there. 

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/mitchellh/packer/issues
IRC: #packer-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Packer" group.
To unsubscribe from this group and stop receiving emails from it, send an email to packer-tool...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/packer-tool/c8aaf5e9-d4ff-4ac7-9cf7-e695954cc694%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Megan Marsh]

I can try to look into this today but it'll be easier if I have debug logs to look at.

[John Roh]

Hi Magan, 

I have updated the ticket at https://github.com/hashicorp/packer/issues/6177 with the link with logs and template.

Thanks for your attention.

John.

On Thu, Aug 30, 2018 at 9:47 AM Megan Marsh <megan....@gmail.com> wrote:

I can try to look into this today but it'll be easier if I have debug logs to look at.

--

This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/mitchellh/packer/issues
IRC: #packer-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Packer" group.
To unsubscribe from this group and stop receiving emails from it, send an email to packer-tool...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/packer-tool/CAGPV1uqJu1vtG%2Bw-Ba%3DqZpVxsASFcum7W0nSr0qXy%3Dq4FBebrg%40mail.gmail.com.

[John Roh]

Hi Richard,

Okay, I will try with v1.2.0 and let you know. What drives me crazy is that packer v1.2.3 works fine in other jenkins-slave build instances (I run 4 different parallel builds in jenkins on 4 different jenkins slaves located in 4 different accounts). Simply I have added new jenkins slave machine in a new account where the problem occurs.  

I have even tried packer v1.3.0 attached at https://github.com/hashicorp/packer/issues/6177 and the debug log is located at https://github.com/WeekendsBull/packerbuild-error/tree/master/packer_debug for both centos and windows that the error occurs while it copies the encrypted AMIs.

These failed debug logs generated from the new jenkins slave and the successfully debug logs got generated from the other jenkins-slave machine. But it ran from the packer version (v1.2.3). 

So far I have tested out different packer version and the consistently spotted the errors while copying AMI from non encrypted to encrypted one fails for sure.  

Thanks,

John.

To view this discussion on the web visit https://groups.google.com/d/msgid/packer-tool/CALz9Rt-jWf1RaoVWPoNoo9Ye%3Dkd35wTszmq9XjV7prc%3DB5KgKw%40mail.gmail.com.

[Megan Marsh]

OH! Sorry I didn't notice this out sooner. You don't export those environment variables as part of the provisioner; you set them when Packer runs. For example:

export AWS_MAX_ATTEMPTS=300; packer build example_template.json

To view this discussion on the web visit https://groups.google.com/d/msgid/packer-tool/CAPxBRp7smPnmcte_DZ5WrkSXHot9rrYqH-N0yCAcDD6r_OP3Uw%40mail.gmail.com.

[John Roh]

I see. I will test further today and keep you posted, Megan. 

To view this discussion on the web visit https://groups.google.com/d/msgid/packer-tool/CAGPV1upO08z%3DyQie7FkcnQZXQaWuw4xNzjSrnSubwm7x96r4gg%40mail.gmail.com.

[John Roh]

Hi Megan,

I have tested out as below 

"AWS_POLL_DELAY_SECONDS=30 AWS_MAX_ATTEMPTS=300 AWS_TIMEOUT_SECONDS=3000 PACKER_LOG=1 packer build packer_template.json"

[Pipeline] [BETA: CentOS7-AWS-Int] }

[BETA: CentOS7-AWS-il2] 2018/08/30 13:46:51 [INFO] Packer version: 1.3.0-dev

[BETA: CentOS7-AWS-il2] 2018/08/30 13:46:51 Packer Target OS/Arch: linux amd64

[BETA: CentOS7-AWS-il2] 2018/08/30 13:46:51 Built with Go Version: go1.10.1

[BETA: CentOS7-AWS-il2] 2018/08/30 13:46:51 Detected home directory from env var: /home/jenkins-slave

 

[BETA: CentOS7-AWS-il2] 2018/08/30 13:58:35 ui: [1;32m==> aws-ext-il2: Creating the AMI: 000XXXX00000-centos7-v2.2.78-test[0m

[BETA: CentOS7-AWS-il2] [1;32m==> aws-ext-il2: Creating the AMI: 000XXXX00000-centos7-v2.2.78-test[0m

[BETA: CentOS7-AWS-il2] 2018/08/30 13:58:36 ui: [0;32m    aws-ext-il2: AMI: ami-0861a1dbd16be33c6[0m

[BETA: CentOS7-AWS-il2] [0;32m    aws-ext-il2: AMI: ami-0861a1dbd16be33c6[0m

[BETA: CentOS7-AWS-il2] 2018/08/30 13:58:36 ui: [1;32m==> aws-ext-il2: Waiting for AMI to become ready...[0m

[BETA: CentOS7-AWS-il2] [1;32m==> aws-ext-il2: Waiting for AMI to become ready...[0m

[BETA: CentOS7-AWS-il2] 2018/08/30 13:58:36 packer: 2018/08/30 13:58:36 No AWS timeout and polling overrides have been set. Packer will defalt to waiter-specific delays and timeouts. If you would like to customize the length of time between retries and max number of retries you may do so by setting the environment variables AWS_POLL_DELAY_SECONDS and AWS_MAX_ATTEMPTS to your desired values.

[BETA: CentOS7-AWS-il2] [1;32m==> aws-ext-il2: Creating Encrypted AMI Copy[0m

[BETA: CentOS7-AWS-il2] [1;32m==> aws-ext-il2: Copying AMI: us-west-2(ami-0861a1dbd16be33c6)[0m

[BETA: CentOS7-AWS-il2] 2018/08/30 14:01:22 ui: [1;32m==> aws-ext-il2: Creating Encrypted AMI Copy[0m

[BETA: CentOS7-AWS-il2] 2018/08/30 14:01:22 ui: [1;32m==> aws-ext-il2: Copying AMI: us-west-2(ami-0861a1dbd16be33c6)[0m

[BETA: CentOS7-AWS-il2] 2018/08/30 14:01:22 ui: [1;32m==> aws-ext-il2: Encrypting with KMS Key ID: arn:aws:kms:us-west-2:xxxxxxxxx:key/xxxxx-a7ed-xxxx-b8cd-xxxxxxx[0m

[BETA: CentOS7-AWS-il2] [1;32m==> aws-ext-il2: Encrypting with KMS Key ID: arn:aws:kms:us-west-2:xxxxxxx:key/xxxxx-a7ed-xxxx-b8cd-xxxxxxx[0m

[BETA: CentOS7-AWS-il2] [1;32m==> aws-ext-il2: Waiting for AMI copy to become ready...[0m

[BETA: CentOS7-AWS-il2] 2018/08/30 14:01:22 ui: [1;32m==> aws-ext-il2: Waiting for AMI copy to become ready...[0m

[BETA: CentOS7-AWS-il2] 2018/08/30 14:01:22 packer: 2018/08/30 14:01:22 No AWS timeout and polling overrides have been set. Packer will defalt to waiter-specific delays and timeouts. If you would like to customize the length of time between retries and max number of retries you may do so by setting the environment variables AWS_POLL_DELAY_SECONDS and AWS_MAX_ATTEMPTS to your desired values.

[BETA: CentOS7-AWS-il2] 2018/08/30 14:01:38 ui error: [1;31m==> aws-ext-il2: Error waiting for AMI Copy: ResourceNotReady: failed waiting for successful resource state[0m

[BETA: CentOS7-AWS-il2] [1;31m==> aws-ext-il2: Error waiting for AMI Copy: ResourceNotReady: failed waiting for successful resource state[0m

[BETA: CentOS7-AWS-il2] 2018/08/30 14:01:38 ui: [1;32m==> aws-ext-il2: Deregistering the AMI because cancellation or error...[0m

[BETA: CentOS7-AWS-il2] [1;32m==> aws-ext-il2: Deregistering the AMI because cancellation or error...[0m

[BETA: CentOS7-AWS-il2] [1;32m==> aws-ext-il2: Terminating the source AWS instance...[0m

[BETA: CentOS7-AWS-il2] 2018/08/30 14:01:38 ui: [1;32m==> aws-ext-il2: Terminating the source AWS instance...[0m

[BETA: CentOS7-AWS-il2] 2018/08/30 14:01:38 packer: 2018/08/30 14:01:38 No AWS timeout and polling overrides have been set. Packer will defalt to waiter-specific delays and timeouts. If you would like to customize the length of time between retries and max number of retries you may do so by setting the environment variables AWS_POLL_DELAY_SECONDS and AWS_MAX_ATTEMPTS to your desired values.

[BETA: CentOS7-AWS-il2] [1;32m==> aws-ext-il2: Cleaning up any extra volumes...[0m

[BETA: CentOS7-AWS-il2] 2018/08/30 14:01:53 ui: [1;32m==> aws-ext-il2: Cleaning up any extra volumes...[0m

[BETA: CentOS7-AWS-il2] 2018/08/30 14:01:53 ui: [1;32m==> aws-ext-il2: No volumes to clean up, skipping[0m

[BETA: CentOS7-AWS-il2] [1;32m==> aws-ext-il2: No volumes to clean up, skipping[0m

[BETA: CentOS7-AWS-il2] 2018/08/30 14:01:53 ui: [1;32m==> aws-ext-il2: Deleting temporary security group...[0m

[BETA: CentOS7-AWS-il2] [1;32m==> aws-ext-il2: Deleting temporary security group...[0m

[BETA: CentOS7-AWS-il2] 2018/08/30 14:01:53 ui: [1;32m==> aws-ext-il2: Deleting temporary keypair...[0m

[BETA: CentOS7-AWS-il2] [1;32m==> aws-ext-il2: Deleting temporary keypair...[0m

[BETA: CentOS7-AWS-il2] 2018/08/30 14:01:53 [INFO] (telemetry) ending amazon-ebs

[BETA: CentOS7-AWS-il2] 2018/08/30 14:01:53 ui error: [1;31mBuild 'aws-ext-il2' errored: Error waiting for AMI Copy: ResourceNotReady: failed waiting for successful resource state[0m

[BETA: CentOS7-AWS-il2] 2018/08/30 14:01:53 Builds completed. Waiting on interrupt barrier...

[BETA: CentOS7-AWS-il2] 2018/08/30 14:01:53 machine readable: error-count []string{"1"}

[BETA: CentOS7-AWS-il2] 2018/08/30 14:01:53 ui error: 

[BETA: CentOS7-AWS-il2] ==> Some builds didn't complete successfully and had errors:

[BETA: CentOS7-AWS-il2] 2018/08/30 14:01:53 machine readable: aws-ext-il2,error []string{"Error waiting for AMI Copy: ResourceNotReady: failed waiting for successful resource state"}

[BETA: CentOS7-AWS-il2] 2018/08/30 14:01:53 ui error: --> aws-ext-il2: Error waiting for AMI Copy: ResourceNotReady: failed waiting for successful resource state

[BETA: CentOS7-AWS-il2] 2018/08/30 14:01:53 ui: 

[BETA: CentOS7-AWS-il2] ==> Builds finished but no artifacts were created.

[BETA: CentOS7-AWS-il2] [1;31mBuild 'aws-ext-il2' errored: Error waiting for AMI Copy: ResourceNotReady: failed waiting for successful resource state[0m

[BETA: CentOS7-AWS-il2] 

[BETA: CentOS7-AWS-il2] ==> Some builds didn't complete successfully and had errors:

[BETA: CentOS7-AWS-il2] --> aws-ext-il2: Error waiting for AMI Copy: ResourceNotReady: failed waiting for successful resource state

[BETA: CentOS7-AWS-il2] 

[BETA: CentOS7-AWS-il2] ==> Builds finished but no artifacts were created.

[BETA: CentOS7-AWS-il2] 2018/08/30 14:01:53 [INFO] (telemetry) Finalizing.

[BETA: CentOS7-AWS-il2] 2018/08/30 14:01:54 waiting for all plugin processes to complete...

[BETA: CentOS7-AWS-il2] 2018/08/30 14:01:54 /usr/bin/packer: plugin process exited

[BETA: CentOS7-AWS-il2] 2018/08/30 14:01:54 /usr/bin/packer: plugin process exited

[BETA: CentOS7-AWS-il2] 2018/08/30 14:01:54 /usr/bin/packer: plugin process exited

[BETA: CentOS7-AWS-il2] 2018/08/30 14:01:54 /usr/bin/packer: plugin process exited

On Thu, Aug 30, 2018 at 12:45 PM Megan Marsh <megan....@gmail.com> wrote:

To view this discussion on the web visit https://groups.google.com/d/msgid/packer-tool/CAGPV1upO08z%3DyQie7FkcnQZXQaWuw4xNzjSrnSubwm7x96r4gg%40mail.gmail.com.

[Megan Marsh]

This is really strange, and I can't reproduce what I'm seeing in your logs. How old is your 1.3.0-dev version? It may have a bug in it that's since been resolved. Can you try a more recent build?  I can't attach one to this email because it is too big but I will upload a recent build to https://github.com/hashicorp/packer/issues/6569.

Specifically, I can't get this log message when running the above line of code you're sharing: 

No AWS timeout and polling overrides have been set. Packer will defalt to waiter-specific delays and timeouts. If you would like to customize the length of time between retries and max number of retries you may do so by setting the environment variables AWS_POLL_DELAY_SECONDS and AWS_MAX_ATTEMPTS to your desired values.

Instead, I get the correct log message:

WARNING: AWS_MAX_ATTEMPTS and AWS_TIMEOUT_SECONDS are both set. Packer will be using AWS_MAX_ATTEMPTS and discarding AWS_TIMEOUT_SECONDS. If you have not set AWS_POLL_DELAY_SECONDS, Packer will default to a 2 second poll delay.

On Thu, Aug 30, 2018 at 2:32 PM Megan Marsh <megan....@gmail.com> wrote:

This is really strange, and I can't reproduce what I'm seeing in your logs. How old is your 1.3.0-dev version? I'm attaching a linux build of today's Master branch because it looks like you're running on CentOS; let me know if the problem persists with this build.

Specifically, I can't get this log message when running the above line of code you're sharing: 

No AWS timeout and polling overrides have been set. Packer will defalt to waiter-specific delays and timeouts. If you would like to customize the length of time between retries and max number of retries you may do so by setting the environment variables AWS_POLL_DELAY_SECONDS and AWS_MAX_ATTEMPTS to your desired values.

Instead, I get the correct log message:

WARNING: AWS_MAX_ATTEMPTS and AWS_TIMEOUT_SECONDS are both set. Packer will be using AWS_MAX_ATTEMPTS and discarding AWS_TIMEOUT_SECONDS. If you have not set AWS_POLL_DELAY_SECONDS, Packer will default to a 2 second poll delay.

[John Roh]

So far, I haven't found out the version it works yet. I'm going to try with v1.3.0 I got from Megan.

https://releases.hashicorp.com/packer/1.2.0/packer_1.2.0_linux_amd64.zip

ERROR! the playbook: packer_builder_type=amazon-ebs could not be found

ERROR! the playbook: packer_builder_type=amazon-ebs could not be found

https://releases.hashicorp.com/packer/1.2.1/packer_1.2.1_linux_amd64.zip

ui error: --> aws-ext-il2: Error waiting for AMI Copy: unexpected state 'failed', wanted target 'available'

https://releases.hashicorp.com/packer/1.2.2/packer_1.2.2_linux_amd64.zip

errored: Error waiting for AMI Copy: unexpected state 'failed', wanted target 'available'

thanks,

John.

[Megan Marsh]

Sounds to me like this is a problem with your machine, not with Packer. It may be that the bandwidth from your new machine is limited or something.

On Thu, Aug 30, 2018 at 3:58 PM John Roh <johnr...@gmail.com> wrote:

Megn,

"Copy image failed with an internal error" is what I spotted on ec2 console.

 

I have re-run the build with AWS_POLL_DELAY_SECONDS=50 AWS_MAX_ATTEMPTS=500 PACKER_LOG=1 packer build but still not passing and stuck with the same error message.

I wonder what the issue is occurring all of sudden on new jenkins slave machine  that the other build machines are working fine with packer v1.2.3 but why not on this new machine which created from the puppetizing it.

John.

--

This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/mitchellh/packer/issues
IRC: #packer-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Packer" group.
To unsubscribe from this group and stop receiving emails from it, send an email to packer-tool...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/packer-tool/bf2dabb7-2275-475c-8c67-6c8d40d37ecb%40googlegroups.com.

[John Roh]

Thank you so much for looking into the issue and now things are working fine.

Have great long weekends, Megan and Richard.

John.