Packer build failure (base ami with cmk encrypted snapshots)

[Vivek]

My packer build keeps failing when using a base ami which has cmk encrypted snapshots. If launched from the console, the instance passes all status checks hence there doesn't seem to be a problem with the ami or snapshots. The error I am seeing is:

01:47:25  [1;32m==> amazon-ebs: Prevalidating AMI Name... [0m
01:47:26  [0;32m    amazon-ebs: Found Image ID: ami-01544b839262649a2 [0m
01:47:26  [1;32m==> amazon-ebs: Using existing SSH private key [0m
01:47:26  [1;32m==> amazon-ebs: Launching a source AWS instance... [0m
01:47:27  [0;32m    amazon-ebs: Instance ID: i-0570fffad0bc0f0a8 [0m
01:47:27  [1;32m==> amazon-ebs: Waiting for instance (i-0570fffad0bc0f0a8) to become ready... [0m
01:47:30  [1;31m==> amazon-ebs: Error waiting for instance (i-0570fffad0bc0f0a8) to become ready: unexpected state 'shutting-down', wanted target 'running' [0m
01:47:30  [1;32m==> amazon-ebs: Terminating the source AWS instance... [0m
01:47:38  [1;32m==> amazon-ebs: No volumes to clean up, skipping [0m
01:47:38  [1;31mBuild 'amazon-ebs' errored: Error waiting for instance (i-0570fffad0bc0f0a8) to become ready: unexpected state 'shutting-down', wanted target 'running' [0m
01:47:38 
01:47:38 ==> Some builds didn't complete successfully and had errors:
01:47:38 --> amazon-ebs: Error waiting for instance (i-0570fffad0bc0f0a8) to become ready: unexpected state 'shutting-down', wanted target 'running'
01:47:38 
01:47:38 ==> Builds finished but no artifacts were created.
01:47:38 Finished: FAILURE
01:47:38 Build step 'Packer' marked build as failure
01:47:38 Finished: FAILURE

I have tried adding in the launch_block_device_mappings and ami_block_device_mappings into my packer.json file but that does not seem to fix the error. Any help would be appreciated. Please let me know if the contents of packer.json would be helpful. 

[Rickard von Essen]

Can you share your template, source AMI-ID, and region you are using?

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/mitchellh/packer/issues
IRC: #packer-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Packer" group.
To unsubscribe from this group and stop receiving emails from it, send an email to packer-tool...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/packer-tool/e0364207-ad7f-493d-856e-dfe9920ab773%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Vivek]

sure. 

"builders": [{

    "type": "amazon-ebs",

    "region": "us-east-1",

    "source_ami": "ami-01544b839262649a2",

    "ami_virtualization_type": "hvm",

    "kms_key_id": "{{`keyid`}}",

    "instance_type": "m3.xlarge",

    "ssh_username": "ec2-user",

    "ami_name": "ooh-api-{{user `branch_name`}} {{isotime \"02-Jan-06 03-04-05 MST\"}}",

    "iam_instance_profile": "DOOH-CW",

    "launch_block_device_mappings": [

    {

      "device_name": "/dev/sdb",

      "snapshot_id": "snap-037ea359f8b20a1cf"

    },

    {

      "device_name": "/dev/sda1",

      "snapshot_id": "snap-0cb01242fb7d1c3e0"

    }]

[Rickard von Essen]

$ AWS_DEFAULT_REGION=us-east-1 aws ec2 describe-images --image-ids ami-01544b839262649a2

{

    "Images": []

}

This seems to be a private AMI, can you provide the output of the above command, using the same credentials as you run packer with.

--

This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/mitchellh/packer/issues
IRC: #packer-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Packer" group.
To unsubscribe from this group and stop receiving emails from it, send an email to packer-tool...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/packer-tool/dd7af0f1-a6fa-4a16-9707-a2116af2670f%40googlegroups.com.

[Vivek]

yes it is a private AMI

Output is:

{

    "Images": [

        {

            "VirtualizationType": "hvm",

            "Name": "ooh-test-encry-tomcat7",

            "Hypervisor": "xen",

            "EnaSupport": true,

            "SriovNetSupport": "simple",

            "ImageId": "ami-01544b839262649a2",

            "State": "available",

            "BlockDeviceMappings": [

                {

                    "DeviceName": "/dev/sda1",

                    "Ebs": {

                        "DeleteOnTermination": false,

                        "SnapshotId": "snap-0cb01242fb7d1c3e0",

                        "VolumeSize": 10,

                        "VolumeType": "gp2",

                        "Encrypted": true

                    }

                },

                {

                    "DeviceName": "/dev/sdb",

                    "Ebs": {

                        "DeleteOnTermination": false,

                        "SnapshotId": "snap-037ea359f8b20a1cf",

                        "VolumeSize": 10,

                        "VolumeType": "gp2",

                        "Encrypted": true

                    }

                }

            ],

            "Architecture": "x86_64",

            "RootDeviceType": "ebs",

            "RootDeviceName": "/dev/sda1",

            "CreationDate": "2018-10-22T19:58:27.000Z",

            "Public": false,

            "ImageType": "machine",

            "Description": "ooh-test-encry-tomcat7"

        }

    ]

}

[Rickard von Essen]

Didn't see anything obvious wrong but the launch_block_device_mappings section in your template is unnecessary, remove it and try to run with -on-error=ask. That will give you an opportunity to review the state of the failing instance.

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/mitchellh/packer/issues
IRC: #packer-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Packer" group.
To unsubscribe from this group and stop receiving emails from it, send an email to packer-tool...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/packer-tool/c7bc3fbb-62af-4395-8480-2eafe74e58b4%40googlegroups.com.

[Vivek]

Thanks i did that and it seems it fails on 'StepRunSourceInstance'. Any insight on how to fix this or what causes this?

16:17:18  [1;32m==> amazon-ebs: Pausing after run of step 'StepKeyPair'. Press enter to continue. [0m  [1;32m==> amazon-ebs: Pausing after run of step 'StepSecurityGroup'. Press enter to continue. [0m  [1;32m==> amazon-ebs: Pausing after run of step 'stepCleanupVolumes'. Press enter to continue. [0m  [1;32m==> amazon-ebs: Launching a source AWS instance... [0m
16:17:19  [0;32m    amazon-ebs: Instance ID: i-049c0ecd1153fc438 [0m
16:17:19  [1;32m==> amazon-ebs: Waiting for instance (i-049c0ecd1153fc438) to become ready... [0m
16:17:21  [1;31m==> amazon-ebs: Error waiting for instance (i-049c0ecd1153fc438) to become ready: unexpected state 'shutting-down', wanted target 'running' [0m
16:17:21  [1;32m==> amazon-ebs: Step "StepRunSourceInstance" failed [0m
16:17:21  [1;32m==> amazon-ebs: [c] Clean up and exit, [a] abort without cleanup, or [r] retry step (build may fail even if retry succeeds)? [0m  [1;32m==> amazon-ebs: Terminating the source AWS instance... [0m

[Rickard von Essen]

Capture the system console output it will probably show the cause, see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instance-console.html

You can send it here if you want advice.

--

This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/mitchellh/packer/issues
IRC: #packer-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Packer" group.
To unsubscribe from this group and stop receiving emails from it, send an email to packer-tool...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/packer-tool/147d1f61-64f8-4122-bb51-1e24b2e0a7a8%40googlegroups.com.

[Vivek]

I tried this previously but nothing shows up in the system logs,. makes me wonder if ssh does not even work

[Rickard von Essen]

Ssh is not yet involved in anything. The way to interpret these log lines:

01:47:26  [1;32m==> amazon-ebs: Launching a source AWS instance... [0m

01:47:27  [0;32m    amazon-ebs: Instance ID: i-0570fffad0bc0f0a8 [0m
01:47:27  [1;32m==> amazon-ebs: Waiting for instance (i-0570fffad0bc0f0a8) to become ready... [0m
01:47:30  [1;31m==> amazon-ebs: Error waiting for instance (i-0570fffad0bc0f0a8) to become ready: unexpected state 'shutting-down', wanted target 'running' [0m

Is that first packer sends the run instance API call which returns an instance-id for the new instance. 

Packer then loops doing describe instance API calls waiting for instance state to become running, but instead it gets shutting-down which means that the instance automatically shut down. The reason for this is most likely available in the system console log from the machine. 

On Tue, Oct 23, 2018, 19:02 'Vivek' via Packer <packe...@googlegroups.com> wrote:

I tried this previously but nothing shows up in the system logs,. makes me wonder if ssh does not even work

--

This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/mitchellh/packer/issues
IRC: #packer-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Packer" group.
To unsubscribe from this group and stop receiving emails from it, send an email to packer-tool...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/packer-tool/23ac8e7e-8d08-4a85-84f3-9630d4d68730%40googlegroups.com.

[Vivek]

how would you proceed if nothing shows up in the system logs?

[Rickard von Essen]

Launch an EC2 with the cli using the exact same settings (this should have the exact same effect, but this will double check your settings).

Set delete on termination false on all ebs volume mappings and and mount the volume on another running ec2 after a failed launch and examine the content. Especially the log files. 

On Tue, Oct 23, 2018, 19:17 'Vivek' via Packer <packe...@googlegroups.com> wrote:

how would you proceed if nothing shows up in the system logs?

--

This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/mitchellh/packer/issues
IRC: #packer-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Packer" group.
To unsubscribe from this group and stop receiving emails from it, send an email to packer-tool...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/packer-tool/7174b573-92c0-4dad-b68c-62accbda1c59%40googlegroups.com.

[Vivek]

will do that. thanks for your help.

[John Roh]

I have similar error all of sudden after we have successful release about a month ago. 

From CloudTrail, I have found out IAM permission required for KMS encryption and which is kms:GenerateDataKeyWithoutPlainText. 

Hope that it helps. 

John.

On Tue, Oct 23, 2018 at 10:28 AM 'Vivek' via Packer <packe...@googlegroups.com> wrote:

will do that. thanks for your help.

--

This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/mitchellh/packer/issues
IRC: #packer-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Packer" group.
To unsubscribe from this group and stop receiving emails from it, send an email to packer-tool...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/packer-tool/098c5d77-3482-446d-a36b-a9d79548dfbd%40googlegroups.com.