Bundling the volume fails on AWS Instance Based AMI

[Kyle Gochenour]

Howdy,

We are trying to make an Instance Based (not EBS) AMI from the Ubuntu 12.04.3 base ami (ami-5ffaa636) and have an issue in Packer.

Here is the output:

==> web.ami: Uploading X509 Certificate...

==> web.ami: Bundling the volume...

    web.ami: bash: sudo ec2-bundle-vol: command not found

    web.ami:

    web.ami:

    web.ami:

==> web.ami: Volume bundling failed. Please see the output above for more

==> web.ami: details on what went wrong.

We saw the documentation mentioned that ec2-bundle-vol command may have issues with .pem files, which we are using. So we added this to our template:

"bundle_vol_command": "sudo ec2-bundle-vol --no-filter"

 Still returning the same as above.

Since we are using a different AMI that the one in the documentation(http://www.packer.io/docs/builders/amazon-instance.html), I went back and pointed to the AMI there (ami-d9d6a6b0), but still had the same error as above.

What am I missing?

Thanks.

[Kyle Gochenour]

Also, we are using packer 0.3.11.

[Mitchell Hashimoto]

Kyle,

Did you install the EC2 AMI tools? It looks like "ec2-bundle-vol" isn't found.

On Fri, Nov 8, 2013 at 11:09 AM, Kyle Gochenour <amazin...@gmail.com> wrote:
> Also, we are using packer 0.3.11.
>

> --
> You received this message because you are subscribed to the Google Groups
> "Packer" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to packer-tool...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.

[Kyle Gochenour]

Hey Mitchell,

We did not. We are working on a chef cookbook for that, but just wanted to test the Instance Builder in the meantime. We didn't see any mention of the AMI Tools in the documentation, and didn't have the issue in the EBS so we went on with build testing. We will hold off then till we get a chef cookbook up and running.

Thanks!

-Kyle

[Mitchell Hashimoto]

Kyle,

Sorry, I'll update the documentation to note this requirement.

[Kyle Gochenour]

Hey Mitchell,

We are still having an issue with this. We are using a packer recipe to install the EC2-ami-tools only (not the full AWS CLI Tools), but Packer is still returning the above error of the command not found.

For testing, I ran the same template to an EBS AMI then booted up an instance to check to see if the "ec2-bundle-vol" command was found. It was.

We ran an instance based builder using "packer build -debug" and we stopped the build after the error. We then tried to get into the temporary instance that was launched but because it uses that temporary security group and key we were not able to, nor could we find the key.

Comparing the chef output from the EBS and Instance builders, the output is the same and the chef install appears successful for the ami tools. So we would like to get into the temporary instance created by the instance builder to see if that command is actually installing and were wondering where the temporary key gets stored so we could connect to the instance. We want to determine if there is an issue with the chef cookbook on instance based (but note there is no issue on EBS based) or if we may have stumbled across a bug in Packer.

Thanks,

-Kyle

[Kyle Gochenour]

Howdy,

Just wanted to follow up to see if anyone may know more about this (see below for full explantation), but we are not having any luck still getting Instance Backed builds from Packer even with ec2-ami-tools installed. If anyone has gotten this working successfully or can shine some light on it, it would be much appreciated.

-Kyle

[Mitchell Hashimoto]

Kyle,

Hm, Packer runs this command by default to bundle the volume: https://github.com/mitchellh/packer/blob/5bec1d9ef2d9a63ae4d5c5c80ed0ea14d3819d2c/builder/amazon/instance/builder.go#L80-L90

With that command, it may be possible it isn't inheriting the SSH user's environmental variables. Maybe we should modify it to include the login shell flag...

Best,

Mitchell

[Kyle Gochenour]

Hey Mitchell,

Thanks for the reply.

Using that info I added this line to my template to use the environmental variables with -E, but still got the same error of command not found.

"bundle_vol_command": "sudo -E -n ec2-bundle-vol -k {{.KeyPath}} -u {{.AccountId}} -c {{.CertPath}} -r {{.Architecture}} -e {{.PrivatePath}} -d {{.Destination}} -p {{.Prefix}} --batch"

Is there a way to find the private key generated by Packer for setting up the AMI, as we would like to get in and double check that the ami-tools are installed. Everything is coming through OK for that install from Packer, and the chef install is working in all our other builds.

-Kyle 

[Mitchell Hashimoto]

Kyle,

Run the build with the "-debug" flag. It should drop the SSH key in your current directory after the step that creates the keypair.

Best,

Mitchell

[Kyle Gochenour]

Mitchell,

We got success, kinda. Armed with the SSH key we were able to troubleshoot what was going on.

We did this on the running instance to troubleshoot:

ubuntu@ip-10-180-138-109:~$ # Try to run the command

ubuntu@ip-10-180-138-109:~$ sudo -E -n ec2-bundle-vol

sudo: ec2-bundle-vol: command not found

ubuntu@ip-10-180-138-109:~$ # Find out where command is for ubuntu user

ubuntu@ip-10-180-138-109:~$ which ec2-bundle-vol

/usr/local/share/ec2_tools/bin/ec2-bundle-vol

ubuntu@ip-10-180-138-109:~$ # Look at ubuntu user's path

ubuntu@ip-10-180-138-109:~$ echo $PATH

/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/share/ec2_tools/bin

ubuntu@ip-10-180-138-109:~$ # Become root

ubuntu@ip-10-180-138-109:~$ sudo bash

ubuntu@ip-10-180-138-109:~$ # Try to run command as root without sudo

root@ip-10-180-138-109:~# ec2-bundle-vol

The program 'ec2-bundle-vol' is currently not installed.  You can install it by typing:

apt-get install ec2-ami-tools

You will have to enable the component called 'multiverse'

ubuntu@ip-10-180-138-109:~$ # Compare root's path

root@ip-10-180-138-109:~# echo $PATH

/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

From there we found we actually needed to drill down and specify where exactly the AMI Tools were located on the instance. They seem to be in the Ubuntu's users path but not the sudoer's path. So we manually specified it in the template.

We also set the environment variables to be used as well. We also noticed that it still wasn't working. Thinking that the command might be running in dash not bash we added in a "sh" tag.

These are our bundle and upload commands in the template:

"bundle_vol_command": "sudo sh -E -n /usr/local/share/ec2_tools/bin/ec2-bundle-vol -k {{.KeyPath}} -u {{.AccountId}} -c {{.CertPath}} -r {{.Architecture}} -e {{.PrivatePath}} -d {{.Destination}} -p {{.Prefix}} --batch"

"bundle_upload_command": "sudo sh -E -n /usr/local/share/ec2_tools/bin/ec2-upload-bundle -b {{.BucketName}} -m {{.ManifestPath}} -a {{.AccessKey}} -s {{.SecretKey}} -d {{.BundleDirectory}} --batch --retry"

The cookbook we are using for the AMI Tools install is here:

https://github.com/Lytro/aws_developer_tools

We tried writing our own, but ended up going with Lytro's as it fit our needs.

So with all of that working, we then hit another snag. Packer was now going through the BundleVolume and UploadBundle steps without errors.

But when we hit the RegisterAmi step it fails with this error:

==> web.ami: Error registering AMI: HTTP 404 (Not Found) response for URL http://s3.amazonaws.com:80/<our_s3_bucket_name>/image-1384869546.manifest.xml: check your manifest path is correct and in the correct region. (InvalidManifest)

We checked the S3 bucket and there were no uploaded bundle parts or a manifest there. We looked on the running temporary instance, it did not have any bundle part files either. It was like Packer did not do the bundle. But it also threw no errors either.

To test, we copied our bundle and upload commands from above and manually typed in the variables specified and ran them on the running temporary instance. The image was bundled into a bunch of part files with a manifest as expected. We then ran the upload command manually, and our S3 bucket filled with the part files as expected. We could then register the AMI manually by pointing to the manifest.

So something with Packer might be causing the bundling to not occur. Even though the logs said it did:

==> web.ami: Pausing after run of step 'StepUploadX509Cert'. Press any key to continue. 

==> web.ami: Bundling the volume...

==> web.ami: Pausing after run of step 'StepBundleVolume'. Press any key to continue. 

==> web.ami: Uploading the bundle... 

-Kyle 

[Kyle Gochenour]

Got it working.

So when we added the "sh" to try and drop to shell that was causing the task to run in Packer's eyes, but nothing was actually occurring. So we went back to the drawing board. Packer still would not run the command with the -E, fully. It would run it, but wouldnt grab the environmental variables from the ubuntu user. But if we got ourselves into the box and ran the same command it would still work. So to deal with that we re-wrote our bundle and upload commands to this:

"bundle_vol_command": "echo \"env EC2_HOME=/usr/local/share/ec2_tools/ /usr/local/share/ec2_tools/bin/ec2-bundle-vol -k {{.KeyPath}} -u {{.AccountId}} -c {{.CertPath}} -r {{.Architecture}} -e {{.PrivatePath}} -d {{.Destination}} -p {{.Prefix}} --batch --no-filter\"> /tmp/bundle_cmd;sudo -E sh /tmp/bundle_cmd",

 

"bundle_upload_command": "echo \"env EC2_HOME=/usr/local/share/ec2_tools/ /usr/local/share/ec2_tools/bin/ec2-upload-bundle -b {{.BucketName}} -m {{.ManifestPath}} -a {{.AccessKey}} -s {{.SecretKey}} -d {{.BundleDirectory}} --batch --retry\"> /tmp/upload_cmd;sudo -E sh /tmp/upload_cmd"

It's not pretty, but it is working successfully. We are finally building instance based AMIs with Packer. 

[Mitchell Hashimoto]

Kyle,

Super odd that it wouldn't see the `-E`. Well, if you figure out a less specific approach to it, I'd be happy to merge in some changes to make this easier.

Best,

Mitchell

--

[Glenn Poston]

I ran into the same issues and I found that editing the following line in /etc/sudoers solved the issue.

Defaults    secure_path = /sbin:/bin:/usr/sbin:/usr/bin

...changed to...

Defaults    secure_path = /sbin:/bin:/usr/sbin:/usr/bin:/opt/aws/bin/

credits go to http://unix.stackexchange.com/a/151188

[Sean Kane]

I have been struggling with a similar problem and used this solution, inspired by this thread, that feels a little cleaner:

"bundle_vol_command": "sudo ruby -I /usr/lib64/ruby/site_ruby /usr/lib/ruby/site_ruby/ec2/amitools/bundlevol.rb -k {{.KeyPath}} -u {{.AccountId}} -c {{.CertPath}} -r {{.Architecture}} -e {{.PrivatePath}} -d {{.Destination}} -p {{.Prefix}} --batch --no-filter",

Sean