Ephemeral disks in BlockDeviceMapping()

[Victor Trac]

I've got an EBS-backed AMI that I want to use for different instance types. I want to be able to use ephemeral disks in my instances, but the problem is that the different instance types have varying numbers of ephemeral disks available (e.g. 1 for m1.smalls, 2 for m1.large, and 4 for m1.xlarge). Using the command line tool ec2-register, I can set up an AMI for each type using the '--block-device-mapping' flag with the ephemeral disks predefined - e.g.:

ec2-register --snapshot snap-abc123ab --kernel aki-2a42a043 --ramdisk ari-2c42a045 --description "my AMI with two ephemeral disks" --name my-AMI --architecture x86_64 --root-device-name /dev/sda1 -b '/dev/sdb1=ephemeral0' -b '/dev/sdc1=ephemeral1'

This works, but I don't want to have to maintain an AMI for each instance type if I want to maximize my ephemeral disk access. I tried setting this up programmatically during instance run, doing something like:

instanceType = 'm1.large'
instanceTypes = {"m1.small": {"/dev/sdb1": "ephemeral0"},
"m1.large": {"/dev/sdb1": "ephemeral0",
"/dev/sdc1": "ephemeral1"},
"m1.xlarge": {"/dev/sdb1": "ephemeral0",
"/dev/sdc1": "ephemeral1",
"/dev/sdd1": "ephemeral2",
"/dev/sde1": "ephemeral3"},
block_map = BlockDeviceMapping()
block_map.update(instanceTypes[instanceType])

Doing this, I run into an "'str' object has no attribute 'snapshot_id'" error because it seems like BlockDeviceMapping.build_list_params() assumes that each device listed is a an EBSBlockDeviceType object with the snapshot_id field. Am I understanding this correctly and this is just something that boto doesn't support at the moment?

Thanks,
Victor

[Mitchell Garnaat]

Hi Victor -

Just wanted to give a quick reply to say that I'm not ignoring you.  It's just that the EC2 API has become so fricking complicated that trying to chase down a question like this requires a substantial amount of time.  I should be able to spend some time on it later today.  It's important that it be possible to do what you are trying to do in boto so if there is a problem I'm eager to get it fixed.

Mitch

--
You received this message because you are subscribed to the Google Groups "boto-users" group.
To post to this group, send email to boto-...@googlegroups.com.
To unsubscribe from this group, send email to boto-users+...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/boto-users?hl=en.

[Mitchell Garnaat]

Hi Victor -

I have checked in a change to ec2/blockdevicemapping.py that should allow you to achieve what you want.  However, each entry within the BlockDeviceMapping object must be of type EBSBlockDeviceType.  What I have done is add a virtual_name attribute to that type so you can do something like this:

In [7]: from boto.ec2.blockdevicemapping import EBSBlockDeviceType, BlockDeviceMapping

In [8]: map = BlockDeviceMapping()

In [9]: t = EBSBlockDeviceType()

In [10]: t.virtual_name = 'ephemeral1'

In [11]: map['/dev/sdc'] = t

In [12]: img.run(key_name='cloudright', instance_type='m1.large', block_device_map=map)

Out[12]: Reservation:r-c85523a0

So, in this case "img" is an EBS-based AMI and I have created a BlockDeviceMapping object and added an EBSBlockDeviceType to it that maps the "ephemeral1" local block device to the device name "/dev/sdc".  I verified that once the instance starts up, the /dev/sdc device is available for use:

root@domU-12-31-39-06-B8-11:~# fdisk -l

Disk /dev/sda1: 10.7 GB, 10737418240 bytes

255 heads, 63 sectors/track, 1305 cylinders

Units = cylinders of 16065 * 512 = 8225280 bytes

Disk identifier: 0x00000000

Disk /dev/sda1 doesn't contain a valid partition table

Disk /dev/sdc: 450.9 GB, 450934865920 bytes

255 heads, 63 sectors/track, 54823 cylinders

Units = cylinders of 16065 * 512 = 8225280 bytes

Disk identifier: 0x00000000

Disk /dev/sdc doesn't contain a valid partition table

root@domU-12-31-39-06-B8-11:~# logout

Does that allow you to do what you want to do?

Mitch

On Tue, Feb 9, 2010 at 3:05 PM, Victor Trac <victo...@gmail.com> wrote:

[Victor Trac]

Hey Mitch,

That works, but IMO it's a bit confusing to anyone newly looking at using the blockdevicemapping module.  Would it be beneficial to rename the EBSBlockDeviceType class to BlockDeviceType to make it more generic and exposing a "ephemeral_name" field instead of "virtual_name" ?  This would make boto field names more aligned with the official AMI API command line tools, which uses just "ephemeral[0-3]".  If you agree, I've attached a patch that works:

import boto

ec2conn = boto.connect_ec2()

from boto.ec2.blockdevicemapping import BlockDeviceType, BlockDeviceMapping

map = BlockDeviceMapping()

sdb1 = BlockDeviceType()

sdc1 = BlockDeviceType()

sdd1 = BlockDeviceType()

sde1 = BlockDeviceType()

sdb1.ephemeral_name = 'ephemeral0'

sdc1.ephemeral_name = 'ephemeral1'

sdd1.ephemeral_name = 'ephemeral2'

sde1.ephemeral_name = 'ephemeral3'

map['/dev/sdb1'] = sdb1

map['/dev/sdc1'] = sdc1

map['/dev/sdd1'] = sdd1

map['/dev/sde1'] = sde1

img = ec2conn.get_all_images(image_ids=['ami-f61dfd9f'])[0]

img.run(key_name='id_bv-keypair', instance_type='c1.xlarge', block_device_map=map)

Something else I also noticed is that you can apply a block device mapping using all 4 possible ephemeral drives, and depending on the instance type, they just show up if they exist, so one doesn't need to pre-create a mapping of EC2 instance types to number of included ephemeral disks.

In any case, your changes work great.  Thanks for the quick response.  Boto rocks. :)

-- Victor

[Mitchell Garnaat]

Yes, I agree that BlockDeviceType is a better name than EBSBlockDeviceType.  I'll make that change today.

It's less clear on the ephemeral_name/virtual_name.  The API actually uses virtual_name but I also think ephemeral_name name makes more sense, especially since the values are of the form "ephemeral[0-4]".  I usually try to stick close to the API type names but in this case I think it makes sense to diverge.

Thanks for the patch and for bringing the issue to light.

Mitch

[Mitchell Garnaat]

I just checked in a slight variation of your patch.  Could you check it out when you get a chance and let me know if you run into any problems?

Thanks,

Mitch

[Victor Trac]

Mitch - latest version works great.  Thanks!

-- Victor