New Releases of Ubuntu Images for Amazon EC2 (Kernels, Security, Tools, PPA, runurl)

311 views
Skip to first unread message

Eric Hammond

unread,
Oct 12, 2009, 9:47:31 AM10/12/09
to ec2ubuntu
New updates have been released for the Ubuntu and Debian AMIs (EC2
images) published on:

http://alestic.com

Please read the details here:

http://alestic.com/2009/10/ec2-ubuntu-ami-release-kernel

Enjoy,
--
Eric Hammond

Dmitriy

unread,
Oct 12, 2009, 4:14:55 PM10/12/09
to ec2ubuntu
Hi Eric,

On Oct 12, 8:47 am, Eric Hammond wrote:
>
>  http://alestic.com/2009/10/ec2-ubuntu-ami-release-kernel
>

Thanks for putting these together.

I just noticed that kernel module files under /lib/modules/*/kernel
are owned by uid=1000/gid=1000.

The workaround is to do "chown -R root.root /lib/modules" on boot.

ami-ccf615a5

- Dmitriy

Eric Hammond

unread,
Oct 12, 2009, 8:07:30 PM10/12/09
to ec2ubuntu
Dmitriy:

Thanks for finding and reporting this. I've fixed the build script so
the next release will have correct ownership.

The cause of this bug was the steps I took to reduce the size of the
Amazon provided kernel modules from 135 MB down to 14 MB. I had
tested modprobe of various kernel modules, which worked, but I didn't
review file ownership.

Other than the potential security risk of the first created user being
able to modify kernel modules, are there any additional problems
caused by this?

Does anybody need a new cut of the AMIs right away or might this wait
for a future update?

--
Eric Hammond

Eric Hammond

unread,
Oct 14, 2009, 6:14:42 AM10/14/09
to ec2ubuntu
In the 20091011 batch of AMIs published on Alestic.com I switched to
Amazon's kernel modules instead of building my own from source.

llinder noticed that these new AMIs are missing the "fuse" kernel
module:
http://alestic.com/2009/10/ec2-ubuntu-ami-release-kernel#comment-122

This module is pretty important in an AWS environment (e.g., s3fs) so
I'll be compiling the fuse kernel module from source and releasing new
AMIs when I get a chance.

--
Eric Hammond

dtornkaew

unread,
Oct 13, 2009, 11:32:41 PM10/13/09
to ec2ubuntu
Hey Eric,

Actually I posted this as a comment on your blog but probably would
have been better to post here.

I noticed the fuse module doesn't exist anymore. In the previous
Jaunty 32bit AMI fuse worked fine after running modprobe fuse. Is
there anyway to have this included in 32 and 64 bit versions again or
was this removed for specific reasons?

Thanks

Robert Coup

unread,
Oct 14, 2009, 7:57:35 PM10/14/09
to ec2u...@googlegroups.com
Hey folks,

On Wed, Oct 14, 2009 at 11:14 PM, Eric Hammond <eham...@thinksome.com> wrote:

In the 20091011 batch of AMIs published on Alestic.com I switched to
Amazon's kernel modules instead of building my own from source.


I'm getting kernel crashes with XFS on the newest Alestic Intrepid images:
  ami-c0f615a9
  32bit/m1.small instance
  default aki/ari, reported as 2.6.21.7-2.fc8xen-ec2-v1.0

Boot with the following user-data script:
  #!/bin/bash
  apt-get update
  apt-get install xfsprogs
  umount /mnt
  mkfs.xfs /dev/sda2 -f
  FSTAB_ENTRY='/dev/sda2 /mnt xfs defaults,relatime 0 0'
  perl -pi -e "s%^.* /mnt .*%${FSTAB_ENTRY}%" /etc/fstab
  mount /mnt
  touch /mnt/some-file
  reboot

kapow! kernel oops every time when the system starts up 2nd time (and subsequently)

I guess it's related to one of the other recent threads, but i thought they were all 2.6.18 related...

Console log (initial + reboot): http://pastebin.com/fc54044b

Should i be running a different aki/ari?

Rob :)

Eric Hammond

unread,
Oct 15, 2009, 4:21:35 AM10/15/09
to ec2u...@googlegroups.com
Robert:

Thanks for providing such an easy way to reproduce this problem.

In retrospect, I guess we should have seen this coming. With Amazon's
previous 2.6.21 kernel, the original xfs kernel module worked fine with
Hardy and Intrepid, but caused problems with Jaunty. Because of this,
Amazon build a special xfs.ko which I installed with Jaunty images.

With Amazon's newest 2.6.21 kernel and kernel modules, XFS seems to work
fine on Jaunty but breaks with Hardy and Intrepid (after reboot).

I've posted a request for help from Amazon:

http://developer.amazonwebservices.com/connect/thread.jspa?threadID=37436

--
Eric Hammond

Gorism

unread,
Oct 19, 2009, 5:46:44 PM10/19/09
to ec2ubuntu
I'm getting a crash with the base 32 bit image (ami-ccf615a5) by
simply:

1. Starting a fresh one
2. Either mounting an existing -or- creating a new XFS based volume
3. Unmounting the volume

At that point the instance is unresponsive and must be terminated. No
reboot required.

This doesn't occur with ext3 based volumes.

A kernel bug shows up in the console, same one as Robert provided
above.

Mark

------------[ cut here ]------------
kernel BUG at arch/i386/mm/hypervisor.c:185!
invalid opcode: 0000 [#1]
SMP
last sysfs file: /block/sda1/uevent
Modules linked in: ipv6(F)(U) xfs(F)(U) xennet(F)(U) xenblk(F)(U) ext3
(F)(U) jbd(F)(U) mbcache(F)(U)
CPU: 0
EIP: 0061:[<c101bf0e>] Tainted: GF VLI
EFLAGS: 00210282 (2.6.21.7-2.fc8xen-ec2-v1.0 #2)
EIP is at xen_pgd_pin+0x54/0x5e
eax: ffffffea ebx: c143eef8 ecx: 00000001 edx: 00000000
esi: 00007ff0 edi: 00000000 ebp: c19cef70 esp: c143eef8
ds: 007b es: 007b fs: 00d8 gs: 0033 ss: 0069
Process bash (pid: 1200, ti=c143e000 task=c140cef0 task.ti=c143e000)
Stack: 00000002 0020aa88 016f6000 0002dec0 00000000 c101816d c14b0480
c1018196
c15e31d4 c102250e c15e38d0 c143efb8 c106e111 c143efb8 bfd0a71c
01200011
00000000 c19cef70 c14b0480 c1346640 c15e31e0 c15e31f4 c15e31f0
c14b0480
Call Trace:
[<c101816d>] __pgd_pin+0x2f/0x3c
[<c1018196>] mm_pin+0x1c/0x23
[<c102250e>] copy_process+0xac3/0x10bc
[<c106e111>] kmem_cache_alloc+0x23/0x98
[<c1022b58>] do_fork+0x51/0x13a
[<c100321e>] sys_clone+0x36/0x3b
[<c1005688>] syscall_call+0x7/0xb
=======================
Code: eb fe a1 24 c2 38 c1 8b 14 90 81 e2 ff ff ff 7f 89 54 24 04 89
e3 b9 01 00 00 00 31 d2 be f0 7f 00 00 e8 36 54 fe ff 85 c0 79 04 <0f>
0b eb fe 83 c4 0c 5b 5e c3 56 89 c2 53 83 ec 0c c1 ea 0c 80
EIP: [<c101bf0e>] xen_pgd_pin+0x54/0x5e SS:ESP 0069:c143eef8
------------[ cut here ]------------
kernel BUG at arch/i386/mm/hypervisor.c:185!
invalid opcode: 0000 [#2]
SMP
last sysfs file: /block/sda1/uevent
Modules linked in: ipv6(F)(U) xfs(F)(U) xennet(F)(U) xenblk(F)(U) ext3
(F)(U) jbd(F)(U) mbcache(F)(U)
CPU: 0
EIP: 0061:[<c101bf0e>] Tainted: GF VLI
EFLAGS: 00210282 (2.6.21.7-2.fc8xen-ec2-v1.0 #2)
EIP is at xen_pgd_pin+0x54/0x5e
eax: ffffffea ebx: c1411ef8 ecx: 00000001 edx: 00000000
esi: 00007ff0 edi: 00000000 ebp: c140f490 esp: c1411ef8
ds: 007b es: 007b fs: 00d8 gs: 0033 ss: 0069
Process syslogd (pid: 1031, ti=c1411000 task=c140d450
task.ti=c1411000)
Stack: 00000002 002916f9 2d176000 005a2ec0 00000000 c101816d c14b0280
c1018196
c15f10cc c102250e c15f15b8 c1411fb8 c106e111 c1411fb8 bff296b4
01200011
00000000 c140f490 c14b0280 c14b0080 c15f10d8 c15f10ec c15f10e8
c14b0280
Call Trace:
[<c101816d>] __pgd_pin+0x2f/0x3c
[<c1018196>] mm_pin+0x1c/0x23
[<c102250e>] copy_process+0xac3/0x10bc
[<c106e111>] kmem_cache_alloc+0x23/0x98
[<c1022b58>] do_fork+0x51/0x13a
[<c10e48a8>] copy_to_user+0x3c/0x50
[<c100321e>] sys_clone+0x36/0x3b
[<c1005688>] syscall_call+0x7/0xb
=======================
Code: eb fe a1 24 c2 38 c1 8b 14 90 81 e2 ff ff ff 7f 89 54 24 04 89
e3 b9 01 00 00 00 31 d2 be f0 7f 00 00 e8 36 54 fe ff 85 c0 79 04 <0f>
0b eb fe 83 c4 0c 5b 5e c3 56 89 c2 53 83 ec 0c c1 ea 0c 80
EIP: [<c101bf0e>] xen_pgd_pin+0x54/0x5e SS:ESP 0069:c1411ef8

On Oct 15, 1:21 am, Eric Hammond <ehamm...@thinksome.com> wrote:
> Robert:
>
> Thanks for providing such an easy way to reproduce this problem.
>
> In retrospect, I guess we should have seen this coming.  With Amazon's
> previous 2.6.21 kernel, the original xfs kernel module worked fine with
> Hardy and Intrepid, but caused problems with Jaunty.  Because of this,
> Amazon build a special xfs.ko which I installed with Jaunty images.
>
> With Amazon's newest 2.6.21 kernel and kernel modules, XFS seems to work
> fine on Jaunty but breaks with Hardy and Intrepid (after reboot).
>
> I've posted a request for help from Amazon:
>
> http://developer.amazonwebservices.com/connect/thread.jspa?threadID=3...
>
> --
> Eric Hammond
>
>
>
> Robert Coup wrote:
> > Hey folks,
>

Gorism

unread,
Oct 19, 2009, 9:06:41 PM10/19/09
to ec2ubuntu
Quick follow up: going back to ami-ed46a784 (the 20090804 32 bit
Jaunty image from Eric) doesn't exhibit this behavior. XFS seems to
behave fine using that image.

Mark

Eric Hammond

unread,
Oct 19, 2009, 9:16:54 PM10/19/09
to ec2u...@googlegroups.com
Mark:

Yep, just be aware that reverting to the previous AMI downgrades the
kernel to one which includes the security hole where users on the system
can gain root.

At the moment we're stuck with two kernel choices from Amazon for
2.6.21, both of which have problems (XFS vs. security).

If your application has any chance of running on Karmic, then I
recommend trying out the Beta AMIs from Canonical which run on 2.6.31.

There has been a little movement on Hardy from Canonical, but I don't
expect a lot to get done there in the short term since Karmic is
launching in a couple weeks.

I have also heard some folks having success running the Alestic.com
Jaunty on the 2.6.27 kernel from Canonical, but that kernel was built a
while back and I don't know how reliable or secure it is.

--
Eric Hammond

Gorism

unread,
Oct 19, 2009, 9:39:17 PM10/19/09
to ec2ubuntu
Thank for the follow up, Eric.

I saw that you reached out to Amazon with a request for help. In that
request you indicated that XFS worked fine with Jaunty with the 2.6.21
kernel, which wasn't the case when I tried it. That's why I did the
follow up post; in case you weren't aware of that.

Sounds like you are aware of the problems with Jaunty + 2.6.21 + XFS,
though.

Have you heard anything back from Amazon regarding your suggestion to
compile a new xfs driver? I didn't see any replies to your original
post, so I'm guessing not.

Mark

Eric Hammond

unread,
Oct 20, 2009, 1:10:25 PM10/20/09
to ec2u...@googlegroups.com
Gorism:

Thanks for the reminder. I've posted an update on the forum thread.

Russ Ferriday

unread,
Oct 20, 2009, 7:40:29 PM10/20/09
to ec2u...@googlegroups.com
Thanks to you both for this thread, and related traffic.

I'm preparing to try the canonical karmic image -- canonical-testing-eu/karmic-i386-20090929.manifest.xml -- for use in EU-west, and I'm doing pre-tests on the AWS Management Console.

When I try to run image ami-404f6434 that requires kernel aki-6e4f641a, I get 

       "Nonexistent default AKI 'aki-6e4f641a' for AMI 'ami-404f6434'."

But running in us-east the counterpart ami-145ebe7d that uses aki-841efeed there is no problem.

Is this related to Eric's thread relating to ambiguity in selection of AKIs?:


And am I wasting my time trying bleeding-edge tests in the EU region? 

--r.


Russ Ferriday - Topia Systems - Open Source content management with Plone and Zope
ru...@topia.com - mobile: +1 805 234 6915 - skype: ferriday

Scott Moser

unread,
Oct 20, 2009, 8:07:31 PM10/20/09
to ec2u...@googlegroups.com
On Tue, 20 Oct 2009, Russ Ferriday wrote:

> Thanks to you both for this thread, and related traffic.
>
> I'm preparing to try the canonical karmic image -- canonical-testing-
> eu/karmic-i386-20090929.manifest.xml -- for use in EU-west, and I'm
> doing pre-tests on the AWS Management Console.
>
> When I try to run image ami-404f6434 that requires kernel
> aki-6e4f641a, I get
>
> "Nonexistent default AKI 'aki-6e4f641a' for AMI 'ami-404f6434'."
>
> But running in us-east the counterpart ami-145ebe7d that uses
> aki-841efeed there is no problem.

The above is an error, probably on my part. The error is correctly
stating that, while aki-6e4f641a is referenced by ami-404f6434, there is
no such registered kernel. I've removed the AMI in question to remove
any confusion. There will probably be further cleanup that needs to be
done at some point in the near future.

I've been getting automated publishing of our daily builds working over
the last 3 weeks or so.

We're fairly well set now. Automated daily build output is published to
'ubuntu-images-testing-us' and 'ubuntu-images-testing-eu' at the same
time. You can see builds named such as:
ubuntu-images-testing-eu/ubuntu-karmic-daily-i386-server-20091020.1.manifest.xml
ubuntu-images-testing-eu/ubuntu-karmic-daily-amd64-server-20091020.1.manifest.xml

There is more information on the publishing policy at
https://wiki.ubuntu.com/UEC/Images/NamingConvention . The above represent
our RC candidate builds. If successful testing, they'll be promoted this
thursday (10/22) to the 'ubuntu-images-milestone-eu' bucket as:
ubuntu-images-milestone-eu/ubuntu-karmic-daily-i386-server-20091020.1.manifest.xml
ubuntu-images-milestone-eu/ubuntu-karmic-daily-amd64-server-20091020.1.manifest.xml

On promotion, their AMI ids and name will change, but their content will
not. The new AMIs will be released as RC, and will not be removed. The
nightly AMI's further above *will* be removed.

> And am I wasting my time trying bleeding-edge tests in the EU region?

I surely hope not. I apologize for the snafu that you've seen and the
lack of ext4 support in the karmic kernel. I can only promise that we're
getting better, and the NamingConvention doc above should be consistent
from here out.

Also, please see https://wiki.ubuntu.com/UEC/Images/RefreshPolicy for
information about post-release refreshing of images.

Scott Moser

unread,
Oct 20, 2009, 8:42:06 PM10/20/09
to ec2u...@googlegroups.com
On Tue, 20 Oct 2009, Scott Moser wrote:
>
> I surely hope not. I apologize for the snafu that you've seen and the
> lack of ext4 support in the karmic kernel. I can only promise that we're
> getting better, and the NamingConvention doc above should be consistent
> from here out.

Regarding the ext4 support in karmic, that bug 428692 is set for 'karmic
updates' , so it is expected that at some point soon after release we will
get ext4 support in AMIs.
https://bugs.launchpad.net/ubuntu/+bug/428692

Russ Ferriday

unread,
Oct 20, 2009, 9:12:02 PM10/20/09
to ec2u...@googlegroups.com
Scott, thank you for your prompt response, and for clarifying your policies. 

I had just gone back to testing a Hardy version for use in eu, but will now give it another shot with one of the these ubuntu daily karmic builds. I'll check that I can get XFS working, ready for a switch to a milestone image.

Best wishes,

--r

devon

unread,
Dec 13, 2009, 12:00:55 AM12/13/09
to ec2ubuntu
greetings Eric

I'm trying to use s3fs on an "Amazon EC2 Ubuntu 9.04 jaunty" AMI
instance, and am running into the "FATAL: Module fuse not found"
problem.

I'm wondering what the latest word on this is -- is there indeed a
missing fuse.ko file? If so, can it be built and installed? If not,
what-to-do?

Thanks for any guidance,

d

Randy Harmon

unread,
Dec 15, 2009, 12:58:37 PM12/15/09
to ec2ubuntu
Nearly 2 months without clear closure on this topic is fairly
worrisome. Did I miss an update that actually resolves the reported
XFS-related kernel instability?

Randy

Eric Hammond

unread,
Dec 15, 2009, 1:14:27 PM12/15/09
to ec2ubuntu
Randy:

You have a couple options:

(1) Stick with the 20090804 AMIs in the "alestic" buckets (not
directly listed on http://alestic.com). These work well with XFS but
the kernel has a security hole which allows normal users to become
root.

(2) Switch to the Ubuntu Karmic and Hardy AMIs published by
Canonical. These use official Ubuntu kernels which have no bug
reports against them on EC2.

I recommend (2) since those AMIs are likely to be better supported
going forward (including by me).

Advanced users may wish to try "(3) Run Alestic AMIs with the
Canonical kernels" but that's not much different from (2) since the
Alestic AMIs are so close to the official Canonical AMIs except for
the kernels.

--
Eric Hammond

devon girard

unread,
Dec 15, 2009, 2:27:41 PM12/15/09
to ec2u...@googlegroups.com
eric, thanks for your thoughts,

perhaps unfortunately, I just upgraded my ubuntu server 9.04 instance to to 9.10 (via network upgrade a la http://www.ubuntu.com/getubuntu/upgrading )

of course I'm receiving the same error : when I run "# modprobe fuse" I get "# FATAL: Module fuse not found."


so it seems and in-place kernel upgrade is impossible on an ec2 instance? kind of annoying, yet good to know!

I will start from a fresh instance and see how things go,

Thanks again,

--
You received this message because you are subscribed to the "ec2ubuntu" Google Group:
 http://groups.google.com/group/ec2ubuntu
To unsubscribe, send email to ec2ubuntu-...@googlegroups.com

brian bulkowski

unread,
Dec 15, 2009, 3:01:16 PM12/15/09
to ec2u...@googlegroups.com
Devon, et al,

When launching in us-west I notice there's a spot in the instance launch where you can specify a different AKI. I don't remember seeing this field before yesterday, but it might allow trying a different AKI without rebundling - and if the launch works right, I would expect that bundling an AMI from that running instance would point by default at the AKI you intend.

Let us know if you try it out.....

-brian

Michael Wood

unread,
Dec 16, 2009, 9:18:29 AM12/16/09
to ec2u...@googlegroups.com
2009/12/15 brian bulkowski <br...@bulkowski.org>:
> Devon, et al,
>
> When launching in us-west I notice there's a spot in the instance launch
> where you can specify a different AKI. I don't remember seeing this field
> before yesterday, but it might allow trying a different AKI without
> rebundling - and if the launch works right, I would expect that bundling an
> AMI from that running instance would point by default at the AKI you intend.

This is more or less option 3 that Eric mentioned above. It was
available before the us-west zone was available. I've never tried it
before, though.

--
Michael Wood <esio...@gmail.com>

Michael Wood

unread,
Dec 16, 2009, 9:26:38 AM12/16/09
to ec2u...@googlegroups.com
2009/12/16 Michael Wood <esio...@gmail.com>:
Of course, this doesn't help with upgrading an existing instance.
Only when starting new instances, in which case you may as well start
with one of the Canonical instances that already have the correct
kernel/initrd.

--
Michael Wood <esio...@gmail.com>

Scott Moser

unread,
Dec 16, 2009, 12:02:27 PM12/16/09
to ec2u...@googlegroups.com
What *is* new, is that ec2-register will now take '--kernel' and
'--ramdisk'. Previously you could only supply ramdisk id and kernel id at
bundle-image time or launch time. Now, you can register a new image with
a different kernel/ramdisk as the "default".
Reply all
Reply to author
Forward
0 new messages