Downloading images from Utah

306 views
Skip to first unread message

Leigh Stoller

unread,
Feb 16, 2016, 12:49:54 PM2/16/16
to emulab...@googlegroups.com
Just a heads up that getting standard (supported) images from Utah has 
changed quite a bit. In the past you would ftp them from the images-STD 
directory on our server. This is no longer the method to use, and in fact 
that directory is gone. The new method is to use “image_import” which will 
bring in an image via URL and add it to your testbed. The benefit of this 
method is that it is a lot simpler, and when we update the images in Utah, 
you can more easily update the image at your site. In order to use image import,
your site needs to be running the latest version of the emulab-stable otr
the emulab-devel repository. 

The basic command for importing a standard image is: 

        boss> wap image_import -g ‘https://...' 

And to update the image at some laster date: 

        boss> wap image_import -g -r emulab-ops,XXX 

See below for the images we support, but for a specific example, consider 
the UBUNTU14-64-STD image. 

First off, if you already have an old version of this image, that you 
imported from images-STD, be sure to go into the web interface and delete 
it. Do NOT purge the disk file, just in case you need back up, so uncheck 
the check box on the web page. 

Then do this (be sure to use quotes around the URL): 

        boss> wap image_import -g 'https://www.emulab.net/image_metadata.php?uuid=0a29c738-32b3-11e4-b30a-001143e453fe' 

This might take a while, depending on how fast your your connection is. 
Images tend to be on the order of 500MB to 1GB in size. 

Once you have the image downloaded, you will want to go back into the web 
interface and edit the node types for your site, to reset the default 
image. Note that you only need to do this when you import a brand new image. 

Now try swapping in an experiment with a single node, to confirm this new 
image works on your various node types. If it does, you are good to go.  If 
not, you might need to modify the image in place to make it work. Note that 
future updates from Utah will overwrite your modifications, but if its 
something simple like adding drivers, we are generally happy to add those 
to the standard images, just let us know. 

To update your image from Utah: 

        boss> wap image_import -g -r emulab-ops,UBUNTU14-64-STD 

If there is an update in Utah, the new version will be downloaded to your 
site. If there is no output, you are up to date. As mentioned above, if you 
have local changes to your image, you might want to contact us first to see 
if we can make this process easier for you. 

We currently have very good support for these images: 

UBUNTU16-64-STD : https://www.emulab.net/image_metadata.php?uuid=7683ca8e-5e37-11e6-ac8a-90e2ba22fee4
UBUNTU14-64-STD : https://www.emulab.net/image_metadata.php?uuid=0a29c738-32b3-11e4-b30a-001143e453fe 
CENTOS7-64-STD   : https://www.emulab.net/image_metadata.php?uuid=6fa68fd6-9163-11e6-ac8c-90e2ba22fee4
XEN46-64-STD        : https://www.emulab.net/image_metadata.php?uuid=61799d52-4dff-11e6-ac8a-90e2ba22fee4

More details can be found in the Emulab installation documentation at:

Leigh 


Bouabid Amine

unread,
Feb 21, 2016, 7:38:35 AM2/21/16
to emulab-admins
Hi,

Thanks a lot for these useful information

By the way, could you please give us also the URL of XEN44-64-STD

Another question, are all these images tested under Xen hypervisor (especially Freebsd 10.2) ?

Thanks again

-- 
Dr. Mohamed El Amine BOUABID
Senior Researcher (Class B)
Networking R&D Division
CERIST, Algiers, Algeria
Phone: +213 21 91 62 05
Mobile: +213 (0) 661 925 677
E-Mail: bou...@dtri.cerist.dz

Leigh Stoller

unread,
Feb 22, 2016, 10:26:17 AM2/22/16
to emulab...@googlegroups.com
> Thanks a lot for these useful information
>
> By the way, could you please give us also the URL of XEN44-64-STD
>
> Another question, are all these images tested under Xen hypervisor (especially Freebsd 10.2) ?

Hi, have you installed the XEN subsystem on boss already? If not, you need
to do that as well. Are you running from the latest version of
emulab-stable or emulab-devel?

Thanks!
Leigh




Bouabid Amine

unread,
Feb 22, 2016, 10:44:27 AM2/22/16
to emulab...@googlegroups.com
Hi, it's already installed (i am using emulab-devel) and the version of
xen image i have is Xen-43-64-STD

Leigh Stoller

unread,
Feb 22, 2016, 11:34:59 AM2/22/16
to emulab...@googlegroups.com
> Hi, it's already installed (i am using emulab-devel) and the version of
> xen image i have is Xen-43-64-STD

Ah, that is good. So first thing to do is:

boss> wap image_import -g 'https://www.emulab.net/image_metadata.php?uuid=6f5e573e-83f3-11e5-bf03-38eaa71273fa'

Then you want to change your default XEN parent:

boss> wap setsitevar general/default_xen_parentosid emulab-ops,XEN44-64-STD

And lastly, you want to change your existing images such that they run on
the new XEN image:

boss> wap runsonxen -a -p emulab-ops,XEN44-64-STD

Leigh




Bouabid Amine

unread,
Feb 23, 2016, 9:28:34 AM2/23/16
to emulab...@googlegroups.com
Hi,

Thanks, it's working very well :)

regards

Chuck Cranor

unread,
Feb 23, 2016, 9:43:10 AM2/23/16
to emulab...@googlegroups.com
On Tue, Feb 16, 2016 at 09:49:50AM -0800, Leigh Stoller wrote:
> Just a heads up that getting standard (supported) images from Utah has
> changed quite a bit. In the past you would ftp them from the images-STD
> directory on our server. This is no longer the method to use, and in fact
> that directory is gone. The new method is to use ?image_import? which will
> bring in an image via URL and add it to your testbed. The benefit of this
> method is that it is a lot simpler, and when we update the images in Utah,
> you can more easily update the image at your site.


Questions/feedback:

- how do I get a list of available images? I used to be able
to go to http://www.emulab.net/downloads/images-STD/ to get a
list of images.

- can I specify images by name instead of long random hex string?

- what flag will download the image NDZ and corresponding XML file
without installing it? we have scripts that customize the STD
images before installing that i would want to run them through.

chuck

Leigh Stoller

unread,
Feb 23, 2016, 11:01:18 AM2/23/16
to emulab...@googlegroups.com
> - how do I get a list of available images? I used to be able
> to go to http://www.emulab.net/downloads/images-STD/ to get a
> list of images.

Hi. That is our current list of supported, updated images. The URL
refers to whatever the latest version of the image is, so as we update it
in Utah, you can update it locally with the command I provided.

Any other images in images-STD were totally out of date and not
very safe to use. Let us know if there is an old image you need access to
and we will send you a URL privately.

> - can I specify images by name instead of long random hex string?

Sorry, we do not support that at this time.

> - what flag will download the image NDZ and corresponding XML file
> without installing it? we have scripts that customize the STD
> images before installing that i would want to run them through.

There is a -i option to image_import that you can use to change the local
name of the image. You can do your cusomizations on that and then move it
into place?

You can also use the URL to download the metadata, and inside is the image
file URL, which should give you the functionality you have now? Let us know
if not.

Leigh




Chuck Cranor

unread,
Jun 10, 2016, 12:38:57 PM6/10/16
to emulab...@googlegroups.com
On Tue, Feb 23, 2016 at 08:01:13AM -0800, Leigh Stoller wrote:
> > - how do I get a list of available images? I used to be able
> > to go to http://www.emulab.net/downloads/images-STD/ to get a
> > list of images.
>
> Hi. That is our current list of supported, updated images.


Where is the official place to find the list of available
www.emulab.net URLs that can be used with scripts like "image_import"?

google gives me:
https://wiki.emulab.net/wiki/ImageImport
https://wiki.emulab.net/wiki/NewStuff/Images
but those point to http://www.emulab.net/downloads/images-STD/

how do i discover the available image uuids that you serve?



> You can also use the URL to download the metadata, and inside is the image
> file URL, which should give you the functionality you have now? Let us know
> if not.

that's what I ended up doing and it was fairly easy, but there is a
problem with the metadata from image_metadata.php: "load_descriptors"
complains "pid: Required value not provided"

chuck@boss:/tmp/i % wap /usr/testbed/sbin/load-descriptors FBSD.xml
pid: Required value not provided

Am I supposed to manually add in the pid:

<attribute name="pid"><value>emulab-ops</value></attribute>

or should it come with the MD from image_metadata.php?


chuck

Chuck Cranor

unread,
Jun 10, 2016, 12:56:40 PM6/10/16
to emulab...@googlegroups.com
hi-

should linux_mod/ipod.c compile on ubuntu16? it doesn't seem to.
is there a newer version floating around?

chuck

chuck@h0:/tmp/linux_mod % make
make -C /lib/modules/4.4.0-22-generic/build M=/tmp/linux_mod modules
make[1]: Entering directory '/usr/src/linux-headers-4.4.0-22-generic'
CC [M] /tmp/linux_mod/ipod.o
/tmp/linux_mod/ipod.c:127:13: warning: initialization from incompatible pointer type [-Wincompatible-pointer-types]
.hook = ipod_hook_fn,
^
/tmp/linux_mod/ipod.c:127:13: note: (near initialization for 'ipod_hook_ops.hoo')
/tmp/linux_mod/ipod.c:128:5: error: unknown field 'owner' specified in initializer
.owner = THIS_MODULE,
^
In file included from include/linux/linkage.h:6:0,
from include/linux/kernel.h:6,
from include/linux/list.h:8,
from include/linux/module.h:9,
from /tmp/linux_mod/ipod.c:24:
include/linux/export.h:34:21: warning: initialization from incompatible pointer type [-Wincompatible-pointer-types]
#define THIS_MODULE (&__this_module)
^
/tmp/linux_mod/ipod.c:128:14: note: in expansion of macro 'THIS_MODULE'
.owner = THIS_MODULE,
^
include/linux/export.h:34:21: note: (near initialization for 'ipod_hook_ops.dev')
#define THIS_MODULE (&__this_module)
^
/tmp/linux_mod/ipod.c:128:14: note: in expansion of macro 'THIS_MODULE'
.owner = THIS_MODULE,
^
scripts/Makefile.build:264: recipe for target '/tmp/linux_mod/ipod.o' failed
make[2]: *** [/tmp/linux_mod/ipod.o] Error 1
Makefile:1396: recipe for target '_module_/tmp/linux_mod' failed
make[1]: *** [_module_/tmp/linux_mod] Error 2
make[1]: Leaving directory '/usr/src/linux-headers-4.4.0-22-generic'
Makefile:30: recipe for target 'all' failed
make: *** [all] Error 2
chuck@h0:/tmp/linux_mod %

David M. Johnson

unread,
Jun 10, 2016, 3:53:07 PM6/10/16
to emulab...@googlegroups.com
Sorry, we haven't built our Ubuntu 16 standard image yet (although we're
actively discussing it!). Because of that, ipod hasn't been ported
forward yet (I assume what you are seeing below is a slight change in
the netfilter hook mechanism, which has happened before). It will be a
few weeks at least before we have a beta image; but hopefully less than
two weeks on the updated ipod module. If you haven't heard from me by
then, feel free to ping again!

David

David M. Johnson

unread,
Aug 9, 2016, 11:05:55 AM8/9/16
to emulab...@googlegroups.com
Chuck, I've updated the ipod module; should work fine through 4.4.x as
of emulab-devel f214a3ec .

David
Reply all
Reply to author
Forward
0 new messages