Sage virtual appliance (VirtualBox)

[Volker Braun]

I wrote a script to create a virtual machine with Sage and export it as an OVA appliance. I'm using VirtualBox but I think VMware should support it, too. At this point it would be useful if people could try it out on a variety of host machines. I built the following Sage appliance to test:

http://www.stp.dias.ie/~vbraun/Sage/Sage-4.7.rc2.ova

The virtual machine starts up in text mode and automatically runs the Sage notebook server. At first startup you are prompted to enter a password, as usual. Once you set up your admin password you have to use your ***host OS*** web browser to open http://localhost:8000. The guest Sage install does not have any GUI in the virtual machine, though you can of course use the Sage command line.

http://trac.sagemath.org/sage_trac/ticket/11330

[Kelvin Li]

On May 13, 8:15 am, Volker Braun <vbraun.n...@gmail.com> wrote:
> I wrote a script to create a virtual machine with Sage and export it as an
> OVA appliance. I'm using VirtualBox but I think VMware should support it,
> too. At this point it would be useful if people could try it out on a
> variety of host machines.

Functionally, it works. Sage starts up, I could connect to the
Notebook from my host's Firefox, do "2+2", and surf the documentation.
I didn't run doctests or stress test it in any way.

I have VirtualBox 4.0.6 (non-free extension pack not installed) on
Windows XP SP3. I did not touch the configuration when importing, so
the guest was set to use 2048 MB of memory, 2 CPUs, and was 64-bit.
The host has 4 GB of memory and an Intel Core 2 Duo.

Although nothing seems broken, I do have a few complaints. First, the
boot time is quite terrible. The time from my pressing "Start" on the
VirtualBox interface to seeing the login prompt was 1 minute 43
seconds. By comparison, my 32-bit Debian installation (in the same
VirtualBox) takes only 30 seconds; it is a rather minimal
installation.

Secondly, this Fedora installation seems rather bloated. When I
shutdown the guest, there are messages saying that the sshd and
sendmail daemons are stopping; but why should they be running in the
first place? I believe that is part of the reason the booting is so
slow.

Thirdly, could the space in the name of the OVA appliance (Sage
4.7.rc2) be removed? When importing, the VM is placed in a directory
called "Sage 4.7.rc2", so I end up with a space in the directory name.

I also tried VMware Player, but upon attempting to open the OVA
appliance, I got an error message: "Failed to open virtual machine:
Failed to query source for information." Same thing happened in VMware
Workstation.

After a brief Bing search, I could not figure out what these messages
meant.

I have only briefly glanced over the build script, but I think this is
a good first iteration!

-- Kelvin

[Dima Pasechnik]

probably VMWare requires a slightly different directory layout:
http://gvaro.wordpress.com/2010/05/07/vmware-player-error-“failed-to-open-virtual-machine-failed-to-query-source-for-information-”/

(don't know if Bing (ahem...) can find this link though :-))

[Volker Braun]

Thanks for the suggestions. I haven't tried to make a hand-crafted absolutely minimal Linux install because that would be a hassle to maintain. I'm only installing the Fedora "Core" packages with a handful of extra stuff (fortran, for example). That way upgrading the Guest OS should just work as easily as upgrading the Sage source release.

I agree that sendmail can be turned off. I like sshd, though, because its a nice way to interact with the virtual machine. You can scp files in/out, for example. By default the VM doesn't punch the ssh port through its NAT for security, but that can be easily done in the VirtualBox settings.

[Emil Widmann]

> Thanks for the suggestions. I haven't tried to make a hand-crafted
> absolutely minimal Linux install because that would be a hassle to maintain.

Having a automated script to produce virtual machine images is a huge
step in the right direction.
Personally I think size (1.4 GB) could be optimized but it was
pointed out in previous discussions that size is not a major concern
for mainstream development. On the other hand with virtual machines
resources of the host are split.
Maybe some ideas from the sage-lithe VM prototype (407 MB)
http://sage.math.washington.edu/home/emil/sagelithe/vm-sage-lithe-a1.zip
could be utilized. In itself this version is not optimized yet (e.g.
full xorg and xvesa)
regards
emil

[Kelvin Li]

> > > probably VMWare requires a slightly different directory layout:
> > > http://gvaro.wordpress.com/2010/05/07/vmware-player-error-“failed-to-open-
> > > virtual-machine-failed-to-query-source-for-information-”/
> > >
> > > (don't know if Bing (ahem...) can find this link though :-))

That was one of the first hits I got. :-P But it does not help me. In
this case, everything is contained in a single OVA file. Why would the
location of an imported applicance make a difference?

> > Thanks for the suggestions. I haven't tried to make a hand-crafted
> > absolutely minimal Linux install because that would be a hassle to maintain.

I wasn't saying that manually tweaking is better; that would defeat
the whole point of the "automated" script! I was wondering whether
Fedora allows you to (automatically) install an even more reduced base
system than what you have right now. Or maybe you can specify which
packages you want to exclude at install time.

> > I agree that sendmail can be turned off. I like sshd, though, because its a
> > nice way to interact with the virtual machine. You can scp files in/out, for
> > example. By default the VM doesn't punch the ssh port through its NAT for
> > security, but that can be easily done in the VirtualBox settings.

I think this is an interesting debate. I see the whole virtual machine
solution as primarily targeting Windows, and most Windows users do not
have ssh or even know about it. Not to say that Sage-on-Windows users
are dumb, but I think that if such a user knows enough about ssh to
want to use it, he/she can install it manually. On the other hand,
instructions on transferring files in/out of VMs using the pre-
installed sshd are more likely to confuse users than to help them.

I also figure that such motivated/knowledgeable users are more likely
to install and configure their own VMs and just build Sage from
source.

I would like to keep things lean because long boot times and gigantic
disk images (about 4.5 GB uncompressed, and the downloading took over
2 hours on my internet connection) could be annoying for users who are
trying to take the shortest possible route to getting Sage running on
Windows.

> Maybe some ideas from the sage-lithe VM prototype (407 MB)http://sage.math.washington.edu/home/emil/sagelithe/vm-sage-lithe-a1.zip

> could be utilized. In itself this version is not optimized yet (e.g.
> full xorg and xvesa)
> regards
> emil

Was this assembled by hand or by a script?

-- Kelvin

[Andrey Novoseltsev]

I have tried to run it on my "desktop notebook": Athlon 64 3000
(1.8GHz, 1 core), 2 Gb RAM (maximum possible), Windows XP Professional
32-bit. I was pretty sure it will not run given the guest memory size,
but I got the following, which does not refer to memory:
================
Failed to open a session for the virtual machine Sage 4.7.rc2

AMD-V is not available. (VERR_SVM_NO_SYM).

Unknown error creating VM (VERR_SVM_NO_SYM).
================
Then I changed guest RAM to 1Gb. It started and showed during boot
that it requires 64-bit and detected only 32.

I think that the machine should be configured to run "as is" on as
many computers, including old ones, as possible. Perhaps 1 core, 512Mb
RAM, 32-bit should be the default and installation instructions can
mention that you can easily adjust the first two parameters according
to your particular hardware, but it is optional. As for 32/64 I think
the best option is to have two images, since with automatic script it
should not be a problem to maintain both.

Thank you,
Andrey

[Volker Braun]

The OVA file format is just a tar archive of a 2-file OVF format. VMware might need you to place the contained files in the same directory. 

[vbraun@volker-desktop data]$ tar tf Sage-4.7.rc2.ova 

Sage-4.7.rc2.ovf

Sage-4.7.rc2-disk1.vmdk

The VM is always going to be painful to download with your modem. We can certainly hand-craft a Fedora install that is even smaller or manually delete/strip/dedup the Sage install. But then that'll just cause problems when we upgrade the base Fedora install or if Sage moves files around. Right now the compressed virtual machine is 1.4GB. Thats not excessive nowadays. The contained virtual hdd is actually 7GB.

[Volker Braun]

I don't think that running a virtual machine on yesteryear's hardware is a viable option. Yes you can make it boot but it'll be too slow for productive use. The virtual machine is really only useful on modern hardware with at least 4GB ram and hardware virtualization support (Intel VT-x or AMD-V). If you have the hardware support then you supposedly can run 64-bit guests on 32-bit hosts, though I haven't tried. 

[Andrey Novoseltsev]

What do you mean here by "productive use"? If you are computing
numerous high-dimensional Mori cones, you will probably work on some
Unix server anyway ;-) While if you are an undergrad with an oldish
Windows computer (especially in less financially fortunate countries),
plotting sines and experimenting with Newton's method for a cubic - I
don't think that there will be an issue with not having enough cores/
memory. Maybe 512Mb is too harsh and will interfere with normal
working of the guest operating system, but 1Gb is certainly enough (if
I recall correctly, student version of MAGMA is limited to 100Mb only)
and it is easy to increase if necessary. It is also just as easy to
decrease it, but the error message that I got didn't say anything
about memory and if it was my first encounter with virtual machines
(which is likely to be the case for many potential users), I wouldn't
know what to do. I am pretty sure that <=2Gb machines are still quite
common around the world and on a 2-core 4Gb 2009-laptop it still was
more comfortable for me to use 1 core for the guest system and 1.5Gb
memory.

On an unrelated note, the 2004-laptop (with an awesome screen) used in
the experiment is my primary machine for Sage development and
SageTeXing and it works great, although for these purposes I boot it
directly into Linux rather than messing with virtual machines. That's
the ultimate form of recycling - just continue using it since it is
not broken and works fine ;-) For CPU/memory/time demanding
computations there are perfect shared servers that beat any personal
setup!

[Kelvin Li]

On May 14, 2:07 pm, Volker Braun <vbraun.n...@gmail.com> wrote:
> The virtual machine is really only useful on modern hardware with at
> least 4GB ram and hardware virtualization support (Intel VT-x or AMD-V).

My personal experience differs. I have run windows VMware guests with
512 MB of memory on top of old hardware (no VT-x/AMD-V) that had only
1 GB of memory in total, and it suited my needs at the time for
experimenting with software.

I think it would be nice if our Sage VM appliance could run on a wide
variety of hardware, much as Sage itself does not require high-end
machines to work.

-- Kelvin

[Emil Widmann]

> I think it would be nice if our Sage VM appliance could run on a wide
> variety of hardware, much as Sage itself does not require high-end
> machines to work.
>
> -- Kelvin

Very true - if Sage is primarily a research tool for mathematicians
then one doesn't need to worry about ressources to much.
If there is a vision that sage could become a basic application for
teaching or as a math tool for science and engineering then it is a
different story. Download size and minimal specs become an issue.

Students might have "older" machines. Schools may have older machines
depending on their funding situation. Universities in poorer countries
may have older machines. You might have an older machine at home - why
dump it every 2 years? Run sage on your small netbook? - Most probably
there will be "limited" resources. I dare to say that lots of people
who try sage for the first time will try it on rather "underpowered"
machines. I don't need to stress the significance of first
impressions.

I think the approach to have automated VM generation via a script is
great and should be the priority. But efficient size (I avoid to say
small) could become an additional goal for the VM release.

>> Maybe some ideas from the sage-lithe VM prototype (407 MB)http://sage.math.washington.edu/home/emil/sagelithe/vm-sage-lithe-a1.zip
>> could be utilized. In itself this version is not optimized yet (e.g.
>> full xorg and xvesa)
>> regards
>> emil

> Was this assembled by hand or by a script?

This was purely by hand, because it was my first attempt to create a
working VM image using the live CD iso.
It is noteworthy that the unzipped image is same size than the zipped
(around 400 MB) - I zipped it only to have the same format as the
official sage VM images. This small size is possible because most is
contained in a squashed file system and therefore is already
compressed. The base OS including Java for 3D plotting, full xorg and
many applications (browser, abiword, gnumeric etc...) is around 190
MB, Sage was stripped down to 210 MB (robbing it of the development
capability - but I don't know if this is a necessary feature in a VM
image). The full sage was 400 MB (squashed) in release 4.6.

So I guess the following Limits for the VM image size would be a
challenge but are reachable:
1) minimal and stripped (application only) version < 300 MB
2) Full Sage with gcc and make tools for sage development capability <
800 MB
Of course built automated and reproducible, not as hand tuned
curiosity.

I don't know if it is useful for Fedora, but there is an Article about
using Squasfs to reduce space on the Sage Wikki
http://wiki.sagemath.org/UsingSquashFS

[Volker Braun]

Fair enough. I think 32-bit guests can work with software virtualization irregardless of the processor/host OS bitwidth, so thats probably safer. I made another VM with F14 i386 + Sage-4.6.2:

http://www.stp.dias.ie/~vbraun/Sage/Sage-4.6.2.ova

Volker

[Emil Widmann]

I tried to download it but I get "forbidden" permission conflict.
:-)

[Volker Braun]

Sorrry %-) fixed now.

[Stephen Nuchia]

I'm new to this group, have been working with Sage on Amazon "EC2"
cloud platforms. It costs several dollars to build Sage from source
there and I couldn't find a binary that would install on their current
recommended Amazon Linux starter image. It's a stripped-down CentOS-
derived thingy.

You can't build ATLAS on a "micro" instance because the effective CPU
speed varies wildly, so you can't use your free-for-a-year micro
instance to build it (not completely). It does run quite well in a
micro instance once built -- it just slows way down if you run more
than about 5 CPU-seconds of work at a time.

Seeking opinions from those interested in such things: I can put up a
pre-configured AMI (Amazon Machine Image) based on a good Amazon
starter image with Sage, LaTex, and other good bits installed. Doing
so will cost me a few buck up front and a few bucks a month, would it
be reasonable to post the image through the Amazon "Dev Pay" service
and charge a little bit for the use of it? Maye a $2.50 one-time fee?

Thanks!
-swn

[Volker Braun]

You might want to make your own thread ;-)

My new ATLAS spkg will be much friendlier to build. It can also more easily use the OS-provided ATLAS libraries. In particular, I'm using them all the time on Fedora. Amazon probably doesn't include the libraries in their base install, but you could probably just copy the RHEL/CentOS library.

See http://trac.sagemath.org/sage_trac/ticket/10226

[Emil Widmann]

I finally installed VirtualBox and tested the Sage-4.6.2.ova.
unfortunatly I get:
python:error while loading shared libraries: /home/sage/sage/local/lib/
libpython2.6.si.1.0: invalid ELF header.
after booting, so I can't start sage.

It took me a moment to figure out that one has to rename the *.ova
into a *.tar and then untar it to get the *.ovf file.
Also the Virtual machine needs 4GB on harddisk. Mine is chronically
full so I had to clean up a bit to get it running.
Initial startup needed > 5 minutes for me with my low spec desktop
(including the conversion from ovf to vdmk)

As comparison I run the official sage 4.6 vm image. I was able to run
it in Vbox, but couldn't get a network connection for the sage server
to the host. The same happened with my sage light vm image.

Howto run vmdk files in Vbox:
press New button -> Create New Virtual Machine Window opens
press Next button -> VM Name and OS Type Window opens
type in a Name, select Operating System: Linux, Version: Linux 2.6
press Next -> Memory Window opens, select 512 MB
press Next -> Virtual Hard Disk Window opens
check "Boot Hard Disk" and select "Use existing hard disk"
Select the vmdk file you want to run (e.g. sage-vmware.vmdk)
press Next -> Summary Window opens
press Finish -> your machine will appear in the left column of the
main Window
Click it to highlight it and then select "Start"

Maybe somebody can also give a clue how to fix the networking with the
host machine, so that you can access the sage server from the host
browser? Maybe somewhere in the settings of Vbox?

Some comparison:
all machines were run with 512 MB of Base memory and 1 processor

Sage 4.6.2:
Base Fedora
Download Size 1.4 GB (as ova)
vmdk size installed: 3.9 GB
Max Disk Size 7.81 GB
Boots into CLI desktop
Boot time: 1 min 25 sec
Shut down time: 13 sec

Sage 4.6 official VM image:
Base Ubuntu
Download Size 1 GB
vmdk size installed 4.5 GB
Max Disk Size 20 GB
Boots into GUI desktop
Boot time 1 min 10 sec
Shut down time: couldn't "Shutdown", had to power off (0 sec)

Sage 4.6 VM light (sage stripped)
Base Puppy
Download size 400 MB
vmdk size installed 478 MB
Max Disk Size 4GB
boots into GUI desktop
Boot time: ca 1 min 40 sec
Shut-down time: 15 sec

just my feedback :-)

[Volker Braun]

It sounds like you have an old version of VirtualBox installed. Which version are you using? I strongly suggest you upgrade. VirtualBox 4 has a GUI to configure the TCP/IP ports to connect between host and guest VM. Also, you can use the OVA simply via File->Import appliance. If you use my VM then you don't need to fiddle with the network setting as it has them already set.

[Emil Widmann]

On May 17, 10:49 am, Volker Braun <vbraun.n...@gmail.com> wrote:
> It sounds like you have an old version of VirtualBox installed. Which
> version are you using?

You were right I had only v 3.2.8 - I upgraded to 4.0.8 now!

I strongly suggest you upgrade. VirtualBox 4 has a
> GUI to configure the TCP/IP ports to connect between host and guest VM.
> Also, you can use the OVA simply via File->Import appliance.

That's great! It makes it easier for inexperienced people.

> If you use my
> VM then you don't need to fiddle with the network setting as it has them
> already set.

I am sorry but I still get this odd error about this invalid ELF
header.

On the other hand you were right, the new version has a nicer GUI for
the networking, I used "internal network" instead of the preconfigured
NAT, so I could connect to the sage server from the host with the
others.

[Emil Widmann]

with the command readelf -h libpython2.6.so.1.0
i get::
Error: Not and Elf file - it has the wrong magic byte at the start
Error: libpython2.6.so.1.0; Failed to read file header

maybe it is just some sort of corruption, so I download file again and
the import OVA as you suggested (will need 1 hour)

[Volker Braun]

You should be getting:

[root@localhost ~]# readelf -h /home/sage/sage/local/lib/libpython2.6.so.1.0 

ELF Header:

  Magic:   7f 45 4c 46 01 01 01 00 00 00 00 00 00 00 00 00 

  Class:                             ELF32

  Data:                              2's complement, little endian

  Version:                           1 (current)

  OS/ABI:                            UNIX - System V

  ABI Version:                       0

  Type:                              DYN (Shared object file)

  Machine:                           Intel 80386

  Version:                           0x1

  Entry point address:               0x1f6a0

  Start of program headers:          52 (bytes into file)

  Start of section headers:          4278584 (bytes into file)

  Flags:                             0x0

  Size of this header:               52 (bytes)

  Size of program headers:           32 (bytes)

  Number of program headers:         5

  Size of section headers:           40 (bytes)

  Number of section headers:         38

  Section header string table index: 35

[root@localhost ~]# md5sum /home/sage/sage/local/lib/libpython2.6.so.1.0

65ef868e1368ffd208e5bc8b77aa7826  /home/sage/sage/local/lib/libpython2.6.so.1.0

[vbraun@volker-desktop data]$ md5sum Sage-4.6.2.ova 

932e62a0ee0fa1f8d74d4bbcef1f324e  Sage-4.6.2.ova

[Emil Widmann]

Downloaded again and imported as "ova". It works fine!

Small issue: I guess you don't plan to support different keyboard
maps - mine is german quertz. But one shouldn't need to type much in
the VM anyway.

what settings would you recommend for a small sage server, like for a
class with 20 people? Is it possible at all to use the vm image that
way?

[Volker Braun]

Using dvorak personally, I know the keyboard settings can be a pain. Though you really only need it for setting the admin password. In the worst case (azerty) you'll probably have to set an initial numeric password. Alternatively, we could set a default admin password that then can be used with the browser interface. 

I don't have any experience running a class room on top of a VM. Should work fine if you give it enough resources. 

[Emil Widmann]

> I don't have any experience running a class room on top of a VM.
Should work
> fine if you give it enough resources.

I thought about something similar to
http://wiki.sagemath.org/DanDrake/JustEnoughSageServer