VM images for FreeBSD

[Alexander Yerenkow]

Hello all.
I'm currently made set of scripts, which builds FreeBSD from svn sources,
and packing it in VirtualBox (*.vdi) compatible images.
It's working now, and producing something like

FreeBSD-9-i386-r226409-2011-10-16.vdi.xz (also .vdi, .vdi.zip and plain .img
which can be dd to USB flash).

I'm developing this here:
https://github.com/yerenkow/freebsd-vm-image

I have more goals to do (like producing more images, with a installed sets
of packages, like KDE-from-ports, KDE-from-area51, with experimental GEM
drivers etc.)

Why I'm writing this letter:

1. Get some help from peoples who interesting in this;
2. Introduce myself in -current (I wasn't active here before)
3. Discuss current bug behavior (or bug in my scripts/mind) of boot loader
PR/161677
4. Also, I'd like to see producing these images semi- or fully- official :)
(Lacking of virtual images to test even of major releases is not good for
project like FreeBSD.)


--
Regards,
Alexander Yerenkow
_______________________________________________
freebsd...@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-curre...@freebsd.org"

[Warren Block]

On Mon, 17 Oct 2011, Alexander Yerenkow wrote:

> Hello all.
> I'm currently made set of scripts, which builds FreeBSD from svn sources,
> and packing it in VirtualBox (*.vdi) compatible images.
> It's working now, and producing something like
>
> FreeBSD-9-i386-r226409-2011-10-16.vdi.xz (also .vdi, .vdi.zip and plain .img
> which can be dd to USB flash).
>
> I'm developing this here:
> https://github.com/yerenkow/freebsd-vm-image
>
> I have more goals to do (like producing more images, with a installed sets
> of packages, like KDE-from-ports, KDE-from-area51, with experimental GEM
> drivers etc.)

Excellent! If live CD/memdisk features are added, this could also be
useful for testing large xorg port updates before commit.

PS: why bash for cron-auto-action.sh?

[Alexander Yerenkow]

2011/10/17 Warren Block <wbl...@wonkity.com>

> On Mon, 17 Oct 2011, Alexander Yerenkow wrote:
>
> Hello all.
>> I'm currently made set of scripts, which builds FreeBSD from svn sources,
>> and packing it in VirtualBox (*.vdi) compatible images.
>> It's working now, and producing something like
>>
>> FreeBSD-9-i386-r226409-2011-10-16.vdi.xz (also .vdi, .vdi.zip and plain
>> .img
>> which can be dd to USB flash).
>>
>> I'm developing this here:
>> https://github.com/yerenkow/freebsd-vm-image
>>
>> I have more goals to do (like producing more images, with a installed sets
>> of packages, like KDE-from-ports, KDE-from-area51, with experimental GEM
>> drivers etc.)
>>
>
> Excellent! If live CD/memdisk features are added, this could also be
> useful for testing large xorg port updates before commit.
>
> PS: why bash for cron-auto-action.sh?
>

Hello all!

I'm working currently on creating images with a set pre-installed packages.
I looked at project pkgng (candidate for replacing current pkg_* subsystem),
and also I have some thought about current packages/ports system.

1. pkg_add can be launched with parameter -p $PREFIX. So, my first thought
was: I create empty directory structure with mtree, and I'll install there
all required packages; after that I need only update this installation tree
(manually by pkg_delete $old pkg_add $new, or with some tool). But I cannot
specify to pkg_add relative root, instead of real one.

Let me show example:
PKG_DBDIR=/zpool0/testroot/var/db/pkg pkg_add -p /zpool0/testroot/usr/local
ubench-0.32.tbz
installs package, and in /zpool0/testroot/var/db/pkg/ubench-0.32/+CONTENTS
there will be such record:
@cwd /zpool0/testroot/usr/local

I can't specify to pkg_add that it should treat /zpool0/testroot as root, as
I need (so record really should be @cwd /usr/local)
Instead, pkg_add allows me to make chroot, which as you understand is not
good (In specified chroot all required by pkg* binaries/libraries must
exists, unfortunately I can't specify some empty dir and install there).

Why is that? Because there is +INSTALL script in packages, in which
package/port system allows execute any code/script written by porter.

2. In ports enhancements task list (somewhere i read it) there was one item:
Make packages non-executable (or something similar). To do this properly, we
must get rid of of free-form post-install post-deinstall scripts.
To do this, we need some deep analysis of what types of actions there
happening, formalize them and provide some way to porters specify all needed
actions in Makefile.
I downloaded all packages for 9-current i386, found all +INSTALL scripts,
and kinda categorized them, you can get all of them here:
http://www.box.net/shared/ieovjj7l8omkrm3l21xb

To summarize my efforts:
I checked 21195 packages;
I found 880 install scripts;

3 scripts contains plain "exit 0"
8 install scripts contains some perl code;
17 scripts contains some additional "install" commands;
70 scripts contains some chgroup/chown actions (which probably could be done
by specifying mtree file?...)
75 contains uncategorized actions (print of license, some interactive
questions, ghostscript actions, tex, fonts etc.)
161 scripts contains some file commands, like (ld / cp / mv, creating
backups, creating configs if they aren't exists etc. )
166 scripts contains useradd/groupadd commands (many similar constructions,
not too hard to move this to .mk, in pkgng group/users can be specified in
yaml config)
380 contains pear component registration (md5 -q * | uniq - produces
exactly one result, so these all scripts are really one, could be moved to
some pear.mk)

Why I'm interested in non-executable install of package (e.g. simple unpack
+ execute some typical actions based on package description):
- Unpacking of hundreds Mb packages takes several minutes (to mdconfig-ed
filesystem)
- Installation of these packages via pkg_add (they downloads from local ftp)
took hours in my case (to mdconfig-ed filesystem)

As you understand, to make efficient image building system, I need to deal
with package installation without spending too many cpu/disk resources.
Ideally I consider all required packages are extracted to some their own
directory, like for ubench:
$X/packages/ubench/ (and here goes all directory structure which should be
copied to new root)

plus separated info of new users/groups (maybe there need some additional
data to make package installed in such way fully working).

So, maybe someone working in this direction, or have any comments?

3. Other "ports" ideas/thoughts.
I proposed small enahcement to pkgng, but instead in pkgng this should be
implemented in ports subsystem, it's about specifying abstract dependencies,
and correct resolving of them:
https://github.com/pkgng/pkgng/issues/100

Who can comment/elaborate about this? It shouldn't be very complicated,
since currently almost same functionality provided in .mk. files ( like
USE_PERL etc)


4. Where's the "right" place to discuss ports system? :)

Thanks.

--
Regards,
Alexander Yerenkow

[Benjamin Kaduk]

On Wed, 19 Oct 2011, Alexander Yerenkow wrote:

> Hello all!
>
> I'm working currently on creating images with a set pre-installed packages.
> I looked at project pkgng (candidate for replacing current pkg_* subsystem),
> and also I have some thought about current packages/ports system.
>
> 1. pkg_add can be launched with parameter -p $PREFIX. So, my first thought
> was: I create empty directory structure with mtree, and I'll install there
> all required packages; after that I need only update this installation tree
> (manually by pkg_delete $old pkg_add $new, or with some tool). But I cannot
> specify to pkg_add relative root, instead of real one.
>
> Let me show example:
> PKG_DBDIR=/zpool0/testroot/var/db/pkg pkg_add -p /zpool0/testroot/usr/local
> ubench-0.32.tbz
> installs package, and in /zpool0/testroot/var/db/pkg/ubench-0.32/+CONTENTS
> there will be such record:
> @cwd /zpool0/testroot/usr/local
>
> I can't specify to pkg_add that it should treat /zpool0/testroot as root, as
> I need (so record really should be @cwd /usr/local)
> Instead, pkg_add allows me to make chroot, which as you understand is not
> good (In specified chroot all required by pkg* binaries/libraries must
> exists, unfortunately I can't specify some empty dir and install there).
>
> Why is that? Because there is +INSTALL script in packages, in which
> package/port system allows execute any code/script written by porter.

This is indeed a frustrating problem.

Thank you for doing this analysis/breakdown!
However, I worry that it may have missed @exec statements in pkg-plist
files ... for example, net/openafs (which I maintain) runs kldxref in
/boot/modules after installing a kernel module, which is needed in order
for kldload to find the module. Now, this is clearly a case that a
potential nonexecutable package framework could handle, checking for
installed kernel modules and acting accordingly. However, having not done
the survey of the sort you did for install scripts, it is an uneasy
dangling unknown.

> Why I'm interested in non-executable install of package (e.g. simple unpack
> + execute some typical actions based on package description):
> - Unpacking of hundreds Mb packages takes several minutes (to mdconfig-ed
> filesystem)
> - Installation of these packages via pkg_add (they downloads from local ftp)
> took hours in my case (to mdconfig-ed filesystem)
>

This is quite a telling statistic :)

> As you understand, to make efficient image building system, I need to deal
> with package installation without spending too many cpu/disk resources.
> Ideally I consider all required packages are extracted to some their own
> directory, like for ubench:
> $X/packages/ubench/ (and here goes all directory structure which should be
> copied to new root)
>
> plus separated info of new users/groups (maybe there need some additional
> data to make package installed in such way fully working).

There would certainly be additional data needed, e.g. for installing
sample configuration files and copying to the real location, and removing
both copies on uninstallation if the "real" file is unchanged from the
sample. I'm sure there are others, too.

>
> So, maybe someone working in this direction, or have any comments?
>

I would be very hesitant to proceed in this direction without doing some
investigation of other package-based systems.
For example, Debian packages are inherently binary-based, there is not a
real parallel to our ports framework. Yet if anything, I think that
"executable packages" may be even more heavily used in Debian than in
FreeBSD. In addition to the tarball of files to extract, the maintainer
can also supply "maintainer scripts" which run before and/or after
installation and/or uninstallation. (Not to mention the infrastructure
components which implement things like diversions.) I have an incomplete
survey of a biased sample of Debian(-style) packages in my slides at
http://web.mit.edu/kaduk/Public/bsdcan-ports-talk-20110511.pdf , which
shows that in addition to being used to manage users and groups, these
maintainer scripts also are used to start/stop services, update gconf
keys, the PAM stack, and more. It quickly becomes quite a pile of
"additional data needed" (per the above) that I fear would be too much
infrastructure to safely maintain in a non-executable package framework.

Another incredibly useful (though hopefully infrequently used) feature of
maintainer scripts is the ability they give to recover from packaging
errors. The first example that comes to mind is unfortunately not a very
good one, but recently here at Athena we had a bug in our TeX
configuration package which resulted in a dangling symlink from a broken
diversion (which has no direct parallel in FreeBSD, making this a bad
example). In any case, this packaging bug made the package uninstallable!
However, we could produce an updated version of the package which had a
preinst that corrected for the previous packaging error, offering us a way
out that did not require manual user intervention, which I feel is
something that we should try very hard to avoid.

Because of this, I don't think that having it be impossible for a package
to have a custom executable component is a realistic goal. (Which is not
to say it is not a goal worth having.) However, it would probably be
feasible to add pieces to our framework (e.g. USERS/GROUPS) that make it
easier for packages to avoid executable components. If appropriately
flagged, then a package could just be an "unpack this tarball" operation,
possibly with a couple hooks (e.g. users/groups) from the packaging
system.

> 3. Other "ports" ideas/thoughts.
> I proposed small enahcement to pkgng, but instead in pkgng this should be
> implemented in ports subsystem, it's about specifying abstract dependencies,
> and correct resolving of them:
> https://github.com/pkgng/pkgng/issues/100
>
> Who can comment/elaborate about this? It shouldn't be very complicated,
> since currently almost same functionality provided in .mk. files ( like
> USE_PERL etc)

Interesting, though I don't think I'm really one to comment/elaborate on
it. It does seem vaguely analogous to the concept of a (Debian) "virtual
package", which can be depended on like a first-class package, but is
actually "provide"d by any number of candidate packages.
I don't have a sense of how hard it would be to implement for us, though,
and I have not had any time to look at pkgng at all. :(

>
>
> 4. Where's the "right" place to discuss ports system? :)

Presumably freebs...@freebsd.org, though I alas do not regularly read
there.

-Ben Kaduk

[Daniel O'Connor]

On 19/10/2011, at 21:19, Alexander Yerenkow wrote:
> I can't specify to pkg_add that it should treat /zpool0/testroot as root, as
> I need (so record really should be @cwd /usr/local)
> Instead, pkg_add allows me to make chroot, which as you understand is not
> good (In specified chroot all required by pkg* binaries/libraries must
> exists, unfortunately I can't specify some empty dir and install there).

Hmmm, why is it empty?
When I have made something analogous I did an installkernel/world into a directory and then chroot'd in there and built ports. There is no reason you couldn't pkg_add from a local mirror (or nullfs mount a local package mirror directory into the chroot).

> Why is that? Because there is +INSTALL script in packages, in which
> package/port system allows execute any code/script written by porter.

This is a feature ;)

> To summarize my efforts:
> I checked 21195 packages;
> I found 880 install scripts;
>
> 3 scripts contains plain "exit 0"
> 8 install scripts contains some perl code;
> 17 scripts contains some additional "install" commands;
> 70 scripts contains some chgroup/chown actions (which probably could be done
> by specifying mtree file?...)
> 75 contains uncategorized actions (print of license, some interactive
> questions, ghostscript actions, tex, fonts etc.)
> 161 scripts contains some file commands, like (ld / cp / mv, creating
> backups, creating configs if they aren't exists etc. )
> 166 scripts contains useradd/groupadd commands (many similar constructions,
> not too hard to move this to .mk, in pkgng group/users can be specified in
> yaml config)
> 380 contains pear component registration (md5 -q * | uniq - produces
> exactly one result, so these all scripts are really one, could be moved to
> some pear.mk)

Interesting stats, thanks for taking the time to do the analysis.

I think one of the reasons pkg_add is so slow is that it copies everything to a staging directory, then copies the files.. This is very tedious (obviously). I wonder if it could be modified to have a "stream" mode where it unpacks directly into the target FS.

Alternatively you could cut it in 2 conceptually and modify pkg_add so it can run it a mode where it just unpacks to a staging area, and another mode where it copies from the staging area to the destination.

--
Daniel O'Connor software and network engineer
for Genesis Software - http://www.gsoft.com.au
"The nice thing about standards is that there
are so many of them to choose from."
-- Andrew Tanenbaum
GPG Fingerprint - 5596 B766 97C0 0E94 4347 295E E593 DC20 7B3F CE8C

[Alexander Yerenkow]

2011/11/4 Daniel O'Connor <doco...@gsoft.com.au>

>
> On 19/10/2011, at 21:19, Alexander Yerenkow wrote:
> > I can't specify to pkg_add that it should treat /zpool0/testroot as
> root, as
> > I need (so record really should be @cwd /usr/local)
> > Instead, pkg_add allows me to make chroot, which as you understand is not
> > good (In specified chroot all required by pkg* binaries/libraries must
> > exists, unfortunately I can't specify some empty dir and install there).
>
> Hmmm, why is it empty?
> When I have made something analogous I did an installkernel/world into a
> directory and then chroot'd in there and built ports. There is no reason
> you couldn't pkg_add from a local mirror (or nullfs mount a local package
> mirror directory into the chroot).
>

>From beginning I thought about having a lot of directories, which contains
one installed package;
I assume using plain copy all requried data, to have requried packages
installed in new chroot env. Not via install, but simply by copy.
Reason was - to make composing of images with pre-installed software faster
(avoiding pkg_add/unpack/mtree/etc. steps).
I could easily use unionfs, if only it could work under zfs :)
In any way, all package installation process, all running scripts ends and
leaving a bunch of new files (links too), and some changed files (like
added groups/users etc), that's all.
I just wanted have unpacked and initialized packages in directories, which
I could use as puzzle parts, to build image with pre-installed packages. Of
course, I understand that there required some tricks to make it all works
(like adding users/groups/ X config etc.), but many straightforward
software will just work, which is in mostly cases enough to test another
release.

Currently I'm using pretty slow way to make pre-installed images, new fresh
copy of base world for each package-set, after that created install script,
and runned in chroot.

for i in `cat test-package-list` ; do
echo "env PACKAGESITE=$packagesite pkg_add -rifF $i" >>
$blank/root/install.sh
done

parameters used:
-r, --remote
Use the remote fetching feature.
-i, --no-deps
Install the package without fetching and installing dependencies.
-I, --no-script
If any installation scripts (pre-install or post-install) exist for a
given package, do not execute them.
-f, --force
Force installation to proceed even if prerequisite packages are not
installed or the requirements script fails.
-F Already installed packages are not an error

So, package not getting dependencies (they all must be specified too), but
all went installed and even working somehow.

But all this is pretty rough :)

About long way to standardize installation process - I think even if it's
so complex, it should just start somewhere. Best candidate - all pear
install scripts; they can easily be moved to *.mk. A bit more complex is
standardizing user/groups.

--
Regards,
Alexander Yerenkow

[Garrett Cooper]

On Nov 3, 2011, at 9:31 PM, Daniel O'Connor wrote:

>
> On 19/10/2011, at 21:19, Alexander Yerenkow wrote:
>> I can't specify to pkg_add that it should treat /zpool0/testroot as root, as
>> I need (so record really should be @cwd /usr/local)
>> Instead, pkg_add allows me to make chroot, which as you understand is not
>> good (In specified chroot all required by pkg* binaries/libraries must
>> exists, unfortunately I can't specify some empty dir and install there).
>
> Hmmm, why is it empty?
> When I have made something analogous I did an installkernel/world into a directory and then chroot'd in there and built ports. There is no reason you couldn't pkg_add from a local mirror (or nullfs mount a local package mirror directory into the chroot).

The chroot option via the pkg_install tools is broken. Look through the PR database for more details.

>> Why is that? Because there is +INSTALL script in packages, in which
>> package/port system allows execute any code/script written by porter.
>
> This is a feature ;)

Except it doesn't work too terribly well on cross-compiled images or in installed worlds where the image version and the host system may not match and the script digs into the info that it needs from the kernel (one example is available here: http://unix.derkeiler.com/Mailing-Lists/FreeBSD/current/2007-07/msg00597.html , but I've seen other issues if things are executing something like ifconfig, geom, etc). This would probably be less of an issue if BUILD_DEPENDS was compiled with the host architecture instead of the target architecture, so the tools could be run on the build host, but assuming that that level of intelligence exists within ports is incorrect.

>> To summarize my efforts:
>> I checked 21195 packages;
>> I found 880 install scripts;
>>
>> 3 scripts contains plain "exit 0"
>> 8 install scripts contains some perl code;
>> 17 scripts contains some additional "install" commands;
>> 70 scripts contains some chgroup/chown actions (which probably could be done
>> by specifying mtree file?...)
>> 75 contains uncategorized actions (print of license, some interactive
>> questions, ghostscript actions, tex, fonts etc.)
>> 161 scripts contains some file commands, like (ld / cp / mv, creating
>> backups, creating configs if they aren't exists etc. )
>> 166 scripts contains useradd/groupadd commands (many similar constructions,
>> not too hard to move this to .mk, in pkgng group/users can be specified in
>> yaml config)
>> 380 contains pear component registration (md5 -q * | uniq - produces
>> exactly one result, so these all scripts are really one, could be moved to
>> some pear.mk)
>
> Interesting stats, thanks for taking the time to do the analysis.
>
> I think one of the reasons pkg_add is so slow is that it copies everything to a staging directory, then copies the files.. This is very tedious (obviously). I wonder if it could be modified to have a "stream" mode where it unpacks directly into the target FS.
>
> Alternatively you could cut it in 2 conceptually and modify pkg_add so it can run it a mode where it just unpacks to a staging area, and another mode where it copies from the staging area to the destination.

This is also why "ports" is "faster" -- it hacks around the double piped tar copy and installs _directly_ to the live system and sets up the metadata after the fact, which needless to say.. isn't very atomic :/..
That and while .bz2 compresses better, for most cases than .gz, compressing/decompressing is slower with .bz2 than with .gz.
Thanks,
-Garrett_______________________________________________

[Alexander Yerenkow]

If anyone interested, I got here [1] VirtualBox Image:

FreeBSD-10-amd64-r228694-2011-12-19.vdi.xz

Anyone who's looking to test 10 can get to test it :)

It contains package-installed partial system with openbox;

Image configured to run DHCP on em0, you can change this in /etc/rc.conf,
as usually.

When you'll get internet working, you can add packages (9 ones), running
/root/addpackage.sh $1
To get X, login as root, start
/root/runx.sh

In a few seconds (there's delays for safety) you should get X with openbox.


BTW, it contains also qt 4.8.0 and qtcreator 2.4.0, you can test something
and help a bit for KDE/QT team with any feedbacks. It's installed with
default settings in their default prefixes (qt in /usr/local/Trolltech, and
qtcreator in / ), so, to run something you probably must set correct
LD_PATH.
As for qtcreator, I created script for launch it, placed in root, which is
also launched when you start X.

1. http://gits.kiev.ua/FreeBSD/

P.S. As always, I'm looking for anyone who will lend me a hand in enhancing
build scripts.

--
Regards,
Alexander Yerenkow
_______________________________________________

[Adrian Chadd]

Hi,

Hm, so this lets us create a virtualbox image from what, a set of
install tarballs? Or /usr/src build?



Adrian

[Alexander Yerenkow]

2011/12/19 Adrian Chadd <adr...@freebsd.org>

> Hi,
>
> Hm, so this lets us create a virtualbox image from what, a set of
> install tarballs? Or /usr/src build?
>

> I'm using cross-build and installation from sources dir (which is after
that got svn-up'ed and all goes again).
It shouldn't be complex to install to image from installation media and/or
tarballs, but mine main idea is to have rolling image for making some
automated tests.
Currently I'm establishing building and providing images scheme, will do
images with KMS+small graphical programs, with qt+unstable KDE, and
probably with BHyVe. I think that's most useful setups currently. And maybe
some image for benchmarking :)


>
> Adrian
>



--
Regards,
Alexander Yerenkow

[Baptiste Daroussin]

On Mon, Dec 19, 2011 at 10:31:51PM +0200, Alexander Yerenkow wrote:
> 2011/12/19 Adrian Chadd <adr...@freebsd.org>
>
> > Hi,
> >
> > Hm, so this lets us create a virtualbox image from what, a set of
> > install tarballs? Or /usr/src build?
> >
> > I'm using cross-build and installation from sources dir (which is after
> that got svn-up'ed and all goes again).
> It shouldn't be complex to install to image from installation media and/or
> tarballs, but mine main idea is to have rolling image for making some
> automated tests.
> Currently I'm establishing building and providing images scheme, will do
> images with KMS+small graphical programs, with qt+unstable KDE, and
> probably with BHyVe. I think that's most useful setups currently. And maybe
> some image for benchmarking :)
>
>

FYI I have been working on a ova file generator for the release, I manage to
create ova images that do work on VirtualBox without problems, there are still
some problems with vmware for now, the goal is to have a standard vm ready image
(ova is standard) that would work on every system that do support it.

the image can be easily created from the release.

regards,
Bapt

[Alexander Yerenkow]

2011/12/21 Baptiste Daroussin <ba...@freebsd.org>

This is good :)
Do you have some scripts left?

> the image can be easily created from the release.
>

Well, I'm trying to build custom images, like rolling from svn [+ patches]
[+ packages] [+ some initial config].
A way for building images from release I think can be taken from PC-BSD,
they distribute upcoming 9 in VM formats too.



>
> regards,
> Bapt

[Outback Dingo]

How about in XEN 4.1 a .vhd image would be awesome, especially boot from zfs

[Baptiste Daroussin]

You can try it here: http://git.etoilebsd.net/vmdkimage/ currently this is a
quick and dirty make-vmdk.sh script I imported vmdkimage.c from Haiku, converted
it from C++ to C and make some changes so that it is able to build scsi vmdk
images (vmware needs scsi disk).

You can use it that way: ./make-vmdk.sh 4 90-RC3 test9

You will get a tes9.ova.xz image the disk size will be a 4G one, the xz will
take 100M. you can import it in vbox it will work ootb, it should work on vmware
I hope, currently no feedback on vmware for the last changes.

Here is an example
http://files.etoilebsd.net/ova/test9.ova.xz

No tunning on the OS yet

regards,
Bapt