testing bup with an automated build environment

[Elyse M. Grasso]

Hi --
I just started testing bup on Ubuntu 14.04.
For some reason the

apt-get build-dep bup

did not pull down python-fuse, but once I got that sorted out bup built and tested clean.

I have what amounts to a double use case for testing:

1) My network backups need bunches of deduplication.

My 'desktop' machine is multi-cored and intermittently supports a few virtualbox VMs, as well as images, music files and miscellaneous stuff including imap email files for thunderfird that take forever to back up and some files backed up from an Android tablet that have messed up  . My 'travel' machine  is a netbook, also running ubuntu 14.04, that deliberately mirrors a subset of the stuff on the main machine.

rsync is not cutting it for backing up this mess in a reasonable amount of time.

2) I need to exercise and evaluate a git-related CI/automated-build tool.

Are there a couple of accessible active branches that I can set up in the automation tool (in addition to master) that would benefit from being built and regression tested frequently? (I realize this is a small team and changes will be sporadic, but building real changes will give better information than trying to set up a fake build environment).


Infrastructure (besides the 2 Linux boxes mentioned above)
My backup storage locations currently include USB connected drives, and router-connected USB drives that present as CIFS or via FTP. I've got a SOHO 4Tbyte NAS that should arrive today -- not sure how that will present (probably CIFS unless it is more configurable than I expect).

I also have NFS mounts and rsync/ssh connections between the big machine and Linux VM images and between the physical machines, though these are not currently being used for backups on a regular basis.

My VM images include Centos 7 and Windows server, so I can do occasional testing of bup built in those environments as well. I can add a few VM images of different types if they would be useful for checking known problem cases.

-- 
Elyse Grasso

[Rob Browning]

"Elyse M. Grasso" <elyse....@gmail.com> writes:

> Hi --
> I just started testing bup on Ubuntu 14.04.

...hey, just wanted to let you know you're not being ignored, but I've
been tied up. In any case, I should be able to reply to your message on
the list in the next few days.

Thanks
--
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

[Elyse Grasso]

No problem. I'm setting up infrastructure and tools (and learning way more than I wanted to know about SSL/TLS and LDAP trusts) and trying to get a full rsync backup before I start playing with bup in detail.

[Rob Browning]

"Elyse M. Grasso" <elyse....@gmail.com> writes:

> 2) I need to exercise and evaluate a git-related CI/automated-build tool.
> ||
> Are there a couple of accessible active branches that I can set up in
> the automation tool (in addition to master) that would benefit from
> being built and regression tested frequently? (I realize this is a small
> team and changes will be sporadic, but building real changes will give
> better information than trying to set up a fake build environment).

In the long run, I'd love to have automated testing for at least the
master branch for all our "notable" platforms.

Right now I can easily test Linux, and I've created a lightweight
headless QEMU/KVM VM I can use to test more safely and conveniently as
root, so I usually test "make check", "fakeroot make check", and then
"make check" (as root) in the VM. I also push master to Travis CI
occasionally to double-check OSX (and Linux/Clang).

More ideally, I'd like to have some way to test any given commit against
a reasonable set of platforms before I push it to master (and to allow
others to do the same), but even just having master tested after the
fact would be handy.

And I'd like to have more coverage, i.e. I'd like to have the relevant
*BSDs included, Red Hat (variants), Ubuntu, Cygwin, etc.

Of course if we're wishing for things, I'd also like to have even
broader testing, say of the top few filesystems on each platform, and
all of the relevant archtectures (i.e. for Debian:
https://buildd.debian.org/status/package.php?p=bup&suite=sid). It's no
fun to find out s390x is broken well after the offending commit was
added.

Right now I always test Debian/amd64/ext4, and sometimes test ext3,
ext4, ext4-strictatime, btrfs, and xfs. I have a (hacked up) script to
help with that, but it'd be nice to automate it more generally.

But I'm not really sure *how* we should try to do any/all of that.
Whatever we do, we'll need to divide the work. There's very little
chance I'll have time to handle the administration of anything more than
the Debian cases myself (if that). We'll almost certainly need help
from those who are interested in each of the various platforms.

In truth, there's a fairly ginormous test space:

platform x architecture x compiler x src-filesystem x target-filesystem

And so even just hitting the "high spots" is likely to be expensive.

One option I've pondered is setting up a central buildbot (or similar),
assuming it'd then be possible for others who are interested to run
(latent) buildslaves for various platforms. i.e. I might provide some
for Debian, someone else provides *BSD, etc.

> I also have NFS mounts and rsync/ssh connections between the big machine
> and Linux VM images and between the physical machines, though these are
> not currently being used for backups on a regular basis.

Note that bup should normally behave best if you can arrange it so that
the repository itself is local (i.e. not accessed via a network
filesystem).

[Greg Troxel]

I think more automated testing would be great, especially on more
platforms.

I can see three separate things going on here:

1) tests should be sufficiently easy to run, so that one can build and
with the equivalent of "make test" check everything. This includes
tests being robust about requirements; sometimes tests require
unreasonable free disk space, and I don't remember if this is an issue
with bup.

2) have some way for people who have VM resources, such as (more or less
portably) xen dom0 or qemu or (linux only) KVM to be able to create
machines that can grab bup from a repo and run tests.

3) have some way for people to make test machines available to run tests
for others, perhaps via testing branches in the public repo and
reporting, or something else.

The problems with 3 is that it runs into policy issues, and the easier 2
is the less necessary 3 is.


First, it's huge that bup has such great automated tests.

For 1), the thing that I think would help is to have more explicit
support for filesystem types, perhaps both for the fs to be backed up
and restored, and for where the BUP_DIR is. Although perhaps it's only
the former that is thought to matter.

I just ran the tests on bup 0.26 in pkgsrc, and they all passed
(netbsd-6 i386, python 2.7). That's great, because they used to run out
of space, so maybe they are thriftier. On my system /tmp is "tmpfs" ram
FS that is only 256M. That's arguably kind of small, but it's basically
ok for almost everything. But, I'm left unclear which FS is being
tested. So maybe outputting some test metadata at the beginning would
help.

2) It's not really super useful for bup, but NetBSD has something called
anita that can with qemu or xen script the entire OS installation
process and run the OS-level tests, starting from a full OS build. It
shouldn't be too hard to script geting pkgsrc and building bup's prereqs
and git and then getting bup source and building it, all automatically.
But that would only give you a netbsd test platform, unless someone
extended this to install and cope with other systems. It seems like
this is a common problem for everyone, not just bup, but I am unaware of
general solutions.

Some of the machines are crazy slow. I just built bash on a Sparc
Classic:
real 175m58.965s
user 109m23.612s
sys 32m57.331s
but that's much more bloated than bup.

3) It may be possible to get various people to host a vm that will
basically only talk to a CI server and central git repo, and let it
run. That's probably easier policy-wise than a machine that arbitrary
other people can access and run code on.

[Rob Browning]

Greg Troxel <g...@lexort.com> writes:

> For 1), the thing that I think would help is to have more explicit
> support for filesystem types, perhaps both for the fs to be backed up
> and restored, and for where the BUP_DIR is. Although perhaps it's only
> the former that is thought to matter.

For now, the easiest thing to do is just clone or copy the source tree
onto the relevant filesystem and run "make check" from there. (That's
what my hacked up test script does.)

I'm not sure I've caught every case, but I've been trying to shift all
the tests to use ./t/tmp.

> I just ran the tests on bup 0.26 in pkgsrc, and they all passed
> (netbsd-6 i386, python 2.7). That's great, because they used to run out
> of space, so maybe they are thriftier.

I suspect that's due to the t/tmp migration. (And that reminds me of
another potential axis for the build space: python X.Y for each
supported X.Y.)

> It seems like this is a common problem for everyone, not just bup, but
> I am unaware of general solutions.

It's definitely not a small-scale problem.

I suppose you could get by if you had a set of qemu images for the
platforms of interest -- perhaps each might already have the bup build
deps installed and allow root ssh logins via a known key -- or perhaps
the images could run buildbot. They'd also need a known, non-root user
for non-root tests.

(...and then we store all the VM images in a central bup repo via
split?)

Of course this approach won't work for non-free platforms since you
probably can't share the images.

For what it's worth, right now, I just have a host I can fire up via kvm
on the command line, and it's set to use a serial console (so I don't
have to deal with a separate window). I originally built the image via
vmdebootstrap.

Once it's running, it'll accept root logins on a forwarded port via a
known key. So to run a test there, I just have a command that rsyncs my
bup working dir to /root/bup on the VM, and then runs "make check" (or
whatever command I specify).

> 3) It may be possible to get various people to host a vm that will
> basically only talk to a CI server and central git repo, and let it
> run. That's probably easier policy-wise than a machine that arbitrary
> other people can access and run code on.

Right - that's one of the options I was pondering, i.e. perhaps
volunteer maintained buildbot buildslaves, though I'm not deeply
familiar with buildbot yet. Of course, if we eventually have the qemu
images mentioned above, then hosting an instance might be fairly easy.
But that raises some additional questions, i.e. regarding security,
security updates, etc.?

I suspect you'd really want the build VMs to be roughly inaccessible
from the net, only communicating results, etc. But I don't know how
easy that would be to arrange.

And of course, as you say, little of this is really a bup specific
problem.

[Greg Troxel]

Rob Browning <r...@defaultvalue.org> writes:

> For now, the easiest thing to do is just clone or copy the source tree
> onto the relevant filesystem and run "make check" from there. (That's
> what my hacked up test script does.)

That's certainly reasonable. Not that there's a "what kind are you"
introspection available, but it would be cool if the test output somehow
said what was tested.

>> I just ran the tests on bup 0.26 in pkgsrc, and they all passed
>> (netbsd-6 i386, python 2.7). That's great, because they used to run out
>> of space, so maybe they are thriftier.
>
> I suspect that's due to the t/tmp migration. (And that reminds me of
> another potential axis for the build space: python X.Y for each
> supported X.Y.)

The t/tmp move sounds good, not just for space but for making what's
tested more obvious.

> Right - that's one of the options I was pondering, i.e. perhaps
> volunteer maintained buildbot buildslaves, though I'm not deeply
> familiar with buildbot yet. Of course, if we eventually have the qemu
> images mentioned above, then hosting an instance might be fairly easy.
> But that raises some additional questions, i.e. regarding security,
> security updates, etc.?

I should learn buildbot; it feels like it's solving the same problem
(that every project has).