Ubuntu PPA for tup tool

[Anatol Pomozov]

Hi,

I just created a PPA project for tup, so those people who uses Ubuntu can install tup even easier than before.

Here is the page https://launchpad.net/~anatol/+archive/tup, just add

deb http://ppa.launchpad.net/anatol/tup/ubuntu lucid main

section to /etc/apt/sources.list file and then execute "sudo apt-get update && sudo apt-get install tup".

The PPA currently contains HEAD version of tup and I am going to update it periodically (~ once a week).

Enjoy!

[Mike Shal]

Cool, thanks! I put instructions for this on the main page.

-Mike

[Anatol Pomozov]

Hi

Any Debian derivative (Debian/Ubuntu/Mint/...) can use this repo as well.

Here is the plumbing code for Debian package system https://github.com/anatol/tup/tree/debian

The package deployments consists of 2 phases:

1) User packs sources with debian information into source package (using 'debuild -S").

2) User deploys the source package to repository where build farm compiles and creates binary packages for all architectures.

The main issue here is that all packages are build on isolated environment with minimum stuff (see https://wiki.ubuntu.com/PbuilderHowto).

1) I was not able to use fuse - it looks like it is not possible to load fuse kernel module with pbuilder. I don't really understand what happens here, but if I try to run "./bootstrap.sh" I endup with error "cannot open file /dev/fuse" error.

2) Because we deploy source package to the build server it means we cannot use 'git describe' during the build process.

Currently I pack tup binary generated on the first step of the bootstrap process (the /build/tup binary). And I use sed tool to set correct git version in sources. Here is the script that builds source package and deploys it to PPA http://pastie.org/2848226

Feel free to use/improve it.

[Ben Boeckel]

Hi,

I'm a packager for Fedora, so I might be able to help a bit.

On Fri, Nov 11, 2011 at 09:44:55 -0800, Anatol Pomozov wrote:
> The main issue here is that all packages are build on isolated environment
> with minimum stuff (see https://wiki.ubuntu.com/PbuilderHowto).
> 1) I was not able to use fuse - it looks like it is not possible to load
> fuse kernel module with pbuilder. I don't really understand what happens
> here, but if I try to run "./bootstrap.sh" I endup with error "cannot open
> file /dev/fuse" error.

Missing a BuildRequires on fuse-utils (or the Debian equivalent)?

> 2) Because we deploy source package to the build server it means we cannot
> use 'git describe' during the build process.

Release tarballs of tup shouldn't need to run 'git describe' I would
expect. Snapshots should be slightly different. The build could also
accept a TUP_HASH= or something during configure and use that instead
of calling 'git describe'.

IIRC, Github can also 'pre-configure' tarballs when downloaded. I can't
find an example, but using something like an environment variable in
some file in the repo will be replaced when a tarball of the repository
when downloaded. I'll see if I can find where I saw it being used. This
does, however, tie Github to (at least some part of) the release
infrastructure though.

> Currently I pack tup binary generated on the first step of the bootstrap
> process (the /build/tup binary). And I use sed tool to set correct git
> version in sources. Here is the script that builds source package and
> deploys it to PPA http://pastie.org/2848226

I don't know Debian's policies, but I know Fedora would only accept it
bootstrapping itself (it's also a great test of the current build).

--Ben

[Elliott Hird]

On 11 November 2011 20:35, Ben Boeckel <math...@gmail.com> wrote:
> Release tarballs of tup

There is no such thing; apart from Windows builds and now this Debian
package, tup is only distributed in git repo form, to my knowledge.

[Ben Boeckel]

True for now, but in the future, I would hope that releases are tagged.

--Ben

[Mike Shal]

On Fri, Nov 11, 2011 at 3:35 PM, Ben Boeckel <math...@gmail.com> wrote:
> Hi,
>
> I'm a packager for Fedora, so I might be able to help a bit.
>
> On Fri, Nov 11, 2011 at 09:44:55 -0800, Anatol Pomozov wrote:
>> The main issue here is that all packages are build on isolated environment
>> with minimum stuff (see https://wiki.ubuntu.com/PbuilderHowto).
>> 1) I was not able to use fuse - it looks like it is not possible to load
>> fuse kernel module with pbuilder. I don't really understand what happens
>> here, but if I try to run "./bootstrap.sh" I endup with error "cannot open
>> file /dev/fuse" error.
>
> Missing a BuildRequires on fuse-utils (or the Debian equivalent)?

Is it a permissions issue? Is the user that pbuilder runs as in the fuse group?

And does the package install automatically put the user into the fuse
group? Or how is that normally handled?

>
>> 2) Because we deploy source package to the build server it means we cannot
>> use 'git describe' during the build process.
>
> Release tarballs of tup shouldn't need to run 'git describe' I would
> expect. Snapshots should be slightly different. The build could also
> accept a TUP_HASH= or something during configure and use that instead
> of calling 'git describe'.
>
> IIRC, Github can also 'pre-configure' tarballs when downloaded. I can't
> find an example, but using something like an environment variable in
> some file in the repo will be replaced when a tarball of the repository
> when downloaded. I'll see if I can find where I saw it being used. This
> does, however, tie Github to (at least some part of) the release
> infrastructure though.

I'd be happy to adjust the bootstrap.sh script to make it easier to
set the correct version without git. Maybe it could accept an
environment variable as you suggest, plus skip the ./build/tup init
and ./build/tup upd steps, since there's no need to have tup build
itself if the directory is just going to be blown away anyway. It
should probably also accept CFLAGS since right now the version built
from bootstrap.sh does not enable compiler optimizations. What would
be easiest for you guys?

-Mike

[Mike Shal]

There are currently several versions tagged in the git repository. Or
are you referring to some other kind of tag?

-Mike

[Ben Boeckel]

On Mon, Nov 14, 2011 at 11:27:22 -0500, Mike Shal wrote:
> There are currently several versions tagged in the git repository. Or
> are you referring to some other kind of tag?

That would likely be sufficient. In Fedora, actual releases aren't given
a git hash since, given the git repository, the version number should be
sufficient as-is. Provided tags aren't moved (or announced when they
are), those tarballs should be fine.

--Ben

[Anatol Pomozov]

Hi,

> Missing a BuildRequires on fuse-utils (or the Debian equivalent)?

The package depends on libfuse-dev. pbuilder sees libfuse.so, but fails to see /dev/fuse. Here is error message:

[ tup ] Parsing Tupfiles...

fuse: device not found, try 'modprobe fuse' first

fuse_mount: No such file or directory

tup error: Unable to mount FUSE on .tup/mnt

make: *** [build-stamp] Error 255

dpkg-buildpackage: error: debian/rules build gave error exit status 2

I suspect the problem is in chroot that pbuilder uses. After chroot the build process does not see /dev/fuse file. Here is full debug log in case if you can spot anything strange: https://gist.github.com/1383660

> Is it a permissions issue? Is the user that pbuilder runs as in the fuse group?

> And does the package install automatically put the user into the fuse group? Or how is that normally handled?

Maybe. I see that it runs compilation as a 'pbuilder' user.

Here is the command: chroot /var/cache/pbuilder/build//2625 env LOGNAME=pbuilder su -p pbuilder

>  Maybe it could accept an environment variable as you suggest, plus skip the ./build/tup init and ./build/tup upd steps

Let's wait until I resolve the 'fuse device' issue. I also filed question to ubuntu forum http://ubuntuforums.org/showthread.php?p=11477405#post11477405

> I'd be happy to adjust the bootstrap.sh script to make it easier to set the correct version without git.

debian build system does not allow to compile project from git repo. Instead it requires maintainer to load source package, and thus 'git describe' does not work.

>  It should probably also accept CFLAGS

Sounds right to me. From build logs I see that pbuilder on PPA passes following flags

dpkg-buildpackage: set CFLAGS to default value: -g -O2
dpkg-buildpackage: set CPPFLAGS to default value:
dpkg-buildpackage: set LDFLAGS to default value: -Wl,-Bsymbolic-functions
dpkg-buildpackage: set FFLAGS to default value: -g -O2
dpkg-buildpackage: set CXXFLAGS to default value: -g -O2

See full build log here https://launchpadlibrarian.net/85136903/buildlog_ubuntu-lucid-amd64.tup_0.4.1.149%2Bg8ca8067_BUILDING.txt.gz

[Ben Boeckel]

On Mon, Nov 21, 2011 at 11:47:23 -0800, Anatol Pomozov wrote:
> > Missing a BuildRequires on fuse-utils (or the Debian equivalent)?
> The package depends on libfuse-dev. pbuilder sees libfuse.so, but fails to
> see /dev/fuse. Here is error message:
>
> [ tup ] Parsing Tupfiles...
> fuse: device not found, try 'modprobe fuse' first
> fuse_mount: No such file or directory
> tup error: Unable to mount FUSE on .tup/mnt
> make: *** [build-stamp] Error 255
> dpkg-buildpackage: error: debian/rules build gave error exit status 2
>
>
> I suspect the problem is in chroot that pbuilder uses. After chroot the
> build process does not see /dev/fuse file. Here is full debug log in case
> if you can spot anything strange: https://gist.github.com/1383660

Does Debian do *everything* in a chroot or just the build step? I don't
know if/how mock (Fedora's tool) works with udev rules in the chroot.

--Ben

[Mike Shal]

Can you try the latest master? Instead of bootstrap.sh, use build.sh.
You can pass in CFLAGS, LDFLASG, and TUP_LABEL (defaults to
"bootstrap") through the environment. It won't run tup afterward, so
it shouldn't need to use fuse (just access to the fuse dev
headers/libs). If you need any other changes to support this let me
know.

Thanks,
-Mike

[Anatol Pomozov]

Hi

Thanks everything seems fine. I just uploaded HEAD version to PPA.

As of the fuse issue - it seems that the problem in permissions. But because I have zero control on how the package is built remotely I doubt that it can be fixed (add user to fuse group). Thus the only way for PPA is to use build.sh instead of bootstrap.sh

[Mike Shal]

On Mon, Nov 28, 2011 at 8:57 PM, Anatol Pomozov

Ok, thanks! Though I guess this would be a problem in the future if
someone wants to create a package for a program that uses tup to build
it. What would we have to do in that case?

-Mike

[Mike Shal]

After thinking about it some more, tup could probably support a
one-time build so that it doesn't do any dependency tracking at all.
This would effectively turn off the fuse watcher, database, etc. It is
somewhat analogous to something like 'svn export', in that the result
is no longer usable by tup. But in the case of a packager the whole
tree is just going to be blown away anyway, so there's no real need
for all of that...

-Mike

[Ben Boeckel]

On Mon, Dec 05, 2011 at 16:34:19 -0500, Mike Shal wrote:
> After thinking about it some more, tup could probably support a
> one-time build so that it doesn't do any dependency tracking at all.
> This would effectively turn off the fuse watcher, database, etc. It is
> somewhat analogous to something like 'svn export', in that the result
> is no longer usable by tup. But in the case of a packager the whole
> tree is just going to be blown away anyway, so there's no real need
> for all of that...

Debuggability of the build system inside the buildroot would be nice.
CMake supports a VERBOSE=1 setting in the generated Makefiles. An
equivalent for these "stripped down" tup builds would be nice since the
build is a one-shot.

--Ben

[Mike Shal]

I think the command could take a --verbose flag similar to 'tup upd'
so that it would print out the full command-line. Or is there
something else you would be debugging at that point?

-Mike

[Ben Boeckel]

On Mon, Dec 05, 2011 at 16:45:37 -0500, Mike Shal wrote:
> I think the command could take a --verbose flag similar to 'tup upd'
> so that it would print out the full command-line. Or is there
> something else you would be debugging at that point?

That should be enough...I can't remember what else I've used in the
past.

--Ben