Anybody using tup for really large/complex builds ?

[Drew Marold]

I have inherited a rather large, complex, recursive make based build system. I've been looking at ways to improve or replace it, and tup is one of the tools that has caught my eye. At my last job I wrote the build using SCons, and I like the whole Python-based build tool thing, but I don't think it would deal well with what I have here. I'm curious to see if anyone has used tup on really large/complex projects and if so, any advice ?

The current build system builds webapps using java/ant, c & c++ code, builds stuff (sometimes the same stuff) for i686, x86_64 SLES, x86_64 Ubuntu (jobs farmed off to another build machine), and PPC. All the various build artifacts are packaged up as rpms or tarballs depending on the system, then all that is wrapped up into platform-specific deployment tarballs. It's pretty epic, terribly inefficient, and mostly single-threaded due to the recursive make.

So I'm looking at a couple of approaches. 1) stick with make, but redo it to be non-recursive. 2) rewrite the whole thing in something else. Right now in the something else category I'm looking at fabricate, tup, or trying SCons again. It's such a huge undertaking though, I'm at a bit of a loss as to where you would start with a tool like tup. Really appreciate any advice or suggestions you guys can offer.

Thanks,

   Drew

[Eriks Latosheks]

Depending on your project(s) structure and processes tup could problematic due to inflexibility running sub-tools. It requires you to list all command outputs, and outputs should go in single directory. Something as simple as applying patches during build, or unpacking archive, is not achievable (or problematic) with current tup design. 

After playing with tup I migrated to CMake + ninja. CMake brings strong multi-platform and high-level scripting support, IDE support (integration with Eclipse), and ninja builds everything on par with tup performance. The only thing I am missing from CMake is ability to build for multiple targets in single shot - this part was easy with tup but not with CMake (it forces you to do independent passes for every target compiler). I.e. if you project have compiler-compiler, i.e. you need to generate something on host, and then compile for target, with tup this is easy and can be done in one script, with CMake it requires multiple tool invocations + different build directories and no proper parallelization between them. 

[Andrew Wagner]

I have about 60 targets, mostly python scripts doing code generation,
then C++ compilation. The hierarchy is very shallow. I have the
luxury of only needing to work on one platform, so I'm not using build
variants.

Once I proposed a minor feature and Mike implemented and committed it
the same night, which is both awesome, and a bit scary.

I feel very safe using tup since it is able to nag you about missing
dependencies and blow away obsolete targets.

In a previous life I also used the perl based Makepp. If you are
using a lot of make black magic, it may be a smoother upgrade path for
you.

> --
> --
> tup-users mailing list
> email: tup-...@googlegroups.com
> unsubscribe: tup-users+...@googlegroups.com
> options: http://groups.google.com/group/tup-users?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "tup-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to tup-users+...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[Mike Shal]

Hi Drew,

On Tue, Feb 19, 2013 at 1:29 PM, Drew Marold <unstopp...@gmail.com> wrote:
> I have inherited a rather large, complex, recursive make based build system.
> I've been looking at ways to improve or replace it, and tup is one of the
> tools that has caught my eye. At my last job I wrote the build using SCons,
> and I like the whole Python-based build tool thing, but I don't think it
> would deal well with what I have here. I'm curious to see if anyone has used
> tup on really large/complex projects and if so, any advice ?
>
> The current build system builds webapps using java/ant, c & c++ code, builds
> stuff (sometimes the same stuff) for i686, x86_64 SLES, x86_64 Ubuntu (jobs
> farmed off to another build machine), and PPC. All the various build
> artifacts are packaged up as rpms or tarballs depending on the system, then
> all that is wrapped up into platform-specific deployment tarballs. It's
> pretty epic, terribly inefficient, and mostly single-threaded due to the
> recursive make.

The parts here that might be problematic with tup are the Java
compilation, and (maybe?) building tarballs. The problem with Java (or
really, javac) is that javac tries to be a build system in addition to
a compiler. You can see some of my comments about that here:

https://groups.google.com/d/msg/tup-users/mdAG-BT4lcw/0GpRSZLdgwgJ

>
> So I'm looking at a couple of approaches. 1) stick with make, but redo it to
> be non-recursive. 2) rewrite the whole thing in something else. Right now in
> the something else category I'm looking at fabricate, tup, or trying SCons
> again. It's such a huge undertaking though, I'm at a bit of a loss as to
> where you would start with a tool like tup. Really appreciate any advice or
> suggestions you guys can offer.

With tup, just run 'tup init' and 'tup upd' in your project root
directory. It will run a little bit as it tries to find Tupfiles (the
yellow progress bar), but since you don't have any yet, nothing will
actually run (the blue progress bar).

Next, go to a subdirectory where you expect to create a file. If you
have lots of auto-generated sources, start with one of those. Or if
sources are static and you can start compiling, start compiling one of
those. Create a Tupfile and dump in the command-line you want to run.
Eg if we have a shell-script to create foo.h:

: |> sh generate-script.sh |> foo.h

Then run 'tup upd' again and that command will run. In make, this
would look like:

foo.h: generate-script.sh (other dependencies listed? who knows)
sh generate-script.sh
clean:
rm foo.h

But you have no idea if you are missing input dependencies, or if the
script changes more than just foo.h. Additionally, incremental builds
will fail if you modify the Makefile, or delete the Makefile, or look
at it funny.

Now you just iterate by adding commands to Tupfiles, or refactoring
Tupfiles into variables, and running 'tup upd' to update it. Tup will
return an error message if you miss a dependency, which is a
significant advantage that it has over make and SCons (not sure about
fabricate).

So I'd recommend trying to build a small part of your project with tup
and seeing if it suits your needs or not. For the cases I mentioned
above where you may run into trouble, you could stick with Ant for the
Java stuff if the Java part of your project is largely independent of
the rest. For building rpms / tarballs, you may want to defer that to
a release machine or continuous integration server (assuming
developers run from the project directory, rather than installing an
rpm for each incremental build they do).

Hope that helps,
-Mike