is there anything Tup can't build

[Gour]

Hello!

In my (long) search for adequate build system going from cmake, Waf,
SCons, even considering Bento for Python project, we arrived to Tup and
it's great to see something looking very powerful and still
human-friendly. :-)

I plan to write multi-platform desktop GUI application - strongly
considering Nimrod (http://nimrod-code.org/) language along with GTK+
bindings.

The application will consist of:

- included 3rd party C library which has Makefile for building

- Nimrod bindings for C lib to be built by using included c2nim tool
which then has to be built using Nimrod compiler

- several custom libraries calling C library

- GUI part of application and main application containing business logic

- reST/Sphinx documentation sources to build HTML/PDF versions of
documentation

- capability to build variants for different OS-es (Linux, Mac OS,
Windows)

Now, not seeing examples of some bigger projects using Tup, I wonder if
there is something in the above description with which we could have
problem incorporating in Tup build to have complete build solution?



Sincerely,
Gour

--
Perform your prescribed duty, for doing so is better than not
working. One cannot even maintain one's physical body without work.

http://www.atmarama.net | Hlapicina (Croatia) | GPG: 52B5C810

[Freddie Chopin]

In case of problems you can easily use make for part of the build and then invoking tup for the "real job". For example building docs will be probably done only sometimes, so you don't need to use tup for it - use make (or sth else). If you have a lib that uses makefile to build - just use it, you probably won't be rebuilding it each time you compile your project, etc. etc. etc.

4\/3!!

[Gour]

On Tue, 18 Dec 2012 11:23:44 -0800 (PST)
Freddie Chopin <freddie...@gmail.com> wrote:

> In case of problems you can easily use make for part of the build and
> then invoking tup for the "real job".

That's OK; but, according to the subject line, I wonder if there is
some scenario where Tup would have problem?

> For example building docs will be probably done only sometimes, so you
> don't need to use tup for it - use make (or sth else).

Again, "don't need" is one thing, while I wonder about "can't". :-)


Sincerely,
Gour

--
The senses are so strong and impetuous, O Arjuna,
that they forcibly carry away the mind even of a man
of discrimination who is endeavoring to control them.

[Freddie Chopin]

From what I've found out so far there is at least one thing tup cannot do - it can't deal with cases where you need to incrementally modify a file (including a case where you read from file and overwrite it with something new), so for example you cannot do:

ar lib.a objA.o

ar lib.a objB.o

ar lib.a objC.o

...

you need to do:

ar lib.a objA.o objB.o objC.o ...

From what I've heard such scenario also happens when you build pdfs with LaTeX - in "normal" software compilation it's really rare (I cannot think of any other common example other than archiver).

4\/3!!

[Gour]

On Tue, 18 Dec 2012 23:29:52 -0800 (PST)
Freddie Chopin <freddie...@gmail.com> wrote:

> From what I've found out so far there is at least one thing tup
> cannot do - it can't deal with cases where you need to incrementally
> modify a file (including a case where you read from file and
> overwrite it with something new), so for example you cannot do:
> ar lib.a objA.o
> ar lib.a objB.o
> ar lib.a objC.o
> ...
>
> you need to do:
> ar lib.a objA.o objB.o objC.o ...

I believe we won't need it.

> From what I've heard such scenario also happens when you build pdfs
> with LaTeX

We might use rst2pdf or simply ship generated PDF then.

So, Tup looks very promising...


Sincerely,
Gour


--
One who is able to withdraw his senses from sense objects,
as the tortoise draws its limbs within the shell,
is firmly fixed in perfect consciousness.

[encodr]

I'm not sure why you would base your choice of build tool on how often you run (part of) it.

My experience is that if your code relationship is 1{many-source -> one product}N then tup can build it 

- at least, as long as you stay on linux. I've had an "interesting" journey on other platforms and on OpenVZ - anything that complicates the use of Fuse.

But I do use 'make' for those aspects of build management for which (it seems to me) tup is not well suited,

e.g. tup has no real equivalent of 'make clean',  nor 'make install' (where the target directory is "over there").

e

On 18 Dec 2012, at 19:23, Freddie Chopin wrote:

In case of problems you can easily use make for part of the build and then invoking tup for the "real job". For example building docs will be probably done only sometimes, so you don't need to use tup for it - use make (or sth else). If you have a lib that uses makefile to build - just use it, you probably won't be rebuilding it each time you compile your project, etc. etc. etc.

4\/3!!

--
tup-users mailing list
email: tup-...@googlegroups.com
unsubscribe: tup-users+...@googlegroups.com
options: http://groups.google.com/group/tup-users?hl=en

[Freddie Chopin]

W dniu środa, 19 grudnia 2012 09:52:09 UTC+1 użytkownik encodr napisał:

I'm not sure why you would base your choice of build tool on how often you run (part of) it.

It's not the reason, but when you have a component (docs, library, whatever) that has a working makefile based system there's no point in converting that to tup if you'll be compiling that once a week... How often will you build the docs and some external lib, what gain will tup give you and how long will the conversion process take? It's completely opposite if you're starting a new project that you'll be doing incremental builds twice in a minute - tup will give you a speed gain, and you have to prepare a new build system anyway.

4\/3!!

[encodr]

Ah, I see. 

And from that point of view, I agree. 

I have no pure-tup builds anyway. 

Always need 'make' for something :-(

e

[Gour]

On Wed, 19 Dec 2012 08:52:09 +0000
encodr <enc...@googlemail.com> wrote:

> But I do use 'make' for those aspects of build management for which
> (it seems to me) tup is not well suited, e.g. tup has no real
> equivalent of 'make clean', nor 'make install' (where the target
> directory is "over there").

Is 'make clean' needed with Tup?

'make install' seems another story?

Any plan to make Tup capable to handle different targets?

Going to Make for such things does not sound so nice.

Otoh, SCons/Waf/Cmake are also not pretty...and we like that Tup is
domain-agnostic allowing to add support for new languages/compilers
very easily.


Sincerely,
Gour

--
One who works in devotion, who is a pure soul, and who controls
his mind and senses is dear to everyone, and everyone is dear to
him. Though always working, such a man is never entangled.

[Andrew Wagner]

On Wed, Dec 19, 2012 at 3:59 PM, Gour <go...@atmarama.net> wrote:
> On Wed, 19 Dec 2012 08:52:09 +0000
> encodr <enc...@googlemail.com> wrote:
>
>> But I do use 'make' for those aspects of build management for which
>> (it seems to me) tup is not well suited, e.g. tup has no real
>> equivalent of 'make clean', nor 'make install' (where the target
>> directory is "over there").
>
> Is 'make clean' needed with Tup?

If you're cleaning just because you don't trust the correctness of
your own build setup (I have found this to usually be the case with
people using CMake), it is usually unnecessary with tup, since tup can
nag you about several common errors (i.e. unstated dependencies, often
exacerbated by make not suppoorting proper recursive builds). It also
deletes extra crap when you switch between git trees and rebuild. If
you just want to save space or something, I think most people using
tup are probably already already using git, which does a great job of
cleaning.

> 'make install' seems another story?

Nope, tup doesn't know anything platform specific aside from the
filesystem magic it depends on.

> Any plan to make Tup capable to handle different targets?

If you put targets in diferent subtrees, you can use "tup upd ." or
whatever to just rebuild targets in a subtree.

> Going to Make for such things does not sound so nice.

You might need to clarify your use case a bit more.

[Ben Boeckel]

On Wed, Dec 19, 2012 at 21:09:25 +0100, Andrew Wagner wrote:
> If you're cleaning just because you don't trust the correctness of
> your own build setup (I have found this to usually be the case with
> people using CMake)

I'd be interested in seeing these case(s). If it's sloppy CMake writing,
that's understandable, but if CMake is generating improper Makefiles,
that's a different story. If it's due to external_project_add, this does
not surprise me at all.

I have pretty complex CMake setups and the only problems I hit are when
I miss dependencies with add_custom_(command|target) calls.

> it is usually unnecessary with tup, since tup can
> nag you about several common errors (i.e. unstated dependencies, often
> exacerbated by make not suppoorting proper recursive builds).

I clean fairly regularly in some projects to help catch warnings in the
source code (they're not warning-clean, so -Werror isn't an option).

> It also deletes extra crap when you switch between git trees and
> rebuild. If you just want to save space or something, I think most
> people using tup are probably already already using git, which does a
> great job of cleaning.

I vastly prefer keeping builds (and I'm disappointed that Tup didn't go
this route rather than variants, but oh well) out of the source tree
(the source tree should ideally be buildable from a read-only
directory), so git can't help there.

> > 'make install' seems another story?
>
> Nope, tup doesn't know anything platform specific aside from the
> filesystem magic it depends on.

There's some projects which might need to relink or modify on install
(OS X seems to be the common one that I've seen due to its...odd rpath
support, but using rpath in the build tree and stripping it at install
is common elsewhere too). Is there a way to have an 'on install' command
for a target in Tup?

> > Any plan to make Tup capable to handle different targets?
>
> If you put targets in diferent subtrees, you can use "tup upd ." or
> whatever to just rebuild targets in a subtree.
>
> > Going to Make for such things does not sound so nice.
>
> You might need to clarify your use case a bit more.

Here are a few targets I have in my CMake builds which don't fit the
'many -> one' target mapping:

- A target to generate a tarball+patch for the current tree (tarball
has HEAD; patch has `git diff HEAD` output iff there's a diff);
- Doxygen (many -> many indeterminate filepaths);
- Updating .git/hooks;
- Dynamic test generation (for each shared library created, make sure
that it can be dlopen'd without error).

I'd be interested to know what Tup can do about these cases.

For the record, I'm interested in build systems as a packager; I don't
believe Tup can fulfill the use cases I currently require that CMake can
do for most of my personal projects.

Lacking a proper install framework is going to make any Tup projects a
nightmare for packagers. The nicest systems I've seen that aren't
essentially language-specific (e.g., Python's distutils or Haskell's
cabal) are CMake and autoconf (much as I dislike it, controlling its
install directories are a breeze). Custom Makefiles span the entire
spectrum from worst to best mainly because developers have to write
stuff from nothing. I'd like Tup to help make things easy for packagers
of Tup-using projects.

--Ben

[Freddie Chopin]

Thanks to the last answer I actually thought about more things you cannot do with tup - you cannot strip a binary and you cannot add index to a static lib. Generally tup won't be able to do things that require one of following things:

- modifying an existing file (stripping, ranlib, making static lib iteratively)

- outputting to unknown filenames (with tools that create some randomly named temporary/intermediate files)

- outputting to "other" directory (which would be required for "install" target)

The doxygen example is not very convincing, as I cannot imagine why would anyone want to build doxygen docs not by just executing doxygen (which obviously cannot be done from a Tupfile - see points 2 and 3 above).

BTW - from what I've seen tup variants are essentially out-of-tree builds with some more (optional) features, so where's the problem with the lack of out-of-source builds?

4\/3!!

[Ben Boeckel]

On Wed, Dec 19, 2012 at 13:55:38 -0800, Freddie Chopin wrote:
> The doxygen example is not very convincing, as I cannot imagine why would
> anyone want to build doxygen docs not by just executing doxygen (which
> obviously cannot be done from a Tupfile - see points 2 and 3 above).

I build Doxygen separately for each library (some options can turn off
libraries, so making documentation for those libraries is less than
helpful) and use CMake to setup dependencies and tag file references.
CMake uses a template Doxyfile which it does a configure_file() on to
set library-specific strings and paths.

> BTW - from what I've seen tup variants are essentially out-of-tree builds
> with some more (optional) features, so where's the problem with the lack of
> out-of-source builds?

My understanding of variants might be lacking, but with variants, you
can offer a e.g., a Debug or Release mode. What if I want to do Release
with debug info (as packagers tend to do)? CFLAGS might work for this,
but I also test some projects against an array of the versions of the
dependencies. With one source tree I have 40+ build trees to test
against the matrix (though not the complete cross product):

- Coverage
- Valgrind

- GCC 4.7.2
- Clang 3.1
- Clang 3.2svn

- Static
- Shared

- No Python
- Python 2.7
- Python 3.2
- Python 3.3

- Boost 1.47.0
- Boost 1.48.0
- Boost 1.48.0 (mangled)
- Boost 1.49.0
- Boost 1.50.0
- Boost 1.51.0
- Boost 1.52.0

Setting up umpteen source trees to test this is much more complicated
than keeping one, then running all the tests against the new revision.
How could I do this with variants without explicitly hard-coding all the
versions of things into the Tupfiles or using a pile of separate source
trees? These build trees also take quite a bit of diskspace (20+ GB in
all) and a 'clean' target is run in each after the tests to minimize
their impact for normal machine usage.

--Ben

[Rendaw]

I'm not familiar with those sorts of tests. If I understand
correctly, you have 1 project source tree with 1 source version, and
you build/test with a range of compiler versions and external build
dependencies.

If that's the case, I think variants will work. A variant is just a
folder with a tup.config that defines various (Tupfile-defined)
configuration parameters for a build. The configuration parameters
can be used to identify external dependencies, compiler, build flags,
etc. Each variant is defined solely by build the tup.config, so you'd
have a separate variant directory/config file for each testing
scenario. Tup duplicates the source tree's directory structure for
each variant, but nothing other than the directories. To save disk
space, you can delete the variant directory afterward.

If you have a single copy of the dependencies in the source tree or
you're testing with different versions of the same source, I don't
think you can (easily) use variants. Tup prevents you from modifying
the source during a build, so you'd have to do a prep-step to change
versions or make multiple copies so that the required versions exist
simultaneously.

[Gour]

On Wed, 19 Dec 2012 21:09:25 +0100
Andrew Wagner <drew...@gmail.com> wrote:

> If you're cleaning just because you don't trust the correctness of
> your own build setup (I have found this to usually be the case with
> people using CMake), it is usually unnecessary with tup, since tup can
> nag you about several common errors (i.e. unstated dependencies, often
> exacerbated by make not suppoorting proper recursive builds). It also
> deletes extra crap when you switch between git trees and rebuild. If
> you just want to save space or something, I think most people using
> tup are probably already already using git, which does a great job of
> cleaning.

OK. This one is clear.

> If you put targets in diferent subtrees, you can use "tup upd ." or
> whatever to just rebuild targets in a subtree.

Thanks. That seems to be enough.

> You might need to clarify your use case a bit more.

Well, I'd develop my project on Linux and would need different targets
like native executable, (maybe) different package format, source
tarball, release/debug version, doc (html/pdf) as well as different
binaries - e.g. Windows installer and bundle or whatever for Mac OS.


Sincerely,
Gour


--
As the embodied soul continuously passes, in this body,
from boyhood to youth to old age, the soul similarly passes
into another body at death. A sober person is not bewildered
by such a change.

[Ben Boeckel]

On Wed, Dec 19, 2012 at 19:01:42 -0800, Rendaw wrote:
> I'm not familiar with those sorts of tests. If I understand
> correctly, you have 1 project source tree with 1 source version, and
> you build/test with a range of compiler versions and external build
> dependencies.

Correct. It's to make sure that we work with older versions and any
features that get added and used in future versions are properly worked
around on all previous versions. I forgot to mention that these tests
are currently being run on OS X and Linux, but a Windows setup will be
required as well.

> If that's the case, I think variants will work. A variant is just a
> folder with a tup.config that defines various (Tupfile-defined)
> configuration parameters for a build. The configuration parameters
> can be used to identify external dependencies, compiler, build flags,
> etc. Each variant is defined solely by build the tup.config, so you'd
> have a separate variant directory/config file for each testing
> scenario. Tup duplicates the source tree's directory structure for
> each variant, but nothing other than the directories. To save disk
> space, you can delete the variant directory afterward.

Can the tup.config exist anywhere or does it have to live underneath the
root Tupfile's directory?

> If you have a single copy of the dependencies in the source tree or
> you're testing with different versions of the same source, I don't
> think you can (easily) use variants.

> Tup prevents you from modifying the source during a build

I'm fine with this and actually am glad that this is enforced :) .

--Ben

[Freddie Chopin]

W dniu czwartek, 20 grudnia 2012 22:49:11 UTC+1 użytkownik Ben Boeckel napisał:

Can the tup.config exist anywhere or does it have to live underneath the
root Tupfile's directory?

I think it has to be in the root dir, but that's only for non-variant build. If you have variants than each of them has it's own, completely separate tup.config file and the one in top-level folder is ignored.

Tup manual says:

> @-variables are specified in the tup.config file at the top of the tup hierarchy or in a variant directory.

http://gittup.org/tup/manual.html

4\/3!!

[encodr]

tup does not require 'make clean'. My build system does.

Targets: tup provides some control over variants (not sure if you meant that),
but AFAIK by design tup's output is always "beside" the Tupfile.

e

[Mike Shal]

On Wed, Dec 19, 2012 at 4:55 PM, Freddie Chopin
<freddie...@gmail.com> wrote:
> Thanks to the last answer I actually thought about more things you cannot do
> with tup - you cannot strip a binary and you cannot add index to a static
> lib. Generally tup won't be able to do things that require one of following
> things:
> - modifying an existing file (stripping, ranlib, making static lib
> iteratively)

There's no point to making a static lib iteratively. If you want to
strip a binary, you can do:

: |> link; strip |> binary-stripped

or:

: |> link |> binary-unstripped
: binary-unstripped |> cp %f %o; strip %o |> binary-stripped

if you want both stripped and unstripped versions. You can't do:

: |> link |> binary
: |> strip |> binary

because that makes no sense. You should get an error message in this
case, which is what tup does. Same goes for ranlib:

: |> ar cr %o; ranlib %o |> libfoo.a

Or if you have an 'ar' written sometime in the last century:

: |> ar crs %o %f |> libfoo.a

-Mike

[Andrew Wagner]

On Wed, Dec 19, 2012 at 9:49 PM, Ben Boeckel <math...@gmail.com> wrote:
> On Wed, Dec 19, 2012 at 21:09:25 +0100, Andrew Wagner wrote:
>> If you're cleaning just because you don't trust the correctness of
>> your own build setup (I have found this to usually be the case with
>> people using CMake)
>
> I'd be interested in seeing these case(s). If it's sloppy CMake writing,
> that's understandable, but if CMake is generating improper Makefiles,
> that's a different story. If it's due to external_project_add, this does
> not surprise me at all.

CMake works well enough for simple cases, that, for better or worse,
people start using it and keep using it without ever understanding how
it works. Something like make or tup has a steeper initial learning
curve, but during that time you learn how the tool works and have a
shallow learning curve for the more complex cases.

> I have pretty complex CMake setups and the only problems I hit are when
> I miss dependencies with add_custom_(command|target) calls.

Yes, this is one of the prime examples we've been bitten by. It is
exacerbated by the out-of-source build stuff since it's non-obvious
how to refer to dependencies or build targets in CMake. I'm sure it's
learnable, but I've yet to run into someone IRL who was really
comfortable with it. That said, I mostly hang around with academics
who are by nature inexperienced programmers learning the ropes. My
programming guru is a guy who was a pro programmer in telecom, and
then hacked on the linux kernel for his PhD, and swears by make.

>> it is usually unnecessary with tup, since tup can
>> nag you about several common errors (i.e. unstated dependencies, often
>> exacerbated by make not suppoorting proper recursive builds).
>
> I clean fairly regularly in some projects to help catch warnings in the
> source code (they're not warning-clean, so -Werror isn't an option).

Ah. Of course, best practice is to fix the causes of the warnings...
but I totally understand if you inherited a pile of non-clean code.

>> It also deletes extra crap when you switch between git trees and
>> rebuild. If you just want to save space or something, I think most
>> people using tup are probably already already using git, which does a
>> great job of cleaning.
>
> I vastly prefer keeping builds (and I'm disappointed that Tup didn't go
> this route rather than variants, but oh well) out of the source tree
> (the source tree should ideally be buildable from a read-only
> directory), so git can't help there.

Well, git can do a local clone of the code, but then you really need a
script of mechanism outside of your code that is pulling changes into
your variants. I think it's customary for buildbots to do clean
checkouts of source code, so there is at least precedence for this
sort of thing. Remember, local clones with git by default use hard
links for the objects, so local cloning is faster and doesn't
unnecessarily duplicate files.

>> > 'make install' seems another story?
>>
>> Nope, tup doesn't know anything platform specific aside from the
>> filesystem magic it depends on.
>
> There's some projects which might need to relink or modify on install
> (OS X seems to be the common one that I've seen due to its...odd rpath
> support, but using rpath in the build tree and stripping it at install
> is common elsewhere too). Is there a way to have an 'on install' command
> for a target in Tup?

Tup itself doesn't even have an install target, so I wouldn't hold my breath :)

> Here are a few targets I have in my CMake builds which don't fit the
> 'many -> one' target mapping:
>

> - Doxygen (many -> many indeterminate filepaths);

> - A target to generate a tarball+patch for the current tree (tarball
> has HEAD; patch has `git diff HEAD` output iff there's a diff);

This is meta-version control stuff. I would keep it out of your build setup.

> - Updating .git/hooks;

Ah yes. The git devs (linus himself?) make a good case for custom
hooks not being versioned (or rather, not having them installed by
default). I keep my hooks in shell scripts. If you run them manually
they set up simlinks to themselves as hooks. If run by git, they act
as hooks.

> - Dynamic test generation (for each shared library created, make sure
> that it can be dlopen'd without error).

You could have your tests in a separate subdirectory. Then you can
just build the main part of your program when you want in its
subdirectory. For this application I would think you'd want your test
program to spit out reports that are known to tup. Then when you
change something, only the relevant tests get re-run. (running the
tests becomes the same as building the test subdirectory)

> I'd be interested to know what Tup can do about these cases.
>
> For the record, I'm interested in build systems as a packager; I don't
> believe Tup can fulfill the use cases I currently require that CMake can
> do for most of my personal projects.
>
> Lacking a proper install framework is going to make any Tup projects a
> nightmare for packagers. The nicest systems I've seen that aren't
> essentially language-specific (e.g., Python's distutils or Haskell's
> cabal) are CMake and autoconf (much as I dislike it, controlling its
> install directories are a breeze). Custom Makefiles span the entire
> spectrum from worst to best mainly because developers have to write
> stuff from nothing. I'd like Tup to help make things easy for packagers
> of Tup-using projects.
>
> --Ben

I think the future for open source software will be to migrate to
something like nix/nixos plus something for much better
automated/distributed monitoring of package dependencies and
conflicts. The current model of having separate communities of people
doing packaging manually (i.e. all the awesome debian folks) seems
tragically wasteful to me, and results in an extremely brittle
software ecosystem. Hopefully someday soon Linus will get sick of
seeing everyone wasting their time on different linux distributions,
and create (or bless) a system that will unify them all.