The Tupfiles/Tuprules.tup files are limited in such a way that it
should always be possible to compose a superproject using subprojects
that individually build with tup (though there may be issues with
conflicting variables, as you alluded to). This is why tup doesn't
define a TUP_ROOT variable that points to where the .tup directory is
initialized, but instead forces you do define your own _ROOT variables
using $(TUP_CWD) (which is always relative). For example, you can
build tup by running 'tup init' inside the tup/ directory, or you
could run 'tup init' in a directory above tup, and it will still build
successfully.
However, tup assumes there is only one database for the whole project.
It is not able to compose multiple databases (that would probably be a
nightmare given tup's implementation). Although it could probably
ignore subdirectories that have .tup directories in them, you'd drive
yourself crazy trying to remember in which subdirectories you have to
run 'tup upd' in order to update things.
There are two options I would suggest trying - I don't know what other
issues you might encounter with them:
1) Keep tup as-is, and move the top of your project down one level (so
you'd have tup_src/.tup, and my_src/.tup)
2) Change tup's bootstrap.sh so it doesn't run ./build/tup init and
./build/tup/upd. You would just end up with the bootstrapped tup in
tup_src/build, and you should be able to have a single .tup directory
under $myprog.
I'd like to know what you end up doing. Also, did you encounter any
specific issues with the Tupfiles under tup_src including the
Tuprules.tup file from your main project?
>
> [1] Since $myprog/tup_src is not a git repository, the 'git describe' at the
> end of tup's Tupfile seemed to fail due to a missing chdir() implementation
> in tup-ldpreload.so.
> I just fixed this problem locally by changing "version=`git describe`" into
> a hardcoded value corresponding to the tup version that I imported into my
> program's repository: "version="v0.3-148-ge66f2dd"".
Even though it doesn't support chdir(), the git describe would
probably still fail (since git is just searching up the tree for a
.git directory). Are you using git for your project? If so, you can
make tup's git repository a submodule of your main git repository.
-Mike
The Tupfiles/Tuprules.tup files are limited in such a way that itshould always be possible to compose a superproject using subprojects
that individually build with tup (though there may be issues with
conflicting variables, as you alluded to).
However, tup assumes there is only one database for the whole project.
It is not able to compose multiple databases (that would probably be a
nightmare given tup's implementation).
There are two options I would suggest trying - I don't know what other
issues you might encounter with them:
1) Keep tup as-is, and move the top of your project down one level (so
you'd have tup_src/.tup, and my_src/.tup)
2) Change tup's bootstrap.sh so it doesn't run ./build/tup init and
./build/tup/upd. You would just end up with the bootstrapped tup in
tup_src/build, and you should be able to have a single .tup directory
under $myprog.
I'd like to know what you end up doing. Also, did you encounter any
specific issues with the Tupfiles under tup_src including the
Tuprules.tup file from your main project?
Even though it doesn't support chdir(), the git describe would
> I just fixed this problem locally by changing "version=`git describe`" into
> a hardcoded value corresponding to the tup version that I imported into my
> program's repository: "version="v0.3-148-ge66f2dd"".
probably still fail (since git is just searching up the tree for a
.git directory). Are you using git for your project? If so, you can
make tup's git repository a submodule of your main git repository.
I'd like to know what you end up doing. Also, did you encounter anyspecific issues with the Tupfiles under tup_src including the
Tuprules.tup file from your main project?
Yes, I think so. For example, I use a different !cc macro and different CFLAGS.
It seems I'm still running into the same issue, even with just one tup database:
As it currently is, the Tuprules.tup files are parsed in order of the
directory hierarchy, so the variables are assumed to go from more
generic to more specific. If you define a !cc macro at the top of your
project, it can be overridden by any sub-project. That doesn't
necessarily mean they conflict, though. Anything inside that
sub-project will use it's particular !cc macro, and anything outside
of it will use the default top-level macro. I'm not claiming it's
perfect, so if you come across any specific issues feel free to report
them.
>
>> However, tup assumes there is only one database for the whole project.
>> It is not able to compose multiple databases (that would probably be a
>> nightmare given tup's implementation).
>
> This may be a stupid idea, but what if tup would always try to go up the
> directory tree up until the root directory (/) trying to find a database, so
> that it always finds the topmost one, regardless if there are others
> in-between?
>
> Not that this is a big issue for my specific case. As you mentioned, for my
> case I can just delete the "tup init" from bootstrap.sh.
Sounds reasonable to me. Though the test harness would have to be
changed, because each test case creates its own database underneath
the one used to build tup itself. Might be more effort than it's
worth.
As you mention later, this is not because of re-using !cc and such.
Whenever you wipe out a database, you clear out all record that tup
ever did any work, so you need to start over from scratch with a fresh
checkout of your project. (Wiping the database is not a normal
use-case of tup). The particular error message you are hitting is in
place so you don't accidentally create a rule where you overwrite a
file you just spent a long time creating but haven't bothered to
check-in yet. Tup will only automatically delete files that it has
created and are no longer used.
>
>>
>> > I just fixed this problem locally by changing "version=`git describe`"
>> > into
>> > a hardcoded value corresponding to the tup version that I imported into
>> > my
>> > program's repository: "version="v0.3-148-ge66f2dd"".
>>
>> Even though it doesn't support chdir(), the git describe would
>> probably still fail (since git is just searching up the tree for a
>> .git directory). Are you using git for your project? If so, you can
>> make tup's git repository a submodule of your main git repository.
>
> Indeed. I'm using git as well, so using a git submodule would likely work.
> That said, after reading the submodule documentation, I made a deliberate
> decision to import the code into my repo and hardcode the version, rather
> than use a submodule.
>
> I'd probably use a submodule if there was a way for a "git clone" of my
> repository to automatically clone the tup subrepo as well.. Perhaps someday
> I will switch to submodules if/when they support that.
You mean you just want 'git clone' to do everything, instead of 'git
clone && git submodule init && git submodule update' or whatever it
is? The submodule will be tied to a specific version in git, so your
users won't be somehow pulling in later versions of tup than what you
have tested with (unless they do so intentionally).
-Mike
Cool.. if defining a macro in a directory replaces the same macro that
was defined in an upper directory, then that should work pretty well.
The only potential problem that I see in this particular case would be
when the sub-project uses a macro that he didn't define, but the
super-project did (e.g. someone makes a typo, and instead of writing "!
gcc" he writes "!cc" instead).
If they were separate projects, it would perhaps lead to an undefined
macro error, I guess? But with both projects combined, it wouldn't lead
to such an error (but could lead to other errors if e.g. the "!cc" in
the super-project is not equivalent to "!gcc" in the sub-project that
the user intended to use).
Personally, I don't think this is a big problem, though... you can
always make sure that the sub-project works by itself, by for example,
moving its directory to /tmp and compiling there.
> I'm not claiming it's
> perfect, so if you come across any specific issues feel free to report
> them.
There's one issue that I potentially see in my particular case: for
example, tup's own "Tuprules.tup" starts defining CFLAGS with this line:
CFLAGS += -Os
The way I understand it, this means that it won't override my
super-project's CFLAGS, but instead it will just append tup's CFLAGS to
my project's CFLAGS.
This means that my super-project is influencing the sub-project's flags,
which may be a problem depending on the flags that I decide to use for
my project.
That said, I'm not running into any issue because of this (in fact, I'm
circumventing this problem, more below), it's just an observation that
I'm making.
> As you mention later, this is not because of re-using !cc and such.
> Whenever you wipe out a database, you clear out all record that tup
> ever did any work, so you need to start over from scratch with a fresh
> checkout of your project. (Wiping the database is not a normal
> use-case of tup). The particular error message you are hitting is in
> place so you don't accidentally create a rule where you overwrite a
> file you just spent a long time creating but haven't bothered to
> check-in yet. Tup will only automatically delete files that it has
> created and are no longer used.
Interesting... I'm still trying to get used to tup's philosophy of not
allowing to clean build products.. maybe this is just a mental barrier
that we'll all overcome someday :-)
The other mental barrier I have is not being able to do partial builds,
i.e. telling tup to build only one thing, rather than everything.
This means that things like "make dist" can't be done with tup, but need
to be done in a separate script instead. Otherwise, tup would always
build a tarball of the project whenever I changed a single line of
code.. :-)
> > I'd probably use a submodule if there was a way for a "git clone" of my
> > repository to automatically clone the tup subrepo as well.. Perhaps someday
> > I will switch to submodules if/when they support that.
>
> You mean you just want 'git clone' to do everything, instead of 'git
> clone && git submodule init && git submodule update' or whatever it
> is?
Exactly.
> The submodule will be tied to a specific version in git, so your
> users won't be somehow pulling in later versions of tup than what you
> have tested with (unless they do so intentionally).
Right. I'm just importing the code into my repo to simplify things for
potential users, not that there's anything technically wrong with
submodules.
Thanks,
Ricardo
The other mental barrier I have is not being able to do partial builds,
i.e. telling tup to build only one thing, rather than everything.
I think Ricardo's case of building or not building a tarball can be
adequately handled with a tup.config variable. For example, by default
tup's own Tupfiles do not build the web pages, but can be enabled by
creating tup.config and setting CONFIG_TUP_WWW=y. You could make up a
similar variable, like CONFIG_MYPROJ_TARBALL, and only create the rule
to make the tarball if that variable is set.
Jed, for your case, I don't think config variables would help much.
There isn't any technical reason why tup can't support a partial
update, it just hasn't been done yet. It would just need to build the
partial DAG as it does now, and then I guess prune it to contain only
the paths that have the desired targets. Trying to work out the
details...
-Mike
Ok, this turned out to be a bit easier than I expected (at least,
after staring at a DAG for an hour). The latest has support for
partial updates. They aren't documented, but should behave as you'd
expect (ie: 'tup upd foo.o bar.o' will only build what is necessary to
get those two objects). If the file you request wasn't going to be
built anyway, it will tell you that. Also the number of commands that
were skipped gets displayed at the end (assuming the part you
requested built successfully). Let me know if you come across any
issues with it.
-Mike
Thanks, this works for me and makes Tup a much more useful tool.
I think the last particularly notable limitation relative to other
systems is a way to handle multiple independent build directories,
sometimes called "variant builds". The issue is that I may want to
build a debug and optimized version, or two versions compiled using
different compilers or different implementations of dependent libraries,
or just with different configuration options. Tup currently insists
that targets be produced in the source tree which pretty much means that
you can only have one build at a time.
Jed
Hi Jed,
I've been trying to think of a good way to do variants for a while,
but haven't actually tried implementing it yet. Based on the way tup
is currently, I think it would take the form of separate @-variable
namespaces, which would then get built into separate subdirectories
(the default would use tup.config and build in the source tree as
now). So if you wanted an in-source optimized build, and a debug build
in a separate tree, you might have:
tup.config:
CONFIG_MY_PROJ_CFLAGS="-O2"
debug.config:
CONFIG_MY_PROJ_CFLAGS="-g"
There might need to be some file describing where the variants should go, like:
tup.variants:
default=tup.config
debug=debug.config
I don't know how that would play out in practice though. Maybe I'll
give it a whirl when I have some spare time. What are your thoughts on
it?
-Mike
Hi Jed,
I've been trying to think of a good way to do variants for a while,
but haven't actually tried implementing it yet. Based on the way tup
is currently, I think it would take the form of separate @-variable
namespaces, which would then get built into separate subdirectories
(the default would use tup.config and build in the source tree as
now). So if you wanted an in-source optimized build, and a debug build
in a separate tree, you might have:
tup.config:
CONFIG_MY_PROJ_CFLAGS="-O2"
debug.config:
CONFIG_MY_PROJ_CFLAGS="-g"
There might need to be some file describing where the variants should go, like:
tup.variants:
default=tup.config
debug=debug.config
I don't know how that would play out in practice though. Maybe I'll
give it a whirl when I have some spare time. What are your thoughts on
it?
Not entirely sure what you mean here. The tup.config file format is
from kconfig, so while you could edit them directly, it'd probably be
nicer to use kconfig to write them. That way you can list out all of
the options for your project in a Kconfig file and provide help text
and such. But if the developer didn't have support in the Tupfiles for
supplying CFLAGS in tup.config, then an end-user wouldn't be able to
create a variant with different CFLAGS (unless they also modify the
Tupfiles).
I don't know if tup.variants would be necessary or not - maybe it
could just wildcard files with a certain extension instead.
>
>>
>> I don't know how that would play out in practice though. Maybe I'll
>> give it a whirl when I have some spare time. What are your thoughts on
>> it?
>
> In the general case where compiler support and properties of dependencies
> need to be discovered (supported features, how to link, etc), I think that
> some configure process in unavoidable. This does not need to be autoconf,
> which I generally hate, but something is needed. The model I prefer is
> cd project
> mkdir build-foo && cd build-foo
> ../configure --lots-of-options --specific-to-this --variant # Writes a
> "config.h" and some variables for Tup.
> tup upd # all build output goes below the current directory
I think in tup this would end up being something like:
cd project
tup variant add build-foo MY_PROJ_CFLAGS=-blah
(or maybe this brings up a kconfig editor... in any case, a new
.config file and tup.variants file are created as necessary)
tup upd
(now files are created in build-foo with @(MY_PROJ_CFLAGS) set to -blah)
You could still rm -rf the directory and corresponding .config file,
or maybe a 'tup variant rm build-foo' would be a wrapper for that.
> This allows an arbitrary number of variants to coexist, with options as
> defined by the user, and leaves the source directory pristine which makes it
> easy to blast it away when you no longer need it (just just delete
> build-foo).
> Note that this inverts the control model your proposed above. Instead of
> having one top-level that needs to know about all possible variants, each
> build directory has a link back to the source directory. With one project
> that I work on, I have over twenty of these variant builds, 6-10 of which I
> use on any particular day, and I guarantee that they are sufficiently
> diverse that nobody would want to maintain a single file describing all of
> them.
> Jed
>
The top-level wouldn't need to know about all possible variants, just
whichever ones are active. In your project you might just have a
default variant, or maybe a few templates, but the end-user could
always create new ones.
What sort of variants are you using? Is it a combination of different
compiler flags and feature sets?
Thanks,
-Mike
I think in tup this would end up being something like:
cd project
tup variant add build-foo MY_PROJ_CFLAGS=-blah
(or maybe this brings up a kconfig editor... in any case, a new
.config file and tup.variants file are created as necessary)
tup upd
(now files are created in build-foo with @(MY_PROJ_CFLAGS) set to -blah)
You could still rm -rf the directory and corresponding .config file,
or maybe a 'tup variant rm build-foo' would be a wrapper for that.
The top-level wouldn't need to know about all possible variants, just
whichever ones are active. In your project you might just have a
default variant, or maybe a few templates, but the end-user could
always create new ones.
What sort of variants are you using? Is it a combination of different
compiler flags and feature sets?
At least initially I wouldn't try to support sharing files between
variants due to the added complexity (tup doesn't have any knowledge
to do that sort of thing at the moment). The primary benefit to having
them inside one tup database would be the parallel updates.
>
>>
>> You could still rm -rf the directory and corresponding .config file,
>> or maybe a 'tup variant rm build-foo' would be a wrapper for that.
>
> Could tup garbage collect the .config when the build directory is deleted?
> Or even put the .config into the build directory? The reason I ask is
> because we already have some complexity through git rm and such, I would be
> cautious about adding additional special commands to remove something.
That's a good idea. Putting the file in the build directory sounds
like the right approach to me.
Thanks for the feedback!
-Mike