Linking and directories

[booga]

Hey folks, I was wondering how you deal with the case where a binary
needs to be generated from objects in multiple directories? Also, It
seems wrong that I can't have the output rules put files in a new
directory?

[Mike Shal]

Linking objects from multiple directories should work. Use a rule like:

: *.o foo/*.o |> gcc %f -o %o |> prog

This will link all the objects in the current directory (*.o) and the
foo/ directory (foo/*.o) into 'prog'. You will of course need rules in
the current directory and in foo to compile those files before trying
to link them. Are you having specific issues with this?

Tup doesn't create directories or write files to different
directories. If you want to generate a file in a particular directory,
make the directory and put a Tupfile there. I haven't found a need to
write files to other directories yet.

-Mike

[Abdulla Kamar]

I'd actually like to see support for output in to separate directories. I use this to separate debug and release builds, but also to do things like tell Time Machine not to back up the build directory (which would be in high churn during development). It also means I don't have object files littering the directories with source code in them.

I see what you mean by just putting a Tupfile in the directory and telling it to build the binaries in there, but I like the idea of having a central point of control without Tupfiles litterring the directories.

--
tup-users mailing list
email: tup-...@googlegroups.com
unsubscribe: tup-users+...@googlegroups.com
options: http://groups.google.com/group/tup-users?hl=en

--
Thank you
Abdulla

[Mike Shal]

On 3/12/10, Abdulla Kamar <abdull...@gmail.com> wrote:
> I'd actually like to see support for output in to separate directories. I
> use this to separate debug and release builds, but also to do things like
> tell Time Machine not to back up the build directory (which would be in high
> churn during development). It also means I don't have object files littering
> the directories with source code in them.

I don't think tup will ever be able to handle having a Tupfile write
the output of a command into a different directory. For example, this
will probably never work:
: input |> command |> foo/output
but you can always do this in the foo/ directory:
: ../input |> command |> output

Though this is different from the concept of a completely separate
build directory, where a mirrored directory structure is setup for all
generated files. That may be possible, but it seems tricky to get the
paths right for generating the commands. Also the Tupfiles would need
knowledge of the separate build directory, since passing in a compiler
flag of "-I." may really need to be "-I. -I$(BUILD_DIR)" if a header
file is generated in a separate build directory.

Do you actually use Time Machine for your source code? I've never used
it myself so I don't know of its capabilities, but I would think an
actual version control system would be more useful in that you'd have
commit messages and branching and such. In other words, why not ignore
the whole development tree instead of just the build directory?

>
> I see what you mean by just putting a Tupfile in the directory and telling
> it to build the binaries in there, but I like the idea of having a central
> point of control without Tupfiles litterring the directories.

One man's litter is another's gold mine of info about what actually got built :)

-Mike

[Abdulla Kamar]

Do you actually use Time Machine for your source code? I've never used
it myself so I don't know of its capabilities, but I would think an
actual version control system would be more useful in that you'd have
commit messages and branching and such. In other words, why not ignore
the whole development tree instead of just the build directory?

I use git for version control, but Time Machine is backs up my whole hard drive every hour. In order to not pollute the backups with unneeded files, I tell it to ignore the build output directory. And sadly it seems to have limited granularity with exclusions (directory based, no regular expressions).

 

One man's litter is another's gold mine of info about what actually got built :)

Oh don't get me wrong, I want to know what got built, I just like having one build script that I can modify. Otherwise I find things seem to get out of sync because there's always one file I missed, or something I didn't notice.

--
Thank you
Abdulla

[William Egert]

Yea, I had figured out the answer to my linking question after I posted this :P
That is an interesting way around the separate directories issue.  One downfall i see to this is there is no easy way to "make install" from tup.  At least nothing obvious to me yet..

-Bill

[Mike Shal]

On 3/15/10, William Egert <beg...@gmail.com> wrote:
> Yea, I had figured out the answer to my linking question after I posted this
> :P
> That is an interesting way around the separate directories
> issue. One downfall i see to this is there is no easy way to "make install"
> from tup. At least nothing obvious to me yet..

Nope, there is no concept of targets in tup. If you want it to handle
installing the end results of things into a final directory (like all
binaries in a bin/ directory), then you'll need to put a Tupfile there
that pulls everything in using 'cp' or something. Otherwise this is
probably better handled by the package manager for your system.

-Mike

[Jed Brown]

On Mar 15, 6:52 am, Mike Shal <mar...@gmail.com> wrote:
> I don't think tup will ever be able to handle having a Tupfile write
> the output of a command into a different directory. For example, this
> will probably never work:
> : input |> command |> foo/output
> but you can always do this in the foo/ directory:
> : ../input |> command |> output

I like my builds to be completely outside of the source tree because I
usually have several of them (different compiler suites, different MPI
implementations, debug and optimized, plus different subsets of about
50 optional dependencies). It would seem that the tup model requires
that my configure system create a build tree populated with tupfiles
that build sources residing in the source tree. But what happens when
I update a Tupfile "template" in the source tree, or even add a new
directory with a new tupfile? Seems like I would need the tupfile
copies in the build tree to depend on these templates in the source
tree. Is it a problem to have the tupfiles themselves depend on
filesystem metadata from outside of the build tree? Or are we with
manually rerunning configure (for every configuration) if these
templates are modified?

Jed

[Mike Shal]

On 5/6/10, Jed Brown <j...@59a2.org> wrote:
> On Mar 15, 6:52 am, Mike Shal <mar...@gmail.com> wrote:
> > I don't think tup will ever be able to handle having a Tupfile write
> > the output of a command into a different directory. For example, this
> > will probably never work:
> > : input |> command |> foo/output
> > but you can always do this in the foo/ directory:
> > : ../input |> command |> output
>
>
> I like my builds to be completely outside of the source tree because I
> usually have several of them (different compiler suites, different MPI
> implementations, debug and optimized, plus different subsets of about
> 50 optional dependencies). It would seem that the tup model requires
> that my configure system create a build tree populated with tupfiles
> that build sources residing in the source tree. But what happens when
> I update a Tupfile "template" in the source tree, or even add a new
> directory with a new tupfile? Seems like I would need the tupfile
> copies in the build tree to depend on these templates in the source
> tree. Is it a problem to have the tupfiles themselves depend on
> filesystem metadata from outside of the build tree? Or are we with
> manually rerunning configure (for every configuration) if these
> templates are modified?

Tup doesn't support generating the Tupfiles themselves from other
programs. If you do that, you would need to run a separate program to
generate them before running 'tup upd'. I would hope that tup is
expressive enough that you don't feel the need to generate the
Tupfiles, though.

When you are building these variants (like debug vs optimized) do you
generally only build one of them at a time? Or do you build one
variant, and then re-configure to the other variant? In either case
you'd probably want to have your configuration system (like kconfig)
write out a tup.config file, and then use @-variables in the Tupfiles
to determine what to build. This way tup can parse the minimal amount
of Tupfiles and re-build the minimum amount of files based on which
@-variables actually changed. See this thread:
http://groups.google.com/group/tup-users/browse_frm/thread/043f0bd5994259be#

-Mike

[Jed Brown]

On Thu, 6 May 2010 11:47:53 -0400, Mike Shal <mar...@gmail.com> wrote:
> Tup doesn't support generating the Tupfiles themselves from other
> programs. If you do that, you would need to run a separate program to
> generate them before running 'tup upd'. I would hope that tup is
> expressive enough that you don't feel the need to generate the
> Tupfiles, though.

There are related issues of portability (e.g. obscure Unix without
shared libraries, Windows, etc) and using the native build systems from
various IDEs.

> When you are building these variants (like debug vs optimized) do you
> generally only build one of them at a time? Or do you build one
> variant, and then re-configure to the other variant?

I usually keep the variants in separate build directories and update
whichever one I'm actively testing. When I'm ready for a production
run, or want to test with a different configuration, I run make from the
other directory. I sometimes run programs from these different trees
simultaneously (e.g. the same program built two different ways open in a
debugger).

> In either case you'd probably want to have your configuration system
> (like kconfig) write out a tup.config file, and then use @-variables
> in the Tupfiles to determine what to build. This way tup can parse the
> minimal amount of Tupfiles and re-build the minimum amount of files
> based on which @-variables actually changed. See this thread:
> http://groups.google.com/group/tup-users/browse_frm/thread/043f0bd5994259be#

I saw that, it's not really a solution because the whole project has to
be rebuilt if you change from debug to optimized. Also, we normally
generate a "config.h" in each build tree (rather than pass hundreds of
-D options on the command line). Since the whole project includes that,
changing to a different configuration would result in a full rebuild
even if the command line arguments didn't change.

Jed

[Mike Shal]

On 5/6/10, Jed Brown <j...@59a2.org> wrote:

> On Thu, 6 May 2010 11:47:53 -0400, Mike Shal <mar...@gmail.com> wrote:
> > Tup doesn't support generating the Tupfiles themselves from other
> > programs. If you do that, you would need to run a separate program to
> > generate them before running 'tup upd'. I would hope that tup is
> > expressive enough that you don't feel the need to generate the
> > Tupfiles, though.
>
>
> There are related issues of portability (e.g. obscure Unix without
> shared libraries, Windows, etc) and using the native build systems from
> various IDEs.

Tup itself isn't that portable yet - it only runs on linux and
opensolaris I think. It is tricky getting the file accesses of
subprograms on other OSes.

>
>
> > When you are building these variants (like debug vs optimized) do you
> > generally only build one of them at a time? Or do you build one
> > variant, and then re-configure to the other variant?
>
>
> I usually keep the variants in separate build directories and update
> whichever one I'm actively testing. When I'm ready for a production
> run, or want to test with a different configuration, I run make from the
> other directory. I sometimes run programs from these different trees
> simultaneously (e.g. the same program built two different ways open in a
> debugger).
>
>
> > In either case you'd probably want to have your configuration system
> > (like kconfig) write out a tup.config file, and then use @-variables
> > in the Tupfiles to determine what to build. This way tup can parse the
> > minimal amount of Tupfiles and re-build the minimum amount of files
> > based on which @-variables actually changed. See this thread:
> > http://groups.google.com/group/tup-users/browse_frm/thread/043f0bd5994259be#
>
>
> I saw that, it's not really a solution because the whole project has to
> be rebuilt if you change from debug to optimized. Also, we normally
> generate a "config.h" in each build tree (rather than pass hundreds of
> -D options on the command line). Since the whole project includes that,
> changing to a different configuration would result in a full rebuild
> even if the command line arguments didn't change.

I don't understand - currently if you change config.h, do you not
rebuild everything that includes it?

-Mike

[Jed Brown]

On Thu, 6 May 2010 14:21:46 -0400, Mike Shal <mar...@gmail.com> wrote:
> On 5/6/10, Jed Brown <j...@59a2.org> wrote:
> > On Thu, 6 May 2010 11:47:53 -0400, Mike Shal <mar...@gmail.com> wrote:
> > > Tup doesn't support generating the Tupfiles themselves from other
> > > programs. If you do that, you would need to run a separate program to
> > > generate them before running 'tup upd'. I would hope that tup is
> > > expressive enough that you don't feel the need to generate the
> > > Tupfiles, though.
> >
> >
> > There are related issues of portability (e.g. obscure Unix without
> > shared libraries, Windows, etc) and using the native build systems from
> > various IDEs.
>
> Tup itself isn't that portable yet - it only runs on linux and
> opensolaris I think. It is tricky getting the file accesses of
> subprograms on other OSes.

This is one reason to generate tupfiles rather than write them by hand.

> I don't understand - currently if you change config.h, do you not
> rebuild everything that includes it?

I have a *different* config.h in each build tree. I'm not changing
config.h.in or whatever, just switching to a different build tree. If I
only have one build tree, then switching to the other build means
updating the one true config.h which necessarily triggers a full
rebuild.

Jed

[Mike Shal]

On 5/6/10, Jed Brown <j...@59a2.org> wrote:
> On Thu, 6 May 2010 14:21:46 -0400, Mike Shal <mar...@gmail.com> wrote:
> > On 5/6/10, Jed Brown <j...@59a2.org> wrote:
> > > On Thu, 6 May 2010 11:47:53 -0400, Mike Shal <mar...@gmail.com> wrote:
> > > > Tup doesn't support generating the Tupfiles themselves from other
> > > > programs. If you do that, you would need to run a separate program to
> > > > generate them before running 'tup upd'. I would hope that tup is
> > > > expressive enough that you don't feel the need to generate the
> > > > Tupfiles, though.
> > >
> > >
> > > There are related issues of portability (e.g. obscure Unix without
> > > shared libraries, Windows, etc) and using the native build systems from
> > > various IDEs.
> >
> > Tup itself isn't that portable yet - it only runs on linux and
> > opensolaris I think. It is tricky getting the file accesses of
> > subprograms on other OSes.
>
>
> This is one reason to generate tupfiles rather than write them by hand.

I don't follow your logic here. I mean that the tup executable isn't
completely portable, not that Tupfiles can't be portable. I see no
reason to generate Tupfiles from a configure step (from a Tupfile.in
or whatever). I have no issue if you want to generate them from a
build-configuration-generator like CMake, but all that buys you is
that you can switch between build systems more easily.

>
>
> > I don't understand - currently if you change config.h, do you not
> > rebuild everything that includes it?
>
>
> I have a *different* config.h in each build tree. I'm not changing
> config.h.in or whatever, just switching to a different build tree. If I
> only have one build tree, then switching to the other build means
> updating the one true config.h which necessarily triggers a full
> rebuild.

Ahh, I see. I don't think there is an easy way to do this in tup yet.

-Mike

[Jed Brown]

On Fri, 7 May 2010 10:15:53 -0400, Mike Shal <mar...@gmail.com> wrote:
> On 5/6/10, Jed Brown <j...@59a2.org> wrote:
> > This is one reason to generate tupfiles rather than write them by hand.
>
> I don't follow your logic here. I mean that the tup executable isn't
> completely portable, not that Tupfiles can't be portable. I see no
> reason to generate Tupfiles from a configure step (from a Tupfile.in
> or whatever). I have no issue if you want to generate them from a
> build-configuration-generator like CMake, but all that buys you is
> that you can switch between build systems more easily.

It doesn't do much good to write Tupfiles by hand if you need to build
places that tup itself doesn't run.

> > I have a *different* config.h in each build tree. I'm not changing
> > config.h.in or whatever, just switching to a different build tree. If I
> > only have one build tree, then switching to the other build means
> > updating the one true config.h which necessarily triggers a full
> > rebuild.
>
> Ahh, I see. I don't think there is an easy way to do this in tup yet.

Okay, thanks. I think this is a pretty important feature. (For tools
like CMake as well.)

Jed