Make vs tup - again

[Gerardo Delgadillo]

I'm porting our ugly C++ Eclipse projects to tup. We have many-many libraries and executables. So, as a test, I ported a few projects to tup and it works great. However, a co-worker said why not use make instead, and I answered with "auto-dependencies and speed." He agrees with the speed part, but he states that make can do the dependency business, if you provide the right info, like *.cpp, *.h, *.d, etc. and using the GNU -MD flag to generate .d files.

What's your take on this eternal-debate?

[Ben Boeckel]

You also miss reruns on flag changes and command line updates with Make. Getting that right is tricky and can make things even slower (file per compilation -> slow due to umpteen thousand tiny files, file per target -> inefficient build potential, but ok if you have uniform flags per library/executable).

I'll note that, AFAIK, Tup has no answer for C++20 modules yet. Unity builds could help, but consuming external modules it going to need build system support. I posted about it a few months back.

(Full disclosure: I'm a CMake developer and am also involved with the C++ standards committee; here because I'm interested in build tools generally and can at least help answer this.)

--Ben

[Ben Boeckel]

On December 19, 2019 8:15:36 PM EST, Ben Boeckel <math...@gmail.com> wrote:
>You also miss reruns on flag changes and command line updates with
>Make. Getting that right is tricky and can make things even slower
>(file per compilation -> slow due to umpteen thousand tiny files, file
>per target -> inefficient build potential, but ok if you have uniform
>flags per library/executable).

Ah, Tup also handles environment variable changed. Not sure I know of other build executors which do that off-hand.

--Ben

[Robb Matzke]

Hi Gerardo,

Besides knowing about the dependencies that "gcc -MD" would
report, Tup's build artifacts also depend on the commands that
build them -- change a command in the Tup file or through
configuration or environment variables and the artifact is
rebuilt. Even better, if the command is some script that's in your
build tree, changing the script itself (or even anything in the
build tree that the script uses!) will rebuild the artifact. Tup
also handles dependencies well for generated source code, if you
do that. And none of this is dependent on some magic,
compiler-specific command-line switch.

--Robb

> --

[Gerardo Delgadillo]

First, let me apologize for posting the same question twice--didn't realize that the system takes a while to authorize new posts.

Thank you for replying! Interesting that you mention you're a CMake developer. I'm also considering CMake. I like that it abstracts the underlying tools and architecture, so I can chose Ninja, for example. But I don't know how CMake handles dependencies, so there's that.

[Gerardo Delgadillo]

This is excellent info I can use to defend my use case with my co-worker! Thanks so much.

[Ben Boeckel]

On Fri, Dec 20, 2019 at 07:23:15 -0800, Gerardo Delgadillo wrote:
> First, let me apologize for posting the same question twice--didn't realize
> that the system takes a while to authorize new posts.
>
> Thank you for replying! Interesting that you mention you're a CMake
> developer. I'm also considering CMake. I like that it abstracts the
> underlying tools and architecture, so I can chose Ninja, for example. But I
> don't know how CMake handles dependencies, so there's that.

Dependencies in what way? External package dependencies or internal
inter-target dependencies?

--Ben

[Gerardo Delgadillo]

Mostly h and cpp files, really. The Eclipse build system is horrible at that, and we want to get out of that nightmare. I.e. change a library cpp file and executables using that library may or may not get re-linked. Even worse: Sometimes Eclipse builds everything just because!

[Ben Boeckel]

On Fri, Dec 20, 2019 at 07:24:31 -0800, Gerardo Delgadillo wrote:
> This is excellent info I can use to defend my use case with my co-worker!
> Thanks so much.

One caveat to be aware of with Tup: commands with unknown outputs are
not supported. For example, tar/zip extraction, git checkout, git clone,
other VCS operations cannot happen within Tup. I believe there have been
issues with things like ccache getting caught up in there, but I think
you can now poke holes for that in Tup so it ignores cache directories.

--Ben

[Ben Boeckel]

On Fri, Dec 20, 2019 at 08:26:41 -0800, Gerardo Delgadillo wrote:
> Mostly h and cpp files, really. The Eclipse build system is horrible at
> that, and we want to get out of that nightmare. I.e. change a library cpp
> file and executables using that library may or may not get re-linked. Even
> worse: Sometimes Eclipse builds everything just because!

The Ninja generator is definitely the most well-behaved CMake generator
for efficient builds. It uses `-MD` and `/showIncludes`, so its C++
dependency list computation is "exact" in that sense. The Makefiles
generator has an over-building tendency, but that's due to the command
line tracking making per-source flags cause rebuilds of all objects in
the library or executable (the "lots of tiny files" problem I mentioned
before).

--Ben

[Gerardo Delgadillo]

Oh, good to know!

[Robb Matzke]

Hi Ben,

Out of curiosity and since the OP is comparing Tup with CMake, do you
have any experience with CMake+Ninja compiling generated source code?

For instance, my project has some large C++ files that get generated as
part of the build process, and those source files #include additional
project headers, some of which might also be generated. In fact, the
generator itself is built as part of the project. Does CMake + Ninja
discover that the generated C++ files depend on certain headers so that
if I change one of the headers then the generated C++ gets recompiled
without necessarily regenerating it first? In other words, does
CMake+Ninja run the dependency-discovery step just once at the beginning
(before generated files are generated) or does it know to run the
dependency-discovery during the build whenever source is newly generated?

--Robb

[Ben Boeckel]

On Fri, Dec 20, 2019 at 12:02:17 -0500, Robb Matzke wrote:
> Out of curiosity and since the OP is comparing Tup with CMake, do you
> have any experience with CMake+Ninja compiling generated source code?

They're comparing it with hand-written Makefiles I believe, not CMake
specifically.

> For instance, my project has some large C++ files that get generated as
> part of the build process, and those source files #include additional
> project headers, some of which might also be generated. In fact, the
> generator itself is built as part of the project. Does CMake + Ninja
> discover that the generated C++ files depend on certain headers so that
> if I change one of the headers then the generated C++ gets recompiled
> without necessarily regenerating it first? In other words, does
> CMake+Ninja run the dependency-discovery step just once at the beginning
> (before generated files are generated) or does it know to run the
> dependency-discovery during the build whenever source is newly generated?

In all the generators, dependencies of sources are discovered at build
time. Ninja uses `-MD` or `/showIncludes`. For historical reasons, the
Makefiles generators use a built-in mechanism by CMake that is only
inaccurate in that it overguesses if anything (there is a plan for
updating them to work like Ninja, but that isn't on a schedule yet). The
IDE generators do what the IDEs do, but have the same behavior if not
mechanism.

I recommend further CMake-specific discussion on the CMake Discourse
rather than on the Tup list.

https://discourse.cmake.org/

--Ben