Issues with a sub-project/module

[Lars Conrad]

Hello, :-)

I'm a "tup first timer”. I decided to give it a try, after I got fed up with cmake taking more time than gcc in many daily tasks.

I stumbled into some problems right away, while trying it on a rather simple project (see the attached archive, source files have been zeroed). The structure is this:

"config_module" is the main (top) directory containing everything. As the name suggests, this is to be ported to C++20 modules in the near future.

Within this directory is a file called "test_config.cpp”, which is a unit test. This is to build the only executable called "test_config", preferably in three possible ways (debug, release and static).

Then there is the directory "src”. In this directory we have the actual config module. This is to be compiled into objects only:

config.hpp

config.cpp

platform_folders.hpp

platform_folders.cpp

I would like to have three build configurations, which are the same for the directories containing the "test_config", as well as the "src" directory:

debug.config: CONFIG_BUILD_TYPE=debug

release.config: CONFIG_BUILD_TYPE=release

static.config: CONFIG_BUILD_TYPE=static

Then I have my universal build rules in a file ("Tuprules.tup") in the "src" directory, which is to be included where it's needed:

# Tuprules.tup in config_module/src

ifeq (@(BUILD_TYPE),debug)

# debug build:

  CPPFLAGS += -march=native

  CPPFLAGS += -ggdb3

  CPPFLAGS += -D_GLIBCXX_DEBUG

else

  ifeq (@(BUILD_TYPE),release)

    CPPFLAGS += -march=native

    CPPFLAGS += -O2

    LDFLAGS += --as-needed -flto

  else

    ifeq (@(BUILD_TYPE),static)

      CPPFLAGS += -static -Os

      LDFLAGS += --as-needed -flto

    else

      error "Build type not set or wrong!"

    endif

  endif

endif

I did put a "Tupfile" in the "src" directory, which contained this:

# Tupfile in config_module/src

CPPFLAGS += -std=c++17

CPPFLAGS += -Wall

LDFLAGS += -lstdc++fs

include_rules

: foreach *.cpp |> g++ $(CPPFLAGS) -c %f -o %o |> %B.o

This works well when I call "init" in the "src" directory!

In the top directory (for building the unit test executable) I have this in a Tupfile:

# Tupfile in config_module

PROG_NAME = test_config

MY_CONFIG_FOLDER = $(TUP_CWD)/debug_config_folder

CPPFLAGS += -std=c++17

CPPFLAGS += -Wall

CPPFLAGS += -Isrc

include_rules

: foreach *.cpp |> g++ $(CPPFLAGS) -D$(MY_CONFIG_FOLDER) -c %f -o %o |> %B.o

#  doesn't work : *.o |> gcc %f -o %o -Wl,$(LDFLAGS) |> $(PROG_NAME)

: *.o |> gcc %f -o %o -Wl,$(LDFLAGS) |> test_config

When I remove the ".tup" from the "src" directory and instead "init" the top directory, neither work.

To be clear: When running "tup build-debug" in the "src" directory, I want it to compile "config.cpp" and "platform_folders.cpp" into ".o" files within the "build-debug" directory under the "src" directory.

When running "tup build-debug" in the top directory, it should compile "test_config.cpp" into "test_config.o" and then link it to "test_config", both into the top-level "build-debug" directory.

In case the dependencies from the sub-module/project (in "src") aren't compiled, it should do that first.

I won't even go into my attempts to put a "make clean" function in there. Read-only file access - that was funny... ;-)

Thank you very much for any help!

[Mike Shal]

On Tue, Jun 9, 2020 at 9:12 AM Lars Conrad <my...@gmx.de> wrote:

Hello, :-)

I'm a "tup first timer”. I decided to give it a try, after I got fed up with cmake taking more time than gcc in many daily tasks.

Hi Lars, welcome! I found a few potential issues that may be causing you problems. Please give them a shot, and if it's still not doing what you expect then feel free to write back and give another tar.gz of what you're working on (that is very helpful, thanks!)

 

I would like to have three build configurations, which are the same for the directories containing the "test_config", as well as the "src" directory:

debug.config: CONFIG_BUILD_TYPE=debug

release.config: CONFIG_BUILD_TYPE=release

static.config: CONFIG_BUILD_TYPE=static

To use tup's variants, you should just have the build configurations at one level at the top where you run 'tup init'. The symlinks you had for the top-level build configurations were broken, so I moved the tup-configs in src to the top-level to fix them (and remove them in src, since we just want them in the top-level):

$ mv src/tup-configs .

$ rm -rf src/build-*

 

Then I have my universal build rules in a file ("Tuprules.tup") in the "src" directory, which is to be included where it's needed:

# Tuprules.tup in config_module/src

ifeq (@(BUILD_TYPE),debug)

# debug build:

  CPPFLAGS += -march=native

  CPPFLAGS += -ggdb3

  CPPFLAGS += -D_GLIBCXX_DEBUG

else

  ifeq (@(BUILD_TYPE),release)

    CPPFLAGS += -march=native

    CPPFLAGS += -O2

    LDFLAGS += --as-needed -flto

  else

    ifeq (@(BUILD_TYPE),static)

      CPPFLAGS += -static -Os

      LDFLAGS += --as-needed -flto

    else

      error "Build type not set or wrong!"

    endif

  endif

endif

In LDFLAGS here you are mixing flags for ld (--as-needed) and flags for gcc (-flto). In your build rules you then use -Wl,$(LDFLAGS) which won't work if LDFLAGS isn't set (as is the case for the debug build). So I'd recommend doing LDFLAGS += -Wl,--as-needed -flto and just use $(LDFLAGS) in your linker rules.

I did put a "Tupfile" in the "src" directory, which contained this:

# Tupfile in config_module/src

CPPFLAGS += -std=c++17

CPPFLAGS += -Wall

LDFLAGS += -lstdc++fs

include_rules

: foreach *.cpp |> g++ $(CPPFLAGS) -c %f -o %o |> %B.o

Looks good!

In the top directory (for building the unit test executable) I have this in a Tupfile:

# Tupfile in config_module

PROG_NAME = test_config

MY_CONFIG_FOLDER = $(TUP_CWD)/debug_config_folder

CPPFLAGS += -std=c++17

CPPFLAGS += -Wall

CPPFLAGS += -Isrc

include_rules

Note that include_rules includes Tuprules.tup files in the current directory, and all directories up to the root (where .tup is created). Since your main Tuprules.tup file is in the src directory, it won't get included in this top-level Tupfile. I think you should move src/Tuprules.tup to the config_module directory (or higher, if you intend it to be shared by multiple modules).

 

: foreach *.cpp |> g++ $(CPPFLAGS) -D$(MY_CONFIG_FOLDER) -c %f -o %o |> %B.o

What are you intending to do with the -D$(MY_CONFIG_FOLDER)? This is a path name, which gets expanded to something like -D./debug_config_folder, which is probably not what you want. This made the compilation fail for me, and I'm not sure what it's trying to do, so I removed the -D flag.

 

#  doesn't work : *.o |> gcc %f -o %o -Wl,$(LDFLAGS) |> $(PROG_NAME)

: *.o |> gcc %f -o %o -Wl,$(LDFLAGS) |> test_config

As mentioned above, change -Wl,$(LDFLAGS) into just $(LDFLAGS) after making the adjustments in setting LDFLAGS to Tuprules.tup

 

When I remove the ".tup" from the "src" directory and instead "init" the top directory, neither work.

To be clear: When running "tup build-debug" in the "src" directory, I want it to compile "config.cpp" and "platform_folders.cpp" into ".o" files within the "build-debug" directory under the "src" directory.

With tup's variants, you get a build-debug directory at the top-level, with the directory structure duplicated underneath. So instead of build-debug in config_module and build_debug in src, you'd have one build_debug/ at the top, and src underneath (which gets created by tup as part of the build). So if you just want to build stuff in src for debug, run 'tup build_debug/src' at the top-level. To do the whole module as debug, 'tup build_debug', and to build all variants just do 'tup'.

 

When running "tup build-debug" in the top directory, it should compile "test_config.cpp" into "test_config.o" and then link it to "test_config", both into the top-level "build-debug" directory.

In case the dependencies from the sub-module/project (in "src") aren't compiled, it should do that first.

The good news is if you can get it to compile cleanly once, the dependencies part should work automatically :)

This patch applied to your example builds for me when running 'tup init' inside config_module, which includes the changes discussed above. Though again, if you plan to have multiple modules alongside config_module, you may want to run init & have your Tuprules.tup one level higher.

diff --git a/Tupfile b/Tupfile
index 3d46419..d7be7e4 100644
--- a/Tupfile
+++ b/Tupfile
@@ -9,5 +9,5 @@ CPPFLAGS += -Isrc
 
 include_rules
 
-: foreach *.cpp |> g++ $(CPPFLAGS) -D$(MY_CONFIG_FOLDER) -c %f -o %o |> %B.o
-: *.o |> gcc %f -o %o -Wl,$(LDFLAGS) |> $(PROG_NAME)
+: foreach *.cpp |> g++ $(CPPFLAGS) -c %f -o %o |> %B.o
+: *.o |> gcc %f -o %o $(LDFLAGS) |> $(PROG_NAME)
diff --git a/src/Tuprules.tup b/Tuprules.tup
similarity index 83%
rename from src/Tuprules.tup
rename to Tuprules.tup
index cea6f37..34acd3d 100644
--- a/src/Tuprules.tup
+++ b/Tuprules.tup
@@ -9,11 +9,11 @@ else

   ifeq (@(BUILD_TYPE),release)
     CPPFLAGS += -march=native
     CPPFLAGS += -O2

-    LDFLAGS += --as-needed -flto
+    LDFLAGS += -Wl,--as-needed -flto

   else
     ifeq (@(BUILD_TYPE),static)
       CPPFLAGS += -static -Os

-      LDFLAGS += --as-needed -flto
+      LDFLAGS += -Wl,--as-needed -flto

     else
       error "Build type not set or wrong!"
     endif

diff --git a/src/build-debug/tup.config b/src/build-debug/tup.config
deleted file mode 120000
index 84c56af..0000000
--- a/src/build-debug/tup.config
+++ /dev/null
@@ -1 +0,0 @@
-../tup-configs/debug.config
\ No newline at end of file
diff --git a/src/build-release/tup.config b/src/build-release/tup.config
deleted file mode 120000
index 98b9f34..0000000
--- a/src/build-release/tup.config
+++ /dev/null
@@ -1 +0,0 @@
-../tup-configs/release.config
\ No newline at end of file
diff --git a/src/build-static/tup.config b/src/build-static/tup.config
deleted file mode 120000
index 6bdbdf7..0000000
--- a/src/build-static/tup.config
+++ /dev/null
@@ -1 +0,0 @@
-../tup-configs/static.config
\ No newline at end of file
diff --git a/test_config.cpp b/test_config.cpp
index e69de29..b552c8e 100644
--- a/test_config.cpp
+++ b/test_config.cpp
@@ -0,0 +1 @@
+int main(void) {}
diff --git a/src/tup-configs/debug.config b/tup-configs/debug.config
similarity index 100%
rename from src/tup-configs/debug.config
rename to tup-configs/debug.config
diff --git a/src/tup-configs/release.config b/tup-configs/release.config
similarity index 100%
rename from src/tup-configs/release.config
rename to tup-configs/release.config
diff --git a/src/tup-configs/static.config b/tup-configs/static.config
similarity index 100%
rename from src/tup-configs/static.config
rename to tup-configs/static.config

-Mike

[Lars Conrad]

Thank you very much for your help, Mike! :-)

This now works. More about that below...

I can see a few problems in implementing tup as a build-system in IDEs. The "pushing everything to the top-level" approach works against best practices.

With object orientation we moved strongly into "divide and conquer". A problem too big, gets split into smaller problems, which then are split into even smaller problems, etc...

This is being reflected by the paradigm of modern programming languages, such as C++. Hence, you will find complex classes that are built upon inheritance and containment. These are typically managed as a project, which contains sub-projects. The IDEs support this paradigm.

It is important that the sub-projects can be seen as "independent units". Typically, they come with their own unit tests. They might also be used by other code.

Likewise, this is being reflected when working on projects. While you work on a sub-project of a sub-project, you don't expect the whole (top-level) project to compile (yet) necessarily.

How about having a tag/command/variable that defines a directory as project, module, unit or whatever you may call it?

The beauty of this would be that you could, henceforth, reference this "project" by name. You could even move it and tup would still assign the paths correctly.

This should make it easier to keep local things local.

Furthermore, you may need to be able to compile a single file from a certain project, with all the specified options and flags. In an IDE this is one of the most common tasks: "Try compiling what I got in front of my nose here; I wanna see what errors pop up." (compile current file). In other words: "Build the file xyz.cpp from project abc" with the project's flags and options.

I case you're wondering about the fixes – Mike already fixed it up pretty well:

I moved the tup-configs to the top level and had it create the build-* folders there. Therefore, in src there is now only this Tupfile (plus sources and headers, of course):

# Tupfile in config_module/src

CPPFLAGS += -std=c++17

CPPFLAGS += -Wall

include_rules

: foreach *.cpp |> g++ $(CPPFLAGS) -c %f -o %o |> %B.o

The Tuprules.tup got moved to the top-level as well and changed to this:

# Tuprules.tup in config_module

LDFLAGS += -Wl,-lstdc++fs

ifeq (@(BUILD_TYPE),debug)

# debug build:

  CPPFLAGS += -march=native

  CPPFLAGS += -ggdb3

  CPPFLAGS += -D_GLIBCXX_DEBUG

else

  ifeq (@(BUILD_TYPE),release)

    CPPFLAGS += -march=native

    CPPFLAGS += -O2 -s -flto

    LDFLAGS += -Wl,--as-needed

  else

    ifeq (@(BUILD_TYPE),static)

      CPPFLAGS += -static -Os -s -flto

      LDFLAGS += -Wl,--as-needed

    else

      error "Build type not set or wrong!"

    endif

  endif

endif

… and the top-level Tupfile now looks like that:

# Tupfile in config_module

MY_CONFIG_FOLDER = $(TUP_CWD)/debug_config_folder

CPPFLAGS += -std=c++17

CPPFLAGS += -Wall

CPPFLAGS += -Isrc

include_rules

: foreach *.cpp |> g++ $(CPPFLAGS) -DMY_CONFIG_FOLDER=$(MY_CONFIG_FOLDER) -c %f -o %o |> %B.o

: src/*.o *.o |> g++ $(CPPFLAGS) $(LDFLAGS) %f -o %o |> test_config

This seems to work nicely. :-)