Compile a program from all C and C++ files in current folder

[Frederick Gotham]

This has taken me a lot longer than it should have, but I think I've put together a Makefile that will compile a program from all the C and C++ files in the current directory.

The C source files, *.c, become object files *.c.o

The C++ source files, *.cpp, become object files *.cpp.o

The very first time you compile your program, it creates dependency files (*.c.d and *.cpp.d), so if you change a header file and then go to recompile the program, it should only recompile the source files that included that one header file.

Here's what I've got so far:

PROGRAM = super_cool_prog

FLAGS_IN_COMMON = -pedantic -Wall -rdynamic -funwind-tables -fno-omit-frame-pointer -fdump-rtl-expand -Og -g
CXXFLAGS = $(FLAGS_IN_COMMON) -std=c++11
CFLAGS = $(FLAGS_IN_COMMON) -std=c99

CC = gcc
CXX = g++

SRC_FILES_C := $(shell find -maxdepth 1 -name '*.c')
SRC_FILES_CXX := $(shell find -maxdepth 1 -name '*.cpp')

OBJECTS_C := $(patsubst %.c,%.c.o,$(SRC_FILES_C))
OBJECTS_CXX := $(patsubst %.cpp,%.cpp.o,$(SRC_FILES_CXX))

OBJECTS = $(OBJECTS_C) $(OBJECTS_CXX)

.PHONY: all
all: $(PROGRAM)

$(PROGRAM): $(OBJECTS)
$(CXX) $(FLAGS_IN_COMMON) -o $@ $^

#This rule compiles the source files to object files
%.c.o: %.c %.c.d
$(CC) $(CFLAGS) -o $@ -c $<

%.cpp.o: %.cpp %.cpp.d
$(CXX) $(CXXFLAGS) -o $@ -c $<

#This rule makes dependencies files for the source files
.PRECIOUS: %.c.d
%.c.d: %.c
$(CC) -MM -MT '$(patsubst %.c,%.c.o,$<)' $< -MF $@

.PRECIOUS: %.cpp.d
%.cpp.d: %.cpp
$(CXX) -MM -MT '$(patsubst %.cpp,%.cpp.o,$<)' $< -MF $@

#This next line pulls in all the dependency files
include $(shell find -maxdepth 1 -name '*.d')

.PHONY: clean
clean:
rm -f $(OBJECTS) $(PROGRAM)
rm -f $(patsubst %.o,%.d,$(OBJECTS))
rm -f $(patsubst %.o,%.229r.expand,$(OBJECTS))

.PHONY: install
install: $(PROGRAM)
cp $< /usr/bin/

.PHONY: uninstall
uninstall:
rm -f /usr/bin/$(PROGRAM)

[David Brown]

On 01/10/2019 13:37, Frederick Gotham wrote:
>
> This has taken me a lot longer than it should have, but I think I've put together a Makefile that will compile a program from all the C and C++ files in the current directory.
>
> The C source files, *.c, become object files *.c.o
>
> The C++ source files, *.cpp, become object files *.cpp.o
>
> The very first time you compile your program, it creates dependency files (*.c.d and *.cpp.d), so if you change a header file and then go to recompile the program, it should only recompile the source files that included that one header file.
>
> Here's what I've got so far:

It's looking very good so far. I have made a few suggestions. They are
untested, but based on my own existing and working Makefiles. They may
also be dependent on GNU Make (but who uses anything else?).

>
> PROGRAM = super_cool_prog
>
> FLAGS_IN_COMMON = -pedantic -Wall -rdynamic -funwind-tables -fno-omit-frame-pointer -fdump-rtl-expand -Og -g
> CXXFLAGS = $(FLAGS_IN_COMMON) -std=c++11
> CFLAGS = $(FLAGS_IN_COMMON) -std=c99
>
> CC = gcc
> CXX = g++
>
> SRC_FILES_C := $(shell find -maxdepth 1 -name '*.c')
> SRC_FILES_CXX := $(shell find -maxdepth 1 -name '*.cpp')

Make's own file search will probably be more efficient:

SRC_FILES_C := $(wildcard *.c)

(Your method might be best for a more complex directory structure of
source files.)

>
> OBJECTS_C := $(patsubst %.c,%.c.o,$(SRC_FILES_C))

Shorter and faster alternative:

OBJECTS_C := $(SRC_FILES_C:.c=.o)

> OBJECTS_CXX := $(patsubst %.cpp,%.cpp.o,$(SRC_FILES_CXX))
>
> OBJECTS = $(OBJECTS_C) $(OBJECTS_CXX)
>
> .PHONY: all
> all: $(PROGRAM)
>
> $(PROGRAM): $(OBJECTS)
> $(CXX) $(FLAGS_IN_COMMON) -o $@ $^
>
> #This rule compiles the source files to object files
> %.c.o: %.c %.c.d
> $(CC) $(CFLAGS) -o $@ -c $<
>
> %.cpp.o: %.cpp %.cpp.d
> $(CXX) $(CXXFLAGS) -o $@ -c $<
>
> #This rule makes dependencies files for the source files
> .PRECIOUS: %.c.d
> %.c.d: %.c
> $(CC) -MM -MT '$(patsubst %.c,%.c.o,$<)' $< -MF $@
>
> .PRECIOUS: %.cpp.d
> %.cpp.d: %.cpp
> $(CXX) -MM -MT '$(patsubst %.cpp,%.cpp.o,$<)' $< -MF $@
>
> #This next line pulls in all the dependency files
> include $(shell find -maxdepth 1 -name '*.d')

DEPENDS = $(OBJECTS:.o=.d)

-include $(DEPENDS)

Since the "-include" command ignores errors, you no longer need the
".PRECIOUS" and you don't have problems if some of your compilations are
failing to generate dependency files.

>
> .PHONY: clean
> clean:
> rm -f $(OBJECTS) $(PROGRAM)
> rm -f $(patsubst %.o,%.d,$(OBJECTS))
> rm -f $(patsubst %.o,%.229r.expand,$(OBJECTS))
>
> .PHONY: install
> install: $(PROGRAM)
> cp $< /usr/bin/
>
> .PHONY: uninstall
> uninstall:
> rm -f /usr/bin/$(PROGRAM)
>

You have two challenges left. First, the dependency files should be
updated automatically when their C or /header/ files change. Currently,
your dependencies are updated if the matching .c or .cpp file changes.
A simple method is:


HEADER_FILES := $(wildcard *.h)

%.c.d: %.c $(HEADER_FILES)

$(CC) -MM -MT '$(patsubst %.c,%.c.o,$<)' $< -MF $@

%.cpp.d: %.cpp $(HEADER_FILES)

$(CXX) -MM -MT '$(patsubst %.cpp,%.cpp.o,$<)' $< -MF $@

That will be a bit pessimistic and update dependency files more than
necessary, but that's better than doing so /less/ than necessarily.

A more advanced step would be something like:

%.c.d: %.c $(HEADER_FILES)

$(CC) -MM -MT '$(patsubst %.c,%.c.o,$<)' $< -MF $@

@sed -i 's,\(*\)\.o[ :]*,\1.o $@ : ,g' $@

I have not checked this "sed" line (and I can't read sed commands
fluently - this is based on lines from my own Makefiles, adapted as best
I can out of my head. I made my own "sed" commands based mainly on
googling and trial and error).

The idea is that you make the .d file itself dependent on the same files
the .o file is dependent upon.


I also like to make all the file generations dependent on "Makefile" too.



Your next challenge is to separate the source code directory (or
directories) from the directories for working files (.o, .d). I'll
leave that one for homework.

[Frederick Gotham]

On Tuesday, October 1, 2019 at 12:37:59 PM UTC+1, Frederick Gotham wrote:

> The very first time you compile your program, it creates dependency files (*.c.d and *.cpp.d), so if you change a header file and then go to recompile the program, it should only recompile the source files that included that one header file.

Actually this doesn't work. If you delete all the ".d" files then it doesn't re-create them like it should.

[Frederick Gotham]

Okay what do you think of this?

PROGRAM = super_cool_program

FLAGS_IN_COMMON = -pedantic -Wall -rdynamic -funwind-tables -fno-omit-frame-pointer -fdump-rtl-expand -Og -g
CXXFLAGS = $(FLAGS_IN_COMMON) -std=c++11
CFLAGS = $(FLAGS_IN_COMMON) -std=c99

CC = gcc
CXX = g++

SRC_FILES_C := $(wildcard *.c)
SRC_FILES_CXX := $(wildcard *.cpp)

OBJECTS = $(SRC_FILES_C:.c=.c.o) $(SRC_FILES_CXX:.cpp=.cpp.o)

DEPENDENCIES = $(OBJECTS:.o=.d)

.PHONY: all
all: $(PROGRAM) $(DEPENDENCIES)

$(PROGRAM): $(OBJECTS)
$(CXX) $(FLAGS_IN_COMMON) -o $@ $^

#This rule makes dependencies files for the source files
.PRECIOUS: %.c.d
%.c.d: %.c

$(CC) -MM -MT $(<:.c=.c.o) $< -MF $@

.PRECIOUS: %.cpp.d
%.cpp.d: %.cpp
$(CXX) -MM -MT $(<:.cpp=.cpp.o) $< -MF $@

#This rule compiles the source files to object files
%.c.o: %.c %.c.d
$(CC) $(CFLAGS) -o $@ -c $<

%.cpp.o: %.cpp %.cpp.d
$(CXX) $(CXXFLAGS) -o $@ -c $<

#This next line pulls in all the dependency files (if they exist)
-include $(DEPENDENCIES)

.PHONY: clean
clean:
rm -f $(DEPENDENCIES) $(OBJECTS) $(PROGRAM) $(OBJECTS:.o=*.expand)

[Manfred]

On 10/1/2019 1:37 PM, Frederick Gotham wrote:
>
> This has taken me a lot longer than it should have, but I think I've put together a Makefile that will compile a program from all the C and C++ files in the current directory.
>
> The C source files, *.c, become object files *.c.o
>
> The C++ source files, *.cpp, become object files *.cpp.o
>
> The very first time you compile your program, it creates dependency files (*.c.d and *.cpp.d), so if you change a header file and then go to recompile the program, it should only recompile the source files that included that one header file.

GNU make has support for generating dependencies automatically in
combination with the -M option of gcc.
You may want to look at
https://www.gnu.org/software/make/manual/html_node/Automatic-Prerequisites.html

[Frederick Gotham]

On Wednesday, October 2, 2019 at 12:35:39 PM UTC+1, Manfred wrote:

> > The very first time you compile your program, it creates dependency files (*.c.d and *.cpp.d), so if you change a header file and then go to recompile the program, it should only recompile the source files that included that one header file.
>
> GNU make has support for generating dependencies automatically in
> combination with the -M option of gcc.
> You may want to look at
> https://www.gnu.org/software/make/manual/html_node/Automatic-Prerequisites.html

I don't understand.

[Frederick Gotham]

On Wednesday, October 2, 2019 at 9:16:10 AM UTC+1, Frederick Gotham wrote:

> .PHONY: clean
> clean:
> rm -f $(DEPENDENCIES) $(OBJECTS) $(PROGRAM) $(OBJECTS:.o=*.expand)

I'm getting strange behaviour when I run "make clean". Instead of just running that one-liner above, it is creating the dependency files and then running that one-liner.

[David Brown]

It is hardly strange. You are trying to include the dependency files -
and Make will try to build them if they don't exist. That applies even
with a "make clean".


You can try this:

ifneq ($(MAKECMDGOALS),clean)
-include $(DEPENDENCIES)
endif

[David Brown]

The referenced link, and related pages, give quite a bit of information
about making smart Makefiles (I have got a lot from these pages).

But what might be confusing you here is that you already use "gcc -M"
runs to generate dependency information - a point that Manfred perhaps
missed.

I can recommend the link, however, as you might get other ideas even
though you already generate dependency files with gcc.

[Frederick Gotham]

On Wednesday, October 2, 2019 at 6:48:46 PM UTC+1, David Brown wrote:

> ifneq ($(MAKECMDGOALS),clean)
> -include $(DEPENDENCIES)
> endif

I'm going to spend another week or two perfecting this because I'm considering using it to replace over 4,000 makefiles in the current firmware project I'm working on.

I'm trying to use "+=" instead of "=" where I can just in case the user does something like this at the command line:

LDLIBS=lboost_filesystem make

Here's what I have now:

PROGRAM = my_super_cool_program

INC_DIRS += -I./
CFLAGS_CXXFLAGS_IN_COMMON += -pedantic -Wall -rdynamic -funwind-tables -fno-omit-frame-pointer -fdump-rtl-expand -Og -g -pthread
CFLAGS += $(INC_DIRS) $(CFLAGS_CXXFLAGS_IN_COMMON) -std=c99
CXXFLAGS += $(INC_DIRS) $(CFLAGS_CXXFLAGS_IN_COMMON) -std=c++11

LDFLAGS += $(CFLAGS_CXXFLAGS_IN_COMMON)
LDLIBS += -lboost_system

SRC_FILES_C := $(wildcard *.c)
SRC_FILES_CXX := $(wildcard *.cpp)

OBJECTS += $(SRC_FILES_C:.c=.c.o) $(SRC_FILES_CXX:.cpp=.cpp.o)

DEPENDENCIES += $(OBJECTS:.o=.d)

.PHONY: all
all: $(PROGRAM) $(DEPENDENCIES)

#This rules runs the linker to combine object files into one executable binary file
$(PROGRAM): $(OBJECTS)
$(CXX) $(LDFLAGS) $^ $(LDLIBS) -o $@

#This rule makes dependencies files for the source files
.PRECIOUS: %.c.d
%.c.d: %.c
$(CC) -MM -MT $(<:.c=.c.o) $< -MF $@

.PRECIOUS: %.cpp.d
%.cpp.d: %.cpp
$(CXX) -MM -MT $(<:.cpp=.cpp.o) $< -MF $@

#This rule compiles the source files to object files
%.c.o: %.c %.c.d
$(CC) $(CFLAGS) -o $@ -c $<

%.cpp.o: %.cpp %.cpp.d
$(CXX) $(CXXFLAGS) -o $@ -c $<

#This next line pulls in all the dependency files (if they exist, and it's not "make clean")

ifneq ($(MAKECMDGOALS),clean)
-include $(DEPENDENCIES)
endif

.PHONY: clean
clean:
$(RM) $(DEPENDENCIES) $(OBJECTS) $(PROGRAM) $(OBJECTS:.o=*.expand)

#These next two rules to (un)install are only for Linux

.PHONY: install
install: $(PROGRAM)
cp $< /usr/bin/

.PHONY: uninstall
uninstall:

$(RM) /usr/bin/$(PROGRAM)

[Frederick Gotham]

On Thursday, October 3, 2019 at 8:39:34 AM UTC+1, David Brown wrote:

> But what might be confusing you here is that you already use "gcc -M"
> runs to generate dependency information - a point that Manfred perhaps
> missed.

That's why I said that I didn't understand -- I didn't get why he was referring me to something that I was already using.

[Jorgen Grahn]

On Tue, 2019-10-01, Frederick Gotham wrote:
>
> This has taken me a lot longer than it should have, but I think I've
> put together a Makefile that will compile a program from all the C
> and C++ files in the current directory.

What's its purpose? It's nice as a demo of Make with dependency
tracking, but I wouldn't want it in projects I work with: I like a
Makefile to be a bill of materials, and build only a fixed set of
source files.

I also tend to split source code in a bunch of subdirectories, and
generate several executables (the unit tests shouldn't be linked into
the main executable).

...

> Here's what I've got so far:
>
> PROGRAM = super_cool_prog
>
> FLAGS_IN_COMMON = -pedantic -Wall -rdynamic -funwind-tables -fno-omit-frame-pointer -fdump-rtl-expand -Og -g

Also consider -Wextra.

> CXXFLAGS = $(FLAGS_IN_COMMON) -std=c++11
> CFLAGS = $(FLAGS_IN_COMMON) -std=c99

...

/Jorgen

--
// Jorgen Grahn <grahn@ Oo o. . .
\X/ snipabacken.se> O o .

[Frederick Gotham]

On Friday, October 4, 2019 at 6:46:02 AM UTC+1, Jorgen Grahn wrote:

> > This has taken me a lot longer than it should have, but I think I've
> > put together a Makefile that will compile a program from all the C
> > and C++ files in the current directory.
>
> What's its purpose? It's nice as a demo of Make with dependency
> tracking, but I wouldn't want it in projects I work with: I like a
> Makefile to be a bill of materials, and build only a fixed set of
> source files.

If a ".cpp" or ".c" file exists in the folder, it's there to be compiled. Otherwise it should have a different extension, e.g. ".cpp.old".

I don't use Makefiles as a bill of materials like that. For such a bill of materials, I would run the command "ls *.{c,cpp}".

> I also tend to split source code in a bunch of subdirectories, and
> generate several executables (the unit tests shouldn't be linked into
> the main executable).

I will be looking at changing "SRC_FILES_C" and "SRC_FILES_CPP" to take in all source files in subdirectories.


> Also consider -Wextra.


Missed that one.

[Hergen Lehmann]

Am 04.10.19 um 08:51 schrieb Frederick Gotham:

> If a ".cpp" or ".c" file exists in the folder, it's there to be compiled.

There may be source files providing interfaces to different OS
platforms, sources related to different configurations of the main
program, sources related to experimental features not being part of the
current release, sources used in debug mode only, sources related to
tools which must be compiled into separate executables, and many more.

In any non-trivial project, you will quickly run into a state, where you
actually NEED to use the makefile as a "bill of materials" rather than
throwing in every file there is.

[David Brown]

On 04/10/2019 07:45, Jorgen Grahn wrote:
> On Tue, 2019-10-01, Frederick Gotham wrote:
>>
>> This has taken me a lot longer than it should have, but I think I've
>> put together a Makefile that will compile a program from all the C
>> and C++ files in the current directory.
>
> What's its purpose? It's nice as a demo of Make with dependency
> tracking, but I wouldn't want it in projects I work with: I like a
> Makefile to be a bill of materials, and build only a fixed set of
> source files.
>

That's fair enough. Personally, I don't - I use Makefiles that build on
the same principles as this example (but a good deal more advanced,
according to need - handling multiple directories, multiple executables,
and whatever else).

To me, the "bill of materials" for a software project is the files in
the directory (and subdirectories). If there is a file there, it is
part of the project. If it is not part of the project, it should not be
in the directory. So "*.c" gives me exactly the C files used for the
project - there is no need to list them.

This makes the build system almost zero maintenance - adding a new file
to the project means adding a new file to the directory and typing "make".

> I also tend to split source code in a bunch of subdirectories, and
> generate several executables (the unit tests shouldn't be linked into
> the main executable).
>
> ...
>> Here's what I've got so far:
>>
>> PROGRAM = super_cool_prog
>>
>> FLAGS_IN_COMMON = -pedantic -Wall -rdynamic -funwind-tables -fno-omit-frame-pointer -fdump-rtl-expand -Og -g
>
> Also consider -Wextra.

Agreed. (I like quite a few more warning flags, but that is a matter of
personal preference, coding standard, etc. People have to decide on
those themselves. The same goes for "-pedantic" and many other flags.)

"-fno-omit-frame-pointer" is a strange choice. It is just a "give me a
bit bigger and slower code" flag. It certainly isn't any help for
debugging with gdb. But maybe the OP has specialist requirements for
which the flag helps.

Another very good choice, IMHO, is "-fno-common".

[Frederick Gotham]

On Friday, October 4, 2019 at 9:02:09 AM UTC+1, David Brown wrote:

> "-fno-omit-frame-pointer" is a strange choice. It is just a "give me a
> bit bigger and slower code" flag. It certainly isn't any help for
> debugging with gdb. But maybe the OP has specialist requirements for
> which the flag helps.

Some of our products are embedded Linux on x86_64, and some are on ARM (aarch64). I'm not sure who to believe, but someone told me that the function call stack on ARM might leave out a frame pointer, meaning that I won't be able to do a stack trace. If this is true, then presumably "-rdynamic" would implicitly include "-fno-omit-frame-pointer" on ARM processors. I don't know who do believe, so I throw in both.

[Ian Collins]

You have -Og, so getting a stack trace won't be a problem...

--
Ian.

[David Brown]

No, you won't. You might find a "bill of materials" makefile to be a
convenient way to handle it, but it most certainly is /not/ the only way
to do it.

There is also endless scope for hybrid solutions. A practical way is to
have different directories for your common code, your test code, your PC
simulator code, your embedded target code, etc. A build for the
embedded target would use all *.c *.cpp files in the "common" and
"embedded" directories, while a simulator build would use all files in
the "common" and "simulator" directories.

There is no one solution that fits all projects, of course. /I/ prefer
a makefile that takes all *.c *.cpp files in the source directories -
but that does not mean I think it is the best choice for everyone or
every project.

[David Brown]

I am afraid I don't know what might be needed for stack tracing on
embedded Linux. My main work is with bare-bones embedded systems, with
gdb connected to the target via a JTAG debugger. There is certainly no
problem looking through stack frames there without forcing a frame pointer.

[Hergen Lehmann]

Am 04.10.19 um 10:49 schrieb David Brown:

> There is also endless scope for hybrid solutions. A practical way is to
> have different directories for your common code, your test code, your PC
> simulator code, your embedded target code, etc. A build for the
> embedded target would use all *.c *.cpp files in the "common" and
> "embedded" directories, while a simulator build would use all files in
> the "common" and "simulator" directories.

That's again a bill of materials, just with directory names instead of
file names. Not much different...

> There is no one solution that fits all projects, of course.

I can agree to that.

[Paavo Helde]

This is just making the life complicated for anyone trying to compile
your code, in particular on those different OS platforms, and in
particular for those who would prefer to use some other build system.

The solution is simple: each source file which is to be conditionally
compiled, e.g. on Mac only, should look like this:

#include "myconfig.h" // defines HAVE_DARWIN and other such macros
#ifdef HAVE_DARWIN
#import <Foundation/NSProcessInfo.h>
// ... rest of Mac specific code
#endif

Voila, now one can just compile *.cpp on any platform, e.g. insert the
whole folder into a Visual Studio or CMake project, done. No make or
makefile needed, no intricate knowledge needed about which files need to
be compiled where.

[Hergen Lehmann]

Am 04.10.19 um 11:36 schrieb Paavo Helde:

> The solution is simple: each source file which is to be conditionally
> compiled, e.g. on Mac only, should look like this:
>
> #include "myconfig.h" // defines HAVE_DARWIN and other such macros
> #ifdef HAVE_DARWIN
> #import <Foundation/NSProcessInfo.h>
> // ... rest of Mac specific code
> #endif

This approach does neither work for multiple target executables, nor for
multiple configurations to be build at the same time, nor for external
dependencies (which you certainly don't want to patch just to follow
some paradigm).

It also quickly gets confusing, when OS dependencies are scattered all
over the source code. I'm certainly not the only one who prefers to have
OS dependencies strictly separate in some interface layer, which is
swapped in it's whole depending on the platform.

> Voila, now one can just compile *.cpp on any platform, e.g. insert the
> whole folder into a Visual Studio or CMake project, done.

Yeah, done with a compilation error, because most external dependencies
coming from other platforms will have to be replaced with hacked-up MSVC
variants. :-(

[David Brown]

On 04/10/2019 11:27, Hergen Lehmann wrote:
> Am 04.10.19 um 10:49 schrieb David Brown:
>
>> There is also endless scope for hybrid solutions.  A practical way is to
>> have different directories for your common code, your test code, your PC
>> simulator code, your embedded target code, etc.  A build for the
>> embedded target would use all *.c *.cpp files in the "common" and
>> "embedded" directories, while a simulator build would use all files in
>> the "common" and "simulator" directories.
>
> That's again a bill of materials, just with directory names instead of
> file names. Not much different...

It is /hugely/ different, because your BOM is just a few lines, with no
dependency information, and you have maintained the one-to-one
relationship between files in the directory and files in the build.

It is all about specifying things manually when you have to, but getting
the setup to work automatically as much as possible.

And it means you have a clear structure in your files and directories -
everything is in a sensible place and it is immediately obvious how it
is put together. (Of course you can still have good directory layout
even with a humongous list in a Makefile. But I prefer the link to be
definitive.)

If you want to know where the function "foo" is used, you can do a
"grep" of the files and know exactly when it is used and in which builds
- you don't need to trawl through huge lists in a makefile to figure out
whether the files containing "foo" are actually used anywhere.

[David Brown]

You can combine this approach with command-line defines, so that your
"myconfig.h" might have:

#if defined(TARGET_MAC)
#define HAVE_DARWIN
#elif defined(TARGET_WIN)
#define HAVE_WIN32
#else
#error Target not defined!
#endif

Then the flags for your "make mac" and "make windows" built targets
would include -DTARGET_MAC or -DTARGET_WIN as appropriate.

[Scott Lurndal]

Frederick Gotham <cauldwel...@gmail.com> writes:
>On Friday, October 4, 2019 at 6:46:02 AM UTC+1, Jorgen Grahn wrote:
>
>> > This has taken me a lot longer than it should have, but I think I've
>> > put together a Makefile that will compile a program from all the C
>> > and C++ files in the current directory.
>>
>> What's its purpose? It's nice as a demo of Make with dependency
>> tracking, but I wouldn't want it in projects I work with: I like a
>> Makefile to be a bill of materials, and build only a fixed set of
>> source files.
>
>
>If a ".cpp" or ".c" file exists in the folder, it's there to be compiled. Otherwise it should have a different extension, e.g. ".cpp.old".
>

That is an opinion, not a hard-and-fast rule. I've never seen a professional project
in over 40 years of programming that didn't explicitly specify all the objects
(and the rules to build them if the default rules were insufficient) necessary to build
the target (whether an executable, a library or both).

[Frederick Gotham]

On Friday, October 4, 2019 at 2:54:58 PM UTC+1, Scott Lurndal wrote:

To have a file in the directory listing for a project is quite explicit. In fact, I could argue that you can't get much more explicit than that.

[Scott Lurndal]

$ ls
ASMBLR Makefile.rules docs main.cpp notes tests
Makefile Makefile.version include main.d pkg utilities
Makefile.ddk common io main.o processor vsim
Makefile.defs data lib mp tapes

How does your approach deal with subdirectories?

$ ls io/dlps
5n_dlp.cpp file_tape_unit.cpp qwik_dlp.h
5n_dlp.d file_tape_unit.d qwik_dlp.o
5n_dlp.h file_tape_unit.h scsi_disk_dlp.cpp
5n_dlp.o file_tape_unit.o scsi_disk_dlp.d
Makefile fips_tape_dlp.cpp scsi_disk_dlp.h
Makefile.ddk fips_tape_dlp.d scsi_disk_dlp.o
buffered_printer_dlp.cpp fips_tape_dlp.h scsi_tape_dlp.cpp
buffered_printer_dlp.d fips_tape_dlp.o scsi_tape_dlp.d
buffered_printer_dlp.h gcr_dlp.cpp scsi_tape_dlp.h
buffered_printer_dlp.o gcr_dlp.d scsi_tape_dlp.o
card_punch_dlp.cpp gcr_dlp.h ssp.cpp
card_punch_dlp.d gcr_dlp.o ssp.d
card_punch_dlp.h ht_dcp_dlp.cpp ssp.h
card_punch_dlp.o ht_dcp_dlp.d ssp.o
card_reader_dlp.cpp ht_dcp_dlp.h ssp_dlp.cpp
card_reader_dlp.d ht_dcp_dlp.o ssp_dlp.d
card_reader_dlp.h ht_dpdc_dlp.cpp ssp_dlp.h
card_reader_dlp.o ht_dpdc_dlp.d ssp_dlp.o
card_unit.cpp ht_dpdc_dlp.h st_tape_unit.cpp
card_unit.d ht_dpdc_dlp.o st_tape_unit.d
card_unit.h htseq_dlp.cpp st_tape_unit.h
card_unit.o htseq_dlp.d st_tape_unit.o
disk_unit.cpp htseq_dlp.h tape_unit.cpp
disk_unit.d htseq_dlp.o tape_unit.d
disk_unit.h iocb.cpp tape_unit.h
disk_unit.o iocb.d tape_unit.o
dlp.cpp iocb.o telcom_dlp.cpp
dlp.d isc_dlp.cpp telcom_dlp.d
dlp.o isc_dlp.d telcom_dlp.h
file_card_unit.cpp isc_dlp.h telcom_dlp.o
file_card_unit.d isc_dlp.o train_printer_dlp.cpp
file_card_unit.h odt_dlp.cpp train_printer_dlp.d
file_card_unit.o odt_dlp.d train_printer_dlp.h
file_disk_unit.cpp odt_dlp.h train_printer_dlp.o
file_disk_unit.d odt_dlp.o uniline_dlp.cpp
file_disk_unit.h printer_unit.cpp uniline_dlp.d
file_disk_unit.o printer_unit.h uniline_dlp.h
file_printer_unit.cpp qwik_dlp.cpp uniline_dlp.o
file_printer_unit.h qwik_dlp.d

A fragment of io/dlps/Makefile:

libdlp_fips_tape.so: fips_tape_dlp.o dlp.o
@echo " BUILD $(LIB)/$@"
$(HUSHCOMPILE)$(CC) -o $(LIB)/$@ $(SOLDFLAGS) $+ -lc

libdlp_htseq.so: htseq_dlp.o disk_unit.o file_disk_unit.o dlp.o
@echo " BUILD $(LIB)/$@"
$(HUSHCOMPILE)$(CC) -o $(LIB)/$@ $(SOLDFLAGS) $+ -lc

libdlp_uniline.so: uniline_dlp.o dlp.o $(LIBCOMMON)
@echo " BUILD $(LIB)/$@"
$(HUSHCOMPILE)$(CC) -o $(LIB)/$@ $(SOLDFLAGS) $+

libdlp_odt.so: odt_dlp.o dlp.o $(LIBCOMMON)
@echo " BUILD $(LIB)/$@"
$(HUSHCOMPILE)$(CC) -o $(LIB)/$@ $(SOLDFLAGS) $+ -lc

libdlp_scsi_disk.so: scsi_disk_dlp.o disk_unit.o file_disk_unit.o dlp.o
@echo " BUILD $(LIB)/$@"
$(HUSHCOMPILE)$(CC) -o $(LIB)/$@ $(SOLDFLAGS) $+ -lc

libdlp_scsi_tape.so: scsi_tape_dlp.o tape_unit.o file_tape_unit.o dlp.o st_tape_unit.o
@echo " BUILD $(LIB)/$@"
$(HUSHCOMPILE)$(CC) -o $(LIB)/$@ $(SOLDFLAGS) $+

libdlp_gcr.so: gcr_dlp.o dlp.o tape_unit.o file_tape_unit.o st_tape_unit.o
@echo " BUILD $(LIB)/$@"
$(HUSHCOMPILE)$(CC) -o $(LIB)/$@ $(SOLDFLAGS) $+ -lc

libdlp_5n.so: 5n_dlp.o disk_unit.o file_disk_unit.o dlp.o
@echo " BUILD $(LIB)/$@"
$(HUSHCOMPILE)$(CC) -o $(LIB)/$@ $(SOLDFLAGS) $+ -lc

libdlp_reader.so: card_reader_dlp.o card_unit.o file_card_unit.o dlp.o
@echo " BUILD $(LIB)/$@"
$(HUSHCOMPILE)$(CC) -o $(LIB)/$@ $(SOLDFLAGS) $+ -lc

libdlp_tpr.so: train_printer_dlp.o dlp.o
@echo " BUILD $(LIB)/$@"
$(HUSHCOMPILE)$(CC) -o $(LIB)/$@ $(SOLDFLAGS) $+ -lc

[David Brown]

I don't mix object files with source code - that quickly gets really
messy, and is terrible if you have multiple builds.

And haven't you read "Recursive make considered harmful" ? A Makefile
in a subdirectory?

My typical setup would be to have "Makefile" and perhaps a few other
files in the project root directory - things like utility scripts used
as part of the build project. (If there are many of them, they go in
their own directory.)

Source is all under a "src" directory - with subdirectories for
different parts of the code, such as third-party code, test code, etc.
This will obviously depend on the size of the project.

Auxiliary files such as linker scripts go in their own directory, which
is sometimes under "src" and sometimes under the project root (I am not
always consistent). Configuration files for debuggers, test scripts,
implementation documentation, etc., have their own directories as needed.

Then there is a "bin" directory for the generated binaries, and a
"build" directory for the built files. Directly under "build" will be
directories named for different build variations - the same source might
be used to build variants for different embedded hardware versions, or
PC test versions, or whatever. Under these, I have directories "dep",
"lst" and "obj" which have sub-directories mirroring the src structure
(this is all created automatically by the Makefile), containing the
dependency files, listing files and object files generated while building.

Everything is handled by a single "make". The Makefile is not
completely automatic or portable - it holds information about the
variants to be built, particular flags, directories, etc. But it avoids
the detailed work of tracking each file.

[Scott Lurndal]

I don't generally; that's an old non-proprietary project; but for that project, it
never caused issues. Generally we have 'release' and 'debug' directories
in each subdirectory to hold the '.o' and '.d' files (compiled for release
and debug respectively). bin and lib directories at the top level. A
top level Makefile.rules and Makefile.defs which is included from every
subdirectory Makefile which makes the subdirectory Makefiles very small,
easy to maintain and suitable for parallel builds.

>
>And haven't you read "Recursive make considered harmful" ? A Makefile
>in a subdirectory?

I've read it, and I completely disagree with almost everything in that document.

There's nothing inherently wrong with recursive make; particularly given
that his primary complaint was slow builds in 1997. Twenty two years
of Moores Law has pretty much killed that argument.

There is no problem supporting parallel makes in a recursive make enviroment
either.

[Ian Collins]

I take it from that you have never use premake? It follows the "if a

".cpp" or ".c" file exists in the folder, it's there to be compiled"

philosophy and generates one fat "Makefile" (in our case, a ninja build
file or visual studio solution).

--
Ian.

[Scott Lurndal]

Ian Collins <ian-...@hotmail.com> writes:
>On 05/10/2019 02:54, Scott Lurndal wrote:
>> Frederick Gotham <cauldwel...@gmail.com> writes:
>>> On Friday, October 4, 2019 at 6:46:02 AM UTC+1, Jorgen Grahn wrote:
>>>
>>>>> This has taken me a lot longer than it should have, but I think I've
>>>>> put together a Makefile that will compile a program from all the C
>>>>> and C++ files in the current directory.
>>>>
>>>> What's its purpose? It's nice as a demo of Make with dependency
>>>> tracking, but I wouldn't want it in projects I work with: I like a
>>>> Makefile to be a bill of materials, and build only a fixed set of
>>>> source files.
>>>
>>>
>>> If a ".cpp" or ".c" file exists in the folder, it's there to be compiled. Otherwise it should have a different extension, e.g. ".cpp.old".
>>>
>>
>> That is an opinion, not a hard-and-fast rule. I've never seen a professional project
>> in over 40 years of programming that didn't explicitly specify all the objects
>> (and the rules to build them if the default rules were insufficient) necessary to build
>> the target (whether an executable, a library or both).
>
>I take it from that you have never use premake?

No. I did use ant back around the turn of the millenium;
I did't like it.

javac has the property, if I recall correctly, that it will
compile all .java files in a directory with a single invocation; That may work for java,
but not for most of the C or C++ projects that I've worked on or
with.

[Ian Collins]

premake doesn't actually compile anything, as it's name suggests it
applies a set of rules (in a lua script) to a project tree and generates
makefile like outputs. By default (at least in our use case) it will
include all sources under a directory in a library, but you can
explicitly include sources from elsewhere and exclude sources you don't
want to build. You can also tune compile options at the library or
source file level. We like it because all of the rules are in one file
and we can generate platform specific outputs (makefiles, ninja files
and Visual Studio solutions). These are big advantages for a fairly
large (3000+ source file) project that gets built on multiple platforms.

--
Ian.

[David Brown]

Fair enough.

>
>>
>> And haven't you read "Recursive make considered harmful" ? A Makefile
>> in a subdirectory?
>
> I've read it, and I completely disagree with almost everything in that document.
>
> There's nothing inherently wrong with recursive make; particularly given
> that his primary complaint was slow builds in 1997. Twenty two years
> of Moores Law has pretty much killed that argument.

I don't see a problem with the speed of make, or starting many recursive
makes - the "make" overhead is not going to be an issue unless you have
/really/ big projects and regularly only need to recompile a very few files.

>
> There is no problem supporting parallel makes in a recursive make enviroment
> either.
>

I think recursive make /can/ be used well - but it is also very easy to
get wrong. I have seen more than enough cases of multi-part projects
where a change in one part of a project does not force a rebuild of
other parts that depend upon it. You make a change somewhere in
subproject A and "cd A; make" handles it. But a couple of files in
subproject B need recompiling to handle the change - "cd B; make" does
nothing. You end up doing lots of "make clean && make" in the
subprojects just to make sure everything is good. Cross-part
dependencies are missing.

So you either risk being incorrect, or you have massive amounts of
unnecessary rebuilds.

(Cross-part dependency tracking can be done, of course - but often it is
not done properly.)

The parallel build scalability problems are a less important issue to
me, but can be trouble for some people. It is far easier to get an
efficient parallel build process under one make than many make's.


There is no problem having local "include" makefiles in subdirectories,
which will be included from the main makefile.

[David Brown]

I haven't used "premake" (though it is on my "things to learn about when
I get the time" list, as a result of previous recommendations from you).
My main point is the principle of identifying "part of the project"
with "in the project directory" - it doesn't really matter if you are
getting it with make, premake, scons, ant, or "gcc *.c".

[Frederick Gotham]

On Friday, October 4, 2019 at 5:14:10 PM UTC+1, Scott Lurndal wrote:

> >And haven't you read "Recursive make considered harmful" ? A Makefile
> >in a subdirectory?
>
> I've read it, and I completely disagree with almost everything in that document.
>
> There's nothing inherently wrong with recursive make; particularly given
> that his primary complaint was slow builds in 1997. Twenty two years
> of Moores Law has pretty much killed that argument.
>
> There is no problem supporting parallel makes in a recursive make enviroment
> either.

Recursive 'make' is ok, but I'm thinking of doing things a little differently.

When I get this Makefile perfect I'm going to have directories nested about 3 or 4 levels deep with programs and libraries to be compiled. So then in the root folder, I might write a script to perform a parallel build something like:

find -name Makefile ! -type d -printf "%d \"%h\"\n" | sort -r -n | cut -d ' ' -f 2 | xargs -n1 -P 0 make -C

This will build the program starting from the deepest folders moving up the directory tree.

Also, instead of copying my 'perfect' Makefile to each folder, I'll just have symbolic links to it so that I can change it in one place.

Of course I have a few things to consider, like what happens if one of the Makfiles higher up the tree gets executed before one of the deeper one's has finished.

[Ian Collins]

Or just use a tool that takes care of this for you!

--
Ian.

[David Brown]

That sounds horrendous, a maintenance disaster, and it is crying out for
problems with cross-directory dependencies. I strongly recommend that
you scrape the whole idea.

Write /one/ Makefile. Put it in the root directory. Call it with
"make", or "make -j" for parallel builds. Have that include
directory-specific parts if you want, but have that one Makefile control
the others. Make sure all your files in all the subdirectories have
dependency files generated automatically so that you get the right
re-compiles when headers change, regardless of the directories.
Ideally, your dependency files should also be updated automatically
rather than needing a manual "make depends" step.

If you really can't avoid recursive makes - perhaps because you are
using third-party project parts that have their own makefiles - then
call these makes from within your main makefile.

The aim should always be that typing "make -j" will give you a correct
build, no matter what has changed, and no matter what order the parallel
tasks run in - all ordering requirements should be given in the makefile.

[Frederick Gotham]

With regard to linking, I want "g++" to be used if there were any C++ source files, otherwise I want to use "gcc" for linking if all the source files were C.

I think I would do this as follows. . .

ifeq ($(SRC_FILES_CXX),)
LINKER = $(CC)
else
LINKER = $(CXX)
endif

[Frederick Gotham]

So here's what I've got now:

PROGRAM = super_cool_program

INC_DIRS += -I./
CFLAGS_CXXFLAGS_IN_COMMON += -pedantic -Wall -Wextra -rdynamic -funwind-tables -fno-omit-frame-pointer -fdump-rtl-expand -fno-common -Og -g -pthread
CFLAGS += $(INC_DIRS) $(CFLAGS_CXXFLAGS_IN_COMMON) -std=c99
CXXFLAGS += $(INC_DIRS) $(CFLAGS_CXXFLAGS_IN_COMMON) -std=c++17

LDFLAGS += $(CFLAGS_CXXFLAGS_IN_COMMON)
LDLIBS +=

SRC_FILES_C := $(wildcard *.c)
SRC_FILES_CXX := $(wildcard *.cpp)

#If there's no C++ files then use the C compiler to link

ifeq ($(SRC_FILES_CXX),)
LINKER = $(CC)
else
LINKER = $(CXX)
endif

OBJECTS += $(SRC_FILES_C:.c=.c.o) $(SRC_FILES_CXX:.cpp=.cpp.o)

DEPENDENCIES += $(OBJECTS:.o=.d)

.PHONY: all
all: $(PROGRAM) $(DEPENDENCIES)

#This rule runs the linker to combine object files into one executable binary file
$(PROGRAM): $(OBJECTS)
$(LINKER) $(LDFLAGS) $^ $(LDLIBS) -o $@

#This rule makes dependencies files for the source files
.PRECIOUS: %.c.d
%.c.d: %.c
$(CC) -MM -MT $(<:.c=.c.o) $< -MF $@

.PRECIOUS: %.cpp.d
%.cpp.d: %.cpp
$(CXX) -MM -MT $(<:.cpp=.cpp.o) $< -MF $@

#This rule compiles the source files to object files
%.c.o: %.c %.c.d
$(CC) $(CFLAGS) -o $@ -c $<

%.cpp.o: %.cpp %.cpp.d
$(CXX) $(CXXFLAGS) -o $@ -c $<

#This next line pulls in all the dependency files (if they exist, and it's not "make clean")
ifneq ($(MAKECMDGOALS),clean)
-include $(DEPENDENCIES)
endif

.PHONY: clean
clean:
$(RM) $(DEPENDENCIES) $(OBJECTS) $(PROGRAM) $(OBJECTS:.o=*.expand)

#These next two rules to (un)install are only for Linux
.PHONY: install
install: $(PROGRAM)
cp $< /usr/bin/

.PHONY: uninstall
uninstall:
$(RM) /usr/bin/$(PROGRAM)

[Frederick Gotham]

Another thing: When creating the dependency files, the header inclusion paths must be passed to the compiler, like so:

%.c.d: %.c
$(CC) $(INC_DIRS) -MM -MT $(<:.c=.c.o) $< -MF $@

%.cpp.d: %.cpp
$(CXX) $(INC_DIRS) -MM -MT $(<:.cpp=.cpp.o) $< -MF $@


So here's what I've got now. . .

- - - - - - - - - - Makefile - - - - - - - -

$(CC) $(INC_DIRS) -MM -MT $(<:.c=.c.o) $< -MF $@

.PRECIOUS: %.cpp.d
%.cpp.d: %.cpp
$(CXX) $(INC_DIRS) -MM -MT $(<:.cpp=.cpp.o) $< -MF $@