Build system proposal

[Malcolm McLean]

How shall we write a better build system?

Current systems use scripts with ad hoc, weird syntax. They don't work
consistently across platforms, they break and are hard to fix, and are
complicated to set up. You even get get one scripting system as a front
end to another, a sort of admission of failure.

But can we do better?

A scripting language provides loops, conditional branches, and
subroutine calls. And we've already got a perfecty good language for
describing that. It is C itself.

So here's the proposal. We implement a C interpreter. Then we provide a
library of facilities - obviously globbing and system calls, maybe also
installed program detection, as C functions called in the normal way.
And the resulting C program is the build script. Because the C languague
is well known and standardised, there's no unfamiliar syntax or
unexpected behaviour.

We've got the skills to do this. Bart writes compilers, and so can
easily modify one to run as an interpreter. And we can develop the
library using normal C compilers. When we're happy with it, it is folded
into the interpreter. A clean, closed, achieveable, but very flexible
design.

It will need to run on Linux, Mac and Windows at a minimum. And
hopefully build on all three with the same script and no platform
conditional code.

--
Check out Basic Algorithms and my other books:
https://www.lulu.com/spotlight/bgy1mm

[bart]

On 04/02/2024 15:39, Malcolm McLean wrote:
> How shall we write a better build system?
>
> Current systems use scripts with ad hoc, weird syntax. They don't work
> consistently across platforms, they break and are hard to fix, and are
> complicated to set up. You even get get one scripting system as a front
> end to another, a sort of admission of failure.
>
> But can we do better?
>
> A scripting language provides loops, conditional branches, and
> subroutine calls. And we've already got a perfecty good language for
> describing that. It is C itself.
>
> So here's the proposal. We implement a C interpreter.

Do we need an interpreter? We need to assume there is a C compiler
otherwise we won't get very far with the build!

If the concern is the time it takes to compile the script, it should be
a small proportion of that needed to build the app. I assume the script
is only built once.

C for scripting builds has been done before. I think Thiago Adams did so.

The Seed7 build process had its own version of an auto-config script
which was written in C, not Bash (but building the app still uses
makefiles, nearly 20 of them for assorted compilers).

[Scott Lurndal]

Malcolm McLean <malcolm.ar...@gmail.com> writes:
>How shall we write a better build system?
>
>Current systems use scripts with ad hoc, weird syntax.

Fact not in evidence. Your opinion thereof does not make it a fact.

[Scott Lurndal]

bart <b...@freeuk.com> writes:
>On 04/02/2024 15:39, Malcolm McLean wrote:
>> How shall we write a better build system?
>>
>> Current systems use scripts with ad hoc, weird syntax. They don't work
>> consistently across platforms, they break and are hard to fix, and are
>> complicated to set up. You even get get one scripting system as a front
>> end to another, a sort of admission of failure.
>>
>> But can we do better?
>>
>> A scripting language provides loops, conditional branches, and
>> subroutine calls. And we've already got a perfecty good language for
>> describing that. It is C itself.
>>
>> So here's the proposal. We implement a C interpreter.

This was tried. It's called the c-shell.

[bart]

This is not a shell program. And it's not proposing using a language
that vaguely looks like C, but /is/ C. I don't think the proposal is to
ever type it live via a REPL, but always from a file.

Although as I understand it, what is proposed could be done right now:
anyone can provide a small C program to orchestrate the building of the
main program.

So perhaps it will be more of a library.

But it will take some care that it doesn't end up doing exactly what
makefiles do (with lots of assumptions about C compilers and the same
failure points), but written as a sequence of C function calls.

I think the main objection will be that it is a two step process. This
may have been the motivation to use an interpreter, but that is too huge
of a task, and too big a dependency, to solve a problem that could be
achieved with shebang lines.

[Malcolm McLean]

tcc has an interpeter mode. So yes, you could put a tcc shebang and
achieve the same thing. But then the library wouldn't be integrated.

[Ben Bacarisse]

csh isn't a C interpreter by any stretch of the imagination!

--
Ben.

[Malcolm McLean]

No. But the concept of a C interpreter isn't new. There's one called
pico C which is open source and available to be incorporated. And a
scripting language is an obvious application for an intepreter.

[Tim Rentsch]

Ben Bacarisse <ben.u...@bsb.me.uk> writes:

>> [...]

>
> csh isn't a C interpreter by any stretch of the imagination!

IT ISN'T??? No wonder I've been having trouble writing csh
scripts...

[Ben Bacarisse]

Malcolm McLean <malcolm.ar...@gmail.com> writes:

> On 04/02/2024 23:32, Ben Bacarisse wrote:
>> sc...@slp53.sl.home (Scott Lurndal) writes:
>>
>>> bart <b...@freeuk.com> writes:
>>>> On 04/02/2024 15:39, Malcolm McLean wrote:
>>>>> How shall we write a better build system?
>>>>>
>>>>> Current systems use scripts with ad hoc, weird syntax. They don't work
>>>>> consistently across platforms, they break and are hard to fix, and are
>>>>> complicated to set up. You even get get one scripting system as a front
>>>>> end to another, a sort of admission of failure.
>>>>>
>>>>> But can we do better?
>>>>>
>>>>> A scripting language provides loops, conditional branches, and
>>>>> subroutine calls. And we've already got a perfecty good language for
>>>>> describing that. It is C itself.
>>>>>
>>>>> So here's the proposal. We implement a C interpreter.
>>>
>>> This was tried. It's called the c-shell.
>>
>> csh isn't a C interpreter by any stretch of the imagination!
>
> No. But the concept of a C interpreter isn't new.

Indeed. But C is not a good fit for interpretation, so there are often
compromises.

> There's one called pico C

Yes, and there's tcc's -run option.

> which is open source and available to be incorporated.

Likewise tcc. So what's wrong (in your opinion) with these two that
caused you to suggest implementing another C interpreter?

> And a scripting
> language is an obvious application for an intepreter.

The term "scripting language" is so vague as to be almost useless, but
in this context the suggestion seems to be to use C as a language to
invoke the commands required to compile and link a C program. But C is
very poorly suited to this task. One would end up writing a library of
functions to do string manipulation and (unless 'system' was deemed
sufficient) program execution. C itself would bring very little to the
party and the resulting scripts would be hard to read.

--
Ben.

[Malcolm McLean]

On 05/02/2024 14:42, Ben Bacarisse wrote:
> Malcolm McLean <malcolm.ar...@gmail.com> writes:
>
>> On 04/02/2024 23:32, Ben Bacarisse wrote:
>>> sc...@slp53.sl.home (Scott Lurndal) writes:
>>>
>>>> bart <b...@freeuk.com> writes:
>>>>> On 04/02/2024 15:39, Malcolm McLean wrote:
>>>>>> How shall we write a better build system?
>>>>>>
>>>>>> Current systems use scripts with ad hoc, weird syntax. They don't work
>>>>>> consistently across platforms, they break and are hard to fix, and are
>>>>>> complicated to set up. You even get get one scripting system as a front
>>>>>> end to another, a sort of admission of failure.
>>>>>>
>>>>>> But can we do better?
>>>>>>
>>>>>> A scripting language provides loops, conditional branches, and
>>>>>> subroutine calls. And we've already got a perfecty good language for
>>>>>> describing that. It is C itself.
>>>>>>
>>>>>> So here's the proposal. We implement a C interpreter.
>>>>
>>>> This was tried. It's called the c-shell.
>>>
>>> csh isn't a C interpreter by any stretch of the imagination!
>>
>> No. But the concept of a C interpreter isn't new.
>
> Indeed. But C is not a good fit for interpretation, so there are often
> compromises.
>

It's less of a problem now. The interpreter can just spin up a virtual
machine and then create a memory space. That was less of a practical
approach when machines were more limited.

>> There's one called pico C
>
> Yes, and there's tcc's -run option.
>
>> which is open source and available to be incorporated.
>
> Likewise tcc. So what's wrong (in your opinion) with these two that
> caused you to suggest implementing another C interpreter?
>

Bart writes compilers. So I assumed, wrongly as it turned out, that it
would be very simple and attractive to him to modify one to be an
interpreter. The system would be core C language, plus a library
callable as C functions with facilities such as "glob". Now the obvious
way to implement that is to write the library in C or some other
language, the fold it into the interpreter. tcc might provide an
alternative way of achieving the same thing, but I haven't looked into
it. To fold the library into the interpreter you need the interpreter
source and to be comfortable with it.

>> And a scripting
>> language is an obvious application for an intepreter.
>
> The term "scripting language" is so vague as to be almost useless, but
> in this context the suggestion seems to be to use C as a language to
> invoke the commands required to compile and link a C program. But C is
> very poorly suited to this task. One would end up writing a library of
> functions to do string manipulation and (unless 'system' was deemed
> sufficient) program execution. C itself would bring very little to the
> party and the resulting scripts would be hard to read.
>

I'd dispute "very poorly". Of course you can devise a language which is
better suited to building programs specifically. CMake takes exactly
that approach. So what happens? It's effectively another programming
language to learn. Someone else wrote elaborate CMakes scripts which I
use at work. Sometimes thing go wrong. And then I'm messing about with a
language I hardly ever use and does things in ways I am unfamilar with,
trying to troubleshoot. It would be easier if the CMake scripts were in
C. It might be bit less convenient. But it's just looping and branching
and calling subroutines at the end of the day. As you know full well,
that's all computers can do.

The library would have facilities for working with lists of strings, of
course. But C can do that perfectly well, and I don't see it as big problem.

[bart]

It's actually quite hard to write an interpreter for a statically typed
language, which has to call FFI routines written in native code, which
include call-back functions that will call back into your program; they
will expect a native code function!

It's extra hard when the language is C, because then it is impossible to
draw a line between routines in language X which is being interpreted,
and language Y which is native code on the other side of the FFI.

Since X and Y are both C.

However, I do have an interpreter for a statically typed IL. To
interpret C code, I need to change the C compiler backend to generate
that IL. (And also tweak that IL because it uses a 64-bit execution
core, but C requires a mixed 32/64-bit core.)

So it's still quite a bit of work.

But I'm still unclear as to why you need a C interpreter, rather than
just using whatever C compiler is provided.

If you provide an interpreter, what form would it take; a binary? If it
can be binary, then why not a binary of the app?

And if you provide source code for the interpreter, theb building the
interpreter might be as much of a job as building the main app!

[Keith Thompson]

Ben Bacarisse <ben.u...@bsb.me.uk> writes:
> Malcolm McLean <malcolm.ar...@gmail.com> writes:
>> On 04/02/2024 23:32, Ben Bacarisse wrote:

[...]

>> No. But the concept of a C interpreter isn't new.
>
> Indeed. But C is not a good fit for interpretation, so there are often
> compromises.
>
>> There's one called pico C
>
> Yes, and there's tcc's -run option.

From the tcc man page:

"tcc -run a.c"
Compile a.c and execute it directly

So it's not exactly an interpreter, though it might be useful in some of
the same contexts where an interpreter would be useful. It would be
straightforward to implement a wrapper for any C compiler that does the
same thing.

--
Keith Thompson (The_Other_Keith) Keith.S.T...@gmail.com
Working, but not speaking, for Medtronic
void Void(void) { Void(); } /* The recursive call of the void */

[Malcolm McLean]

The way if would work is that there would be a program called
"fixedmake", or whatever, which was distributed as an executable exactly
like make. The someone writes program source, packages it up in a zip
file, and instead of a makefile, he puts a "fixedmake" file in the
source directory. However the fixedmake file is formally a C program.
Maybe we could have the convention that it is called buildme.c.

The user types fixedmake buildme.c. fixedmake is effectively a C
intepreter. buildme.c executes, calls C compilers and other things as
externals, and the program is built.

Now because buildme.c is a C program, we could use other approaches. You
could go gcc buildme.c -lfixedmake.lib, prodcue an a.out, and run a.out.
But the snag is that it's meant to be a build system. And launching gcc
is in and of itself a build. So it's not really automated any more.

Or we could say that since fixedmake is really just a C intepreter, we
don't have to be ones to write that. Just use tcc. But the problem is
that whilst buildme.c will uusally be fairly short and simple, to keep
it short and simple there will need to a rich library with facilites for
getting lisrs of files, launching compilers, and so on. So if we use
tcc, we've got to package this library as well as buildme.c and the
actual sources in the distribution.

However tcc is open source, so we might be able to modify tcc.

Before the system has any sort of traction, you can't assume that the
fixedmake executable will be available, however. So people are going to
have to provide it with the distribution. And if you provide as binary,
why not just provide the program as binary? And if you provide as source
and build with make, why not build the program with make? But I don't
see any way round that for any new build system.

[Michael S]

On Mon, 5 Feb 2024 16:00:43 +0000
bart <b...@freeuk.com> wrote:
>
> But I'm still unclear as to why you need a C interpreter, rather than
> just using whatever C compiler is provided.
>

One reason could be that when build system's "meta" code (as opposed to
bulk of project's code) has a bug, especially out of bound array
access, I'd rather want it to fail gracefully, with meaningful error
indication, something that typical compiled C environment unable to
provide.

[bart]

On 05/02/2024 16:51, Malcolm McLean wrote:

One problem as I see is that the build system doesn't know which C
compiler is available, or which one the client wishes to use.

The client will know that, so can simply supply that information.
However it is needed in two places:

(1) The compiler used to build build.c (as it's called in the demo
below).

(2) The compiler used by build.exe to build the app.

(1) is easy, the client is just told to use their prefered compiler here:

tcc build.c

But the next step is harder: build.exe won't know what was typed in step
(1). Even if they do 'tcc -run build.c', 'tcc' does not appear as
'args[0]', it will be 'build.c' (not even build.exe).


Here is a mock-up anyway of a build system:

1) The demo app used has three C files cipher.c, hmac.c,sha2.c

2) There is a program called build.c

3) The data describing the project is in a file called here
filelist.txt, although it contains C source code.

Files (2) and (3) are shown below. It runs like this, using tcc's run
option:

c:\c>tcc -run build.c
Compiler = tcc
Invoking compiler:tcc -o cipher.exe cipher.c hmac.c sha2.c
Finished building: cipher.exe

build.c defaults to using tcc. For use with another compiler, the
process looks like this:

c:\c>gcc build.c -o build

c:\c>build gcc
Compiler = gcc
Invoking compiler:gcc -o cipher.exe cipher.c hmac.c sha2.c
Finished building: cipher.exe

It's a litle untidier. However you can always provide a makefile to shut
people up. The difference here is:

* The makefile only contains the handful of lines needed to implement
the above

* The process can be easily be done manually if needed

* The critical build info, the list of files, is inside the file
filelist.txt. While it is C, it can still be presented in a readable
manner. It can even be incorporated into the user's own programs.

The info in filelist.txt is simplistic. It also needs the locations of
any special headers, the libraries to use and and so on. This is all
info added to the final invocation.

Ideally build.c, knowing the name of the compiler, can select the right
forms of any options needed.



--------------------------------------------------------------
filelist.txt
--------------------------------------------------------------
//#define exeext = ".exe"

char* files[] = {
"cipher.c",
"hmac.c",
"sha2.c"
};

int nfiles = sizeof(files)/sizeof(files[0]);

char* projectname = "cipher";
char* exefilename = "cipher.exe";
--------------------------------------------------------------

--------------------------------------------------------------
build.c
--------------------------------------------------------------
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

#include "filelist.txt"

char* compiler;

void error(char* mess, char* param){
printf("Build error: %s %s\n",mess, param);
exit(1);
}

void compileproject(void) {
int n, length, i;
char* cmdstr;

length=0;
for (i=0; i<nfiles; ++i) length+=strlen(files[i])+1;
cmdstr=malloc(length+100);

strcpy(cmdstr, compiler);
strcat(cmdstr, " ");

strcat(cmdstr, "-o ");
strcat(cmdstr, exefilename);
strcat(cmdstr, " ");

for (i=0; i<nfiles; ++i) {
strcat(cmdstr, files[i]);
strcat(cmdstr, " ");
}

printf("Invoking compiler:");
puts(cmdstr);
if (system(cmdstr)!=0)
error("Error building:", exefilename);
else
printf("Finished building: %s\n", exefilename);
}

int main(int n, char** args) {

if (n>=2)
compiler=strdup(args[1]);
else
compiler="tcc";

printf("Compiler = %s\n", compiler);

compileproject();
}
--------------------------------------------------------------

[Chris M. Thomasson]

On 2/5/2024 9:39 AM, bart wrote:
> On 05/02/2024 16:51, Malcolm McLean wrote:
>> On 05/02/2024 16:00, bart wrote:

[...]

>     printf("Invoking compiler:");
>     puts(cmdstr);
>     if (system(cmdstr)!=0)
>         error("Error building:", exefilename);
>     else
>         printf("Finished building: %s\n", exefilename);
> }

[...]

For some reason this is reminding me of how one compiles and links GLSL
shader programs on the fly wrt OpenGL.

[David Brown]

Have you ever tried /any/ other programming language? Apart from
assembly and Forth, I have not seen a language that is more cumbersome
for string handling than C. Yes, you /can/ do string handling in C, but
no one would /choose/ C for that.

It's not unreasonable to suggest that you'd rather base your build
system on an existing mainstream language than a domain-specific
language (though DSL's have the big advantage of having a syntax and
semantics tuned to the task in question). But why C?

Here's a quick challenge for you.

I have a directory "src". I want to find all the .c files in "src". I
want to make a list of all these files, and a list of matching object
files to make in the "build/obj/src" directory. For each file, I want
to call "gcc -c src/file.c -o build/obj/src/file.o". I want to do so in
parallel, up to 8 commands at a time. (Ignore any possible runtime errors.)


With make, that would be (this is untested) :

default : all
.PHONY all
srcdir = "src"
objdir = "build/obj"

srcfiles = $(wildcard $(srcdir)/*.c)
objfiles_src = $(srcfiles:.c=.o)
objfiles = $(addprefix $(objdir)/, \
$(patsubst ../%,%,$(objfiles_src)))

all : objfiles

$(objdir)/src :
mkdir -p $@

$(objdir)/%.o : %.c | $(objdir)/src
gcc -c $< -o $@

Run with "make -j 8".


With Python, you could have (untested) :

import glob
import os
import multiprocessing
import subprocess

srcdir = "src"
objdir = "build/obj"

srcfiles = glob.glob(srcdir + "/*.c")
objfiles_src = [fn[:-2] + ".o" for fn in srcfiles]
objfiles = [objdir + "/" + fn for fn in srcfiles]

os.mkdirs(objdir + "/" + srcdir, exist_ok = True)

with multiprocessing.Pool(5) as pool :
pool.map(lambda src, obj : subprocess.call("gcc -c " +
src + " -o " + obj, shell = True),
zip(srcfiles, objfiles))

Obviously much of this could be put in a reusable library, so that end
users don't have to know about process pools. (And there are many other
ways to structure such code.)


Your task is to duplicate this in C.

[Malcolm McLean]

The newgroup is comp.lang.c. So if we propose using a generally accepted
and widely understood programming language as our build scripting
language, the choice of language has to be C.

#include "builduils.h"

int main(void)
{
STRINGLIST *sources;
STRINGLIST *objfiles;
int i;


sources = glob("src/*.c");
objfiles = stringlist();
for (i = 0; i < stringlist_Nstrings(sources); i++)
{
char objname[1024];
snprintf(objname, 1024, "%s/%s", objdir,
replaceextension(stringlist_get(sources, i), ".o"));
stringlist_add(objfiles, objname);
}
char outputdir[1024];
snprinf("%s/%s", 1024, objdir, srcdir);
mkdir(outputdir);

callparallel("gcc -c %s %s", srcfiles, objfiles);
}

OK now I've cheated a little bit by inventing buildutils library
functions ad hoc. But I haven't done so outrageously. And we're leaking
memory. But likely we have many gigabytres, so who are about maybe 2K
for a few strings. And yes, maybe we could reoplace snprintf() with
something that creates an arbitrary length output.

The Python is a bit shorter and more concise. But there's not that much
in it.

[Michael S]

On Mon, 5 Feb 2024 20:47:53 +0100
David Brown <david...@hesbynett.no> wrote:
>
> Have you ever tried /any/ other programming language? Apart from
> assembly and Forth, I have not seen a language that is more
> cumbersome for string handling than C.
>

In order of my exposure to them: Pascal, Fortran, Ada.

[bart]

On 05/02/2024 19:47, David Brown wrote:

> On 05/02/2024 16:17, Malcolm McLean wrote:

>> The library would have facilities for working with lists of strings,
>> of course. But C can do that perfectly well, and I don't see it as big
>> problem.
>>
>
> Have you ever tried /any/ other programming language?  Apart from
> assembly and Forth, I have not seen a language that is more cumbersome
> for string handling than C.  Yes, you /can/ do string handling in C, but
> no one would /choose/ C for that.

See the build.c demo elsewhere in the thread.

> I have a directory "src".  I want to find all the .c files in "src".  I
> want to make a list of all these files, and a list of matching object
> files to make in the "build/obj/src" directory.  For each file, I want
> to call "gcc -c src/file.c -o build/obj/src/file.o".  I want to do so in
> parallel, up to 8 commands at a time.  (Ignore any possible runtime
> errors.)

> With Python, you could have (untested) :
>
>     import glob
>     import os
>     import multiprocessing
>     import subprocess
>
>     srcdir = "src"
>     objdir = "build/obj"
>
>     srcfiles = glob.glob(srcdir + "/*.c")
>     objfiles_src = [fn[:-2] + ".o" for fn in srcfiles]
>     objfiles = [objdir + "/" + fn for fn in srcfiles]
>
>     os.mkdirs(objdir + "/" + srcdir, exist_ok = True)
>
>     with multiprocessing.Pool(5) as pool :
>         pool.map(lambda src, obj : subprocess.call("gcc -c " +
>                  src + " -o " + obj, shell = True),
>                 zip(srcfiles, objfiles))
>
> Obviously much of this could be put in a reusable library, so that end
> users don't have to know about process pools.  (And there are many other
> ways to structure such code.)

I had a go at expressing the Python version in my scripting language:

srcdir := "src/"
objdir := "build/obj/"+srcdir

srcfiles := dirlist(srcdir + "*.c")

objfiles_src := mapvs(changeext, srcfiles, "o")
objfiles := mapsv(+, objdir, objfiles_src)

createdir(objdir)

for i, sfile in srcfiles do
execcmd(sfprint("gcc -c # -o #", srcdir+sfile, objfiles[i]))
od

A couple of issues: 'createdir' can only create one directory at a time;
for a chain of them like a/b/c, I'd need to split it up and do it one by
one. (To test this, I created it manually.)

Also, I don't have any features for parallel executions. However
'execcmd' starts a process but then doesn't wait for it to complete. If
I instead use 'system' (or my 'execwait'), then compiling all Lua .c
files in ./src takes 6.5 seconds instead of 2.5 seconds.

But this isn't the point of posting this. My interpreter can be
expressed as a single C file. That allows you to use this kind of
scripting, without adding any extra dependencies. You still only need a
C compiler.

(I tried your Python, but os.mkdir(dir) didn't work, even after fixing
the name typo and removing the named argument. Bypassing that, there
were all sorts of errors to do with pickle.py and 'unbounded' methods.

I believe your glob.glob routine returns filenames with path prepended
(my dirlist doesn't). I suspect that's why you chose a dest path as
build/obj/src rather than build/obj. Anyway, I wasn't able to compare
peformance.)

BTW this scripting is still hard work. For building you want to express
the requires as data, not code. If you can do that (see my build.c
example), then C may be adequate.

[Keith Thompson]

Malcolm McLean <malcolm.ar...@gmail.com> writes:
> On 05/02/2024 19:47, David Brown wrote:

[...]

>> Have you ever tried /any/ other programming language?  Apart from
>> assembly and Forth, I have not seen a language that is more
>> cumbersome for string handling than C.  Yes, you /can/ do string
>> handling in C, but no one would /choose/ C for that.
>> It's not unreasonable to suggest that you'd rather base your build
>> system on an existing mainstream language than a domain-specific
>> language (though DSL's have the big advantage of having a syntax and
>> semantics tuned to the task in question).  But why C?
>>
> The newgroup is comp.lang.c. So if we propose using a generally
> accepted and widely understood programming language as our build
> scripting language, the choice of language has to be C.

Right -- *if* we want to discuss it here in comp.lang.c.

If C is not a suitable language for a new build systems, the solution is
to discuss it somewhere else, not to discuss it here and pretend C is
the best answer.

[Lawrence D'Oliveiro]

On Mon, 5 Feb 2024 15:17:15 +0000, Malcolm McLean wrote:

> It would be easier if the CMake scripts were in
> C. It might be bit less convenient.

Aren’t “easier” and “less convenient” kind of ... opposites?

[Lawrence D'Oliveiro]

On Mon, 5 Feb 2024 21:28:20 +0000, bart wrote:

> (I tried your Python, but os.mkdir(dir) didn't work, even after fixing
> the name typo and removing the named argument.

“mkdirs” should have been “makedirs”
<https://docs.python.org/3/library/os.html#os.makedirs>.

> Bypassing that, there
> were all sorts of errors to do with pickle.py and 'unbounded' methods.

Trouble with your Python install? In other words, Windows trouble?

[Lawrence D'Oliveiro]

On Mon, 5 Feb 2024 17:39:25 +0000, bart wrote:

> strcat(cmdstr, "-o ");
> strcat(cmdstr, exefilename);
> strcat(cmdstr, " ");
>
> for (i=0; i<nfiles; ++i) {
> strcat(cmdstr, files[i]);
> strcat(cmdstr, " ");
> }
>

> ...
> if (system(cmdstr)!=0)

What is wrong with this code?

[Malcolm McLean]

Less convenient for the whizzy suoer skilled build system programmeer to
write. Easier for the humble C programmer who has it fall over on him to
fix.

[Lawrence D'Oliveiro]

On Tue, 6 Feb 2024 01:41:56 +0000, Malcolm McLean wrote:

> On 05/02/2024 23:34, Lawrence D'Oliveiro wrote:
>
>> On Mon, 5 Feb 2024 15:17:15 +0000, Malcolm McLean wrote:
>>
>>> It would be easier if the CMake scripts were in C. It might be bit
>>> less convenient.
>>
>> Aren’t “easier” and “less convenient” kind of ... opposites?
>
> Less convenient for the whizzy suoer skilled build system programmeer to
> write. Easier for the humble C programmer who has it fall over on him to
> fix.

But the C code will be anything up to an order of magnitude larger than
the build rules written in the domain-specific language. So more work
required overall.

[Malcolm McLean]

Well yes. If the program to build the program has to be more elaborate
than the porgram itself, something is going wrong. But whilst the CMake
scripts are quite short, they aren't trivial, and there's a lot of
fussing about needed to pull in the sources, set the IDE uo in the right
way, and then make sure that the non-IDE build is exactly the same. But
if it falls over I can't work, and you don't like to call in someone
else unless it's established that the script itself actually needs
modification. So you tend to trouble shoot. But whilst I do write my own
CMkae scripts, I don't write them often enough to really know the
language, and I just forget all the details. So troubleshooting a CMake
scriot is quite hard work. Not as hard as debugging a complex C
algorithm. But that's my core job, whilst the build system is not.

[Lawrence D'Oliveiro]

On Tue, 6 Feb 2024 02:59:11 +0000, Malcolm McLean wrote:

> But whilst the CMake scripts are quite short, they aren't trivial ...

I know, I’ve fiddled with CMake scripts myself. The power is needed
because builds can be complex.

> But whilst I do write my own CMkae scripts, I don't write them often

> enough to really know the language ...

Same here. That’s why, every time I need to touch a CMake script, I make
sure to have the docs open in a browser window
<https://cmake.org/documentation/>. I don’t fiddle at random: I try to
understand what I am doing. And sometimes I find the person who fiddled
the script before me didn’t understand that there was a simpler way to do
something.

[Chris M. Thomasson]

Putting the build rules in source code, is a bad idea, TM?

[Chris M. Thomasson]

On 2/5/2024 7:09 PM, Lawrence D'Oliveiro wrote:
> On Tue, 6 Feb 2024 02:59:11 +0000, Malcolm McLean wrote:
>
>> But whilst the CMake scripts are quite short, they aren't trivial ...
>
> I know, I’ve fiddled with CMake scripts myself. The power is needed
> because builds can be complex.
>
>> But whilst I do write my own CMkae scripts, I don't write them often
>> enough to really know the language ...
>
> Same here. That’s why, every time I need to touch a CMake script, I make
> sure to have the docs open in a browser window
> <https://cmake.org/documentation/>.

Nothing wrong with that! :^)

[Lawrence D'Oliveiro]

Build rules *are* source code.

[Chris M. Thomasson]

Well, trying to put something that is akin to a makefile into source
code is a bad idea? Any better?

[Malcolm McLean]

CMake is a programming language in its own right and there's no
ambiguity about that, though it is special purpose one unsuitable to
write anything except build scripts. make is syntax for instructions to
a program which builds programs. There might be something in it which is
technically Turing compatible and thus can be used to create arbitrary
behaviour, but if true it will be difficult to do and abuse of the
intended purpose.

C is a general purpose programming language. However if you provide a
library of functions designed to help with building programs then it
becomes effectively a special purpose programming language, but still
with the general purpose grammar. It's still suitable for general
purpose programming and so less suitable for wriitng build scripts than
make, but the language is familar.

CMake and make both gained wide traction. Whilst apparently build
scripts using C as the scripting language have been tried before and
it's not a totally new idea, it's been tried out rather than seriously
tried.

[David Brown]

No, it does not. It is perfectly reasonable to say "not C".

Yes. Of course a practical solution would split the common code into a
library, and not expect people to re-write that all the time. But a key
point here is that in make and in Python - both well-established
languages and tools - this is already in place. But you are skipping
the hard bits. (Good luck implementing "callparallel" with a sane
interface that includes the call you have made here!)

We know it is /possible/ to do this kind of thing in C. We also know it
is ugly and fiddly compared to other languages that handle strings and
lists as first-class types (or at least have them as classes). Even
when you assume the existence of a library with every type and function
you want, you still need a lot of extra boilerplate code, you have
artificial limits, memory leaks, etc.

[David Brown]

Your scripting language is definitely an improvement over C for this
kind of thing. That is as I would expect - scripting languages tend to
be much better at working with strings and external commands.

>
> Also, I don't have any features for parallel executions. However
> 'execcmd' starts a process but then doesn't wait for it to complete. If
> I instead use 'system' (or my 'execwait'), then compiling all Lua .c
> files in ./src takes 6.5 seconds instead of 2.5 seconds.
>
> But this isn't the point of posting this. My interpreter can be
> expressed as a single C file. That allows you to use this kind of
> scripting, without adding any extra dependencies. You still only need a
> C compiler.
>
> (I tried your Python, but os.mkdir(dir) didn't work, even after fixing
> the name typo and removing the named argument. Bypassing that, there
> were all sorts of errors to do with pickle.py and 'unbounded' methods.
>

"os.mkdirs()" was not a typo. "mkdirs()" is a different function from
"mkdir()" - it can make multiple directories as needed. But I may have
had other errors.

> I believe your glob.glob routine returns filenames with path prepended
> (my dirlist doesn't).

Yes.

> I suspect that's why you chose a dest path as
> build/obj/src rather than build/obj.

No. I used that because it is the structure I use for bigger projects.
I generally have multiple source directories, and in my build
directories I have "obj", "deps" (for dependency files), perhaps also
directories for listings files and other bits and pieces, and inside
these the directory structure mirrors the source tree. (I actually
often have another layer between "build" and "obj" because there can be
multiple different builds for the same project.)

> Anyway, I wasn't able to compare
> peformance.)

There was no intention of that. I merely wanted to show that something
that is key to software building can be done relatively easily in a
higher-level language suitable for scripting, in a way that is simpler
and easier than in C.

It was not a full build in any sense, and did not cover dependency graphs.

>
> BTW this scripting is still hard work. For building you want to express
> the requires as data, not code. If you can do that (see my build.c
> example), then C may be adequate.
>

Eventually, when you add all the features and flexibility people want,
you end up with a domain-specific language as you data, interpreted by
code written in some other language (C or whatever - though again I
expect other languages to be more suitable). You've just re-invented
"make".

Of course it is possible to re-invent make in a better way than the
original (at least for some people's preferences and needs - "better" is
always subjective). If I were doing so, I'd want an embedded high-level
scripting language of some sort that was not a variant of Lisp. (Some
people like Lisp, I don't.) Possible options from existing languages
would be Lua, TCL and Python. And I'd want it neatly integrated, not
just a way to make extensions, as that would simplify the main DSL.
(There's plenty of other things I'd change, but I don't want to go
through a list here.)

[David Brown]

On 06/02/2024 00:38, Lawrence D'Oliveiro wrote:

> On Mon, 5 Feb 2024 21:28:20 +0000, bart wrote:
>
>> (I tried your Python, but os.mkdir(dir) didn't work, even after fixing
>> the name typo and removing the named argument.
>
> “mkdirs” should have been “makedirs”
> <https://docs.python.org/3/library/os.html#os.makedirs>.
>

Oh, so it was a typo - the "s" was right, but not the "mk" :-(

The code was just for reference, not tested or complete.