Module libraries

[Thiago Adams]

(I really don't know what title to use.)

I started a topic on comp lang c, but this topic
is also relevant for C++.

This is my suggestion for C and C++ and for tools.

** What problem do I want to solve? **

I want copy-paste libraries from their repository
(with source) and use in my projects as easily as if
they were header only libraries or amalgamated.

The proposal is here

https://github.com/thradams/Modules/blob/master/README.md

Basically, we can add #pragma source in your source code
and this tell the compiler what are the source code necessary
to build the program.

This does NOT solve the same problem of C++ modules.
This does NOT solve the same problem of make ,
or general build systems but it can be complementary.

SAMPLE
Library to use console: (Folders)

Console
Src
ConsoleWin.c
ConsoleLinux.c
UnitTest.c
Include
Console.h


At Console.h

/*
Console.h
*/

#ifdef WIN32
#pragma source "..\Scr\ConsoleWin.c"
#elif LINUX
#pragma source "..\Scr\ConsoleLinux.c"
#endif

...

-------------

To use this library

/*
MyProgram.c
*/

#include "Console.h"

int main()
{
}
--------

to compile this program: (if it was part of the compiler)

ccompiler MyProgram.c

[Jorgen Grahn]

On Fri, 2018-11-02, Thiago Adams wrote:
> (I really don't know what title to use.)
>
> I started a topic on comp lang c, but this topic
> is also relevant for C++.
>
> This is my suggestion for C and C++ and for tools.
>
> ** What problem do I want to solve? **
>
> I want copy-paste libraries from their repository
> (with source) and use in my projects as easily as if
> they were header only libraries or amalgamated.

That's what you want to do, but it doesn't explain what
problem you want to solve.

(Just pointing it out. I suspect I wouldn't see it as a problem,
or as worse than the problems this would cause.)

/Jorgen

--
// Jorgen Grahn <grahn@ Oo o. . .
\X/ snipabacken.se> O o .

[Thiago Adams]

On Friday, November 2, 2018 at 4:52:38 PM UTC-3, Jorgen Grahn wrote:
> On Fri, 2018-11-02, Thiago Adams wrote:
> > (I really don't know what title to use.)
> >
> > I started a topic on comp lang c, but this topic
> > is also relevant for C++.
> >
> > This is my suggestion for C and C++ and for tools.
> >
> > ** What problem do I want to solve? **
> >
> > I want copy-paste libraries from their repository
> > (with source) and use in my projects as easily as if
> > they were header only libraries or amalgamated.
>
> That's what you want to do, but it doesn't explain what
> problem you want to solve.
>
> (Just pointing it out. I suspect I wouldn't see it as a problem,
> or as worse than the problems this would cause.)
>

Yes, thanks for the feedback.

Problem:

- Libraries with multiple source files are much harder to use
compared with header only libraries or amalgamated source
code.

- There is no standard way to say what source files are necessary
to create a library/program. If you want to create a lint program
,for instance, you need to read visual studio solution or xcode solution
or invent a new format to say what files should be parsed.

- Binary libraries generates explosion of combinations for each
platform compiler settings making them hard to use as well.

for those who have already used #pragma comment(lib , "") on VC++
can have the felling of remove outside compiler configuration
to something inside the code , many times surrounding by #ifdef.
The suggested feature #pragma source, #pragma includedir etc also
would give similar power but in a standard way (if possible)

[Paavo Helde]

So you say you don't care how and with what options and other quirks the
library author wanted to build their library, you just want to compile
it afresh with whatever options your project happens to have and
incorporate the result in your project.

Fair enough, I believe this is a legitimate wish, figuring out how to
build yet another special snowflake library can be a PITA. The good news
this functionality is already there:

Step 1:

ConsoleWin.c:
#ifdef WIN32
//....
#endif

ConsoleLinux.c
#ifdef LINUX
// ...
#endif

UnitTest.c
#ifdef SAMPLE_UNITTEST
// ...
#endif

In your project, add ../otherproject/*.c or "Add Existing:Select All" to
your sources.

Voila, done. Works fine for some Boost libraries, for example.

Cheers
Paavo

[Thiago Adams]

The compiler options can be passed at the beginning. But, the same compiler
options will be used in all files unless we can specify this compiler
setting per file. (For instance, this is possible for warnings)

> Fair enough, I believe this is a legitimate wish, figuring out how to
> build yet another special snowflake library can be a PITA. The good news
> this functionality is already there:
>
> Step 1:
>
> ConsoleWin.c:
> #ifdef WIN32
> //....
> #endif
>
> ConsoleLinux.c
> #ifdef LINUX
> // ...
> #endif
>
> UnitTest.c
> #ifdef SAMPLE_UNITTEST
> // ...
> #endif
>
> In your project, add ../otherproject/*.c or "Add Existing:Select All" to
> your sources.
>
> Voila, done. Works fine for some Boost libraries, for example.
>

The real code will have several files. This solution does not scale
and don't compose well compared with this pragma source.

[Paavo Helde]

Of course there are several files. Did you overlook the '*' and 'Select
All'?

> This solution does not scale
> and don't compose well compared with this pragma source.

The pragma source proposal has its charm, but there is a slight problem
in that it does not exist. I also suspect that it would overlap or
conflict with the upcoming C++ modules feature. And it also don't scale
in the sense that pretty often the source code needs some special
compiler flags, prebuild and postbuild steps, etc, which might depend on
the platform, configuration, etc. You cannot replace the myriad of the
existing build systems with a single pragma.

In simpler cases it would work though.

[David Brown]

Libraries come in three varieties.

There are header-only libraries - as you say, these are usually fairly
easy to handle.

There are stand-alone libraries. The usually come with their own build
procedures - ./configure, makefiles, CMake, etc. You handle these by
following the build instructions for the library.

Then there are libraries that come as source code to be compiled and
included along with your own project. These are common in the embedded
world, where you don't have shared or dynamic libraries. Sometimes
these could do with better documentation or information, especially if
they have odd restrictions (such as requiring non-standard C behaviour
like wrapping signed integers or gcc's -fno-strict-aliasing flag). But
generally it is easy to see what source files are needed.


Sometimes libraries have particular needs to make them easy to use -
such as lists of include file directories or lists of static libraries
for linking. The "pkg-config" solution is to have a ".pc" file with the
library that gives this information. I think that if you feel these
files don't have all the details you need, it would make more sense to
extend this existing system instead of trying to invent something new.

[Thiago Adams]

The problem with * is that you don't know what files
are necessary, and adding many #ifdefs SAMPLE_UNITTEST
is a lot of work. Also it is intrusive. The pragma source
can be intrusive or not. In the other words, if you have
an existing library someone can generate the file that
represents the source for that library in a way the orignal
library don´t need to be changed. But if it is something standard
would be nice to have it for all libraries. This is also something
that will not broke old compilers.

> > This solution does not scale
> > and don't compose well compared with this pragma source.
>
> The pragma source proposal has its charm, but there is a slight problem
> in that it does not exist. I also suspect that it would overlap or
> conflict with the upcoming C++ modules feature.

C++ modules feature is similar of binary libs. You need to compile
for each platform. Macros are also not available at the same form,
and it will not be compatible with C.

> And it also don't scale
> in the sense that pretty often the source code needs some special
> compiler flags, prebuild and postbuild steps, etc, which might depend on
> the platform, configuration, etc.

Please note that this feature is not a build system and it doesn't
replace build systems. This has been a source of confusion. The
name of the feature could be "new pragmas" to avoid confusion.
We can have specific platform settings. I also would like to have
compiler setting inside the source. Something similar of what VC++ has
for warnings

#pragma warning( push )
#pragma warning( disable : 4705 )
#pragma warning( disable : 4706 )
#pragma warning( disable : 4707 )

// Some code

#pragma warning( pop )

Some of the compiler flags could be universal, but this
is not necessary. If the compiler had this setting on the source
the proposed feature could be used together with #ifdef for
an specific compiler.
The way it is today, when you specify compiler settings it will be
applied for all sources.

>You cannot replace the myriad of the
> existing build systems with a single pragma.
>
> In simpler cases it would work though.

Usually my build is for windows, and I have to sign the executable,
generate documentation, tests, run the installer program,sign the installer
and publish. Also I have to do some parts on different computers (mac).
This feature will not replace this build.

[Pavel]

Thiago Adams wrote:
> (I really don't know what title to use.)
>
> I started a topic on comp lang c, but this topic
> is also relevant for C++.
>
> This is my suggestion for C and C++ and for tools.
>
> ** What problem do I want to solve? **
>
> I want copy-paste libraries from their repository
> (with source) and use in my projects as easily as if
> they were header only libraries or amalgamated.

This problem is quite old. One issue that will arise with a naive solution is
that the subset of source files needed to build a program or library changes as
new versions appear. Also, the "universal subset" (i.e. the total set of
sources) is usually a set of files by some version control system (VCS) so it is
not surprising the first (known to me) working solution appeared as part of (one
of the very first) VCS, specifically SCCS.

You can declare source file in the file itself (you will usually do it
automatically as you create a new source, probably using same procedure that
adds a copyright header); then use "sccs what" command on a binary and get a set
of all sources. You do not have to use SCCS for actual version control; although
using some keyword-expanding vcs like CVS,Subversion or RCS provides additional
benefits in that you can grab not only the complete file set but also the exact
versions with which a binary was built.

The syntax for this or similar feature is unfortunately different on different
platforms but it often does involve pragmas. E.g. sometimes as #sccs directive
(works on most Unices), #ident (works on gcc), #pragma comment (MSVC), #pragma
ident etc.

See e.g.
https://stackoverflow.com/questions/15773282/access-ident-information-in-an-executable
for some example.

>
> The proposal is here
>
> https://github.com/thradams/Modules/blob/master/README.md
>
> Basically, we can add #pragma source in your source code

According to what rule will you decide to which source file add the pragmas? (I
mean, you don't want to add

#pragma source "..\Scr\ConsoleWin.c"

to every other C++ file, do you? Are you going to designate "a single source
file per program or library to contain them all"? If yes, why does it even has
to be a C/C++ file -- its role is clearly providing list of files rather than
C/C++ code).

-Pavel

[Thiago Adams]

I don't think it a good idea to increase the size to keep
this information.

> for some example.
> >
> > The proposal is here
> >
> > https://github.com/thradams/Modules/blob/master/README.md
> >
> > Basically, we can add #pragma source in your source code
> According to what rule will you decide to which source file add the pragmas? (I
> mean, you don't want to add
> #pragma source "..\Scr\ConsoleWin.c"
> to every other C++ file, do you? Are you going to designate "a single source
> file per program or library to contain them all"? If yes, why does it even has
> to be a C/C++ file -- its role is clearly providing list of files rather than
> C/C++ code).

We have two options using the same feature.

One option is intrusive.
If you create a new library you can just add in your header
the corresponding source.

file1.h
-----------

/*
File1.h
*/

#pragma once
#pragma source "File1.c"
...

---

The other option is non intrusive. You can create a different file
to describe all sources. This could be a build.txt. But .h is a
good extension. The will expand the pragma source.

I also would like pragma includedir , pragma lib .

And pragma once span.

pragma once span is to reuse the parsed header in more than
one source file. It will take in account the macros for the
first inclusion and after that it will not be expanded anymore.

[Christian Gollwitzer]

Am 02.11.18 um 20:16 schrieb Thiago Adams:

> At Console.h
>
> /*
> Console.h
> */
>
> #ifdef WIN32
> #pragma source "..\Scr\ConsoleWin.c"
> #elif LINUX
> #pragma source "..\Scr\ConsoleLinux.c"
> #endif
>

What's the difference to


#ifdef WIN32
#include "..\Scr\ConsoleWin.c"
#elif LINUX
#include "..\Scr\ConsoleLinux.c"
#endif

?
Christian

[Thiago Adams]

Include expand the file. For instance, if you had
two function static void F() in two different files
this would be a problem.

#pragma source collects the source file name and
at the end of the compilation of that unit, it will
compile the sources that have not yet been compiled
until all the sources are compiled.

[David Brown]

On 05/11/18 12:21, Thiago Adams wrote:
> On Monday, November 5, 2018 at 6:37:54 AM UTC-2, Christian Gollwitzer wrote:
>> Am 02.11.18 um 20:16 schrieb Thiago Adams:
>>> At Console.h
>>>
>>> /*
>>> Console.h
>>> */
>>>
>>> #ifdef WIN32
>>> #pragma source "..\Scr\ConsoleWin.c"
>>> #elif LINUX
>>> #pragma source "..\Scr\ConsoleLinux.c"
>>> #endif
>>>
>>
>> What's the difference to
>>
>>
>> #ifdef WIN32
>> #include "..\Scr\ConsoleWin.c"
>> #elif LINUX
>> #include "..\Scr\ConsoleLinux.c"
>> #endif
>>
>
> Include expand the file. For instance, if you had
> two function static void F() in two different files
> this would be a problem.
>

Only one of the two files here would be included - so there would be no
conflict.

[Thiago Adams]

On Monday, November 5, 2018 at 9:51:44 AM UTC-2, David Brown wrote:
> On 05/11/18 12:21, Thiago Adams wrote:
> > On Monday, November 5, 2018 at 6:37:54 AM UTC-2, Christian Gollwitzer wrote:
> >> Am 02.11.18 um 20:16 schrieb Thiago Adams:
> >>> At Console.h
> >>>
> >>> /*
> >>> Console.h
> >>> */
> >>>
> >>> #ifdef WIN32
> >>> #pragma source "..\Scr\ConsoleWin.c"
> >>> #elif LINUX
> >>> #pragma source "..\Scr\ConsoleLinux.c"
> >>> #endif
> >>>
> >>
> >> What's the difference to
> >>
> >>
> >> #ifdef WIN32
> >> #include "..\Scr\ConsoleWin.c"
> >> #elif LINUX
> >> #include "..\Scr\ConsoleLinux.c"
> >> #endif
> >>
> >
> > Include expand the file. For instance, if you had
> > two function static void F() in two different files
> > this would be a problem.
> >
>
> Only one of the two files here would be included - so there would be no
> conflict.

I am answering about the general problem, not about this
specific case with only two files.

[Christian Gollwitzer]

Am 05.11.18 um 13:05 schrieb Thiago Adams:

I was asking because I do not yet fully understand what you are
proposing. What problem are you trying to solve that is not possible at
the moment? So, if in the above case there is no difference, could you
please provide an example where there is a difference between #pragma
source and #include?

#include yould also be accompanied by #pragma once (in the file), if
needed.

Christian

[Pavel]

From my practice, I don't recall when that one string per source file created
any size issues (the overhead is usually less than 1-2%). Your mileage can vary,
of course. A good thing it gives you a certainty that a particular binary is
built from particular set of sources; without injecting a string to the binary,
this cannot be done (you might be able to inject something to debug sections,
then strip these off though; but this is not what #ident / #scsi do).

Are you going to actually compile this file or just pre-process?.

If the latter, you probably don't need new pragmas; in fact, you don't even need
the pre-processing result's to be a valid C/C++ file. You could just have
something like


#ifdef WIN32
..\Scr\ConsoleWin.c
#elif defined(linux)
..\Scr\ConsoleLinux.c
#endif

, compile it, then use the result in your builder (e.g. compile in sub- make and
include the result to a higher level make).

-Pavel

[Thiago Adams]

Let's say you have a library using 4 files:

file1.h file1.c
file2.h file2.c

Then you just copy-paste for your project
and include file1.h.

/*
MyProgram.c
*/
#include "file1.h"

int main()
{
}


To compile this program (if the proposal was implemented
inside the compiler) just do:

ccompiler MyProgram.c


/*file1.h*/
#pragma source "file1.c"

/*file1.c*/
#include "file2.h"


/*file2.h*/
#pragma source "file2.c"

/*file2.c*/
#include "file2.h"

When you compile MyProgram.c it add "file1.c" in a
list "to-compile".
Then it will compile "file1.c" that will add "file2.c" into
the list "to-compile"
Then "file2.c" will be compiled and the list "to-compile" will
be empty.
After that linker can work.

Why not #include?
If file1.c and file2.c had some internal function
like static void F() and you include both you end up
with a problem of duplicated functions.
One problem solved is to make easy and universal the
use of libraries.
Other problem solved is to have an universal description
of what files are used. This is useful for many other
programs like lint, automatic documentation etc..

[Thiago Adams]

Is this question for pragma once span?

The idea is similar of precompiled headers.
In Visual C++ the first included header must be the precompiled
header.
This avoids to have something before the header.
#pragma once span
allow you to add the file in any position, what is important
is to understand that previous macros will not affect the parser
after the first expansion even in different translation units.

Then if you have
#include <stdio.h>
at file1.c

and at file2.c

#include <stdio.h>

the compiler can reuse the parsing of <stdio.h>
at the second file. (just like precompiled headers)

[Pavel]

No, the question is why do you need any pragmas at all if you can use C/C++
preprocessor on your xxxlib.h to generate the list of source files and use it in
your repository retrieval and build systems.