FLTK's CMake FLTK-Targets broken on Windows with MSVC DLLs

[Gonzalo Garramuño]

On Windows, with MSVC as compiler, running on an MSYS64 shell, when
compiling dynamic DLLs (-DFLTK_BUILD_SHARED_LIBS=ON), there are a number
of issues:

- The fltk_gl DLL is not created.  A .lib is (not sure if it is static
or a DLL .stub).
- There's both a fltk.lib and a libfltk.lib in the lib/ directory
(again, not sure which one is static and which one is a DLL stub).
- The DLLs in the bin/ directory are called lib* when the convention on
Windows is to not use the lib prefix.  The lib/ (stub?) files, except
for libfltk.lib do not have the lib prefix.


when running find_package(FLTK REQUIRED NOCONFIG) with the generated
FLTKConfig.cmake and FLTK-Targets, I get:

CMake Error at
D:/code/applications/mrv2/BUILD-Msys-amd64/Release/install/CMake/FLTK-Targets.cmake:151
(message):
  The imported target "fltk_z_SHARED" references the file

"D:/code/applications/mrv2/BUILD-Msys-amd64/Release/install/lib/libfltk_z.lib"

  but this file does not exist.  Possible reasons include:

  * The file was deleted, renamed, or moved to another location.

  * An install or uninstall procedure did not complete successfully.

  * The installation package was faulty and contained

"D:/code/applications/mrv2/BUILD-Msys-amd64/Release/install/CMake/FLTK-Targets.cmake"

  but not all the files it references.

Call Stack (most recent call first):
D:/code/applications/mrv2/BUILD-Msys-amd64/Release/install/CMake/FLTKConfig.cmake:31
(include)
  CMakeLists.txt:34 (find_package)


--
Gonzalo Garramuño
ggar...@gmail.com

[Ian MacArthur]

On Wednesday, 4 October 2023 at 23:04:22 UTC+1  Gonzalo  wrote:

On Windows, with MSVC as compiler, running on an MSYS64 shell, when
compiling dynamic DLLs (-DFLTK_BUILD_SHARED_LIBS=ON), there are a number
of issues:

I don't think I've ever tried that (using cl with cmake from an Msys shell.) 

I have used cl and cmake together from a VS  "developer shell" (or whatever it is they call the DOS box with the MSVC bits set up) and that has pretty much always Just Worked.

And of course recent VS versions seem to support cmake pretty reliably inside the IDE too.

I guess the mixing of MSYS and cl is confusing cmake in some way? (Maybe cmake expects Msys to be mingw perhaps?)

FWIW, a long while ago now, I used a wrapper I'd written that allowed me to invoke cl as if it were gcc/g++ from an Msys makefile, but I doubt very much that would work for cmake!

 

[Gonzalo Garramuño]

El 10/5/2023 a las 9:13 AM, Ian MacArthur escribió:

On Wednesday, 4 October 2023 at 23:04:22 UTC+1  Gonzalo  wrote:

On Windows, with MSVC as compiler, running on an MSYS64 shell, when
compiling dynamic DLLs (-DFLTK_BUILD_SHARED_LIBS=ON), there are a number
of issues:

I don't think I've ever tried that (using cl with cmake from an Msys shell.) 

The problem is described on the CMakeLists.txt files and it has nothing to do with the compilation environment.  It is an issue entirely with FLTK.

First, the issue is that FLTK's CMakeLists.txt files don't properly support creating shared and static libs independently or together.  Currently, statict libs are always created, which, on MSVC, also have a .lib extension.

On Windows, with the MSVC and DLL builds, only one DLL is created that has all the libraries (fltk_gl, fltk, fltk_forms, etc) in it.  Why is that is not clearly explained as it is really easy to have multiple DLLs.   The main issue is the function fl_add_library() in the functions.cmake.

-- 
Gonzalo Garramuño
ggar...@gmail.com

[Gonzalo Garramuño]

El 10/5/2023 a las 9:13 AM, Ian MacArthur escribió:

On Wednesday, 4 October 2023 at 23:04:22 UTC+1  Gonzalo  wrote:

On Windows, with MSVC as compiler, running on an MSYS64 shell, when
compiling dynamic DLLs (-DFLTK_BUILD_SHARED_LIBS=ON), there are a number
of issues:

I don't think I've ever tried that (using cl with cmake from an Msys shell.) 

The problem is not the compile environment but an issue entirely with FLTK's CMakeLists.txt files.  It is, partially, explained there in the comments.

FLTK does not allow turning off static lib compilation and always builds static one libraries.  The problem with MSVC is that static libs and shared lib stubs both carry the .lib extension, which is confusing and since FLTK creates both, you'd get them overwritten.

To work around the issue, Albretch I think, when compiling with MSVC and shared builds, FLTK creates the static libs with a .lib extension, and only *ONE* DLL/.lib that has all the FLTK libs in it (fltk_gl, fltk, fltk_forms and fltk_images) with a "lib" *PREFIX*.  It cannot get more confusing than that!  Why not use a _SHARED or _DLL *SUFFIX* which would make it more clear, is beyond me.

IMHO, the fl_add_library() function defined in fl_add_library.cmake in the FLTK distribution is broken for MSVC.  What's bad about compiling FLTK shared with MSVC is that there's no way to know the flags, compile switches or libraries from anywhere, as fltk-config just records the flags for the static libs and CMake's Config/Targets.cmake imports is built broken as you have only a single DLL, instead of several.

I did a dirty manual hack on my viewer to make it work, by listing all the libs on Windows and had to pass the system DLLs too and the -DFL_DLL flag.  But I would like, with Albretch permission, modify the FLTK CMake files to make them work properly with CMake find_package() and multiple DLLs.  If I can make it work, I would submit a PR.

Gonzalo Garramuño
ggar...@gmail.com

[Gonzalo Garramuno]

> El 5 oct 2023, a las 10:46, Gonzalo Garramuño <ggar...@gmail.com> escribió:

> The problem is described on the CMakeLists.txt files and it has nothing to do with the compilation environment. It is an issue entirely with FLTK.
> First, the issue is that FLTK's CMakeLists.txt files don't properly support creating shared and static libs independently or together. Currently, statict libs are always created, which, on MSVC, also have a .lib extension.
> On Windows, with the MSVC and DLL builds, only one DLL is created that has all the libraries (fltk_gl, fltk, fltk_forms, etc) in it. Why is that is not clearly explained as it is really easy to have multiple DLLs. The main issue is the function fl_add_library() in the functions.cmake.
>

I took a closer look and I see the difficulty. The static libs are always created as the test, examples and more importantly fluid link to them. This is mainly a convenience to avoid users having DSO troubles with them.

However, it should still be possible to create a CMake Targets.cmake file for MSVC that links to the one and only DLL and has the compilation flags.

With that, users of FLTK would then be able to just do:

find_package(FLTK REQUIRED)
Target_link_libraries( myExe WIN32 FLTK::FLTK )

Which is the way CMake is supposed to be used.


—
Gonzalo Garramuno
ggar...@gmail.com

[Matthias Melcher]

ggar...@gmail.com schrieb am Freitag, 6. Oktober 2023 um 00:30:07 UTC+2:

I took a closer look and I see the difficulty. The static libs are always created as the test, examples and more importantly fluid link to them. This is mainly a convenience to avoid users having DSO troubles with them.

However, it should still be possible to create a CMake Targets.cmake file for MSVC that links to the one and only DLL and has the compilation flags.

I am not very experienced with CMake, but I would love to have a DLL/DSO-only option that creates individual libraries. I will need this eventually or cross compiling for Arduino where the static libraries are useless (as is cross compiling FLUID and probably most of the tests). So yes, I would love to see fixes, but can't really do them myself. 

[Albrecht Schlosser]

On 10/6/23 00:29 Gonzalo Garramuno wrote:
>> El 5 oct 2023, a las 10:46, Gonzalo Garramuño <ggar...@gmail.com> escribió:
>> The problem is described on the CMakeLists.txt files and it has nothing to do with the compilation environment. It is an issue entirely with FLTK.
>> First, the issue is that FLTK's CMakeLists.txt files don't properly support creating shared and static libs independently or together. Currently, statict libs are always created, which, on MSVC, also have a .lib extension.

Yes, the '.lib' extension for both the static lib and the dynamic linker
stub is a known problem that causes trouble in many build environments.

>> On Windows, with the MSVC and DLL builds, only one DLL is created that has all the libraries (fltk_gl, fltk, fltk_forms, etc) in it. Why is that is not clearly explained as it is really easy to have multiple DLLs.

Gonzalo and Matthias (I'm replying to him as well), there are many good
points in this thread and I'd like to clarify some of them.

First of all, please be patient, I'm working on this Windows DLL build
stuff but I will need more time do finally do it.

>> I took a closer look and I see the difficulty. The static libs are always created as the test, examples and more importantly fluid link to them. This is mainly a convenience to avoid users having DSO troubles with them.

Well, it's not really intended as a convenience for users, it's "always
been that way" - since we used CMake at least. Michael who wrote our
1.3.x CMakeLists.txt etc. tried to make the CMake build mostly
compatible with the configure/make build system and presumably also with
the provided Windows IDE projects in 1.3.x.

>> However, it should still be possible to create a CMake Targets.cmake file for MSVC that links to the one and only DLL and has the compilation flags.

Yep, that's what I'm working on.

>> With that, users of FLTK would then be able to just do:
>>
>> find_package(FLTK REQUIRED)
>> Target_link_libraries( myExe WIN32 FLTK::FLTK )
>>
>> Which is the way CMake is supposed to be used.

And again, that's what I'm trying to do. I hope I'll soon get this
ready, however it might take 2 weeks or more, depending on other work
I'm doing.

Summary: I'm working on many CMake improvements to move further to
"Modern CMake". I'm not going to explain this term here right now, but
read on if you're interested in details.

@Gonzalo: please don't try to make a PR right now. This would certainly
conflict with my work and would thus not be helpful at this time. I'm
taking all your points into account, thanks for your comments in this
thread.

OK, here are *some* of the issues I'm trying to resolve:

1) One or multiple DLL's ? The problem with building multiple DLL's has
to do with the FL_EXPORT and FL_LIBRARY macros which are essential for
Windows DLL builds. If we had multiple DLL's we needed a specific macro
for each DLL. For instance, symbols in the core fltk.dll would have to
be *exported* as usual, but e.g. fltk_images.dll would have to *import*
the same symbols from fltk.dll. And so on. To do this, each library
would need its own, say, FL_LIBRARY_IMAGES symbol and its own
FL_EXPORT_IMAGES macro (replace 'IMAGES' with 'GL', 'FORMS' etc.) for
the other libs. CMake supports generating appropriate include files
(done in experimental code in CMakeLists.txt line 186++, look for
'generate_export_header') but we'd also have to replace FL_EXPORT with
FL_EXPORT_XXX everywhere in the non-fltk.dll source files. I've been
considering this but kept this for later (if at all).

Note that the 1.3 IDE files also create(d) only one FLTK.DLL file rather
than individual DLL's.

The paragraph above concerns only Visual Studio builds. MinGW and MSYS
builds are not affected and build separate DLL's as they do on other
platforms (Unix, Linux, macOS). As you, Gonzalo, noticed, there are also
problems linking the image libs (jpeg, png, zlib) which I'm also working on.

2) Gonzalo, you wrote in a previous reply:

> when running find_package(FLTK REQUIRED NOCONFIG) with the generated

FLTKConfig.cmake and FLTK-Targets ...

Please note that 'NOCONFIG' is equivalent to 'MODULE' which tries to use
FindFLTK.cmake (provided by CMake) which does not support FLTK 1.4. This
would not use "the generated FLTKConfig.cmake and FLTK-Targets" at all
(at least it shouldn't IMHO). However, if you intended to write "CONFIG"
that *should* work but this is not yet working with the current Windows
builds (CMake files). This is one point I'm also working on.

I'm also considering to write our own FindFLTK.cmake file which we could
distribute with FLTK. The CMake development moves away from using
Find*.cmake however, in favor of providing CONFIG files (which we do and
which I'm going to improve).

3) Using (CMake) namespace. Gonzalo, you wrote above:

> users of FLTK would then be able to just do:
> find_package(FLTK REQUIRED)
> Target_link_libraries( myExe WIN32 FLTK::FLTK )

This will be supported in my development and I have some questions to
those who have more experiences with other projects.

(a) what is the convention for the namespace:
(a1) lowercase 'fltk::'
(a2) uppercase 'FLTK::'
(a3) doesn't matter, the namespace is case insensitive anyway (is it?).

(b) Library naming: our libs (targets) are currently fltk, fltk_images,
fltk_gl etc. If I just added the namespace to exported targets we'd have

(b1) fltk::fltk, fltk::fltk_images, fltk::fltk_gl etc.

I'm thinking about removing the redundant 'fltk_' from the non-core libs
which would result in

(b2) fltk::fltk, fltk::images, fltk::gl etc.

In this case the library filenames would still be for instance fltk.a,
fltk_images.a, fltk_gl.a etc..

Which of these naming conventions (b1 or b2) should we use? Thoughts,
anybody?

(c) Always building the static libraries: We have issue #753: "Give the
option to disable the static library"
<https://github.com/fltk/fltk/issues/753>. I made some progress in
building only the shared libs (linking fluid and all test programs
against the shared libs) but I postponed further work until the other
libs stuff (see above) is done.

Note that some of the CMake code results from experiments how to link
fluid and some of the example programs against the shared FLTK libs.
This will be simplified if we can force building either static or shared
or both libs.

There's maybe more to it but I have to stop here. Feel free to ask
further questions and to post comments here. Thanks.

[Matthias Melcher]

Albrecht Schlosser schrieb am Freitag, 6. Oktober 2023 um 11:23:56 UTC+2:

I made some progress in building only the shared libs (linking fluid and all test programs against the shared libs) but I postponed further work until the other libs stuff (see above) is done.

Thanks for all your work. Two things: IMHO, if a user decides to only compile the shared libraries, I don't think that there is a need to build FLUID or any of the tests. Building "shared only" seems to me a decision to explicitly *not* build anything else. If shared libs are requested, it is certainly useful to build one or two test apps that specifically link dynamically with multiple fltk DSO's, just to verify that the process is working.

With CMake developing very quickly and looking at our limited resources, would it make sense to try to get some external CMake expert involved, helping us to modernise our build setup? Maybe even if that person requires some reasonable pay.

[imm]

Something of an aside, and I see that Albrecht already discussed this,
but my recollection is that the "monolithic" DLL predates cmake and
fltk-1.4.

(IIRC) It was a workaround for linkage issues with the Win32 DLL
builds and the general weirdness of imports/exports on the various
DLL's, that was hard to fix.
Bundling everything into one big DLL "solved" all the problems and
gave us something that worked well enough... and TBH one big DLL does
not take up any more space than several small ones (in fact, IIRC,
it's a bit smaller than all the individual DLLs put together) and it
is only one blob to distribute with the installation, rather than
several.
So maybe it is not a problem?
I have only used it in a limited way, but it worked fine for the cases
I needed, and I was using the image and GL parts anyway so would have
been using / distributing pretty much all of the individual DLLs
anyway, and so one big "all in one" option was actually easier for me.

YMMV...

[Albrecht Schlosser]

On 10/6/23 12:48 'Matthias Melcher' via fltk.general wrote:

Albrecht Schlosser schrieb am Freitag, 6. Oktober 2023 um 11:23:56 UTC+2:

I made some progress in building only the shared libs (linking fluid and all test programs against the shared libs) but I postponed further work until the other libs stuff (see above) is done.

Thanks for all your work.

Welcome, thanks to you as well.

Two things: IMHO, if a user decides to only compile the shared libraries, I don't think that there is a need to build FLUID or any of the tests. Building "shared only" seems to me a decision to explicitly *not* build anything else.

I disagree. I imagine for instance a Linux distribution that installs only shared libs and all FLTK programs included in the distro are linked against the shared libs or a user that "wants" shared libs but also uses fluid and fltk-options.

If shared libs are requested, it is certainly useful to build one or two test apps that specifically link dynamically with multiple fltk DSO's, just to verify that the process is working.

Yep, that's what we're doing now. If everything works like I think it should we could simplify the CMake files a lot and still test both shared and static lib builds.

With CMake developing very quickly and looking at our limited resources, would it make sense to try to get some external CMake expert involved, helping us to modernise our build setup? Maybe even if that person requires some reasonable pay.

You can ask me this question again *if* I failed with my WIP. I'm pretty confident that I can manage this CMake modernization in the short term. Teaching another person what we want and how to do it would also take a lot of time.

The bigger problem with our CMake support is which oldest CMake version we want to support. New features don't matter much *if* we keep cmake_minimum_required() as low as we have now. Note that I raised it recently to 3.12.0 (released 2018-07-17) and we may need 3.13 (2018-11-20) or maybe even higher. CMake versions 3.12 and 3.13 added many of the "Modern CMake" features I'd like to use.

Here are some thoughts regarding cmake_minimum_required():

(1) Try to keep it as low as possible (conservative approach) to enable most of our users to use CMake from their distro
(2) Since CMake can be installed pretty easily (Windows download from their website) and even self-built w/o (m)any prerequisites set cmake_minimum_required() as high as "needed" (whatever this is)
(3) Take an approach between (1) and (2).

I would like to do (1) as much as reasonable w/o sacrificing FLTK development speed, i.e. basically use (3). I'm trying to keep compatibility with CMake released no longer than 4-5 years ago. This means for example that we can support Ubuntu 20.04 LTS but not Ubuntu 18.04 LTS which has been superseded by two consecutive LTS versions.

[Albrecht Schlosser]

On 10/6/23 13:09 imm wrote:
> Something of an aside, and I see that Albrecht already discussed this,
> but my recollection is that the "monolithic" DLL predates cmake and
> fltk-1.4.

Yep, I don't remember how 1.1 did it, but the IDE files in 1.3
definitely build such a monolithic DLL.

> (IIRC) It was a workaround for linkage issues with the Win32 DLL
> builds and the general weirdness of imports/exports on the various
> DLL's, that was hard to fix.

Thanks for confirmation. This is exactly what I assumed. For the linkage
errors see my post about FL_EXPORT etc.

> Bundling everything into one big DLL "solved" all the problems and
> gave us something that worked well enough... and TBH one big DLL does
> not take up any more space than several small ones (in fact, IIRC,
> it's a bit smaller than all the individual DLLs put together) and it
> is only one blob to distribute with the installation, rather than
> several.

True.

> So maybe it is not a problem?

Yes and no, I'd say. If you build one shared lib with "everything" then
this shared lib depends on e.g. libpng and zlib although the application
doesn't use any images (think of hello.cxx linked to such a monolithic
DLL). That would be a potential disadvantage for binary distribution of
application programs.

However, if you build and run your programs on your local machine I
don't see any disadvantages of building a single shared library for
everything.

Another point to mention is that I really don't think that we need to
build separate *static* libs. Building a program from such a static lib
would pull in only used modules which would be equivalent to linking
with only one or two static FLTK libs. There wouldn't be any additional
dependencies - unless I'm missing a point.

> I have only used it in a limited way, but it worked fine for the cases
> I needed, and I was using the image and GL parts anyway so would have
> been using / distributing pretty much all of the individual DLLs
> anyway, and so one big "all in one" option was actually easier for me.

Just to be sure: you did this in 1.3.x and we're talking about building
with Visual Studio using the provided IDE projects, right? I'm asking
this because I don't think that our 1.4 CMake builds with Visual Studio
ever worked with shared libs. I would be surprised if it did - although
ISTR that I could *build* the shared libs at some time but I'm not sure.
If they could be built, then there were probably issues when *using* them.