Requiring CMake 2.8.10 for the multiple export sets feature

[Stephen Kelly]

Hi,

I'm trying to patch the repos up a bit to try out my new cmake target usage
requirements features, but the you-must-run-cmake-twice stuff is getting in
the way and confusing me.

As far as I know, that trick is only needed as a workaround for the lack of
the 'multiple export sets' feature which is new in CMake 2.8.10
(http://public.kitware.com/Bug/view.php?id=12588). If we can bump the
dependency to that version of CMake, we can remove the trick and make the
cmake code more simple.

Thoughts? Volunteers? I might be able to do it, but I'm not sure how much
and how many areas the trick touches in order to remove it.

Thanks,

Steve.

[Dave Abrahams]

on Tue Nov 20 2012, Stephen Kelly <steveire-AT-gmail.com> wrote:

> Hi,
>
> I'm trying to patch the repos up a bit to try out my new cmake target
> usage requirements features, but the you-must-run-cmake-twice stuff is
> getting in the way and confusing me.

Daniel Pfeifer is the expert on that part, but IIUC it is there because
there are dependency cycles among some of the libraries, which only get
fully resolved by CMake if we run it twice.

> As far as I know, that trick is only needed as a workaround for the
> lack of the 'multiple export sets' feature which is new in CMake
> 2.8.10 (http://public.kitware.com/Bug/view.php?id=12588).

I'm not entirely sure, but from reading the ticket it sounds plausible.

> If we can bump the dependency to that version of CMake, we can remove
> the trick and make the cmake code more simple.
>
> Thoughts? Volunteers? I might be able to do it, but I'm not sure how
> much and how many areas the trick touches in order to remove it.

Hopefully Daniel will jump in here soon. I think, except for Daniel,
you're probably the person around here most qualified to do it. :-)

--
Dave Abrahams
BoostPro Computing Software Development Training
http://www.boostpro.com Clang/LLVM/EDG Compilers C++ Boost

[Daniel Pfeifer]

Am Dienstag, 20. November 2012 18:38:30 UTC+1 schrieb Dave Abrahams:

on Tue Nov 20 2012, Stephen Kelly <steveire-AT-gmail.com> wrote:

> Hi,
>
> I'm trying to patch the repos up a bit to try out my new cmake target
> usage requirements features, but the you-must-run-cmake-twice stuff is
> getting in the way and confusing me.

Daniel Pfeifer is the expert on that part, but IIUC it is there because
there are dependency cycles among some of the libraries, which only get
fully resolved by CMake if we run it twice.

> As far as I know, that trick is only needed as a workaround for the
> lack of the 'multiple export sets' feature which is new in CMake
> 2.8.10 (http://public.kitware.com/Bug/view.php?id=12588).

I'm not entirely sure, but from reading the ticket it sounds plausible.

> If we can bump the dependency to that version of CMake, we can remove
> the trick and make the cmake code more simple.

The second CMake-run is a temporary workaround. I am more than happy if we can get rid of it.

The reason for that workaround are circular dependencies. Each project finds its dependencies with `find_package()' which loads the according <project>_config.cmake files for the build tree.

These config files are generated in each CMake-run. In the first run, not all config files are usable yet, hence a second run is required.

Another workaround is that we generate the config files "manually" instead of using `install(EXPORT ...)'. This workaround is obsolete with 2.8.10.

To obsolete the first workaround, I think we need 2.8.11 and your interface targets, Steve.

In my opinion, we can set the minimum required version of CMake to 2.8.11, even if that is not even released yet.

I don't recommend moving Boost officially to CMake as long as these two workarounds are needed. As long as it is not official, we may dare to break it.

 

> Thoughts? Volunteers? I might be able to do it, but I'm not sure how
> much and how many areas the trick touches in order to remove it.

Hopefully Daniel will jump in here soon.  I think, except for Daniel,
you're probably the person around here most qualified to do it.  :-)

:-)

[Dave Abrahams]

on Tue Nov 20 2012, Daniel Pfeifer <purplekarrot-AT-gmail.com> wrote:

> Am Dienstag, 20. November 2012 18:38:30 UTC+1 schrieb Dave Abrahams:
>
>
> In my opinion, we can set the minimum required version of CMake to
> 2.8.11, even if that is not even released yet.

How will we even test it, in that case?

[Mathias Gaunard]

On 20/11/12 22:23, Daniel Pfeifer wrote:

> The reason for that workaround are circular dependencies. Each project
> finds its dependencies with `find_package()' which loads the
> according <project>_config.cmake files for the build tree.
> These config files are generated in each CMake-run. In the first run,
> not all config files are usable yet, hence a second run is required.

If the modules are found in topological order, only circular
dependencies are a problem.

You can deal with circular dependencies in a fairly simple way: just
assume that whatever module you're depending on that hasn't been found
yet -- which therefore is necessarily depended on circularly, since the
topological order would force all dependees to be found before -- has no
dependencies other than itself.
The only thing you may miss are external dependencies that do not appear
in the graph, but you should really make it so that modules with
external dependencies are not depended on circularly.

[Daniel Pfeifer]

On Tuesday, November 20, 2012 10:26:04 PM UTC+1, Dave Abrahams wrote:

on Tue Nov 20 2012, Daniel Pfeifer <purplekarrot-AT-gmail.com> wrote:

> Am Dienstag, 20. November 2012 18:38:30 UTC+1 schrieb Dave Abrahams:
>
>
> In my opinion, we can set the minimum required version of CMake to
> 2.8.11, even if that is not even released yet.

How will we even test it, in that case?

By using a development version of CMake.

[Dave Abrahams]

on Tue Nov 20 2012, Daniel Pfeifer <purplekarrot-AT-gmail.com> wrote:

> On Tuesday, November 20, 2012 10:26:04 PM UTC+1, Dave Abrahams wrote:
>
>
> on Tue Nov 20 2012, Daniel Pfeifer <purplekarrot-AT-gmail.com>
> wrote:
>
> > Am Dienstag, 20. November 2012 18:38:30 UTC+1 schrieb Dave
> Abrahams:
> >
> >
> > In my opinion, we can set the minimum required version of CMake
> to
> > 2.8.11, even if that is not even released yet.
>
> How will we even test it, in that case?
>
>
> By using a development version of CMake.

Where do we get that from?

[Daniel Pfeifer]

On Tuesday, November 20, 2012 11:06:04 PM UTC+1, Dave Abrahams wrote:

on Tue Nov 20 2012, Daniel Pfeifer <purplekarrot-AT-gmail.com> wrote:

> On Tuesday, November 20, 2012 10:26:04 PM UTC+1, Dave Abrahams wrote:
>
>    
>     on Tue Nov 20 2012, Daniel Pfeifer <purplekarrot-AT-gmail.com>
>     wrote:
>    
>     > Am Dienstag, 20. November 2012 18:38:30 UTC+1 schrieb Dave
>     Abrahams:
>     >
>     >
>     > In my opinion, we can set the minimum required version of CMake
>     to
>     > 2.8.11, even if that is not even released yet.
>    
>     How will we even test it, in that case?
>
>
> By using a development version of CMake.

Where do we get that from?

From Steves branch on Gitorious. The interface target feature will change the way we use CMake.

[Stephen Kelly]

I posted a link to the CMake clone here:

 https://groups.google.com/forum/?fromgroups=#!topic/ryppl-dev/4Rqj40hbpaI

I hope to get the branch integrated into CMake master within a week or two anyway, which should simplify testing. The next release of CMake will likely be in early February.

[Stephen Kelly]

Am Dienstag, 20. November 2012 22:23:26 UTC+1 schrieb Daniel Pfeifer:

Am Dienstag, 20. November 2012 18:38:30 UTC+1 schrieb Dave Abrahams:

on Tue Nov 20 2012, Stephen Kelly <steveire-AT-gmail.com> wrote:

> Hi,
>
> I'm trying to patch the repos up a bit to try out my new cmake target
> usage requirements features, but the you-must-run-cmake-twice stuff is
> getting in the way and confusing me.

Daniel Pfeifer is the expert on that part, but IIUC it is there because
there are dependency cycles among some of the libraries, which only get
fully resolved by CMake if we run it twice.

> As far as I know, that trick is only needed as a workaround for the
> lack of the 'multiple export sets' feature which is new in CMake
> 2.8.10 (http://public.kitware.com/Bug/view.php?id=12588).

I'm not entirely sure, but from reading the ticket it sounds plausible.

> If we can bump the dependency to that version of CMake, we can remove
> the trick and make the cmake code more simple.

The second CMake-run is a temporary workaround. I am more than happy if we can get rid of it.

The reason for that workaround are circular dependencies. Each project finds its dependencies with `find_package()' which loads the according <project>_config.cmake files for the build tree.

These config files are generated in each CMake-run. In the first run, not all config files are usable yet, hence a second run is required.

If all targets are part of the same buildsystem (as they currently are), then, yes, my topic should help. If the plan is to eventually build boost libraries standalone, then I think the bootstrapping will remain a bit complex until the dependencies are reduced.

As an experiment I commented all libraries out and then tried to use the boost::any library too see what the headers depend on. I had to uncomment all of the following:


add_subdirectory(boost/algorithm)
add_subdirectory(boost/array)
add_subdirectory(boost/bind)
add_subdirectory(boost/concept_check)
add_subdirectory(boost/config)
add_subdirectory(boost/container)
add_subdirectory(boost/conversion)
add_subdirectory(boost/core)
add_subdirectory(boost/exception)
add_subdirectory(boost/format)
add_subdirectory(boost/function_types)
add_subdirectory(boost/function)
add_subdirectory(boost/functional_hash)
add_subdirectory(boost/fusion)
add_subdirectory(boost/integer)
add_subdirectory(boost/intrusive)
add_subdirectory(boost/io)
add_subdirectory(boost/iterator)
add_subdirectory(boost/lambda)
add_subdirectory(boost/math)
add_subdirectory(boost/move)
add_subdirectory(boost/mpl)
add_subdirectory(boost/numeric_conversion)
add_subdirectory(boost/optional)
add_subdirectory(boost/parameter)
add_subdirectory(boost/preprocessor)
add_subdirectory(boost/range)
add_subdirectory(boost/regex)
add_subdirectory(boost/smart_ptr)
add_subdirectory(boost/static_assert)
add_subdirectory(boost/tr1)
add_subdirectory(boost/tuple)
add_subdirectory(boost/type_traits)
add_subdirectory(boost/typeof)
add_subdirectory(boost/ublas)
add_subdirectory(boost/units)
add_subdirectory(boost/unordered)
add_subdirectory(boost/utility)

Is such a long list expected?
 

Another workaround is that we generate the config files "manually" instead of using `install(EXPORT ...)'. This workaround is obsolete with 2.8.10.

Yes, I think I removed that stuff in my branch, and now rely on cmake generating them. I think that wouldn't have worked before, but I didn't test in any way with 2.8.9.
 

To obsolete the first workaround, I think we need 2.8.11 and your interface targets, Steve.

In my opinion, we can set the minimum required version of CMake to 2.8.11, even if that is not even released yet.

The problem with the FATAL_ERROR in my branch, which is generated by install(EXPORT), is that cmake master does not identify itself as version 2.8.11, but as 2.8.10.20121119-g30ef7. Maybe I can do the feature check another way.
 

[Dave Abrahams]

on Tue Nov 20 2012, Stephen Kelly <steveire-AT-gmail.com> wrote:

> The reason for that workaround are circular dependencies. Each
> project finds its dependencies with `find_package()' which loads
> the according <project>_config.cmake files for the build tree.
> These config files are generated in each CMake-run. In the first
> run, not all config files are usable yet, hence a second run is
> required.
>
>
> If all targets are part of the same buildsystem (as they currently
> are), then, yes, my topic should help. If the plan is to eventually
> build boost libraries standalone, then I think the bootstrapping will
> remain a bit complex until the dependencies are reduced.

We are already building boost libraries standalone. That's what Ryppl
does.

Let's just say it's not too surprising. One recent analysis yielded
this dependency graph:

http://f.cl.ly/items/3p0V1C1Z2N041B2b3P3h/boost-all-dependencies.svg

Scripts that do this analysis and generate the graphs are in
https://github.com/ryppl/ryppl/tree/develop/scripts, FWIW

[Daniel Pfeifer]

Am Dienstag, 20. November 2012 23:38:46 UTC+1 schrieb Stephen Kelly:

Am Dienstag, 20. November 2012 22:23:26 UTC+1 schrieb Daniel Pfeifer:

Am Dienstag, 20. November 2012 18:38:30 UTC+1 schrieb Dave Abrahams:

on Tue Nov 20 2012, Stephen Kelly <steveire-AT-gmail.com> wrote:

> Hi,
>
> I'm trying to patch the repos up a bit to try out my new cmake target
> usage requirements features, but the you-must-run-cmake-twice stuff is
> getting in the way and confusing me.

Daniel Pfeifer is the expert on that part, but IIUC it is there because
there are dependency cycles among some of the libraries, which only get
fully resolved by CMake if we run it twice.

> As far as I know, that trick is only needed as a workaround for the
> lack of the 'multiple export sets' feature which is new in CMake
> 2.8.10 (http://public.kitware.com/Bug/view.php?id=12588).

I'm not entirely sure, but from reading the ticket it sounds plausible.

> If we can bump the dependency to that version of CMake, we can remove
> the trick and make the cmake code more simple.

The second CMake-run is a temporary workaround. I am more than happy if we can get rid of it.

The reason for that workaround are circular dependencies. Each project finds its dependencies with `find_package()' which loads the according <project>_config.cmake files for the build tree.

These config files are generated in each CMake-run. In the first run, not all config files are usable yet, hence a second run is required.

If all targets are part of the same buildsystem (as they currently are), then, yes, my topic should help. If the plan is to eventually build boost libraries standalone, then I think the bootstrapping will remain a bit complex until the dependencies are reduced.

Do you think so? When a library is built standalone, it will call `find_package()' to import the targets it depends on. When the dependencies are part of the same buildsystem, that `find_package()' call is not necessary: At generation time, there will be a target for each dependency.

The library does not know whether it is part of a monolithic build or not, so the `find_package()' function should be used and then be disabled (somehow) in the top level project. Something like:

set(BoostFilesystem_IS_PART_OF_THE_BUILDSYSTEM ON)

set(BoostSystem_IS_PART_OF_THE_BUILDSYSTEM ON)

add_subdirectory(boost/filesystem)

add_subdirectory(boost/system)

The `find_package(BoostSystem)' inside `boost/filesystem/CMakeLists.txt' should do nothing at all, because by setting `BoostSystem_IS_PART_OF_THE_BUILDSYSTEM', we guarantee that at configuration time there will be an interface `boost::system' that can be resolved. All other dependencies are expected to be installed on the system.

That top-level `CMakeLists.txt' file with all the `set(*_IS_PART_OF_THE_BUILDSYSTEM ON)' and `add_subdirectory(*)' calls shall be automatically generated by the Ryppl tool.

Do you thinkt that would be possible?

[Stephen Kelly]

Am Mittwoch, 21. November 2012 08:16:31 UTC+1 schrieb Daniel Pfeifer:

Am Dienstag, 20. November 2012 23:38:46 UTC+1 schrieb Stephen Kelly:

Am Dienstag, 20. November 2012 22:23:26 UTC+1 schrieb Daniel Pfeifer:

Am Dienstag, 20. November 2012 18:38:30 UTC+1 schrieb Dave Abrahams:

on Tue Nov 20 2012, Stephen Kelly <steveire-AT-gmail.com> wrote:

> Hi,
>
> I'm trying to patch the repos up a bit to try out my new cmake target
> usage requirements features, but the you-must-run-cmake-twice stuff is
> getting in the way and confusing me.

Daniel Pfeifer is the expert on that part, but IIUC it is there because
there are dependency cycles among some of the libraries, which only get
fully resolved by CMake if we run it twice.

> As far as I know, that trick is only needed as a workaround for the
> lack of the 'multiple export sets' feature which is new in CMake
> 2.8.10 (http://public.kitware.com/Bug/view.php?id=12588).

I'm not entirely sure, but from reading the ticket it sounds plausible.

> If we can bump the dependency to that version of CMake, we can remove
> the trick and make the cmake code more simple.

The second CMake-run is a temporary workaround. I am more than happy if we can get rid of it.

The reason for that workaround are circular dependencies. Each project finds its dependencies with `find_package()' which loads the according <project>_config.cmake files for the build tree.

These config files are generated in each CMake-run. In the first run, not all config files are usable yet, hence a second run is required.

If all targets are part of the same buildsystem (as they currently are), then, yes, my topic should help. If the plan is to eventually build boost libraries standalone, then I think the bootstrapping will remain a bit complex until the dependencies are reduced.

Do you think so? When a library is built standalone, it will call `find_package()' to import the targets it depends on. When the dependencies are part of the same buildsystem, that `find_package()' call is not necessary: At generation time, there will be a target for each dependency.

Yes, for a non-monolithic build the find_package is not needed.

I guess I'm not familiar enough with the realities and goals of non-monolithic builds. Starting from scratch, how would I install boost::any, given the dependencies I posted earlier? I can't just clone it and cmake && make && make install, right?
 

The library does not know whether it is part of a monolithic build or not, so the `find_package()' function should be used and then be disabled (somehow) in the top level project. Something like:

set(BoostFilesystem_IS_PART_OF_THE_BUILDSYSTEM ON)

set(BoostSystem_IS_PART_OF_THE_BUILDSYSTEM ON)

add_subdirectory(boost/filesystem)

add_subdirectory(boost/system)

The `find_package(BoostSystem)' inside `boost/filesystem/CMakeLists.txt' should do nothing at all, because by setting `BoostSystem_IS_PART_OF_THE_BUILDSYSTEM', we guarantee that at configuration time there will be an interface `boost::system' that can be resolved. All other dependencies are expected to be installed on the system.

Or just put a conditional around the find_package:

if (NOT TARGET boost::system)
    find_package(BoostSystem)
endif()
 

That top-level `CMakeLists.txt' file with all the `set(*_IS_PART_OF_THE_BUILDSYSTEM ON)' and `add_subdirectory(*)' calls shall be automatically generated by the Ryppl tool.

Do you thinkt that would be possible?

Yes. There may be other possible ways to resolve the issue too.
 
Thanks,

Steve.

[Daniel Pfeifer]

Right, that is not possible yet. But that is exactly where we want to go.

In general, if all dependencies are installed (each with its config file), you can just clone a project and build it. This works already, but you have to install the dependencies manually.

In the future, you won't clone the library with git, but setup a workspace with ryppl. Ryppl will clone the library *and* resolve dependencies. Dependencies may be installed though packagekit, downloaded as tarballs, or cloned into the workspace. Ryppl will then create/update the CMakeLists.txt file in the top level.

For each dependency that is ...

a) downloaded as a prebuilt archive, Ryppl will write "set(Dependency_DIR ..)" to the path where the archive was extracted.

b) part of the buildsystem, Ryppl will write "add_subdirectory(..)".

c) installed on the system, Ryppl writes nothing.

For each dependency that is installed on the system, 

 

The library does not know whether it is part of a monolithic build or not, so the `find_package()' function should be used and then be disabled (somehow) in the top level project. Something like:

set(BoostFilesystem_IS_PART_OF_THE_BUILDSYSTEM ON)

set(BoostSystem_IS_PART_OF_THE_BUILDSYSTEM ON)

add_subdirectory(boost/filesystem)

add_subdirectory(boost/system)

The `find_package(BoostSystem)' inside `boost/filesystem/CMakeLists.txt' should do nothing at all, because by setting `BoostSystem_IS_PART_OF_THE_BUILDSYSTEM', we guarantee that at configuration time there will be an interface `boost::system' that can be resolved. All other dependencies are expected to be installed on the system.

Or just put a conditional around the find_package:

if (NOT TARGET boost::system)
    find_package(BoostSystem)
endif()

I am afraid that will not work. A target may be not defined yet, but still be part of the buildsystem.

[Daniel Pfeifer]

a) downloaded as a prebuilt archive, Ryppl will write "set(Dependency_DIR <path>)" where <path> is where the archive was extracted to.

b) part of the buildsystem, Ryppl will write "add_subdirectory(..)".

c) installed on the system, Ryppl writes nothing.

For all three cases, "find_package(Dependency)" should succeed (or be a no-op) and targets should be able to link Foo::Dependency.