boost_config depends on boost_core

[Stephen Kelly]

Hi there,

One of the annoying parts of trying to work on the boost cmake files is that small changes are impossible. I can't just change something in the generated cmake files for boost_any and boost_config, and then test it and see if it works. The reason is that boost_config depends on boost_core, which depends on 35 other libraries: http://thread.gmane.org/gmane.comp.programming.tools.ryppl.devel/9/focus=26 . So, I have to change 35 libraries if I want to try/test something.

My boss has a catchphrase: "This is not a tools problem". He uses it whenever someone tries to avoid facing work head-on by instead creating a tool for the job. For example, in scheduling who is going to work on what, the team-leads have to communicate and agree. No tool is going to make the need for agreement go away. After a sufficiently good and simple tool is used, there's a long tail of things that could be better in the tool, but do not make the task of reaching agreement significantly easier.

My understanding is that the ryppl community already decided that the interdependencies between modules don't matter because ryppl will be configured to download all needed dependencies. However, in my view, this is not a tools problem. I'm sure many circular-dependency issues in boost modules can be solved. The ryppl tool can be helpful, but making it mandatory doesn't seem right. The result is not a 'modular boost' if all of the modules are interdependent.

If all the interdependencies between the headers are going to remain, then why not put the 'core mesh' into one repo? If you want one of the 35 libraries in the list, you need all of them, right?

But a better question is why not remove the interdependencies? Without even trying I can see 3 ways to break the dependency of boost_config on boost_core. That would 'free up' boost_any for example (the only one I checked, there may be others positively affected) and make it plausible to use boost without the ryppl tool.

1. Create a core_types library, and move core/type.hpp and core/non_type.hpp in there. Make boost_core and boost_config depend on the new library.
2. Move type.hpp and non_type.hpp into boost_config.
3. Move the BOOST_EXPLICIT_TEMPLATE{,_NON}_TYPE defines into boost_core.

This is just the first, most obvious and most obviously beneficial change that could be made to make the 'core mesh' smaller. I'm sure that other interdependencies are also not needed. Any reason not to pursue this?

This seems like it should have been a prerequisite to splitting the code into multiple git repos, so I'm assuming the approach of actually modularizing the code was already considered and rejected for some reason?

Thanks,

Steve.

[Vladimir Prus]

On 28.03.2013 16:18, Stephen Kelly wrote:
> One of the annoying parts of trying to work on the boost cmake files is that small changes are impossible. I can't just change something in
> the generated cmake files for boost_any and boost_config, and then test it and see if it works. The reason is that boost_config depends on
> boost_core, which depends on 35 other libraries: http://thread.gmane.org/gmane.comp.programming.tools.ryppl.devel/9/focus=26 . So, I have to
> change 35 libraries if I want to try/test something.

Pardon my ignorance, but why do you need to *change* 35 libraries? Or did you mean "checkout" or something else?

Thanks,

[Stephen Kelly]

On Thursday, March 28, 2013 4:05:07 PM UTC+1, Vladimir Prus wrote:

Pardon my ignorance, but why do you need to *change* 35 libraries? Or did you mean "checkout" or something else?

Sorry, maybe I didn't make that clear.

If I'm working on the ryppl.cmake file, how it works and what it does, then I'll need to also change the CMakeLists.txt files that include it. If it were possible to comment out everything except config and any, that would be easy, but if have to change 35 libraries *just* to develop/test a change, it's easier for me to throw in the towel :).

This is kind of a side-show to the general issue of modularity though.

Thanks,

Steve.

[Dave Abrahams]

on Thu Mar 28 2013, Stephen Kelly <steveire-AT-gmail.com> wrote:

> Hi there,
>
> One of the annoying parts of trying to work on the boost cmake files
> is that small changes are impossible. I can't just change something
> in the generated cmake files for boost_any and boost_config,

I'm sorry, but who's generating cmake files? As far as I know we are
creating cmake files entirely by hand.

> and then test it and see if it works.

It's great to hear that you're trying to work on this stuff, Steve!
I'll do whatever I can to help.

> The reason is that boost_config depends on boost_core, which depends
> on 35 other libraries: http://thread.gmane.org/gmane.comp.programming.tools.ryppl.devel/9/focus=26

> So, I have to change 35 libraries if I want to try/test something.

> On Thursday, March 28, 2013 4:05:07 PM UTC+1, Vladimir Prus wrote:
>
> Pardon my ignorance, but why do you need to *change* 35
> libraries? Or did you mean "checkout" or something else?
>
> Sorry, maybe I didn't make that clear.
>
> If I'm working on the ryppl.cmake file, how it works and what it
> does, then I'll need to also change the CMakeLists.txt files that
> include it. If it were possible to comment out everything except
> config and any, that would be easy, but if have to change 35
> libraries *just* to develop/test a change, it's easier for me to
> throw in the towel :).

That would be a shame. Why don't you just use Ryppl.cmake to put
together some smaller representative examples?

On the other hand, a global search/replace in 35 CMakeLists.txt files
can't be that hard unless... (ahem) you don't have the right tools ;-)

> This is kind of a side-show to the general issue of modularity
> though.

I'm not sure what you mean by that.

> My boss has a catchphrase: "This is not a tools problem". He uses it
> whenever someone tries to avoid facing work head-on by instead
> creating a tool for the job. For example, in scheduling who is going
> to work on what, the team-leads have to communicate and agree. No
> tool is going to make the need for agreement go away. After a
> sufficiently good and simple tool is used, there's a long tail of
> things that could be better in the tool, but do not make the task of
> reaching agreement significantly easier.
>
> My understanding is that the ryppl community already decided that the
> interdependencies between modules don't matter because ryppl will be
> configured to download all needed dependencies.

Uhm, no. I think you have grossly misinterpreted the attitude of the
ryppl community. Interdependencies certainly matter. Boost has too
many of them right now in part because the way we build and develop
Boost doesn't force people to confront them. By modularizing and making
the interdependencies explicit, we hope to encourage developers to
refactor. But we can only solve one problem at a time.

> However, in my view, this is not a tools problem. I'm sure many
> circular-dependency issues in boost modules can be solved. The ryppl
> tool can be helpful, but making it mandatory doesn't seem right.

Making what mandatory?

> The result is not a 'modular boost' if all of the modules are
> interdependent.

Of course not. But they're not *all* interdependent. There are,
unfortunately, a few strongly-connected components in the graph.

> If all the interdependencies between the headers are going to remain,
> then why not put the 'core mesh' into one repo? If you want one of
> the 35 libraries in the list, you need all of them, right?

If we do that, the dependencies would never become untangled.

> But a better question is why not remove the interdependencies?

Good question. It would be great if you could do that!

> Without even trying I can see 3 ways to break the dependency of
> boost_config on boost_core.

Fantastic. I suggest you propose this refactoring on the Boost list and
submit a patch against the Boost SVN. You can test your changes using
the existing (non-CMake) Boost testing tools.

> That would 'free up' boost_any for example (the only one I checked,
> there may be others positively affected) and make it plausible to use
> boost without the ryppl tool.
>
> 1. Create a core_types library, and move core/type.hpp and core/
> non_type.hpp in there. Make boost_core and boost_config depend on the
> new library.
> 2. Move type.hpp and non_type.hpp into boost_config.
> 3. Move the BOOST_EXPLICIT_TEMPLATE{,_NON}_TYPE defines into
> boost_core.
>
> This is just the first, most obvious and most obviously beneficial
> change that could be made to make the 'core mesh' smaller. I'm sure
> that other interdependencies are also not needed. Any reason not to
> pursue this?

No reason at all; I hope you do!

> This seems like it should have been a prerequisite to splitting the
> code into multiple git repos, so I'm assuming the approach of
> actually modularizing the code was already considered and rejected
> for some reason?

Individual maintainers "own" the individual libraries, and it seemed
like too much to take on negotiating all these changes at the same time
as doing the tools work. These dependencies have mostly not been a
problem for us during development of the tools, so there was little
pressure to fix it. There are some SCCs involving binary libraries that
are a bit more complex to deal with, but some simple graph analysis code
was enough to allow me to sort those out without changes to Boost
itself.


--
Dave Abrahams

[Stephen Kelly]

On Saturday, March 30, 2013 7:09:56 PM UTC+1, Dave Abrahams wrote:

on Thu Mar 28 2013, Stephen Kelly <steveire-AT-gmail.com> wrote:

> Hi there,
>
> One of the annoying parts of trying to work on the boost cmake files
> is that small changes are impossible. I can't just change something
> in the generated cmake files for boost_any and boost_config,

I'm sorry, but who's generating cmake files?  As far as I know we are
creating cmake files entirely by hand.

I'm referring to the generation of files done by ryppl/cmake/Modules/RypplExport.cmake and testing modifications to that file.
 

> The reason is that boost_config depends on boost_core, which depends
> on 35 other libraries: http://thread.gmane.org/gmane.comp.programming.tools.ryppl.devel/9/focus=26
> So, I have to change 35 libraries if I want to try/test something.

> On Thursday, March 28, 2013 4:05:07 PM UTC+1, Vladimir Prus wrote:
>
>     Pardon my ignorance, but why do you need to *change* 35
>     libraries? Or did you mean "checkout" or something else?
>
> Sorry, maybe I didn't make that clear.
>
> If I'm working on the ryppl.cmake file, how it works and what it
> does, then I'll need to also change the CMakeLists.txt files that
> include it. If it were possible to comment out everything except
> config and any, that would be easy, but if have to change 35
> libraries *just* to develop/test a change, it's easier for me to
> throw in the towel :).

That would be a shame.  Why don't you just use Ryppl.cmake to put
together some smaller representative examples?

I guess that would be an option, but it would only delay real testing of the changes.
 

On the other hand, a global search/replace in 35 CMakeLists.txt files
can't be that hard unless... (ahem) you don't have the right tools ;-)

> This is kind of a side-show to the general issue of modularity
> though.

I'm not sure what you mean by that.

I mean that modifying all of the necessary files is annoying for me, but indeed possible. What I meant is that the real issue I'm trying to bring up is the underlying issue of modularity and interdependence between the boost repos.

 

> My boss has a catchphrase: "This is not a tools problem". He uses it
> whenever someone tries to avoid facing work head-on by instead
> creating a tool for the job. For example, in scheduling who is going
> to work on what, the team-leads have to communicate and agree. No
> tool is going to make the need for agreement go away. After a
> sufficiently good and simple tool is used, there's a long tail of
> things that could be better in the tool, but do not make the task of
> reaching agreement significantly easier.
>
> My understanding is that the ryppl community already decided that the
> interdependencies between modules don't matter because ryppl will be
> configured to download all needed dependencies.

Uhm, no.  I think you have grossly misinterpreted the attitude of the
ryppl community.

Great! Thanks for clearing this up. My understanding was coming from

 http://thread.gmane.org/gmane.comp.programming.tools.ryppl.devel/9/focus=35

and in particular "In the future, you won't clone the library with git, but setup a workspace with ryppl. Ryppl will clone the library *and* resolve dependencies." and "Ryppl will then create/update the CMakeLists.txt file in the top level." which indicated to me that all use of boost repos with cmake will require the ryppl tool for practical reasons. It also indicated to me that a sufficiently good tool makes any issues with interdependencies vanish. Thanks for clearing up that that's not the case.

 

 Interdependencies certainly matter.  Boost has too
many of them right now in part because the way we build and develop
Boost doesn't force people to confront them.  By modularizing and making
the interdependencies explicit, we hope to encourage developers to
refactor.  But we can only solve one problem at a time.

Are you concerned about modularizing the repos first and the code second, and the effect that has on the history? It's not really easy/practical to move files between different repos. It seems that with a 'one problem at a time' approach, the first problem that would need to be solved would be modularization of the code itself, not splitting into multiple repos. It seems you're saying that that would not be practical for social/community reasons? Any you're using split repos to force people to realize that the interdependencies exist? And you assume that will incentivize people to reduce such interdependencies? Sorry for all the questions. Although I've been reading the boost mailing list for a few months, I'm not familiar with what it's like to try to get something done in the community when collaboration like that is needed.
 

> However, in my view, this is not a tools problem. I'm sure many
> circular-dependency issues in boost modules can be solved. The ryppl
> tool can be helpful, but making it mandatory doesn't seem right.

Making what mandatory?

The ryppl tool. See the above quotes about how in the future it would be needed to clone the right 35 repos and generate a CMakeLists.txt file.
 

> The result is not a 'modular boost' if all of the modules are
> interdependent.

Of course not.  But they're not *all* interdependent.  There are,
unfortunately, a few strongly-connected components in the graph.

> If all the interdependencies between the headers are going to remain,
> then why not put the 'core mesh' into one repo? If you want one of
> the 35 libraries in the list, you need all of them, right?

If we do that, the dependencies would never become untangled.

Indeedm but I didn't know that the plan was to modularize the code at all :). KDE is going through a similar modularization over the past couple of years, which I've been involved with. We're modularizing the code directly in the kdelibs repo, and we'll eventually split it up into multiple independent git repos *after* we've done the modularization of the code.

I assumed, given the above quote again, and given my own opinion of the right order to do things (from my experience in the KDE efforts), that if you were splitting now, then you have already done all the modularization you intend to do. I can accept that that's not the case. Thanks for clearing that up.
 

> But a better question is why not remove the interdependencies?

Good question.  It would be great if you could do that!

 

> This seems like it should have been a prerequisite to splitting the

> code into multiple git repos, so I'm assuming the approach of
> actually modularizing the code was already considered and rejected
> for some reason?

Individual maintainers "own" the individual libraries, and it seemed
like too much to take on negotiating all these changes at the same time
as doing the tools work.

This is a little worrying :). It seems like something with obvious benefits to me, so I'd expect collaboration, not negotiation, unless some of the individual owning maintainers are opposed to the effort for some reason?

If you would have do time-consuming negotiations, what chance do I have? :) I have no credibility in the boost community. I also don't have any familiarity with the codebase.

I'm interested though, so I'll start a discussion there.

Thanks,

Steve.

[Stephen Kelly]

On Sunday, March 31, 2013 8:04:00 PM UTC+2, Stephen Kelly wrote:

I'm interested though, so I'll start a discussion there.

 
It seems the git repo at https://github.com/ryppl/boost-svn has not been updated in several months. Has the script maintaining it been disabled? I know patches have to be submitted through svn, but having a git clone of the history or git-svn is useful too.

Thanks,

Steve.

[Dave Abrahams]

on Sun Mar 31 2013, Stephen Kelly <steveire-AT-gmail.com> wrote:

> On Saturday, March 30, 2013 7:09:56 PM UTC+1, Dave Abrahams wrote:
>>
>>
>> on Thu Mar 28 2013, Stephen Kelly <steveire-AT-gmail.com> wrote:
>>
>> > Hi there,
>> >
>> > One of the annoying parts of trying to work on the boost cmake files
>> > is that small changes are impossible. I can't just change something
>> > in the generated cmake files for boost_any and boost_config,
>>
>> I'm sorry, but who's generating cmake files? As far as I know we are
>> creating cmake files entirely by hand.
>>
>
> I'm referring to the generation of files done by
> ryppl/cmake/Modules/RypplExport.cmake and testing modifications to that
> file.

OK. Sorry, I didn't design most of the CMake stuff in this system, so
I'm not really in a position to (usefully) comment on what makes working
on it annoying. That's really Daniel Pfeifer's department.

However, I do have a theory about what would make it much easier...

>> > The reason is that boost_config depends on boost_core, which depends
>> > on 35 other libraries:
>> > http://thread.gmane.org/gmane.comp.programming.tools.ryppl.devel/9/focus=26
>> > So, I have to change 35 libraries if I want to try/test something.
>>
>> > On Thursday, March 28, 2013 4:05:07 PM UTC+1, Vladimir Prus wrote:
>> >
>> > Pardon my ignorance, but why do you need to *change* 35
>> > libraries? Or did you mean "checkout" or something else?
>> >
>> > Sorry, maybe I didn't make that clear.
>> >
>> > If I'm working on the ryppl.cmake file, how it works and what it
>> > does, then I'll need to also change the CMakeLists.txt files that
>> > include it. If it were possible to comment out everything except
>> > config and any, that would be easy, but if have to change 35
>> > libraries *just* to develop/test a change, it's easier for me to
>> > throw in the towel :).
>>
>> That would be a shame. Why don't you just use Ryppl.cmake to put
>> together some smaller representative examples?
>
> I guess that would be an option, but it would only delay real testing of
> the changes.

Actually, I think that's the problem. We don't have any "real testing"
for Ryppl. Real tests—as opposed to a single, really large (though
important) use-case—would make it much easier to make changes and know
that they're right. I, at least, would be very grateful if you could
write some test cases, and I'd set them up to run at
http://jenkins.boost.org.

>> On the other hand, a global search/replace in 35 CMakeLists.txt files
>> can't be that hard unless... (ahem) you don't have the right tools ;-)

Oh, c'mon, man! I deserve a "touché" for that quip, don't I????

>> > My boss has a catchphrase: "This is not a tools problem"…

And the appropriate version in this case is, "this is not a Ryppl
problem; it's a Boost problem." Boost is merely Ryppl's first, most
important customer. But Ryppl itself is not about Boost at all.

>> > My understanding is that the ryppl community already decided that the
>> > interdependencies between modules don't matter because ryppl will be
>> > configured to download all needed dependencies.
>>
>> Uhm, no. I think you have grossly misinterpreted the attitude of the
>> ryppl community.
>
> Great! Thanks for clearing this up. My understanding was coming from
>
> http://thread.gmane.org/gmane.comp.programming.tools.ryppl.devel/9/focus=35
>
> and in particular "In the future, you won't clone the library with
> git, but setup a workspace with ryppl. Ryppl will clone the library
> *and* resolve dependencies." and "Ryppl will then create/update the
> CMakeLists.txt file in the top level." which indicated to me that all
> use of boost repos with cmake will require the ryppl tool for
> practical reasons. It also indicated to me that a sufficiently good
> tool makes any issues with interdependencies vanish.

Not *all* issues, but some of them. No matter how much the dependencies
of Boost are untangled I'm sure it will be extremely tedious to collect
all the dependendencies of some of these libraries without using a tool
like Ryppl.

> Thanks for clearing up that that's not the case.
>
>> Interdependencies certainly matter. Boost has too
>> many of them right now in part because the way we build and develop
>> Boost doesn't force people to confront them. By modularizing and making
>> the interdependencies explicit, we hope to encourage developers to
>> refactor. But we can only solve one problem at a time.
>
> Are you concerned about modularizing the repos first and the code second,
> and the effect that has on the history?

Not really.

> It's not really easy/practical to move files between different
> repos.

Sure it is. Of course you do end up with unpleasant discontinuities in
revision history. But from what I've seen, I guesstimated that most of
the dependency problems are not going to be solved by moving files, but
by splitting repositories into multiple parts.

That said, the module structure is being determined by
https://github.com/ryppl/Boost2Git/blob/master/repositories.txt and is
still (very) open to change. If you think files belong in different
repos, we can do that without disturbing Boost.

> It seems that with a 'one problem at a time' approach, the first
> problem that would need to be solved would be modularization of the
> code itself, not splitting into multiple repos. It seems you're saying
> that that would not be practical for social/community reasons?

It might be "practical" by some measure, but Boost is a big ship that
takes a long time to turn, so we're trying hard not to have Ryppl depend
on the decisions Boost makes (beyond switching to a modularized Git
structure).

> Any you're using split repos to force people to realize that the
> interdependencies exist?

Not exactly. Initially the split repos will still use the existing
Boost.Build procedure with a monolithic Boost "super-project." However,
integration with Ryppl forces you to declare your dependencies
explicitly. If/when Boost gets there, the dependency knots will become
much more apparent and it will be much easier to make a case for fixing
them.

> And you assume that will incentivize people to reduce such
> interdependencies?

Somewhat.

> Sorry for all the questions. Although I've been reading the boost
> mailing list for a few months, I'm not familiar with what it's like to
> try to get something done in the community when collaboration like
> that is needed.

Questions are good.

>> > However, in my view, this is not a tools problem. I'm sure many
>> > circular-dependency issues in boost modules can be solved. The ryppl
>> > tool can be helpful, but making it mandatory doesn't seem right.
>>
>> Making what mandatory?
>
> The ryppl tool. See the above quotes about how in the future it would be
> needed to clone the right 35 repos and generate a CMakeLists.txt file.

You can always do it manually, but, as I said, it would be tedious.
Even if you have only 5 dependencies. Don't forget, Ryppl will encode,
and use, version dependency information too.

>> > The result is not a 'modular boost' if all of the modules are
>> > interdependent.
>>
>> Of course not. But they're not *all* interdependent. There are,
>> unfortunately, a few strongly-connected components in the graph.
>>
>> > If all the interdependencies between the headers are going to remain,
>> > then why not put the 'core mesh' into one repo? If you want one of
>> > the 35 libraries in the list, you need all of them, right?
>>
>> If we do that, the dependencies would never become untangled.
>
> Indeedm but I didn't know that the plan was to modularize the code at all
> :).

I hope Boost will do that. I will encourage Boost to do that. But
Ryppl is a separate project.

> KDE is going through a similar modularization over the past couple of
> years,

I know; we're using a fork of their tool.

> which I've been involved with. We're modularizing the code directly in
> the kdelibs repo, and we'll eventually split it up into multiple
> independent git repos *after* we've done the modularization of the
> code.

BTW, I've found some serious bugs in their tool which I've reported but
it seems like the reports were ignored.

> I assumed, given the above quote again, and given my own opinion of the
> right order to do things (from my experience in the KDE efforts), that if
> you were splitting now, then you have already done all the modularization
> you intend to do. I can accept that that's not the case. Thanks for
> clearing that up.

It's not that simple. You can feel free to pursue things in the order
you think best. *I* barely have time for the Ryppl side of things.

>> > But a better question is why not remove the interdependencies?
>>
>> Good question. It would be great if you could do that!
>>

However, it has to be said, being able to handle awful dependency knots
is an important test case for Ryppl if people are going to use it to
distribute existing code.

>> > This seems like it should have been a prerequisite to splitting the
>> > code into multiple git repos, so I'm assuming the approach of
>> > actually modularizing the code was already considered and rejected
>> > for some reason?
>>
>> Individual maintainers "own" the individual libraries, and it seemed
>> like too much to take on negotiating all these changes at the same time
>> as doing the tools work.
>
> This is a little worrying :). It seems like something with obvious
> benefits to me, so I'd expect collaboration, not negotiation, unless
> some of the individual owning maintainers are opposed to the effort
> for some reason?

I don't know. I expect that such reorganization could require a lot of
work in some of the libraries, whose maintainers may not have a lot of
time right now, and I don't want it to be a blocker for Ryppl.

> If you would have do time-consuming negotiations, what chance do I
> have? :) I have no credibility in the boost community. I also don't
> have any familiarity with the codebase.
>
> I'm interested though, so I'll start a discussion there.

If you discover the changes that need to be made in the code, I'll be
happy to lend my credibility in support of your proposals to make the
changes.

--
Dave Abrahams

[Dave Abrahams]

on Sun Mar 31 2013, Stephen Kelly <steveire-AT-gmail.com> wrote:

> On Sunday, March 31, 2013 8:04:00 PM UTC+2, Stephen Kelly wrote:
>
> I'm interested though, so I'll start a discussion there.
>
> It seems the git repo at https://github.com/ryppl/boost-svn has not
> been updated in several months. Has the script maintaining it been
> disabled?

Probably.

> I know patches have to be submitted through svn, but having a git
> clone of the history or git-svn is useful too.

I can probably set that back up, but I am not sure it's worth the
effort. Why, exactly, do you want this?

--
Dave Abrahams