[Boost-users] [bcp] [shared_ptr] Extracting subset of boost for shared_ptr - 3.2MB?

[Tyler Mandry]

I extracted a subset of boost using bcp so that my project could use
shared_ptr. The exact line I used is this:

$ bcp --boost=/usr/include boost/shared_ptr.hpp boost/

This gave me more than just the boost headers, but any other way and
it wouldn't work. It wasn't a problem though; I could just use the
files in the newly-created boost/boost directory and wipe out anything
else. So I was left with, presumably, all the boost headers and files
that depended on shared_ptr.hpp.

But, after running du, I found that the extracted directory was a
whopping 3.2MB - bigger than the existing code of my project
altogether. bcp had given me 274 different header files. All for just
one smart pointer class?

Is this correct? Am I using bcp correctly? Are there some paths that
bcp is taking that aren't really neccessary? Can I narrow this huge
vat of files down to the ones that I *really* need, or is this it?
How?

Regards and thanks,
--
Tyler Mandry

_______________________________________________
Boost-users mailing list
Boost...@lists.boost.org
http://lists.boost.org/mailman/listinfo.cgi/boost-users

[Peter Dimov]

Tyler Mandry wrote:

> Are there some paths that bcp is taking that aren't really neccessary?

Unfortunately bcp isn't smart enough to not look into optional dependencies
that are guarded by #ifdef. In this particular case it follows the
dependency to detail/quick_allocator.hpp, which in turn brings in the
type_traits and mpl libraries. But quick_allocator.hpp is only used if you
#define BOOST_SP_USE_QUICK_ALLOCATOR.

> Can I narrow this huge vat of files down to the ones that I *really* need,
> or is this it?
> How?

The only reliable way that I know of is to copy the required headers into a
pristine include directory one by one until your program compiles. Not
particularly elegant, but it works. In shared_ptr's case you need

boost/assert.hpp
boost/current_function.hpp
boost/checked_delete.hpp
boost/throw_exception.hpp

Most of boost/detail/, so I just copy it wholesale
boost/config/*

[John Maddock]

> Is this correct? Am I using bcp correctly? Are there some paths that
> bcp is taking that aren't really neccessary? Can I narrow this huge
> vat of files down to the ones that I *really* need, or is this it?
> How?

I updated the docs recently because of this very case:

"File dependencies are found as follows:

* C++ source files are scanned for #includes, all #includes present in the
boost source tree will then be scanned for their dependencies and so on.
* C++ source files are associated with the name of a library, if that
library has source code (and possibly build data), then include that source
in the dependencies.
* C++ source files are checked for dependencies on Boost.test (for example
to see if they use cpp_main as an entry point).
* HTML files are scanned for immediate dependencies (images and style
sheets, but not links).

It should be noted that in practice bcp can produce a rather "fat" list of
dependencies, reasons for this include:

* It searches for library names first, so using "regex" as a name will give
you everything in the libs/regex directory and everything that depends on.
This can be a long list as all the regex test and example programs will get
scanned for their dependencies. If you want a more minimal list, then try
using the names of the headers you are actually including, or use the --scan
option to scan your source code.
* If you include the header of a library with separate source, then you get
that libraries source and all it's dependencies. This is deliberate and in
general those extra dependencies are needed.
* When you include a header, bcp doesn't know what compiler you're using, so
it follows all possible preprocessor paths. If you're distributing a subset
of Boost with you're application then that is what you want to have happen
in general.

The last point above can result in a substantial increase in the number of
headers found compared to most peoples expectations. For example bcp finds
274 header dependencies for boost/shared_ptr.hpp: by running bcp in report
mode we can see why all these headers have been found as dependencies:

* All of the Config library headers get included (52 headers, would be about
6 for one compiler only).
* A lot of MPL and type traits code that includes workarounds for broken
compilers that you may or may not need. Tracing back through the code shows
that most of these aren't needed unless the user has defined
BOOST_SP_USE_QUICK_ALLOCATOR, however bcp isn't aware of whether that
preprocessor path will be taken or not, so the headers get included just in
case. This adds about 48 headers (type traits), plus another 49 from MPL.
* The Preprocessor library gets used heavily by MPL: this adds another 96
headers.
* The Shared Pointer library contains a lot of platform specific code, split
up into around 22 headers: normally your compiler would need only a couple
of these files.

As you can see the number of dependencies found are much larger than those
used by any single compiler, however if you want to distribute a subset of
Boost that's usable in any configuration, by any compiler, on any platform
then that's exactly what you need. If you want to figure out which Boost
headers are being used by your specific compiler then the best way to find
out is to prepocess the code and scan the output for boost header includes.
You should be aware that the result will be very platform and compiler
specific, and may not contain all the headers needed if you so much as
change a compiler switch (for example turn on threading support)."

So... to answer your specific question: if you want your code to be portable
to *any* compiler then yes you need all those headers, otherwise using sed
to process the preprocessed source would give you a more slimline set of
headers.

HTH, John

[John Maddock]

> Unfortunately bcp isn't smart enough to not look into optional
> dependencies that are guarded by #ifdef. In this particular case it
> follows the dependency to detail/quick_allocator.hpp, which in turn
> brings in the type_traits and mpl libraries. But quick_allocator.hpp
> is only used if you #define BOOST_SP_USE_QUICK_ALLOCATOR.

Peter, this comes up so often, I wonder is it worth while "hiding" that
optional dependency behind a #define, as in:

#define BOOST_SP_QA <some-header.hpp>

#ifdef BOOST_SP_USE_QUICK_ALLOCATOR
#include BOOST_SP_QA
#endif

Of course I have no data on how often that dependency is actually needed (or
not).

John.

[Peter Dimov]

John Maddock wrote:

>> Unfortunately bcp isn't smart enough to not look into optional
>> dependencies that are guarded by #ifdef. In this particular case it
>> follows the dependency to detail/quick_allocator.hpp, which in turn
>> brings in the type_traits and mpl libraries. But quick_allocator.hpp
>> is only used if you #define BOOST_SP_USE_QUICK_ALLOCATOR.
>
> Peter, this comes up so often, I wonder is it worth while "hiding"
> that optional dependency behind a #define, as in:
>
> #define BOOST_SP_QA <some-header.hpp>
>
> #ifdef BOOST_SP_USE_QUICK_ALLOCATOR
> #include BOOST_SP_QA
> #endif
>
> Of course I have no data on how often that dependency is actually
> needed (or not).

How about something along the lines of:

bcp boost/shared_ptr.hpp --exclude=quick_allocator.hpp

Or:

#ifdef BOOST_SP_USE_QUICK_ALLOCATOR

//@bcp optional "quick_allocator is only used if explicitly requested"
#include <boost/detail/quick_allocator.hpp>

#endif

which would then inform the user that an optional dependency wasn't being
followed (if this is the default):

bcp: an optional dependency isn't included in the subset because

`quick_allocator is only used if explicitly requested'

bcp: use `bcp boost/shared_ptr.hpp boost/detail/quick_allocator.hpp' to
include it

The advantage is that we don't have to decide whether to follow a particular
dependency, the choice can be left to the person running bcp.

[David Abrahams]

"John Maddock" <jo...@johnmaddock.co.uk> writes:

>> Unfortunately bcp isn't smart enough to not look into optional
>> dependencies that are guarded by #ifdef. In this particular case it
>> follows the dependency to detail/quick_allocator.hpp, which in turn
>> brings in the type_traits and mpl libraries. But quick_allocator.hpp
>> is only used if you #define BOOST_SP_USE_QUICK_ALLOCATOR.
>
> Peter, this comes up so often, I wonder is it worth while "hiding" that
> optional dependency behind a #define, as in:
>
> #define BOOST_SP_QA <some-header.hpp>
>
> #ifdef BOOST_SP_USE_QUICK_ALLOCATOR
> #include BOOST_SP_QA
> #endif
>

If doing that works to fool bcp, we need a better system. It would be
a lot of work, but maybe it would be the first real use of Wave as a
library rather than just as a preprocessor. We could use the graph
library to do a real search with some knowledge of compiler and
configuration #defines. It would be a conservative search, so it
would follow *all* PP #ifdef branches unless it could prove that they
shouldn't be followed. Wow, that sounds like an interesting but
difficult dynamic programming problem. I wonder if it's even
tractable?

--
Dave Abrahams
Boost Consulting
www.boost-consulting.com

[David Abrahams]

David Abrahams <da...@boost-consulting.com> writes:

> "John Maddock" <jo...@johnmaddock.co.uk> writes:
>
>>> Unfortunately bcp isn't smart enough to not look into optional
>>> dependencies that are guarded by #ifdef. In this particular case it
>>> follows the dependency to detail/quick_allocator.hpp, which in turn
>>> brings in the type_traits and mpl libraries. But quick_allocator.hpp
>>> is only used if you #define BOOST_SP_USE_QUICK_ALLOCATOR.
>>
>> Peter, this comes up so often, I wonder is it worth while "hiding" that
>> optional dependency behind a #define, as in:
>>
>> #define BOOST_SP_QA <some-header.hpp>
>>
>> #ifdef BOOST_SP_USE_QUICK_ALLOCATOR
>> #include BOOST_SP_QA
>> #endif
>>
>
> If doing that works to fool bcp, we need a better system. It would be
> a lot of work, but maybe it would be the first real use of Wave as a
> library rather than just as a preprocessor. We could use the graph
> library to do a real search with some knowledge of compiler and
> configuration #defines. It would be a conservative search, so it
> would follow *all* PP #ifdef branches unless it could prove that they
> shouldn't be followed. Wow, that sounds like an interesting but
> difficult dynamic programming problem. I wonder if it's even
> tractable?

I like Peter's idea much better in the near term ;-)

[John Maddock]

> How about something along the lines of:
>
> bcp boost/shared_ptr.hpp --exclude=quick_allocator.hpp
>
> Or:
>
> #ifdef BOOST_SP_USE_QUICK_ALLOCATOR
>
> //@bcp optional "quick_allocator is only used if explicitly requested"
> #include <boost/detail/quick_allocator.hpp>
>
> #endif
>
> which would then inform the user that an optional dependency wasn't
> being followed (if this is the default):
>
> bcp: an optional dependency isn't included in the subset because
>
> `quick_allocator is only used if explicitly requested'
>
> bcp: use `bcp boost/shared_ptr.hpp
> boost/detail/quick_allocator.hpp' to include it
>
> The advantage is that we don't have to decide whether to follow a
> particular dependency, the choice can be left to the person running
> bcp.

I like it, shouldn't be too hard to add either.....

John.

[John Maddock]

> Or:
>
> #ifdef BOOST_SP_USE_QUICK_ALLOCATOR
>
> //@bcp optional "quick_allocator is only used if explicitly requested"
> #include <boost/detail/quick_allocator.hpp>
>
> #endif

I've just added this to bcp as an experiment (it's in mainline cvs), with
the extra line above added to shared_ptr as well, the full dependency list
is down to:

$bcp --cvs --list shared_ptr.hpp
Optional functionality won't be copied: quick_allocator is only used if
explicitly requested
Add the file boost/detail/quick_allocator.hpp to the list of dependencies to
extract to copy this functionality.
boost/assert.hpp
boost/checked_delete.hpp
boost/config.hpp
boost/config/abi/borland_prefix.hpp
boost/config/abi/borland_suffix.hpp
boost/config/abi/msvc_prefix.hpp
boost/config/abi/msvc_suffix.hpp
boost/config/abi_prefix.hpp
boost/config/abi_suffix.hpp
boost/config/auto_link.hpp
boost/config/compiler/borland.hpp
boost/config/compiler/comeau.hpp
boost/config/compiler/common_edg.hpp
boost/config/compiler/compaq_cxx.hpp
boost/config/compiler/digitalmars.hpp
boost/config/compiler/gcc.hpp
boost/config/compiler/gcc_xml.hpp
boost/config/compiler/greenhills.hpp
boost/config/compiler/hp_acc.hpp
boost/config/compiler/intel.hpp
boost/config/compiler/kai.hpp
boost/config/compiler/metrowerks.hpp
boost/config/compiler/mpw.hpp
boost/config/compiler/sgi_mipspro.hpp
boost/config/compiler/sunpro_cc.hpp
boost/config/compiler/vacpp.hpp
boost/config/compiler/visualc.hpp
boost/config/no_tr1/complex.hpp
boost/config/no_tr1/functional.hpp
boost/config/no_tr1/memory.hpp
boost/config/no_tr1/utility.hpp
boost/config/platform/aix.hpp
boost/config/platform/amigaos.hpp
boost/config/platform/beos.hpp
boost/config/platform/bsd.hpp
boost/config/platform/cygwin.hpp
boost/config/platform/hpux.hpp
boost/config/platform/irix.hpp
boost/config/platform/linux.hpp
boost/config/platform/macos.hpp
boost/config/platform/qnxnto.hpp
boost/config/platform/solaris.hpp
boost/config/platform/win32.hpp
boost/config/posix_features.hpp
boost/config/requires_threads.hpp
boost/config/select_compiler_config.hpp
boost/config/select_platform_config.hpp
boost/config/select_stdlib_config.hpp
boost/config/stdlib/dinkumware.hpp
boost/config/stdlib/libcomo.hpp
boost/config/stdlib/libstdcpp3.hpp
boost/config/stdlib/modena.hpp
boost/config/stdlib/msl.hpp
boost/config/stdlib/roguewave.hpp
boost/config/stdlib/sgi.hpp
boost/config/stdlib/stlport.hpp
boost/config/stdlib/vacpp.hpp
boost/config/suffix.hpp
boost/config/user.hpp
boost/current_function.hpp
boost/detail/atomic_count.hpp
boost/detail/atomic_count_gcc.hpp
boost/detail/atomic_count_pthreads.hpp
boost/detail/atomic_count_win32.hpp
boost/detail/bad_weak_ptr.hpp
boost/detail/interlocked.hpp
boost/detail/shared_count.hpp
boost/detail/shared_ptr_nmt.hpp
boost/detail/sp_counted_base.hpp
boost/detail/sp_counted_base_cw_ppc.hpp
boost/detail/sp_counted_base_gcc_ia64.hpp
boost/detail/sp_counted_base_gcc_ppc.hpp
boost/detail/sp_counted_base_gcc_x86.hpp
boost/detail/sp_counted_base_nt.hpp
boost/detail/sp_counted_base_pt.hpp
boost/detail/sp_counted_base_w32.hpp
boost/detail/sp_counted_impl.hpp
boost/detail/workaround.hpp
boost/non_type.hpp
boost/shared_ptr.hpp
boost/throw_exception.hpp
boost/type.hpp
boost/version.hpp

The list of config files is still rather long, but I'd say that's a result -
certainly not bad for 10 minutes programming :-)

John.

[Jeff Garland]

On Sun, 19 Feb 2006 10:17:12 -0000, John Maddock wrote

> > How about something along the lines of:
> >
> > bcp boost/shared_ptr.hpp --exclude=quick_allocator.hpp
> >
> > Or:
> >
> > #ifdef BOOST_SP_USE_QUICK_ALLOCATOR
> >
> > //@bcp optional "quick_allocator is only used if explicitly requested"
> > #include <boost/detail/quick_allocator.hpp>
> >
> > #endif
> >
> > which would then inform the user that an optional dependency wasn't
> > being followed (if this is the default):
> >
> > bcp: an optional dependency isn't included in the subset because
> >
> > `quick_allocator is only used if explicitly requested'
> >
> > bcp: use `bcp boost/shared_ptr.hpp
> > boost/detail/quick_allocator.hpp' to include it
> >
> > The advantage is that we don't have to decide whether to follow a
> > particular dependency, the choice can be left to the person running
> > bcp.
>
> I like it, shouldn't be too hard to add either.....

As you've already shown...

Is 'optional' a 'fixed tag' or can we use it to slice and dice things? For
example, suppose we wanted to create a boost-wide 'with[out]_serialization'
capability. Could we do this?

//@bcp with_serialization "serialization support for date-time is optional"
#include #include "boost/serialization/split_free.hpp"

There a bunch of boost libraries that would benefit from this type of optional
flagging. For date-time it turns out that the --exclude=xyz.hpp capability
could also serve the same purpose.

Another possible idea. What if we created a new convention: <libname>_bcp.cfg
that could be put in the library header directory. Then the library author
could directly express the options using some trivial syntax of our own
invention that would feed bcp and thus trivialize the command line for users.
This syntax would be something like:

option option_name option_description bcp_command_parameters

option serialization "Provides support for serialization" --exclude
date_time/gregorian/greg_serialize.hpp --exclude
date_time/posix_time/time_serialize.hpp

or even better:

option serialization "Provides support for serialization"
--exclude_library_depend serialization --exclude_library_depend archive

Anyway, just some wild ideas ;-)

Jeff

[David Abrahams]

"Jeff Garland" <je...@crystalclearsoftware.com> writes:

> Another possible idea. What if we created a new convention: <libname>_bcp.cfg
> that could be put in the library header directory. Then the library author
> could directly express the options using some trivial syntax of our own
> invention that would feed bcp and thus trivialize the command line for users.
> This syntax would be something like:
>
> option option_name option_description bcp_command_parameters

We probably ought to be thinking about how this could work with
Boost.Build. I don't see it clearly yet, but I bet there's some kind
of integration opportunity.

--
Dave Abrahams
Boost Consulting
www.boost-consulting.com

_______________________________________________

[Peter Dimov]

John Maddock wrote:
>> Or:
>>
>> #ifdef BOOST_SP_USE_QUICK_ALLOCATOR
>>
>> //@bcp optional "quick_allocator is only used if explicitly
>> requested" #include <boost/detail/quick_allocator.hpp>
>>
>> #endif
>
> I've just added this to bcp as an experiment (it's in mainline cvs),

> with the extra line above added to shared_ptr as well, ...

This extra line doesn't seem to be in CVS yet, did you forget to commit, or
should I add it for you? :-)

[John Maddock]

> This extra line doesn't seem to be in CVS yet, did you forget to
> commit, or should I add it for you? :-)

Apologies: I should have been clearer: bcp has been updated to recognise

//@bcp optional message-text
#include <optional-header.hpp>

But I deliberately haven't changed any headers, please do go ahead and
experiment with this and make changes to whatever headers you see fit to :-)

There have also been some useful suggestions from Jeff Garland that I want
to mull over, conceptually we could have some thing like:

//@bcp default=enable name=Wow Optional feature Wow is enabled by default
use --disable-Wow to suppress it's inclusion

or:

//@bcp default=disable name=Wow Optional feature Wow is disabled by default
use --enable-Wow to force it's inclusion

and

//@bcp suppress

which would unconditionally suppress the inclusion of the following include.

But I haven't really thought all that through yet. Jeff's idea of a
separate file for optional includes would make it easier for lib authors to
exclude inclusion of headers through a whole set of files, but requires
rather more modifications to bcp than may be wise this close to a release?

There may be synergies with Boost.Build as well, as I believe that scans
headers in a similar way to bcp.

John.