[boost] Any interest in static plugins?
Dave van Soest
I created a template-based header-only library for creating static plugin
systems. I'm wondering whether there would be any interest to include such
a library in Boost.
About static plugins:
Static plugins are not as common as dynamic plugins, but they have some
useful applications. Static plugins are a way to extend the functionality
of a program or library without having to change the existing source
files, just by adding a source file or object file, which is then linked
statically against the already existing object files. Some reasons you
might be doing this:
- You don't have permission to change the existing source files.
- You don't have the source files, only the object and header files. - You
don't want to have to change existing sources every time you extend
functionality.
- Simply including or excluding the object file from the linking process
adds or removes functionality from the program or library, leaving no
traces inside.
- Different configurations can be build easily this way.
- Testing only the required functionality without having to build the
full application.
- And some other advantages it shares with dynamic plugins.
If there appears to be any interest I will post more information and some
code and code examples.
Best regards,
Dave van Soest
_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Christian Schladetsch
You may get more response if you posted or linked to some motivational
examples.
Regards,
Christian.
Darryl Green
> Hi Dave,
>
> You may get more response if you posted or linked to some motivational
> examples.
>
> On Fri, Nov 27, 2009 at 11:45 AM, Dave van Soest <dso...@xs4all.nl> wrote:
>
> > Hi all,
> >
> > I created a template-based header-only library for creating static plugin
> > systems. I'm wondering whether there would be any interest to include such
> > a library in Boost.
+1 on the examples. So far I'm merely intrigued to understand what the
library is/does.
Dave van Soest
>> You may get more response if you posted or linked to some motivational
>> examples.
>
> +1 on the examples. So far I'm merely intrigued to understand what the
> library is/does.
Suppose that your program provides this simplified plugin interface
class 'ObjectBase' in the header 'ObjectBase.hpp':
---- Start of code sample 1 ----
class ObjectBase
{
public:
typedef std::string IdType;
ObjectBase() {}
virtual ~ObjectBase() {}
virtual void doSomething() = 0;
};
---- End of code sample 1 ----
Then the static plugin library will allow you to extend the
functionality of this program merely by adding a separate source file
containing the following code:
---- Start of code sample 2 ----
#include <static_plugin.hpp>
#include "ObjectBase.hpp"
class SomeObject : ObjectBase
{
void doSomething() { std::cout << "Doing something" << std::endl; }
static const IdType id;
};
const SomeObject::IdType SomeObject::id = "SomeObject_ID";
static const static_plugin::Creator<SomeObject, ObjectBase> myPlugin;
---- End of code sample 2 ----
The above code causes a factory for the 'SomeObject' class to be created
automatically and registered with the factory manager for classes
implementing the 'ObjectBase' interface. This all happens when the
program is just started (before it enters 'main').
The factory manager for 'ObjectBase' derivatives, which follows the
singleton pattern, can be used from your program as follows (note: again
this code is simplified):
---- Start of code sample 3 ----
#include <static_plugin.hpp>
#include "ObjectBase.hpp"
typedef FactoryManager<ObjectBase> ObjectBaseFM;
int main(void) {
const ObjectBaseFM& fm(getFactoryManager<ObjectBase>());
assert(fm.factoryCount() > 0);
ObjectBase::IdType id(fm.factoryIdList().at(0));
std::auto_ptr<ObjectBase> object(fm.create(id));
object->doSomething();
return 0;
}
---- End of code sample 3 ----
The only requirements are that ObjectBase defines the IdType type and
SomeObject contains a static member of type IdType called 'id'. The
declaration of the static_plugin::Creator<...> causes the SomeObject
plugin to be registered.
Sidenote: I just tested this functionality in a dynamically loaded
library and that works now too. This means that any plugin in your
dynamic library is automatically registered without the need to call
some 'initLibrary' function in the library.
OvermindDL1
Heh, I do the exact same thing for many of my things, especially in
the console to register new commands. If you can make it more simple
and/or efficient then I already have my code, I would use this, but so
far your code is slightly more verbose.
Christian Schladetsch
I appreciate your effort with this.
However, it is uninteresting to me. Relying on static initialiser ordering
is a huge mistake which IMO cannot be fixed.
Factory models based on such approaches are doomed to failure. It is best to
explicitly make a factory and then explicitly add runtime types dynamically
(perhaps via a configuration file).
This is not only more correct (the order or registration is repeatable), it
preserves ideas of scope and removes requirements for globals.
Best,
Christian.
Pierre Morcello
>>>>I just tested this functionality in a dynamically loaded library and that works now too.
Does it work correctly with static member of templated classes for example? I have always had problem of conflicts for those. I mean conflicts like a creation of static members copy instead of a reuse of a shared static instance (when one dll use another one dll) ... -Please note that I am no dll expert -.
Best regards,
Pierre Morcello
OvermindDL1
<pmorcell-...@yahoo.fr> wrote:
> Dave van Soest wrote :
>>>>>I just tested this functionality in a dynamically loaded library and that works now too.
>
> Does it work correctly with static member of templated classes for example? I have always had problem of conflicts for those. I mean conflicts like a creation of static members copy instead of a reuse of a shared static instance (when one dll use another one dll) ... -Please note that I am no dll expert -.
For note, let me give my use case.
I have one class, say ConsoleCommands, and it is a singleton designed
in such a way that it is constructed upon the first call to
GetInstance (and it does not even attempt to construct it, just
returns its pointer if you call GetInstancePtr).
I have things called ConsoleCommand that, as an example, can be called
like this:
ConsoleCommand cmd_help("help", &cmd_helpHandler, "Displays the help menu");
Internally it basically calls
ConsoleCommands::GetInstance.RegisterCommand("help", &cmd_helpHandler,
"Displays the help menu");
That will construct a ConsoleCommands if necessary (which is nothing
but an unordered_map essentially, nothing I need to worry about being
run before main).
It is not necessarily pretty, but I have been using it for near ten
years and never had a problem with it yet. I occasionally do other
things that make such registrations as well, so if he made a more
useful interface for it all so I do not need to make 3 helper structs
(only one of which actually holds data, the others are empty), then I
would be for it, but only if it really is easier. And yes, my method
works across DLL boundaries (since my code is designed to allow the
hot-plugging and removing of commands for the console for example).
Vicente Botet Escriba
Dave van Soest wrote:
>
> Hi Christian, Darryl, and anyone else interested,
>
>>> You may get more response if you posted or linked to some motivational
>>> examples.
>>
>> +1 on the examples. So far I'm merely intrigued to understand what the
>> library is/does.
>
> <snip>
>
> The factory manager for 'ObjectBase' derivatives, which follows the
> singleton pattern, can be used from your program as follows (note: again
> this code is simplified):
>
> ---- Start of code sample 3 ----
> #include <static_plugin.hpp>
> #include "ObjectBase.hpp"
>
> typedef FactoryManager<ObjectBase> ObjectBaseFM;
>
> int main(void) {
> const ObjectBaseFM& fm(getFactoryManager<ObjectBase>());
> assert(fm.factoryCount() > 0);
> ObjectBase::IdType id(fm.factoryIdList().at(0));
> std::auto_ptr<ObjectBase> object(fm.create(id));
> object->doSomething();
> return 0;
> }
> ---- End of code sample 3 ----
>
> The only requirements are that ObjectBase defines the IdType type and
> SomeObject contains a static member of type IdType called 'id'. The
> declaration of the static_plugin::Creator<...> causes the SomeObject
> plugin to be registered.
>
> _______________________________________________
> Unsubscribe & other changes:
> http://lists.boost.org/mailman/listinfo.cgi/boost
>
>
Hi,
I don't know if this is really related or not, there is a Functional/Factory
library accepted in Boost.
In case this can help you,
Vicente
--
View this message in context: http://old.nabble.com/Any-interest-in-static-plugins--tp26536909p26574107.html
Sent from the Boost - Dev mailing list archive at Nabble.com.
Christian Schladetsch
The key things to understand here are construction ordering and namespace
pollution.
When using static initialisers, you have absolutely no control over the
practical ordering of those initialisations in the final image. One build
may be A,B,C, the next may be C,A,B, even on the same platform with the same
set of types. The ordering is effectively arbitrary from build to build. No,
really, this is true.
Given that, using such an approach to build a factory model is irrational.
Many types need other types in their definitions. Without a deterministic
order, using static initialisation to create a factory is impossible. Sure,
you can come up with and practically use toy systems with a given linker in
a given configuration that have no problem. But when you have complex and.or
derived types, or use dynamic linking, it becomes an intractable problem.
I thought that these days, such issues are generally well-known. If you want
to do this sort of thing, *you must do it after main() has been called*.
Ideally, use some code-gen as well to make it type-safe as well as
build-time dynamic.
The second issue is global scope pollution. Systems based on pre-main()
initialisation are necessarily polluting the global namespace (you can hide
the variables in a namespace, but they are still global). As such, they
become impossible to test in isolation and further are problematic with
threads.
This is all just a longer-worded reply with the same answer as my previous
post: such approaches as Dave suggests are relatively common, and all are
doomed to failure in the general case.
My best advice to avoid heartbreak is to move to a localised, scoped,
dynamically driven model, perhaps with an automated code-gen back-end.
Regards,
Christian.
Christian Schladetsch
variables created within the same compilation unit, and its in order of
declaration. But you have no control over the total ordering of the linking
of the compilation units in C++, so this is small compensation.
Stewart, Robert
>
> Greetings,
Please don't top post. See http://www.boost.org/community/policy.html#quoting.
> When using static initialisers, you have absolutely no
> control over the
> practical ordering of those initialisations in the final
> image. One build
> may be A,B,C, the next may be C,A,B, even on the same
> platform with the same
> set of types. The ordering is effectively arbitrary from
> build to build. No, really, this is true.
Depending upon platform, the use of a dynamically linked library (DLL or shared object) may give desirable results, whereas the use of statically linked libraries will not. For example, in our environment, static objects in an object file in a static library *will not* be initialized unless a function in the same translation unit is called (at least indirectly) from main().
> Given that, using such an approach to build a factory model
> is irrational.
Not so much irrational as misinformed or ignorant of portability considerations.
> Many types need other types in their definitions. Without a
> deterministic
> order, using static initialisation to create a factory is
> impossible. Sure,
> you can come up with and practically use toy systems with a
> given linker in
> a given configuration that have no problem. But when you have
> complex and.or
> derived types, or use dynamic linking, it becomes an
> intractable problem.
That problem is worsened when it must be portable.
> I thought that these days, such issues are generally
> well-known. If you want
There's a great deal to know and one has to go looking for it to find it. It is easy to develop something in isolation and think it will work more broadly.
> Ideally, use some code-gen as well to make it type-safe as well as
> build-time dynamic.
Code generation is never an ideal solution, though it might be necessary to best satisfy a given set of criteria.
> The second issue is global scope pollution. Systems based on
> pre-main()
> initialisation are necessarily polluting the global namespace
> (you can hide
> the variables in a namespace, but they are still global). As
That is nonsensical. How can a name in a namespace pollute the global namespace? Perhaps you're trying to say that such variables contribute to the global system state.
> such, they
> become impossible to test in isolation and further are
> problematic with threads.
In the OP's environment, such statics may well be initialized before main() runs, thus avoiding threading problems. That isn't necessarily true, but it is accepted that threads shouldn't be created via static initialization, so it isn't unreasonable to make that a requirement for using such a library.
You are certainly on point about testing being difficult in the OP's scheme.
_____
Rob Stewart robert....@sig.com
Software Engineer, Core Software using std::disclaimer;
Susquehanna International Group, LLP http://www.sig.com
IMPORTANT: The information contained in this email and/or its attachments is confidential. If you are not the intended recipient, please notify the sender immediately by reply and immediately delete this message and all its attachments. Any review, use, reproduction, disclosure or dissemination of this message or any attachment by an unintended recipient is strictly prohibited. Neither this message nor any attachment is intended as or should be construed as an offer, solicitation or recommendation to buy or sell any security or other financial instrument. Neither the sender, his or her employer nor any of their respective affiliates makes any warranties as to the completeness or accuracy of any of the information contained herein or that this message or any of its attachments is free of viruses.
Christian Schladetsch
Rob Stewart wrote:
>
Depending upon platform, the use of a dynamically linked library (DLL or
> shared object) may give desirable results, whereas the use of statically
> linked libraries will not.
>
Not only depending on platform, but also depending on the order of the
linking of compilation units within a given DLL or other library; which is
not deterministic.
[snip]
> The second issue is global scope pollution. Systems based on
> > pre-main()
> > initialisation are necessarily polluting the global namespace
> > (you can hide
> > the variables in a namespace, but they are still global). As
>
> That is nonsensical. How can a name in a namespace pollute the global
> namespace? Perhaps you're trying to say that such variables contribute to
> the global system state.
>
Either or both my and the general terminology is lacking here. I don't know
what the convention is for this, but to clarify: when I say 'global' I mean
'globally accessible, given matching declaration'. A variable in namespace *
foo* can still be globally accessible, and remains traumatic when used in
threaded applications.
> In the OP's environment, such statics may well be initialized before
> main() runs, thus avoiding threading problems.
>
>
How could such a reprehensible act of creating a thread before main() runs
avoid problems?
Regards,
Christian.
Christian Schladetsch
of the initialisation of a global.
On Tue, Dec 1, 2009 at 12:50 AM, Christian Schladetsch <
christian....@gmail.com> wrote:
>
> Dissenting points from my preceding post are commented on:
>
> Rob Stewart wrote:
>>
>
> Depending upon platform, the use of a dynamically linked library (DLL or
>> shared object) may give desirable results, whereas the use of statically
>> linked libraries will not.
>>
>
> Not only depending on platform, but also depending on the order of the
> linking of compilation units within a given DLL or other library; which is
> not deterministic.
>
> [snip]
>
> > The second issue is global scope pollution. Systems based on
>> > pre-main()
>> > initialisation are necessarily polluting the global namespace
>> > (you can hide
>> > the variables in a namespace, but they are still global). As
>>
>> That is nonsensical. How can a name in a namespace pollute the global
>> namespace? Perhaps you're trying to say that such variables contribute to
>> the global system state.
>>
>
> Either or both my and the general terminology is lacking here. I don't know
> what the convention is for this, but to clarify: when I say 'global' I mean
> 'globally accessible, given matching declaration'. A variable in namespace
> *foo* can still be globally accessible, and remains traumatic when used in
Stewart, Robert
> Rob Stewart wrote:
>
> > > The second issue is global scope pollution. Systems based on
> > > pre-main()
> > > initialisation are necessarily polluting the global namespace
> > > (you can hide
> > > the variables in a namespace, but they are still global). As
> >
> > That is nonsensical. How can a name in a namespace pollute
> > the global namespace? Perhaps you're trying to say that such
> > variables contribute to the global system state.
>
> Either or both my and the general terminology is lacking
> here. I don't know
> what the convention is for this, but to clarify: when I say
> 'global' I mean
> 'globally accessible, given matching declaration'. A variable
> in namespace *
> foo* can still be globally accessible, and remains traumatic
> when used in threaded applications.
I'm still not sure I understand your concern. Any namespace name, regardless of how it is nested, is "globally accessible" according to your definition. IOW, unless something is a non-static member of a class, it is always accessible by appropriately resolving the name with enclosing namespace and class names.
Why does that matter?
> > In the OP's environment, such statics may well be initialized before
> > main() runs, thus avoiding threading problems.
> >
> How could such a reprehensible act of creating a thread
> before main() runs avoid problems?
If the OP never creates threads until main(), and all statics are initialized before main() runs, then threading need not be an issue.
_____
Rob Stewart robert....@sig.com
Software Engineer, Core Software using std::disclaimer;
Susquehanna International Group, LLP http://www.sig.com
IMPORTANT: The information contained in this email and/or its attachments is confidential. If you are not the intended recipient, please notify the sender immediately by reply and immediately delete this message and all its attachments. Any review, use, reproduction, disclosure or dissemination of this message or any attachment by an unintended recipient is strictly prohibited. Neither this message nor any attachment is intended as or should be construed as an offer, solicitation or recommendation to buy or sell any security or other financial instrument. Neither the sender, his or her employer nor any of their respective affiliates makes any warranties as to the completeness or accuracy of any of the information contained herein or that this message or any of its attachments is free of viruses.
Jeremy Pack
variables. Based on this difficult experience, I would highly recommend
initializing the factories inside of functions called after main() has
started, rather than in static initializers. The amount of code to make this
work correctly across platforms is nontrivial. In fact, we require that a
certain initialization function be called as soon as main() has started,
just to clean up all of the messes left by doing static initialization of so
many things. I believe that this original design was a mistake - but we have
too much code to change it overnight.
I wouldn't call it "doomed to failure" - just difficult to get right, and
probably not worth the effort to get it right.
I wrote some code for the sandbox a while ago that does this:
It works for both static and dynamic linking, and for shared libraries that
are determined at run time (the shared libraries need to have been compiled
with the same compiler options). It also allows arbitrary constructor
signatures, inheritance across shared libraries, reflection and a few other
features.
I've been waiting to finalize the reflection functionality (it works, but I
don't like parts of the API) until C++0x is more finalized in the various
compilers.
Jeremy Pack
Dave van Soest
> Heh, I do the exact same thing for many of my things, especially in
> the console to register new commands. If you can make it more simple
> and/or efficient then I already have my code, I would use this, but so
> far your code is slightly more verbose.
I think the verbosity of the code could be reduced by making 'id' a
static public member of ObjectBase, then let IdType be an integer and in
the initializer of 'id' call a counter function that generates unique
id's. The disadvantage of this approach is that you don't know
beforehand what the id of a certain plugin will be.
Best regards,
Dave van Soest
_______________________________________________
Dave van Soest
> However, it is uninteresting to me. Relying on static initialiser ordering
> is a huge mistake which IMO cannot be fixed.
I may be wrong, but I think this can be fixed. First of all, the
initialisation order of the plugins don't need any ordering, as they are
retrievable by their id. Secondly, because the initialiser of the plugin
factory uses a constant expression, the plugin factory is always
initialised before main is entered. The initialiser of the factory
manager is a non-const expression and is therefore initialised upon
first usage, which is exactly when the first plugin factory tries to
register itself.
So my point is: this static plugin code doesn't have to rely on static
initialiser ordering.
> Factory models based on such approaches are doomed to failure. It is best to
> explicitly make a factory and then explicitly add runtime types dynamically
> (perhaps via a configuration file).
>
> This is not only more correct (the order or registration is repeatable), it
> preserves ideas of scope and removes requirements for globals.
I agree that the order of registration is not necessarily repeatable (at
least not in different builds), but if the program relies on the
ordering of plugin initialisation, then there might be something wrong
with the program.
>> static const static_plugin::Creator<SomeObject, ObjectBase> myPlugin;
As you can see in the code above, the 'myPlugin; is declared static
(i.e. local to the source file), so there's nothing wrong with its scope.
I agree that global state should be prevented as much as possible, but a
(static) plugin initialisation just is a global thing, and it should be.
Best regards,
Dave van Soest
Dave van Soest
>
> Hi,
> I don't know if this is really related or not, there is a Functional/Factory
> library accepted in Boost.
>
> In case this can help you,
> Vicente
>
Thanks for the suggestion. I'm currently not using Boost Functions or
Factories, but I will look into it.
Kind regards,
Dave van Soest
Dave van Soest
Thanks for sharing your thoughts in this discussion. I'll try to
summarize your posts in this way: you both don't think it is possible to
create this static plugin functionality using static initialization and
get it working on every supported platform.
Well, I've already created a early version of this library and
successfully tested it on two different platforms with three different
compilers (two on each platform). I also think it conforms to the C++
standard in the way it uses static initialization (see my post dated
2009-11-30 14:00:54). Additionally I ran a successful test using shared
libraries and dynamic plugins. Unfortunately I currently don't have more
platforms and compilers to test the library with, but hopefully I will
soon find a way to do this.
I'm very much interested to learn whether this library will work on
other platforms, for example the platform Robert describes in his first
post to this thread.
My (now slightly adjusted) question still is: in the hypothetical
situation that this library works across all supported platforms, would
there be any interest to include it in Boost?
Best regards,
Dave van Soest
Dave van Soest
> My team works on code that uses a lot of static initialization for global
> variables. Based on this difficult experience, I would highly recommend
> initializing the factories inside of functions called after main() has
> started, rather than in static initializers. The amount of code to make this
> work correctly across platforms is nontrivial. In fact, we require that a
> certain initialization function be called as soon as main() has started,
> just to clean up all of the messes left by doing static initialization of so
> many things. I believe that this original design was a mistake - but we have
> too much code to change it overnight.
I want the plugins to work without explicitly initializing them after
main() has started. Because they are plugins, the rest of the program's
code won't know anything about them. It only knows how to call them --
through the specified interface. If it can't work that way, it makes no
sense to have this static plugin library.
I would be interested to learn what your and your team's experiences are
in static initialization across platforms. You say it's nontrivial, can
you please provide me some detail on this?
> I wouldn't call it "doomed to failure" - just difficult to get right, and
> probably not worth the effort to get it right.
Well, for the reasons I provide in my original post, I think it's worth
it to get it right. But hey, you seem to be the first one around here
not convinced that it's not feasible! That gives me some hope.
> I wrote some code for the sandbox a while ago that does this:
>
> http://boost-extension.redshoelace.com/docs/boost/extension/boost_extension/tutorials/tutorial02.html
>
> It works for both static and dynamic linking, and for shared libraries that
> are determined at run time (the shared libraries need to have been compiled
> with the same compiler options). It also allows arbitrary constructor
> signatures, inheritance across shared libraries, reflection and a few other
> features.
Your example uses a centralized place (the type-map) to store the
information about the derived classes/plugins. That is what I'm aiming
to avoid with my static plugin library.
Kind regards,
Dave van Soest
_______________________________________________
Jeremy Pack
> information about the derived classes/plugins. That is what I'm aiming to
> avoid with my static plugin library.
>
>
global to make it global. It is just like your "Factory Manager", but it can
be either global or local (with simple wrapper functions).
You could make it safe with the static initialization by making sure that
the Factory Manager is initialized before it is used. For your use case, I
think it would be sufficient to instantiate the Factory Manager inside of
the function that returns it (Get() below):
template <class Base>
class FactoryManager {
public:
static void RegisterTypeConstructor(Base* (*func)(), const string& id) {
Get()->insert(make_pair(id, func));
}
static map<string, Base*(*)()>* Get() {
static map<string, Base*(*)()> constructors;
return &constructors;
}
static Base* Create(const string& id) {
return (*(*Get())[id])();
}
};
template <class Derived, class Base>
class Factory {
public:
Factory(const string& name) {
FactoryManager<Base>::RegisterTypeConstructor(&Factory::Create, name);
}
static Base* Create() {
return new Derived();
}
};
class MyBaseClass {
public:
virtual void PrintString() {
cout << "MyBaseClass" << endl;
}
};
class MyDerivedClass : public MyBaseClass {
public:
virtual void PrintString() {
cout << "MyDerivedClass" << endl;
}
};
Factory<MyDerivedClass, MyBaseClass> i("derived");
class MyOtherDerivedClass : public MyBaseClass {
public:
virtual void PrintString() {
cout << "MyOtherDerivedClass" << endl;
}
};
Factory<MyOtherDerivedClass, MyBaseClass> j("other_derived");
int main(int argc, char* argv[]) {
cout << "Entering main." << endl;
MyBaseClass* c = FactoryManager<MyBaseClass>::Create("derived");
MyBaseClass* d = FactoryManager<MyBaseClass>::Create("other_derived");
c->PrintString();
d->PrintString();
}
/// End of code
Jeremy
OvermindDL1
>> Your example uses a centralized place (the type-map) to store the
>> information about the derived classes/plugins. That is what I'm aiming to
>> avoid with my static plugin library.
>>
>>
> I did this because I figured it would be trivial for a user who wanted it
> global to make it global. It is just like your "Factory Manager", but it can
> be either global or local (with simple wrapper functions).
>
> You could make it safe with the static initialization by making sure that
> the Factory Manager is initialized before it is used. For your use case, I
> think it would be sufficient to instantiate the Factory Manager inside of
> the function that returns it (Get() below):
Which is what I do in the code I demonstrated, as I said... I am
quite sure my emails are going to the list just fine...
Christian Schladetsch
On Fri, Dec 4, 2009 at 7:17 AM, Dave van Soest <dso...@xs4all.nl> wrote:
> Hi Christian, Robert, others interested,
>
> Thanks for sharing your thoughts in this discussion. I'll try to summarize
> your posts in this way: you both don't think it is possible to create this
> static plugin functionality using static initialization and get it working
> on every supported platform.
>
Actually, what I said was:
Sure, you can come up with and practically use toy systems with a given
> linker in a given configuration that have no problem. But when you have
> complex and.or derived types, or use dynamic linking, it becomes an
> intractable problem.
>
The problem here is that it is just a bad idea to use statics and type
factories in this way. There will be problem after problem. What are these
problems? The first problem is that of non-deterministic ordering of
initialisers. The second problem is that you're doing work and calling
functions before main(). The third problem are all the unknown problems that
will arise by using this approach.
This approach has been tried many times before. AFAICT, it has never ended
well. It requires bad practise (singletons, globals). It is untestable. It
doesn't work well with DLL's. If you use it in a threaded application, there
will be problems because of the globals. It doesn't scale to use types that
have other type components, or derive from other types.
Yes, it can work in a simple, limited case. But because of the scaling
problems, it is inadvisable to do this. I think that perhaps everyone has to
try to do this once, before they realise that it is a bad idea ;)
I could well be wrong, and I don't think I will dissuade you from continuing
your work on this. Best of luck.
Best,
Christian.
Dave van Soest
>> You could make it safe with the static initialization by making sure that
>> the Factory Manager is initialized before it is used. For your use case, I
>> think it would be sufficient to instantiate the Factory Manager inside of
>> the function that returns it (Get() below):
>
> Which is what I do in the code I demonstrated, as I said... I am
> quite sure my emails are going to the list just fine...
Jeremy, your code example also has many similarities with the code in my
library. So, now I'm interested why you aren't using this approach...
What kind of troubles do you have with it?
Jeremy Pack
> On 12/04/2009 03:11 AM, OvermindDL1 wrote:
>
> Jeremy, your code example also has many similarities with the code in my
> library. So, now I'm interested why you aren't using this approach... What
> kind of troubles do you have with it?
1. Boost.Extension is targeted at shared libraries linked at run time,
using functions like dlopen(). This means that in order to use the type of
static initialization that you do, I'd have to merge the Factory Managers
from multiple shared libraries when the libraries are loaded, and unmerge
them when the shared libraries are unloaded. I'd have to track how many
objects had been created from each shared library to decide whether or not a
given shared library could be unloaded.
2. I would have to introduce a dependency on Boost.Thread to guarantee
thread-safety when these libraries are opened.
3. The library is currently header-only. This would be lost.
4. I wanted to allow users the flexibility of either using it in a static
fashion or not. I could certainly provide wrapper functions that would
enable this type of static initialization.
For your goal, these issues aren't really a problem.
Jeremy
Dave van Soest
> Sure, you can come up with and practically use toy systems with a given
>> linker in a given configuration that have no problem. But when you have
>> complex and.or derived types, or use dynamic linking, it becomes an
>> intractable problem.
Can you name any complex or derived types that wouldn't work?
I've run some successful tests with dynamic linking. But I don't know
yet whether it works on all platforms.
> The problem here is that it is just a bad idea to use statics and type
> factories in this way. There will be problem after problem. What are these
> problems? The first problem is that of non-deterministic ordering of
> initialisers. The second problem is that you're doing work and calling
> functions before main(). The third problem are all the unknown problems that
> will arise by using this approach.
The order in which plugin factories are initialized shouldn't matter.
The factory manager is initialized when the first plugin factory tries
to register with it. So I don't see the problem here.
What is wrong with calling functions before main(), if you do it
carefully? The client coder won't get the opportunity to call any
function before main() through this library.
My response to the third problem is, just like the problem itself, yet
unknown.
> This approach has been tried many times before. AFAICT, it has never ended
> well. It requires bad practise (singletons, globals). It is untestable. It
> doesn't work well with DLL's. If you use it in a threaded application, there
> will be problems because of the globals. It doesn't scale to use types that
> have other type components, or derive from other types.
I will be doing more research to see whether my approach is different
from the ones that didn't end well.
I don't have any problems with using singletons, as long as they are
used for the right purposes.
Why is it untestable? I don't have a clue here.
As I said, I ran a successful test using dynamically loaded libraries.
I don't see any problems with threaded applications. The 'global'
factories are not accessible outside there compilation unit. And the
global factory manager(s) don't change any global information.
I don't see why it doesn't scale to certain types. Can you give an
example of a structure that wouldn't work?
> Yes, it can work in a simple, limited case. But because of the scaling
> problems, it is inadvisable to do this. I think that perhaps everyone has to
> try to do this once, before they realise that it is a bad idea ;)
I haven't reached the point that I realised it's a bad idea. I'm seeing
more use-cases each day. That's keeping me from giving up too early.
Moreover, I haven't seen any concrete counter-example yet. And even if
can only prove it to work on just a part of the platforms, it will still
be valuable. But in that case not for inclusion in Boost of course.
> I could well be wrong, and I don't think I will dissuade you from continuing
> your work on this. Best of luck.
Thanks. I really appreciate the time you spent on joining the
discussion. Indeed you can't dissuade me from continuing this work, but
still I really think it's valuable to have a sceptical counterweight to
keep me from turning assumptions into beliefs and pointing me at
potential problems I haven't really well thought out.
Best regards,
Dave van Soest
Pierre Morcello
Does it work with static templated members of templated classes? because if not perfectly prepared, it may end up with copies of the templated member instead of a unique shared member between the dlls. (Or I am missing the goal of your proposed system ?)
Best regards,
Pierre
Christian Schladetsch
> Hi Christian,
>
> Sure, you can come up with and practically use toy systems with a given
>>
>>> linker in a given configuration that have no problem. But when you have
>>> complex and.or derived types, or use dynamic linking, it becomes an
>>> intractable problem.
>>>
>>
> Can you name any complex or derived types that wouldn't work?
>
// this requires that type A is registered before type B
struct A { };
struct B { Pointer<A> a; static void Register(Factory &f) { a =
f.Create<A>(); } };
...
// this also requires that A is registered before B
struct A { };
struct B : DerivesFrom<A> { };
...
>
> What is wrong with calling functions before main(), if you do it carefully?
>
Because "doing it carefully" sometimes isn't enough. For example, if you're
doing your own memory management (a common motivator for using factories at
all), then any allocation done before main() will be using the clib malloc -
and before any of your own custom memory allocation systems have been
initialised. Then you have to work-around that.
Then that becomes a problem because often this allocator has to be guarded
against multiple threads, which requires semaphores or other locking
structures to be setup, which then have to be done statically, and because
of the ordering issue you are back to square one.
> The client coder won't get the opportunity to call any function before
> main() through this library.
>
This is not true; initialisation code in the registration method will be
called before main(), as will anything called from it. Over time you may
want to read configuration files or do memory allocation in these pre-main
systems. It grows. It gets gnarly. You work-around it. It grows more. Then
you get it working, then change linkers and it all breaks again.
Other example gotchas are initialisation of logging systems, string pools
systems, data-structures with private static pools - basically, pre-main is
a bad space in which to try to do anything.
If you kept it extremely simple - no allocation, no threading, no file
handling, no exceptions, no nested or derived types, and in a known, single
development environment then yeah, sure, it can work. But its not portable,
doesn't scale, is untestable, and its extremely limited.
Oh, its not testable because the use of globals means that you cant create
the system, test it and tear it down again in isolation of other systems.
The lack of deterministic ordering means that testing is problematic even in
the basic cases.
> My response to the third problem is, just like the problem itself, yet
> unknown.
The C machine is not meant to be used this way. It can be used this way, but
it invariably results in many subtle problems.
This idea has been kicked around by various people and teams going back
decades. It has always proven to be a neat idea that doesn't work in the
real world. Perhaps it can work; but all I know about are counter-examples.
> Thanks. I really appreciate the time you spent on joining the discussion.
> Indeed you can't dissuade me from continuing this work, but still I really
> think it's valuable to have a sceptical counterweight to keep me from
> turning assumptions into beliefs and pointing me at potential problems I
> haven't really well thought out.
Welcome,
Christian.
Dave van Soest
The C++ standards document states that it is implementation dependent
whether the constructor of a static object is called before entering
main() or just before first use (ISO/IEC 14882:2003 section 3.6.2 part 3).
I can't help wondering why the standard leaves this choice open to
implementation.
The compilers/linkers/platforms I tested my library with all enable
calling the constructors before entering main() by default, but this
behavior is not guaranteed by the language. I currently don't know
whether this behavior can be guaranteed in any way. If so, the library
may still be useful, but perhaps not for Boost.
Kind regards,
Dave van Soest
Emil Dotchevski
> Hi all,
>
> The C++ standards document states that it is implementation dependent
> whether the constructor of a static object is called before entering main()
> or just before first use (ISO/IEC 14882:2003 section 3.6.2 part 3).
>
> I can't help wondering why the standard leaves this choice open to
> implementation.
The idea is to allow for costly initializations to be postponed until
they are actually needed. For example, if a static object allocates a
gig of RAM, that would be wasteful if the static object isn't needed
in a particular run of the program.
> The compilers/linkers/platforms I tested my library with all enable calling
> the constructors before entering main() by default, but this behavior is not
> guaranteed by the language. I currently don't know whether this behavior can
> be guaranteed in any way. If so, the library may still be useful, but
> perhaps not for Boost.
No, the behavior can not be guaranteed. What's more, static objects,
along with all relevant code, can be deadstripped unless you
explicitly call a function from their translation unit (cpp file.)
Emil Dotchevski
Reverge Studios, Inc.
http://www.revergestudios.com/reblog/index.php?n=ReCode