STL and thread-safety
Divya Mali
I am looking for a site that shows in a matrix thread-safety issues
associated with STL from different vendors. Is there one available?
I am planning to use STL heavily in a multi-threaded application and am
looking for any gotchas, things to avoid, optimized way of doing things etc.
I used STL 5 years back, and at that time became aware of static allocators
and the fact that two different instances of a container class could share
the same allocator, and hence calls to certain operations (methods) of these
container classes had to be synchronized. Do such problems still exist?
Thanks,
Ganesh
[ See http://www.gotw.ca/resources/clcm.htm for info about ]
[ comp.lang.c++.moderated. First time posters: Do this! ]
Maxim Yegorushkin
> I am looking for a site that shows in a matrix thread-safety issues
> associated with STL from different vendors. Is there one available?
I doubt that one exists.
STL containers are not thread safe and they are such for very good
reasons. It is safe to read/access concurrently a container unless
somebody is writing/changing it at the same time.
For longer discussion please see
http://www.sgi.com/tech/stl/thread_safety.html
> I am planning to use STL heavily in a multi-threaded application and am
> looking for any gotchas, things to avoid, optimized way of doing things
> etc.
The main gotcha is to avoid sharing data between threads at all. The less
your share data between threads the less synchronization and locking you
need.
> I used STL 5 years back, and at that time became aware of static
> allocators
> and the fact that two different instances of a container class could
> share
> the same allocator, and hence calls to certain operations (methods) of
> these
> container classes had to be synchronized. Do such problems still exist?
Well, you can think of allocators as of monostate pattern: all instances
use the same data and behavior. Often they are implemented as simple
wrappers that call ::operator new() or malloc(), although some
implementations (like STLPort) are more complicated than that. In any
case, they are as thread safe as the underlying ::operator new()/malloc()
are, which are usually safe to call simultaneously from different threads.
--
Maxim Yegorushkin
James Kanze
> Divya Mali wrote:
>>I am looking for a site that shows in a matrix thread-safety
>>issues associated with STL from different vendors. Is there
>>one available?
> I doubt that one exists.
> STL containers are not thread safe and they are such for very
> good reasons.
This depends on the implementation. All of the implementations
I know are thread safe. That is, they define exactly what
guarantees they give with regards to thread safety, and give at
least the guarantees that Posix gives for a basic type. (The
only exception I know of here is std::string in g++. But g++
isn't really very usable in a multithreaded environment anyway.)
> It is safe to read/access concurrently a container unless
> somebody is writing/changing it at the same time.
> For longer discussion please see
> http://www.sgi.com/tech/stl/thread_safety.html
That's the definition of thread safety.
--
James Kanze home: www.gabi-soft.fr
Conseils en informatique orientée objet/
Beratung in objektorientierter Datenverarbeitung
9 pl. Pierre Sémard, 78210 St.-Cyr-l'École, France +33 (0)1 30 23 00 34
Maxim Yegorushkin
>> STL containers are not thread safe and they are such for very
>> good reasons.
>
> This depends on the implementation. All of the implementations
> I know are thread safe. That is, they define exactly what
> guarantees they give with regards to thread safety, and give at
> least the guarantees that Posix gives for a basic type. (The
> only exception I know of here is std::string in g++. But g++
> isn't really very usable in a multithreaded environment anyway.)
Well, it depends on the definition of thread safety. IMO, OP meant if it
is safe to call container's member functions simultaneously without
locking no matter whether these functions change to container or not.
--
Maxim Yegorushkin
James Talbut
problems with concurrent operations on different instances of vectors
(i.e. it's not safe to expand two vectors simultaneously).
Do problems of this kind exist in the implementations that you are
aware of?
"James Kanze" <ka...@none.news.free.fr> wrote in message
news:4234b288$0$13914$626a...@news.free.fr...
> (The only exception I know of here is std::string in g++. But g++
> isn't really very usable in a multithreaded environment anyway.)
Could you either explain that further or direct me to somewhere that
does?
Thanks.
--
J.T.
Please reply via the newsgroup.
ka...@gabi-soft.fr
> My reading of the original question was that he understood
> there to be problems with concurrent operations on different
> instances of vectors (i.e. it's not safe to expand two vectors
> simultaneously). Do problems of this kind exist in the
> implementations that you are aware of?
Only if you compile (or link) them without specifying the
necessary options for a multithreaded environment.
There are problems with concurrent accesses to std::string in
g++, even if no thread is modifying it. Despite the Posix
guarantees that this is allowed.
> "James Kanze" <ka...@none.news.free.fr> wrote in message
> news:4234b288$0$13914$626a...@news.free.fr...
> > (The only exception I know of here is std::string in g++.
> > But g++ isn't really very usable in a multithreaded
> > environment anyway.)
> Could you either explain that further or direct me to
> somewhere that does?
Well, I should have been more precise: the problem is that g++
generally gives significantly less guarantees than the native
system. In my case, under Solaris, I have experienced problems
with std::string and with the Posix function pthread_cancel --
in both cases, one could argue that it is just a case of g++
giving different guarantees, but the g++ guarantee for
std::string is contrary to what Posix guarantees (e.g. for
char[]), and while Posix doesn't say what pthread_cancel should
do in C++, it gives strong hints, and the g++ implementation
under Solaris doesn't provide the guarantees that Solaris
requires (e.g. that destructors will be called).
Globally, they've made a lot of progress in a very short time,
but the implementation still lacks maturity (with regards to
multi-threading -- not over-all).
--
James Kanze GABI Software
Conseils en informatique orientée objet/
Beratung in objektorientierter Datenverarbeitung
9 place Sémard, 78210 St.-Cyr-l'École, France, +33 (0)1 30 23 00 34
ka...@gabi-soft.fr
> James Kanze wrote:
> >> STL containers are not thread safe and they are such for
> >> very good reasons.
> > This depends on the implementation. All of the
> > implementations I know are thread safe. That is, they
> > define exactly what guarantees they give with regards to
> > thread safety, and give at least the guarantees that Posix
> > gives for a basic type. (The only exception I know of here
> > is std::string in g++. But g++ isn't really very usable in
> > a multithreaded environment anyway.)
> Well, it depends on the definition of thread safety. IMO, OP
> meant if it is safe to call container's member functions
> simultaneously without locking no matter whether these
> functions change to container or not.
But that definition is neither reasonable nor usable.
I agree that there could be some problems with what the original
poster understands by "thread safety". But if he's got an
unreasable definition, then you have to correct his definition.
The link you posted doesn't say that the SGI implementation of
the STL isn't thread-safe; it defines the contract the
implementation gives with regards to thread-safety (which is
basically the same contract as for int). The presence of such a
contract means that the implementation IS thread-safe, as long
as you, the user, fulfill your end of the contract.
--
James Kanze GABI Software
Conseils en informatique orientée objet/
Beratung in objektorientierter Datenverarbeitung
9 place Sémard, 78210 St.-Cyr-l'École, France, +33 (0)1 30 23 00 34
Dylan Nicholson
> Maxim Yegorushkin wrote:
>
> > Well, it depends on the definition of thread safety. IMO, OP
> > meant if it is safe to call container's member functions
> > simultaneously without locking no matter whether these
> > functions change to container or not.
>
> But that definition is neither reasonable nor usable.
>
has to be done, the question is whether it has to be done explicitly
by the user, or is handled automatically by the container. In most
cases, given current technology, it's generally better to leave the
user to do the locking, but
certain types of containers, especially message queues (where one
thread adds items and another removes items) can benefit from handling
their own locking (although not the way std::deque() works - where
there is no way of simultaneous removing & accessing the last the
element).
James Kanze
> My reading of the original question was that he understood
> there to be problems with concurrent operations on different
> instances of vectors (i.e. it's not safe to expand two vectors
> simultaneously). Do problems of this kind exist in the
> implementations that you are aware of?
> "James Kanze" <ka...@none.news.free.fr> wrote in message
> news:4234b288$0$13914$626a...@news.free.fr...
>>(The only exception I know of here is std::string in g++. But
>>g++ isn't really very usable in a multithreaded environment
>>anyway.)
> Could you either explain that further or direct me to somewhere that
> does?
I suppose it depends on the platform, but under Solaris, there
are at least two problems:
-- std::string doesn't conform to what Posix requires for basic
types. I guess you could say that this isn't an error --
Posix makes no requirements concerning std::string, but it
is certainly not what the user would expect, nor is it very
convenient when you are trying to replace char[] by
std::string.
-- Solaris says that pthread_cancel calls destructors. (Posix
hints very strongly that it should.) It doesn't with g++.
There are likely more issues. For the moment, those are
sufficient to put any multi-threaded developments with g++ on
hold.
--
James Kanze home: www.gabi-soft.fr
Conseils en informatique orientée objet/
Beratung in objektorientierter Datenverarbeitung
9 pl. Pierre Sémard, 78210 St.-Cyr-l'École, France +33 (0)1 30 23 00 34
[ See http://www.gotw.ca/resources/clcm.htm for info about ]
Maxim Yegorushkin
[]
> ... In my case, under Solaris, I have experienced problems
> with std::string and with the Posix function pthread_cancel --
> in both cases, one could argue that it is just a case of g++
> giving different guarantees, but the g++ guarantee for
> std::string is contrary to what Posix guarantees (e.g. for
> char[])
Could you please provide a link to the Posix guarantees. I failed to
google it.
--
Maxim Yegorushkin
James Kanze
> James Kanze <ka...@none.news.free.fr> writes:
>> -- Solaris says that pthread_cancel calls destructors. (Posix
>> hints very strongly that it should.) It doesn't with g++.
> Interesting, I guess it is reasonable to assume this applies
> to g++ on other platforms as well. Does anybody know if this
> is the case, or whether there are reasons such an assumtion is
> foolish?
> My interests are mainly in Linux as this is the only platform
> on which we use g++.
I don't really know. pthread_cancel is a very particular
function; to implement it correctly needs some sort of compiler
magic, at least in C++.
A quick check on my Linux machine revealed some surprising
behavior concerning pthread_cancel IF the thread terminates
without encountering a cancelation point, but under normal
circomstances, it seems to work; destructors are called.
But that's just a quick check; in all cases (and not just with
g++), I'd recommend a thorough set of very rigorous tests.
--
James Kanze mailto: james...@free.fr
James Kanze
> <ka...@gabi-soft.fr> wrote:
> []
>>... In my case, under Solaris, I have experienced problems
>>with std::string and with the Posix function pthread_cancel --
>>in both cases, one could argue that it is just a case of g++
>>giving different guarantees, but the g++ guarantee for
>>std::string is contrary to what Posix guarantees (e.g. for
>>char[])
> Could you please provide a link to the Posix guarantees. I failed to
> google it.
http://www.opengroup.org/onlinepubs/000095399/basedefs/xbd_chap04.html#tag_04_10
Particularly the final sentence: "Applications may allow more
than one thread of control to read a memory location
simultaneously."
I'm not surprised you had difficulty finding it with Google;
it's not easy to find unless you know exactly where to look, and
even then, I usually have to make several stabs at it before
finding the exact page. (And finding the exact words to search
for isn't obvious.)
Butenhof discusses the issues more in detail in §3.4; in
particular: "As the rules state, there are specific cases where
you do not need to use a mutex to ensure visibility. If one
thread sets a global variable, and then creates a new thread
that reads the same variable, you know that the new thread will
not see the old value." This guarantee doesn't hold for the g++
implementation of std::string.
Taken literally, of course, the Posix guarantees only concern
basic types, and what Butenhof is discussing is basic types,
because that is all you can "access" in C. Logically, however,
things like std::string or std::vector should behave like basic
types (or arrays of basic types); every implementation of
std::vector I know of does, and g++'s implementation of
std::string seems to be the only exception for std::string.
(But don't consider my statement above a guarantee. Obviously,
I've not used every implementation of these classes.) More
generally, I would consider this a good standard for every value
oriented class, and for every container type.
--
James Kanze mailto: james...@free.fr
Conseils en informatique orientée objet/
Beratung in objektorientierter Datenverarbeitung
9 pl. Pierre Sémard, 78210 St.-Cyr-l'École, France +33 (0)1 30 23 00 34
Maxim Yegorushkin
[]
> Butenhof discusses the issues more in detail in §3.4; in
> particular: "As the rules state, there are specific cases where
> you do not need to use a mutex to ensure visibility. If one
> thread sets a global variable, and then creates a new thread
> that reads the same variable, you know that the new thread will
> not see the old value." This guarantee doesn't hold for the g++
> implementation of std::string.
Could you please elaborate on this or provide a link if it has been
discussed already elsewhere. Creating a posix thread constitutes a memory
barrier, so I wonder how std::string can possibly break it.
--
Maxim Yegorushkin
James Kanze
> On 27 Mar 2005 05:20:25 -0500, James Kanze <ka...@none.news.free.fr>
wrote:
> []
>>Butenhof discusses the issues more in detail in §3.4; in
>>particular: "As the rules state, there are specific cases
>>where you do not need to use a mutex to ensure visibility. If
>>one thread sets a global variable, and then creates a new
>>thread that reads the same variable, you know that the new
>>thread will not see the old value." This guarantee doesn't
>>hold for the g++ implementation of std::string.
> Could you please elaborate on this or provide a link if it has
> been discussed already elsewhere. Creating a posix thread
> constitutes a memory barrier, so I wonder how std::string can
> possibly break it.
It's the classical problem where what you don't see can hurt
you. Internally, the g++ implementation modifies things without
it being visible to you, and without taking the necessary
locks. So even if you don't modify anything, just looking at an
std::string (with the g++ implementation) can cause undefined
behavior.
Not all functions are affected. I memory serves me correctly,
the only problems occur in operator[] and at(), and then only on
the first invocation, and if there is no copy of the the string
in existance elsewhere.
--
James Kanze mailto: james...@free.fr
Conseils en informatique orientée objet/
Beratung in objektorientierter Datenverarbeitung
9 pl. Pierre Sémard, 78210 St.-Cyr-l'École, France +33 (0)1 30 23 00 34
[ See http://www.gotw.ca/resources/clcm.htm for info about ]
wa...@stoner.com
James Kanze wrote:
>>> [g++ std::string is particularly MT unfriendly]
> Not all functions are affected. I memory serves me correctly,
> the only problems occur in operator[] and at(), and then only on
> the first invocation, and if there is no copy of the the string
> in existance elsewhere.
Those functions, and a few others (begin(), end(), ...) have the
property that people think of them as const, even when they are calling
the non-const version. If you put a lock around your use of these,
then you're probably ok.
// non-const op[] modifies the string even when it looks like
// it is just a read access.
c = my_str[0];
Note that your lock needs to prevent even true-read-only access from
other threads. The fact that it is so easy to forget the lock, makes
this kind of string pretty much unusable, IMO, in an MT environment.
Also note that both (g++)
string::string(const string&);
string& string::operator=(const string&);
may actually mutate their const argument. It may be that the mutation
is properly locked in the implementation, I don't know.
James Kanze
> James Kanze wrote:
>>>>[g++ std::string is particularly MT unfriendly]
>>Not all functions are affected. I memory serves me correctly,
>>the only problems occur in operator[] and at(), and then only
>>on the first invocation, and if there is no copy of the the
>>string in existance elsewhere.
> Those functions, and a few others (begin(), end(), ...) have
> the property that people think of them as const, even when
> they are calling the non-const version.
It's not a question of whether the object is const or not. The
question is whether I modify the object or not. According to
Posix, if I don't modify the object in any thread, I don't need
the lock.
> If you put a lock around your use of these, then you're
> probably ok.
Obviously.
> // non-const op[] modifies the string even when it looks like
> // it is just a read access.
> c = my_str[0];
> Note that your lock needs to prevent even true-read-only
> access from other threads. The fact that it is so easy to
> forget the lock, makes this kind of string pretty much
> unusable, IMO, in an MT environment.
What makes it "unusable" is the fact that it doesn't conform to
the standard conventions of the system in question.
> Also note that both (g++)
> string::string(const string&);
> string& string::operator=(const string&);
> may actually mutate their const argument. It may be that the
> mutation is properly locked in the implementation, I don't
> know.
The implementation uses a non-locking algorithm. It works (as
far as I have been able to tell) for assignment and copy, as
long as operator[] and at() (and maybe begin() and end()) are
not used. If I remember correctly, the error is in the
functions at() and [], but it is only triggered if another
thread tries to make a copy at the same time. And only on the
first call to one of the these functions.
--
James Kanze mailto: james...@free.fr
Conseils en informatique orientée objet/
Beratung in objektorientierter Datenverarbeitung
9 pl. Pierre Sémard, 78210 St.-Cyr-l'École, France +33 (0)1 30 23 00 34
[ See http://www.gotw.ca/resources/clcm.htm for info about ]
wa...@stoner.com
James Kanze wrote:
>
> It's not a question of whether the object is const or not. The
> question is whether I modify the object or not. According to
> Posix, if I don't modify the object in any thread, I don't need
> the lock.
A convention in C++ is that calls to non-const member functions may
modify objects. D&E: "... const ... allows ... distinction between
functions that modify the state of an object and functions that don't
... ." By that convention (combined with the posix convention that
modifications need synchronization) any call to non-const begin(),
op[], ... needs synchronization, whether or not you later use the
result of the call to make further modifications.
The C++ standard is even more explicit. I'll assert without proof that
"references into a container are invalidated" implies "the container is
modified". C++98 says "[a basic_string is modified by] the first call
to non-const ... op[]". The posix convention is that modifications
require synchronization, and C++ says you modified the object.
Now you and I both "know" that non-const std::vector::op[] doesn't
modify the vector (although it could in a conforming implementation),
and so we don't synchronize
x = v[i];
In that regard, std::string::op[] clearly violates our "convention." I
think we agree that for all practical purposes, passing raw COW
std::string across threads is extremely dangerous.
ka...@gabi-soft.fr
> James Kanze wrote:
> > It's not a question of whether the object is const or not.
> > The question is whether I modify the object or not.
> > According to Posix, if I don't modify the object in any
> > thread, I don't need the lock.
> A convention in C++ is that calls to non-const member
> functions may modify objects.
It is a rule of the language that the choice between a const and
a non-const member depends uniquely on the type (const or not)
of the object, and NOT on whether I intend to modify the object
or not. It is an almost universal convention in C++ that
operator[] (const or not) does NOT modify the object, logically
at least -- for it to do so would be a blatant case of operator
abuse. The distinction between const and non-const here is
simply that the non-const version returns a type which allows
you to modify the object; a const version doesn't.
> D&E: "... const ... allows ... distinction between functions
> that modify the state of an object and functions that don't
> ... ."
I'd have to see that in context, but I doubt that it is the full
story. I might add that D&E explains the history of the
language. I have no doubt that this is the reason why const was
introduced. I don't think it explains all the issues concerning
the established conventions of const today. (I don't think that
that is why the book was written, either. The title is
"Development and Evolution", not "Preferred modern programming
practice".
At any rate, by this logic, you only need on operator[] for
vector, the const one.
> By that convention (combined with the posix convention that
> modifications need synchronization) any call to non-const
> begin(), op[], ... needs synchronization, whether or not you
> later use the result of the call to make further
> modifications.
The problem is that that convention isn't used anywhere, and
hasn't been for years. (When I first started C++, there was
some discussion on the relative values of bit-wise vs. logical
const. Today, there seems to be a concensus for logical const.)
The whole point in overloading operator[] is to make the object
behave like a built-in array, at least with regards to this
operator. If this isn't the case in a multithreaded
environment, either you've abused operator overloading, or
you've violated the multithreaded contract.
> The C++ standard is even more explicit.
The C++ standard doesn't say anything about threading issues.
> I'll assert without proof that "references into a container
> are invalidated" implies "the container is modified". C++98
> says "[a basic_string is modified by] the first call to
> non-const ... op[]". The posix convention is that
> modifications require synchronization, and C++ says you
> modified the object.
For what definition of modified. The standard doesn't allow the
value of the string to change. The standard also allows
modification of internal state at any time, regardless.
>From an implementation point of view, any function (even const)
can modify internal state of the container. From a user point
of view, about the only definition of "modify" which makes sense
is one which may result in a visible change of value -- the
results of a == operator may change, for example.
> Now you and I both "know" that non-const std::vector::op[]
> doesn't modify the vector (although it could in a conforming
> implementation), and so we don't synchronize
> x = v[i];
Actually, I don't know that. I can easily imagine a "profiling"
implementation, which keeps count of the accesses.
> In that regard, std::string::op[] clearly violates our
> "convention." I think we agree that for all practical
> purposes, passing raw COW std::string across threads is
> extremely dangerous.
It works with every implementation I know of but that of g++.
Actually, passing g++ strings between threads works too.
Curiously enough, as long as there is more than one shared copy,
there is no problem. The only problem I know of occurs when one
thread makes the first copy at the same moment that another
thread makes the first indexation. And the real problem is the
poorly designed interface to std::string. But it's all we got,
so we gotta live with it.
(Note that this isn't the only problem with the g++
implementation. Something like std::basic_string<double> will
lead to a core dump right away on my machine. To be perfectly
frank, however, that's one error I really don't care about.
where as the threading issue... It's hard to convince people to
replace char[] with std::string when std::string doesn't work.)
--
James Kanze GABI Software
Conseils en informatique orientée objet/
Beratung in objektorientierter Datenverarbeitung
9 place Sémard, 78210 St.-Cyr-l'École, France, +33 (0)1 30 23 00 34
wa...@stoner.com
ka...@gabi-soft.fr wrote:
> > I'll assert without proof that "references into a container
> > are invalidated" implies "the container is modified". C++98
> > says "[a basic_string is modified by] the first call to
> > non-const ... op[]". The posix convention is that
> > modifications require synchronization, and C++ says you
> > modified the object.
>
> For what definition of modified. The standard doesn't allow the
> value of the string to change.
vector::reserve() doesn't allow the value (op==) of the vector to
change. It does potentially change the state of the vector and I'd
expect to need external synchronization in an MT environment.
The standard doesn't say it in so many words, but it is clearly the
intention that a call to nonconst string::op[] potentially changes the
state of the string from "shared" to "unshared and unsharable", and the
implications are very similar to vector::reserve, existing references
are invalidated. I don't see how the call is any less of a
modification than a call to vector::reserve().
> > I think we agree that for all practical
> > purposes, passing raw COW std::string across threads is
> > extremely dangerous.
>
> It works with every implementation I know of but that of g++.
I don't see how it can work in a COW implementation with dumb
references. Other lazy strategies are possible (copy on any reference,
rather than just copy on non-const reference) but I wouldn't call that
COW. Can you point me at a COW std::string implementation where nc
std::string::op[] never invalidates earlier (other thread) const
references/iterators/... ?
> the real problem is the
> poorly designed interface to std::string.
Yes sir.
ka...@gabi-soft.fr
> ka...@gabi-soft.fr wrote:
> > > I'll assert without proof that "references into a
> > > container are invalidated" implies "the container is
> > > modified". C++98 says "[a basic_string is modified by] the
> > > first call to non-const ... op[]". The posix convention is
> > > that modifications require synchronization, and C++ says
> > > you modified the object.
> > For what definition of modified. The standard doesn't allow
> > the value of the string to change.
> vector::reserve() doesn't allow the value (op==) of the vector
> to change.
It does allow the external state (e.g. vect.capacity()) to
change. The change is visible to the user.
The valid analogy would be with the [] operator of vector and
deque. And these don't require external synchronization, as
long as I don't actually modify any of the contents.
> It does potentially change the state of the vector and I'd
> expect to need external synchronization in an MT environment.
> The standard doesn't say it in so many words, but it is
> clearly the intention that a call to nonconst string::op[]
> potentially changes the state of the string from "shared" to
> "unshared and unsharable", and the implications are very
> similar to vector::reserve, existing references are
> invalidated. I don't see how the call is any less of a
> modification than a call to vector::reserve().
There is no visible change. The standard doesn't require a
change in this case. There is a visible change with
vector::reserve.
I consciously choose to call vector::reserve, because I want
its side effects. I don't choose whether the const or the
non-const version of operator[] is called; the compiler
chooses. And the compiler's critera have nothing to do with
what I want to do with the results.
Operator[] is supported by the built in types: char[], and more
generally T[]. If I overload the operator, it should have the
same semantics as the built-in operator. I can use [] on a C
style array without locking, as long as *I* don't modify the
value.
Note that I can use operator[] on a non-const vector or deque
without problems.
> > > I think we agree that for all practical purposes, passing
> > > raw COW std::string across threads is extremely dangerous.
> > It works with every implementation I know of but that of
> > g++.
> I don't see how it can work in a COW implementation with dumb
> references. Other lazy strategies are possible (copy on any
> reference, rather than just copy on non-const reference) but I
> wouldn't call that COW. Can you point me at a COW std::string
> implementation where nc std::string::op[] never invalidates
> earlier (other thread) const references/iterators/... ?
Maybe that's why other implementations have stopped using
COW:-).
--
James Kanze GABI Software
Conseils en informatique orientée objet/
Beratung in objektorientierter Datenverarbeitung
9 place Sémard, 78210 St.-Cyr-l'École, France, +33 (0)1 30 23 00 34