[Boost-users] Mersenne twister problem
ruya
Hello
I've built a meta-heuristics library in which I use the boost random
library. I am using several independent generators through the application.
The problem is in the core of my library. The tricky part is that I have
three different algorithms using the core, all of them using the class where
a segmentation fault happens, but only one of them crashes when it tries to
use the class, the others run as expected. Gdb points to the boost library,
as you can see in the snippet bellow. I couldn't find any information on
this issue, and I cannot see something wrong with my code (the relevant part
is bellow).
The parent class where generators are declared:
template< class SType, typename MemP >
class SearchFunctor{
(...)
public:
virtual SPtr operator()( SPtr s, int iter ) = 0;
SearchFunctor(){
(...)
//This is how I initialize teh generators, being MASTER SEED 666
generator1 = base_generator_type(MASTER_SEED*13);
uni_dist1 = boost::uniform_real<>(0,1);
uniEmp = new GeneratorI(generator1, uni_dist1);
generator2 = base_generator_type(MASTER_SEED*17);
uni_dist2 = boost::uniform_real<>(0,1);
uniRes = new GeneratorI(generator2, uni_dist2);
}
protected:
base_generator_type generator1, generator2;
boost::uniform_real<> uni_dist1, uni_dist2;
GeneratorI* uniEmp;
GeneratorI* uniRes;
};
The derived class using the generators:
template< class SType, typename MemP >
struct SwapSearch : public SearchFunctor<SType,MemP>{
typedef SearchFunctor<SType,MemP> Parent;
(...)
SPtr operator()( SPtr s, int iter ){
assert(s != NULL);
assert(s->GetDomain()!= NULL);
assert(s->GetDomain()->GetTotalEntities()>1);
(...)
d1 = (*Parent::uniEmp)() * (s->GetDomain()->GetTotalEntities()-1);
d2 = (*Parent::uniEmp)() * (s->GetDomain()->GetTotalEntities()-1);
(...)
return 0;
}
};
The output of the debugger has a lot of lines because of meta-programming
but starts and finishes with this:
#0 0x08071a57 in brickheuristics::SwapSearch (...)
(...) at /usr/include/boost/random/mersenne_twister.hpp:279
279 UIntType z = x[i];
Thanks
--
View this message in context: http://www.nabble.com/Mersenne-twister-problem-tp25382114p25382114.html
Sent from the Boost - Users mailing list archive at Nabble.com.
_______________________________________________
Boost-users mailing list
Boost...@lists.boost.org
http://lists.boost.org/mailman/listinfo.cgi/boost-users
Matthias Troyer
I would write this very differently, and can give you a better version
that should have no problems, but could you please send me the
definition of GeneratorI first?
Matthias
ruya
Hi
Matthias Troyer-2 wrote:
>
> I would write this very differently, and can give you a better version
> that should have no problems, but could you please send me the
> definition of GeneratorI first?
>
Sorry about that, the missing typedefs at SearchFunctor:
typedef boost::mt19937 base_generator_type;
typedef boost::variate_generator<base_generator_type&,
boost::uniform_real<> > GeneratorI;
--
View this message in context: http://www.nabble.com/Mersenne-twister-problem-tp25382114p25432292.html
Sent from the Boost - Users mailing list archive at Nabble.com.
_______________________________________________
Matthias Troyer
On Sep 14, 2009, at 9:40 AM, ruya wrote:
>
> Hi
>
>
> Matthias Troyer-2 wrote:
>>
>> I would write this very differently, and can give you a better
>> version
>> that should have no problems, but could you please send me the
>> definition of GeneratorI first?
>>
> Sorry about that, the missing typedefs at SearchFunctor:
>
> typedef boost::mt19937 base_generator_type;
> typedef boost::variate_generator<base_generator_type&,
> boost::uniform_real<> > GeneratorI;
Another question: what does the destructor of your class look like?
Can you send the full class?
Matthias
ruya
SearchFunctor has a virtual destructor, the derivees have empty destructor.
MemHandler is just a helper does not store anything.
class SearchFunctor{
protected:
typedef SmartPtr<SType> SPtr;
typedef typename SType::Domain Domain;
typedef typename SType::DPtr DPtr;
typedef typename Domain::MType MType;
typedef typename Domain::MPtr MPtr;
typedef typename Domain::EPtr EPtr;
typedef typename Domain::RPtr RPtr;
typedef vector<typename SType::MPtr> MVector;
typedef typename SType::DataType DataType;
typedef boost::mt19937 base_generator_type;
typedef boost::variate_generator<base_generator_type&,
boost::uniform_real<> > GeneratorI;
typedef Private<MemP::ObjectType::_ACTIVE,MemP> MemHandler;
public:
virtual SPtr operator()( SPtr s, int iter ) = 0;
SearchFunctor(){
memh = MemHandler();
generator1 = base_generator_type(MASTER_SEED+13);
uni_dist1 = boost::uniform_real<>(0,1);
uniEmp = new GeneratorI(generator1, uni_dist1);
generator2 = base_generator_type(MASTER_SEED+17);
uni_dist2 = boost::uniform_real<>(0,1);
uniRes = new GeneratorI(generator2, uni_dist2);
}
protected:
virtual ~SearchFunctor(){}
base_generator_type generator1, generator2;
boost::uniform_real<> uni_dist1, uni_dist2;
GeneratorI* uniEmp;
GeneratorI* uniRes;
MemHandler memh;
};
--
View this message in context: http://www.nabble.com/Mersenne-twister-problem-tp25382114p25450374.html
Sent from the Boost - Users mailing list archive at Nabble.com.
_______________________________________________
Matthias Troyer
On Sep 15, 2009, at 10:50 AM, ruya wrote:
>
> SearchFunctor has a virtual destructor, the derivees have empty
> destructor.
> MemHandler is just a helper does not store anything.
Here is my guess as to what happens: SPtr points to a SearchFunctor
that has been deallocated. The generators were never deleted and can
still be used, but the underlying engine got destroyed. Your code
contains memory leaks, and could be simplified a lot.
Matthias
ruya
SType is not a SearchFunctor. It is a template parameter of SearchFunctor.
Look, I am not a professional coder or something. I could accept that my
code has memory leaks, except that I monitored the memory usage and it is
stable and quite small too. I have three other algorithms using the same
SearchFunctors and the behavior is as expected and no memory leaks.
As for the complication that my code appears to be to you, how would you
simplify it? You argued before that you would do it diferently and avoid for
sure the segmentation fault.
Rui
Matthias Troyer-2 wrote:
>
>
> On Sep 15, 2009, at 10:50 AM, ruya wrote:
>
>>
>> SearchFunctor has a virtual destructor, the derivees have empty
>> destructor.
>> MemHandler is just a helper does not store anything.
>
> Here is my guess as to what happens: SPtr points to a SearchFunctor
> that has been deallocated. The generators were never deleted and can
> still be used, but the underlying engine got destroyed. Your code
> contains memory leaks, and could be simplified a lot.
>
> Matthias
>
>
>>
>> template< class SType, typename MemP >
--
View this message in context: http://www.nabble.com/Mersenne-twister-problem-tp25382114p25468208.html
Matthias Troyer
On 16 Sep 2009, at 10:38, ruya wrote:
>
> SType is not a SearchFunctor. It is a template parameter of
> SearchFunctor.
> Look, I am not a professional coder or something. I could accept
> that my
> code has memory leaks, except that I monitored the memory usage and
> it is
> stable and quite small too. I have three other algorithms using the
> same
> SearchFunctors and the behavior is as expected and no memory leaks.
> As for the complication that my code appears to be to you, how would
> you
> simplify it? You argued before that you would do it diferently and
> avoid for
> sure the segmentation fault.
>>>
>>> SearchFunctor(){
>>> memh = MemHandler();
>>> generator1 = base_generator_type(MASTER_SEED+13);
>>> uni_dist1 = boost::uniform_real<>(0,1);
>>> uniEmp = new GeneratorI(generator1, uni_dist1);
>>> generator2 = base_generator_type(MASTER_SEED+17);
>>> uni_dist2 = boost::uniform_real<>(0,1);
>>> uniRes = new GeneratorI(generator2, uni_dist2);
>>> }
>>> protected:
>>> virtual ~SearchFunctor(){}
Just look at this code fragment: you allocate memory in the
constructor using new and never release it using delete
ruya
I probably don't see this leak in the memory usage because only a limited
amount of SearchFunctors are created at the beginning and they live for as
long as the program runs. (I think this makes sense)
Anyhow, writing the proper destructors doesn't solve my problem, but now gdb
points to immediatly before the creation of any of the SearchFunctors. So,
to remember, I have three SearchFunctors (SFs). In three different
applications the three of them work as expected. In this particular app, the
SFs were successfully created, then two ran well, and one would give a seg
fault. Now, with the proper destructor in class SearchFunctor, the fault
comes at the time of creation. gdb points to the line where the following
functions is called:
functors = vector<SearchF>();
TL2FunV<TList, SearchF>::FillVector( functors ); //gdb points fault here but
does not go inside
template< typename TList, class SearchFun >
struct TL2FunV{
static void FillVector( std::vector< SearchFun >& v){
typename TList::Head h;
v.push_back( SearchFun(h) );
TL2FunV<typename TList::Tail, SearchFun>::FillVector( v );
}
};
template< class SearchFun >
struct TL2FunV< Loki::NullType, SearchFun>{
static void FillVector( std::vector< SearchFun >& v ){};
};
Rui
--
View this message in context: http://www.nabble.com/Mersenne-twister-problem-tp25382114p25491944.html
Sent from the Boost - Users mailing list archive at Nabble.com.
_______________________________________________
Matthias Troyer
Since you use pointers inside your class you also need to write a copy
constructor and assignment operator creating new generators. I now
understand the reason for your crash:
- you create a SearchFunctor, this creates the engine as a member and
the generators through pointers
- you copy the SearchFunctor in the FillVector function. Using the
implicitly generated copy constructor this just copies the pointers
- after copying the original class gets destroyed - including the
random number engine.
- you call the generator, but the engine no longer exists ->
segmentation fault.
Whenever you allocate memory on a constructor you need to provide a
special destructor, copy constructor and assignment operator. Either
add those, or - much easier, just use the following class:
template< class SType, typename MemP >
class SearchFunctor{
protected:
typedef boost::mt19937 base_generator_type;
typedef boost::variate_generator<base_generator_type,
boost::uniform_real<> > GeneratorI;
public:
SearchFunctor()
: uniEmp(base_generator_type(MASTER_SEED+13),boost::uniform_real<>
(0,1))
, uniRes(base_generator_type(MASTER_SEED+17),boost::uniform_real<>
(0,1))
{}
protected:
virtual ~SearchFunctor(){}
GeneratorI uniEmp;
GeneratorI uniRes;
}
Note that I removed the & after the generator type to avoid a problem
with the copy constructor. Better yet, make the generator a static
class member and use the &:
template< class SType, typename MemP >
class SearchFunctor{
protected:
typedef boost::mt19937 base_generator_type;
typedef boost::variate_generator<base_generator_type&,
boost::uniform_real<> > GeneratorI;
public:
SearchFunctor()
: uniEmp(generator,boost::uniform_real<>(0,1))
, uniRes(generator,boost::uniform_real<>(0,1))
{}
protected:
virtual ~SearchFunctor(){}
GeneratorI uniEmp;
GeneratorI uniRes;
private:
static base_generator_type generator;
}
template< class SType, typename MemP >
typename SearchFunctor< SType, MemP>::base_generator_type
SearchFunctor< SType, MemP>::generator(MASTER_SEED);
Matthias
ruya
Hi
The problem was the missing copy constructor, which I usually write by
default but for some reason had it commented. I still don't understand why
it didn't happened with the remaining algorithms but anyway, thank you for
the help on the fault.
Now, for the code refactoring, if the generator is static all SFs will share
the same generator and the distributions will not be independent anymore,
which is a requisite in this kind of algorithm.
Regards
Rui Lopes
--
View this message in context: http://www.nabble.com/Mersenne-twister-problem-tp25382114p25505730.html
Sent from the Boost - Users mailing list archive at Nabble.com.
_______________________________________________
Matthias Troyer
On Sep 18, 2009, at 11:30 AM, ruya wrote:
>
> Hi
>
> The problem was the missing copy constructor, which I usually write by
> default but for some reason had it commented. I still don't
> understand why
> it didn't happened with the remaining algorithms but anyway, thank
> you for
> the help on the fault.
You also need to implement the assignment operator
> Now, for the code refactoring, if the generator is static all SFs
> will share
> the same generator and the distributions will not be independent
> anymore,
> which is a requisite in this kind of algorithm.
Actually, as long as the generator is good they are uncorrelated. At
least this is what is tested for. This is tested much better than
correlations between different seeds.
ruya
Matthias Troyer-2 wrote:
>
>
> You also need to implement the assignment operator
>
> Actually, as long as the generator is good they are uncorrelated. At
> least this is what is tested for. This is tested much better than
> correlations between different seeds.
>
>
But isn't the assignment equal to calling the copy constructor by default?
What I mean is: if you have obj1 = obj2 then obj1 is created by
Object(obj2).
As for the generators I was not aware of this. can you give me some link for
a technical paper on this?
Rui
--
View this message in context: http://www.nabble.com/Mersenne-twister-problem-tp25382114p25510845.html
Sent from the Boost - Users mailing list archive at Nabble.com.
_______________________________________________
Matthias Troyer
On 18 Sep 2009, at 17:35, ruya wrote:
>
>
> Matthias Troyer-2 wrote:
>>
>>
>> You also need to implement the assignment operator
>>
>> Actually, as long as the generator is good they are uncorrelated. At
>> least this is what is tested for. This is tested much better than
>> correlations between different seeds.
>>
>>
>
> But isn't the assignment equal to calling the copy constructor by
> default?
> What I mean is: if you have obj1 = obj2 then obj1 is created by
> Object(obj2).
If you have:
Object obj1=obj2;
then indeed the copy constructor is used, but if you have:
Obj obj1;
obj1=obj2;
then the assignment constructor is called. The default is again a
member-wise copy which is wrong in your example.
> As for the generators I was not aware of this. can you give me some
> link for
> a technical paper on this?
All test suites test correlations within a sequence, but tests between
different sequences with different seeds are rare. For lagged
Fibonacci generators there are proofs that uncorrelated seed blocks
can be generated, but not for other generators. Especially your using
MASTER_SEED*13 and MASTER_SEED*17 is completely untested (and in the
case MASTER_SEED=0 it would give identical results!)
Matthias