memkind c++ allocator and the Eigen library

[adrianj.jackson]

Hi,

I want to use the memkind libmemkind::pmem::allocator to map allocation of Eigen matrices from the Eigen library (https://eigen.tuxfamily.org/) on to NVRAM. However, the Eigen matrices are their own types, i.e. Eigen::Matrix<double, Eigen::Dynamic, 1>, with their own allocators, and I can't find any way of getting the memkind allocator to work here.

What I've tried to do is use the memkind allocator on std::vector of Eigen matrices so the allocation override happens at that level, i.e.

template <class T = t_real> using Vector = Eigen::Matrix<T, Eigen::Dynamic, 1>;

typedef Vector<double> vector_double
typedef libmemkind::pmem::allocator<vector_double> vector_double_alloc_type;
typedef std::vector<vector_double, std::scoped_allocator_adaptor<vector_double_alloc_type>> nested_double_vector;

However, this fails at compile time with errors like this:

error: no instance of constructor "libmemkind::pmem::allocator<T>::allocator [with T= vector_double]" matches the argument list

Is this because the base allocator in the scoped series needs to be on a primitive type?

Is there any way to get a class type like the Eigen::Matrix to allocate on NVRAM using the pmem allocator?

thanks

adrianj

[adrianj.jackson]

Hi,

Ok, scratch that question, the issue was I was using one of the types without passing in an allocator at initialisation time, hence the error. I can now get the code to compile and run. However, the allocators never actually get the data on to the NVRAM, it always stays in DRAM.

For nested vectors of primitive types I can get the scoped allocators to work, i.e. the data ends up on NVRAM, but for these more complex types I can't get the allocator to do the job. Any pointers on whether this is possible?

cheers

adrianj

[Jan K]

You know, you already answered your question yourself:

> However, the Eigen matrices are their own types, i.e.
> Eigen::Matrix<double, Eigen::Dynamic, 1>, with their
> own allocators, and I can't find any way of getting
> the memkind allocator to work here.

Just open this file:
https://gitlab.com/libeigen/eigen/-/blob/master/Eigen/src/Core/util/Memory.h
and replace all mallocs and reallocs.


But, there's another question that may be interesting:
can one LD_PRELOAD a custom .so library that replaces
malloc&co with memkind allocations to pmem?

Regards,
Jan

> --
> You received this message because you are subscribed to the Google Groups
> "pmem" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to pmem+uns...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/pmem/d234adda-79c4-4986-a077-9a0f39ec94d4n%40googlegroups.com.
>

[adrianj.jackson]

Hi,

Thanks, however, I want to selectively choose which matrices to put into NVRAM. I don't want all the Eigen matrices to go into NVRAM because that would be bad for performance. Hence the approach using the allocator, so I can provide an allocator for those matrices I want in persistent memory, and not for those I want in DRAM. Likewise, I could do an LD_PRELOAD but that would affect all the allocations in my program, not just the ones I want to put in the other memory space.

thanks

adrianj

[Michal Biesek]

Hi Adrian,

Could you please open an issue with details (possibly reproduction steps) on https://github.com/memkind/memkind/issues? Therefore, would be easier to analyze your issue with pmem::allocator.

Best Regards

[Igor Chorążewicz]

Hi Adrian,

If I understand the code correctly, the Eigen::Matrix class does not support Allocator interface (there is not Allocator template parameter), is that right? If so, using std::allocator_adaptor_type won't help - it can only be used for containers which can be parametrized by an Allocator.

You can see an example here: https://github.com/memkind/memkind/blob/master/examples/pmem_cpp_allocator.cpp

Best regards,
Igor

[adrianj.jackson]

Hi Igor,

That's my understanding as well, but I'm putting the Eigen::Matrix objects into a std::vector that does support the allocator interface. What happens in this scenario? The std::vector should be allocating in persistent memory, and the Eigen::Matrix objects are being copied into that vector. I'd hoped that would mean the Eigen::Matrix would end up in persistent memory, but does it just mean the std::vector internal data ends up in the persistent memory but the object data (the Eigen::Matrix objects themselves) stay in volatile memory? It would seem to be a strange copy into the vector if that's the case, but maybe the std::vector invokes the Eigen::Matrix allocator when doing the copy?

cheers

adrianj

[Jan K]

> I'd hoped that would mean the Eigen::Matrix would end up in
> persistent memory, but does it just mean the std::vector internal
> data ends up in the persistent memory but the object data (the
> Eigen::Matrix objects themselves) stay in volatile memory? It
> would seem to be a strange copy into the vector if that's the
> case, but maybe the std::vector invokes the Eigen::Matrix
> allocator when doing the copy?

All real data are not stored in the Matirx class itself, or else
you'd never be able to allocate one on stack - there's a limit on
stack memory.
The data of a Matrix are stored in a malloc-allocated memory that
is merely pointed to by some field of the class Matrix.

Your Eigen::Matrix object (= its fields and vtable) does end up in
persistent memory. The memory explicitly allocated by member
functions (or constructors) of this object using a malloc() stay
in DRAM.

There's nothing you can do apart form rewriting Eigen library.

Regards,
Jan

> https://groups.google.com/d/msgid/pmem/96163572-c2b4-4897-a0c7-bb493f22d967n%40googlegroups.com.
>