Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss
Allocating dynamically two large memory blocks next to each other.
108 views
Skip to first unread message
pkon...@gmail.com
Oct 17, 2014, 9:56:45 PM10/17/14
to
Hi folks.
Is there a way of allocating two large memory blocks next to each other using 'new', 'malloc' or any other allocation strategy? The trick is that I want to use these two blocks as a single memory space addressed by a pointer, though, later I'll perform a two-fold in-place compression on such a vector and I really have to release the upper half of the allocated memory. We're talking here about more than 100GB assembled from 2x50GB continuous memory blocks.
Thanks in advance,
Pete
Is there a way of allocating two large memory blocks next to each other using 'new', 'malloc' or any other allocation strategy? The trick is that I want to use these two blocks as a single memory space addressed by a pointer, though, later I'll perform a two-fold in-place compression on such a vector and I really have to release the upper half of the allocated memory. We're talking here about more than 100GB assembled from 2x50GB continuous memory blocks.
Thanks in advance,
Pete
Message has been deleted
Robert Wessel
Oct 18, 2014, 12:11:09 AM10/18/14
to
On 18 Oct 2014 02:05:40 GMT, r...@zedat.fu-berlin.de (Stefan Ram)
wrote:
>pkon...@gmail.com writes:
>>Is there a way of allocating two large memory blocks next to
>>each other using 'new', 'malloc' or any other allocation
>>strategy?
>
> Round up both block sizes to account for all alignment
> requirements, then
>
>thetwoblockscanbefoundat = malloc( sizeoffirst + sizeoofsecond ).
>sizeoffirst );
>
> C++ might not guarantee this, but your implementation might
> handle it this way.
Unfortunately realloc() is not required to shrink allocations in
place. It may allocate the new smaller area, copy over the data that
will fit, deallocate the original area, and return the pointer to the
new area.
If the application is depending on addresses in the allocated block,
this will likely break them.
OTOH, most implementations do (usually) shorten blocks in-place, but
they are not required to. And it would not surprise me at all that
they do not in all cases - for example, many allocators go to the OS
for very large allocations, if you realloc'd a multi-megabyte
allocation down to a few bytes, I could easily see that allocation
being moved into the "normal" heap.
That being said, there may be implementation dependent ways of doing
that. For example, in Windows you could VirtualAlloc() a region (this
has page granularity, but that would seem to not be an issue for the
OPs problem), and the VirtualFree() specific pages from with that
allocation later. Alternatively, the OS Heap functions might be
enough, HeapRealloc() allows you to specify
HEAP_REALLOC_IN_PLACE_ONLY, which will cause the function to fail if
it cannot do the reallocation in-place (which would seem to be a
reasonable fallback for the OP's problem - things continue, just the
shrink didn't occur).
wrote:
>pkon...@gmail.com writes:
>>Is there a way of allocating two large memory blocks next to
>>each other using 'new', 'malloc' or any other allocation
>>strategy?
>
> requirements, then
>
>thetwoblockscanbefoundat = malloc( sizeoffirst + sizeoofsecond ).
>
>>The trick is that I want to use these two blocks as a single
>>memory space addressed by a pointer, though, later I'll
>>perform a two-fold in-place compression on such a vector and
>>I really have to release the upper half of the allocated
>>memory.
>
>thelowblockcouldstillbeat = realloc( thetwoblockscanbefoundat,
>>The trick is that I want to use these two blocks as a single
>>memory space addressed by a pointer, though, later I'll
>>perform a two-fold in-place compression on such a vector and
>>I really have to release the upper half of the allocated
>>memory.
>
>sizeoffirst );
>
> C++ might not guarantee this, but your implementation might
> handle it this way.
Unfortunately realloc() is not required to shrink allocations in
place. It may allocate the new smaller area, copy over the data that
will fit, deallocate the original area, and return the pointer to the
new area.
If the application is depending on addresses in the allocated block,
this will likely break them.
OTOH, most implementations do (usually) shorten blocks in-place, but
they are not required to. And it would not surprise me at all that
they do not in all cases - for example, many allocators go to the OS
for very large allocations, if you realloc'd a multi-megabyte
allocation down to a few bytes, I could easily see that allocation
being moved into the "normal" heap.
That being said, there may be implementation dependent ways of doing
that. For example, in Windows you could VirtualAlloc() a region (this
has page granularity, but that would seem to not be an issue for the
OPs problem), and the VirtualFree() specific pages from with that
allocation later. Alternatively, the OS Heap functions might be
enough, HeapRealloc() allows you to specify
HEAP_REALLOC_IN_PLACE_ONLY, which will cause the function to fail if
it cannot do the reallocation in-place (which would seem to be a
reasonable fallback for the OP's problem - things continue, just the
shrink didn't occur).
Öö Tiib
Oct 18, 2014, 6:09:29 AM10/18/14
to
On Saturday, 18 October 2014 04:56:45 UTC+3, pkon...@gmail.com wrote:
> Hi folks.
>
>
> Is there a way of allocating two large memory blocks next to each other
> using 'new', 'malloc' or any other allocation strategy?
> The trick is that I want to use these two blocks as a single memory
> space addressed by a pointer, though, later I'll perform a two-fold
> in-place compression on such a vector and I really have to release
> the upper half of the allocated memory.
Well, we have 'std::vector' and 'std::deque' side-by-side in language.
> Hi folks.
>
>
> Is there a way of allocating two large memory blocks next to each other
> using 'new', 'malloc' or any other allocation strategy?
> The trick is that I want to use these two blocks as a single memory
> space addressed by a pointer, though, later I'll perform a two-fold
> in-place compression on such a vector and I really have to release
> the upper half of the allocated memory.
It may be that you can solve your issue by addressing the problem in
style of 'std::deque'.
> We're talking here about more than 100GB assembled from 2x50GB
> continuous memory blocks.
such case? Does your C++ implementation use that support? Neither C
nor C++ contain guaranteed tools for that.
C++11 'std::vector' provides shrink_to_fit() member function that is
non-binding but still most likely place that uses such a feature if
it is available.
If the feature is not available or not used by 'std::allocator' (that
is default what 'std::vector' uses) then you may have to extend
the memory management of your operating system kernel to support your
desire and/or to write an allocator that uses the support.
BGB
Oct 18, 2014, 10:14:53 AM10/18/14
to
On 10/17/2014 9:05 PM, Stefan Ram wrote:
adjacent memory, is *very* unsafe and non-portable.
if applicable, possible is to just treat them as a single big block, for
example:
pBoth=malloc(szFirst+szSecond);
pFirst=pBoth;
pSecond=(void *)((char *)pBoth+szFirst); //*
*: while GCC and friends allow pointer arithmetic on 'void *', this is
an extension, and better is to cast to/from char or similar for the
arithmetic.
however, it is necessary to free both at the same time in this case.
as a practical example, this is commonly done with VBOs in OpenGL, since
generally you have to do everything in a given DrawArrays or
DrawArrayElements call with a single VBO bound (which is essentially a
memory buffer holding several parallel arrays).
or, in a particularly ugly example:
dumping most of the terrain geometry into a single massive set of
parallel arrays (generally holding around 1M vertices / 340k tris in
stats). as the player moves, the contents of these arrays are repacked
and uploaded to GL, mostly so that the terrain rendering could be cut
down to a more modest number of draw-calls.
so, say, one allocates 64MB for their temporary vertex arrays (using
mostly packed formats), fills them up, then copies some memory to
"collapse" any unused space, then sending the whole mess to GL.
> pkon...@gmail.com writes:
>> Is there a way of allocating two large memory blocks next to
>> each other using 'new', 'malloc' or any other allocation
>> strategy?
>
>> Is there a way of allocating two large memory blocks next to
>> each other using 'new', 'malloc' or any other allocation
>> strategy?
>
> Round up both block sizes to account for all alignment
> requirements, then
>
> thetwoblockscanbefoundat = malloc( sizeoffirst + sizeoofsecond ).
>
> requirements, then
>
> thetwoblockscanbefoundat = malloc( sizeoffirst + sizeoofsecond ).
>
>> The trick is that I want to use these two blocks as a single
>> memory space addressed by a pointer, though, later I'll
>> perform a two-fold in-place compression on such a vector and
>> I really have to release the upper half of the allocated
>> memory.
>
>> memory space addressed by a pointer, though, later I'll
>> perform a two-fold in-place compression on such a vector and
>> I really have to release the upper half of the allocated
>> memory.
>
> thelowblockcouldstillbeat = realloc( thetwoblockscanbefoundat,
> sizeoffirst );
>
> C++ might not guarantee this, but your implementation might
> handle it this way.
>
expecting resizing in-place, and subsequent mallocs to allocate said
> sizeoffirst );
>
> C++ might not guarantee this, but your implementation might
> handle it this way.
>
adjacent memory, is *very* unsafe and non-portable.
if applicable, possible is to just treat them as a single big block, for
example:
pBoth=malloc(szFirst+szSecond);
pFirst=pBoth;
pSecond=(void *)((char *)pBoth+szFirst); //*
*: while GCC and friends allow pointer arithmetic on 'void *', this is
an extension, and better is to cast to/from char or similar for the
arithmetic.
however, it is necessary to free both at the same time in this case.
as a practical example, this is commonly done with VBOs in OpenGL, since
generally you have to do everything in a given DrawArrays or
DrawArrayElements call with a single VBO bound (which is essentially a
memory buffer holding several parallel arrays).
or, in a particularly ugly example:
dumping most of the terrain geometry into a single massive set of
parallel arrays (generally holding around 1M vertices / 340k tris in
stats). as the player moves, the contents of these arrays are repacked
and uploaded to GL, mostly so that the terrain rendering could be cut
down to a more modest number of draw-calls.
so, say, one allocates 64MB for their temporary vertex arrays (using
mostly packed formats), fills them up, then copies some memory to
"collapse" any unused space, then sending the whole mess to GL.
pkon...@gmail.com
Oct 18, 2014, 8:17:12 PM10/18/14
to
Dear Stefan and Robert.
I don't mind if I get this behaviour on a specific platform only. Portability is not an issue at this point at all. I use linux with the following kernel.
3.11.10-100.fc18.x86_64
#1 SMP Mon Dec 2 20:28:38 UTC 2013
x86_64 x86_64 x86_64 GNU/Linux
Any chance I'd get the behaviour of realloc that Stefan talks about on this platform?
Seems like 'memory collapse' would do the job, or some way to force allocation of two memory blocks to be exactly next to each other in the first place.
Another option is to dump all data on a hard drive and reload and shrink on the fly, though, this would be too slow. The main reason for operating on such large continuous blocks is speed.
I don't mind if I get this behaviour on a specific platform only. Portability is not an issue at this point at all. I use linux with the following kernel.
3.11.10-100.fc18.x86_64
#1 SMP Mon Dec 2 20:28:38 UTC 2013
x86_64 x86_64 x86_64 GNU/Linux
Any chance I'd get the behaviour of realloc that Stefan talks about on this platform?
Seems like 'memory collapse' would do the job, or some way to force allocation of two memory blocks to be exactly next to each other in the first place.
Another option is to dump all data on a hard drive and reload and shrink on the fly, though, this would be too slow. The main reason for operating on such large continuous blocks is speed.
Melzzzzz
Oct 18, 2014, 8:59:46 PM10/18/14
to
On Sat, 18 Oct 2014 17:17:00 -0700 (PDT)
pkon...@gmail.com wrote:
> Dear Stefan and Robert.
>
> I don't mind if I get this behaviour on a specific platform only.
> Portability is not an issue at this point at all. I use linux with
> the following kernel.
>
> 3.11.10-100.fc18.x86_64
> #1 SMP Mon Dec 2 20:28:38 UTC 2013
> x86_64 x86_64 x86_64 GNU/Linux
>
> Any chance I'd get the behaviour of realloc that Stefan talks about
> on this platform?
>
> Seems like 'memory collapse' would do the job, or some way to force
> allocation of two memory blocks to be exactly next to each other in
> the first place.
Try to allocate with mmap directly.
pkon...@gmail.com wrote:
> Dear Stefan and Robert.
>
> I don't mind if I get this behaviour on a specific platform only.
> Portability is not an issue at this point at all. I use linux with
> the following kernel.
>
> 3.11.10-100.fc18.x86_64
> #1 SMP Mon Dec 2 20:28:38 UTC 2013
> x86_64 x86_64 x86_64 GNU/Linux
>
> Any chance I'd get the behaviour of realloc that Stefan talks about
> on this platform?
>
> Seems like 'memory collapse' would do the job, or some way to force
> allocation of two memory blocks to be exactly next to each other in
> the first place.
--
Manjaro all the way!
http://manjaro.org/
Paavo Helde
Oct 19, 2014, 3:34:55 AM10/19/14
to
pkon...@gmail.com wrote in
news:c9182655-7c04-4a04...@googlegroups.com:
> Dear Stefan and Robert.
>
> I don't mind if I get this behaviour on a specific platform only.
> Portability is not an issue at this point at all. I use linux with the
On Linux, use mmap() and mremap(). The latter seems like a direct hit for
your concerns.
Cheers
Paavo
news:c9182655-7c04-4a04...@googlegroups.com:
> Dear Stefan and Robert.
>
> I don't mind if I get this behaviour on a specific platform only.
> Portability is not an issue at this point at all. I use linux with the
your concerns.
Cheers
Paavo
me
Nov 24, 2014, 11:55:00 PM11/24/14
to
management your heart desires.
mad-crapper
Nov 24, 2014, 11:55:57 PM11/24/14
to
0 new messages