heap size exceeded for large matrices
John Raymond Dore
1..10000)
I have no difficulty in increasing the storage size of a task.
I cannot do the same for matrices within a procedure which reports
that the heap is exceeded.
I am using AdaCore GNAT GPL Ada and AdaCore GPS under Ubuntu.
I have 8 GB available as static memory plus a large disk space.
Help!
I surely cannot be the first person to face this problem.
I am using Ubuntu linux because it supports wireless 'out of the box'
and the SDK for AMD GPUs is supported.
I would otherwise use Debian to which I plan to migrate at a later
date.
Yannick Duchêne (Hibou57)
<john...@gmail.com> a écrit:
> I have no difficulty in increasing the storage size of a task.
> I cannot do the same for matrices within a procedure
allocated as a local object of that procedure ?
If so, the stack may have a maximum limit which cannot be exceeded, and
dynamic allocation may solve the case.
If not, an excerpt from the source could help to see.
Do you use Ada.Numerics.Generic_Real_Arrays.Real_Matrix ? (that is what I
supposed, may need to be confirmed).
--
“Dual licensing is the Perl's way to disinfect the GNU General Public
Virus!” (anonymous)
Pascal Obry
> I wish to process both real and complex large matrices eg (1..10000,
> 1..10000)
> I have no difficulty in increasing the storage size of a task.
> I cannot do the same for matrices within a procedure which reports
> that the heap is exceeded.
You probably mean stack size and not heap size. In that case you need to
add the proper linker option to increase the stack size for the application.
Pascal.
--
--|------------------------------------------------------
--| Pascal Obry Team-Ada Member
--| 45, rue Gabriel Peri - 78114 Magny Les Hameaux FRANCE
--|------------------------------------------------------
--| http://www.obry.net - http://v2p.fr.eu.org
--| "The best way to travel is by means of imagination"
--|
--| gpg --keyserver keys.gnupg.net --recv-key F949BD3B
Simon Wright
> Le Sun, 29 Aug 2010 10:50:39 +0200, John Raymond Dore
> <john...@gmail.com> a écrit:
>> I have no difficulty in increasing the storage size of a task.
>> I cannot do the same for matrices within a procedure
> Forgive my question is this ever happens to looks stupid: is your
> matrix allocated as a local object of that procedure ?
> If so, the stack may have a maximum limit which cannot be exceeded,
> and dynamic allocation may solve the case.
> If not, an excerpt from the source could help to see.
>
> Do you use Ada.Numerics.Generic_Real_Arrays.Real_Matrix ? (that is
> what I supposed, may need to be confirmed).
The implementation of the Ada.Numerics.Generic*Arrays in GNAT (and in
https://sourceforge.net/projects/gnat-math-extn/) allocates temporary
arrays on the stack, which will be of similar sizes to the original.
Can you call the matrix operations from a task with a large stack?
I'm not sure that generalized algorithms are going to meet your needs
(you're going to be perilously close to your 8GB of real memory, when
you consider temporary space, which is going to risk much swapping; and
won't you need a 64-bit OS?) Are there any sparse matrix algorithms?
Brian Drummond
>John,
>
>> I wish to process both real and complex large matrices eg (1..10000,
>> 1..10000)
>> I have no difficulty in increasing the storage size of a task.
>> I cannot do the same for matrices within a procedure which reports
>> that the heap is exceeded.
>
>You probably mean stack size and not heap size. In that case you need to
>add the proper linker option to increase the stack size for the application.
>
>Pascal.
I have had trouble with this in the past - the documented linker options
apparently didn't work (around the 2008 timeframe - may not still be true) and
Pragma Storage_Size only worked for new tasks, not the main program..
I resorted to dynamic allocation on the heap, and renaming to hide the pointer
dereference. So, given
my_array_type is array (positive range <>, positive range <>) of float;
I replaced
my_array: my_array_type (1..10000, 1..10000);
with
my_array_ptr : access my_array_type := new my_array_type (1..10000, 1..10000);
my_array : my_array_type renames my_array_ptr.all;
The rename meant I didn't have to rewrite anything else in the application.
- Brian
John Raymond Dore
> "Yannick Duchêne (Hibou57)" <yannick_duch...@yahoo.fr> writes:
>
> > Le Sun, 29 Aug 2010 10:50:39 +0200, John Raymond Dore
> >> I have no difficulty in increasing the storage size of a task.
> >> I cannot do the same for matrices within a procedure
> > Forgive my question is this ever happens to looks stupid: is your
> > matrix allocated as a local object of that procedure ?
> > If so, the stack may have a maximum limit which cannot be exceeded,
> > and dynamic allocation may solve the case.
> > If not, an excerpt from the source could help to see.
>
> > Do you use Ada.Numerics.Generic_Real_Arrays.Real_Matrix ? (that is
> > what I supposed, may need to be confirmed).
>
suffice accessible from a procedure.
It does not have to be declared within a procedure.
My computer is running 64bit Ubuntu
> The implementation of the Ada.Numerics.Generic*Arrays in GNAT (and inhttps://sourceforge.net/projects/gnat-math-extn/) allocates temporary
John Raymond Dore
GNAT GPL specifically says heap in error message
John Raymond Dore
wrote:
Brilliant elegant solution to my problem Brian.
It works just fine.
Very many thanks.
John
Pascal Obry
> I have had trouble with this in the past - the documented linker options
> apparently didn't work (around the 2008 timeframe - may not still be true) and
> Pragma Storage_Size only worked for new tasks, not the main program..
Right, that's why you need to tell the linker about the proper stack
size you need.
> I resorted to dynamic allocation on the heap, and renaming to hide the pointer
> dereference. So, given
> my_array_type is array (positive range <>, positive range <>) of float;
>
> I replaced
>
> my_array: my_array_type (1..10000, 1..10000);
>
> with
>
> my_array_ptr : access my_array_type := new my_array_type (1..10000, 1..10000);
> my_array : my_array_type renames my_array_ptr.all;
Ok, fine, that's indeed a workaround to this problem.
Pascal Obry
John,
> GNAT GPL specifically says heap in error message
But it turned out that my guess was correct!
Ludovic Brenta
> John,
>
>> GNAT GPL specifically says heap in error message
>
> But it turned out that my guess was correct!
Here is another guess: the large array was on the secondary stack, which
is taken from the heap?
--
Ludovic Brenta.
Pascal Obry
Ludovic,
> Here is another guess: the large array was on the secondary stack, which
> is taken from the heap?
I don't think the secondary stack is taken from the heap... but I'm far
from expert on this!
Robert A Duff
> Ludovic,
>
>> Here is another guess: the large array was on the secondary stack, which
>> is taken from the heap?
>
> I don't think the secondary stack is taken from the heap... but I'm far
> from expert on this!
It is taken from the heap (by chunks) on some (most?) platforms.
On others, it is allocated on the primary stack.
See package System.Parameters.
As far as I know, the only things allocated on the seconary stack are
function results of caller-unknown size (as in function(...) return
String), and some controlled/finalization-related things.
- Bob
Ludovic Brenta
> Pascal Obry <pas...@obry.net> writes:
>
>> Ludovic,
>>
>>> Here is another guess: the large array was on the secondary stack, which
>>> is taken from the heap?
>>
>> I don't think the secondary stack is taken from the heap... but I'm far
>> from expert on this!
>
> It is taken from the heap (by chunks) on some (most?) platforms.
> On others, it is allocated on the primary stack.
> See package System.Parameters.
This says that on x86_64-linux-gnu (which is both John's and my
platform), the secondary stack is indeed on the heap.
> As far as I know, the only things allocated on the seconary stack are
> function results of caller-unknown size (as in function(...) return
> String), and some controlled/finalization-related things.
Right, that would be the case for a function returning the large array,
which was of an unconstrained type (array (Positive range <>, Positive
range <>) of Long_Float in this case).
--
Ludovic Brenta.
Peter C. Chapin
>> I replaced
>>
>> my_array: my_array_type (1..10000, 1..10000);
>>
>> with
>>
>> my_array_ptr : access my_array_type := new my_array_type (1..10000, 1..10000);
>> my_array : my_array_type renames my_array_ptr.all;
>>
>> The rename meant I didn't have to rewrite anything else in the application.
>>
>> - Brian
>
> Brilliant elegant solution to my problem Brian.
> It works just fine.
Be aware that you may now have to explicitly deallocate the array at
some point. How necessary that is will depend on your program and some
other factors. It's something to keep in mind.
Peter
Brian Drummond
Certainly worth pointing out, depending on the application (and possibly the
compiler?)
In my usage the "new" allocation is usually in the declare... section of a local
block, with no tricks on the access type, so the intended lifetime is the block
itself.
It seems that the deallocation in this case could be automated - and in crude
tests enclosing the block in a loop I couldn't see memory use increasing with
time ... perhaps this is not the case, I wasn't looking carefully enough?
- Brian
Natasha Kerensikova
> Be aware that you may now have to explicitly deallocate the array at
> some point. How necessary that is will depend on your program and some
> other factors. It's something to keep in mind.
Are there some facilities like C++'s auto_ptr (i.e. containers that
behaves like pointers but with a reference counter, so that when the
last container is finalized the pointed data is release automatially)?
I haven't seen any, have I missed it?
I guess it's so easy to write that if it's not standardized, it's
probably not because of complexity. Is there something in Ada language
or in Ada philosophy that would make them almost useless (except in rare
situations like OP's)?
Natasha
Brian Drummond
<lithi...@gmail.com> wrote:
>On 2010-08-29, Peter C. Chapin <cha...@acm.org> wrote:
>> Be aware that you may now have to explicitly deallocate the array at
>> some point. How necessary that is will depend on your program and some
>> other factors. It's something to keep in mind.
>
>Are there some facilities like C++'s auto_ptr (i.e. containers that
>behaves like pointers but with a reference counter, so that when the
>last container is finalized the pointed data is release automatially)?
>I haven't seen any, have I missed it?
Have you looked at controlled types?
- Brian
Brian Drummond
<brian_d...@btconnect.com> wrote:
>On Sun, 29 Aug 2010 18:06:46 -0400, "Peter C. Chapin" <cha...@acm.org> wrote:
>>Be aware that you may now have to explicitly deallocate the array at
>>some point. How necessary that is will depend on your program and some
>>other factors. It's something to keep in mind.
>
>..., so the intended lifetime is the block itself.
>It seems that the deallocation in this case could be automated - and in crude
>tests enclosing the block in a loop I couldn't see memory use increasing with
>time ... perhaps this is not the case, I wasn't looking carefully enough?
Apologies to all: My late night recollection was faulty; my experiment covered a
different case.
If the "new" allocation is in a loop, explicit deallocation (or possibly
controlled types - I haven't tried them yet) is required.
- Brian
Dmitry A. Kazakov
> On 2010-08-29, Peter C. Chapin <cha...@acm.org> wrote:
>> Be aware that you may now have to explicitly deallocate the array at
>> some point. How necessary that is will depend on your program and some
>> other factors. It's something to keep in mind.
>
> Are there some facilities like C++'s auto_ptr (i.e. containers that
> behaves like pointers but with a reference counter, so that when the
> last container is finalized the pointed data is release automatially)?
> I haven't seen any, have I missed it?
Are you looking for a reference-counted GC or for a pointer that brings the
referenced object down with it, when goes out of scope?
> I guess it's so easy to write that if it's not standardized, it's
> probably not because of complexity. Is there something in Ada language
> or in Ada philosophy that would make them almost useless (except in rare
> situations like OP's)?
If you meant the latter, you don't need anything to write as the language
already provides collection through scoped access types (pointers).
Consider this:
with Ada.Finalization;
with Ada.Text_IO; use Ada.Text_IO;
package P is
type Object is new Ada.Finalization.Limited_Controlled with null record;
overriding procedure Finalize (X : in out Object);
end P;
package body P is
procedure Finalize (X : in out Object) is
begin
Put_Line ("I am down");
end Finalize;
end P;
The object Object prints "I am down" when destroyed.
Now if you either declare an access type in a scope or else use an
anonymous type then any object allocated by new and assigned to this
pointer will be automatically destroyed when the scope is left:
declare
type Pointer is access all Object;
X1 : Pointer := new Object;
X2 : access Object := new Object;
begin
... -- Using X1 and X2
end; -- Targets of X1 and X2 are finalized and freed here
For a reference-counted GC there are several implementations of. For
example this:
http://www.dmitry-kazakov.de/ada/components.htm#Objects_etc
--
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de
Natasha Kerensikova
Yes, that's why I said such an auto_ptr would be very easily
implemented, with controlled types and generics. But the auto_ptr
itself, as far as I know, does not exist (yet). There should probably a
reason, shoudn't it?
Natasha
Georg Bauhaus
> For a reference-counted GC there are several implementations of. For
> example this:
>
> http://www.dmitry-kazakov.de/ada/components.htm#Objects_etc
Another expo,
http://www.christ-usch-grein.homepage.t-online.de/Ada/Safe_Pointers.html
Georg
Cyrille
wrote:>
> Apologies to all: My late night recollection was faulty; my experiment covered a
> different case.
>
> If the "new" allocation is in a loop, explicit deallocation (or possibly
> controlled types - I haven't tried them yet) is required.
If you want this behavior (i.e. deallocation of your dynamically
allocated object on scope exit) you can explicitly associate the local
access type to a pool object of type
System.Pool_Local.Unbounded_Reclaim_Pool.
Jacob Sparre Andersen
> If you want this behavior (i.e. deallocation of your dynamically
> allocated object on scope exit) you can explicitly associate the local
> access type to a pool object of type
> System.Pool_Local.Unbounded_Reclaim_Pool.
Checking on Google indicates that this is a GNAT specific package.
Are there any special reasons that the RM doesn't include a collection
of "useful" storage pool types like this one?
Greetings,
Jacob
--
"If you think Tuck has said something that is completely
wrong, you are almost certainly missing something :-)"
Dmitry A. Kazakov
> Cyrille <co...@eu.adacore.com> writes:
>
>> If you want this behavior (i.e. deallocation of your dynamically
>> allocated object on scope exit) you can explicitly associate the local
>> access type to a pool object of type
>> System.Pool_Local.Unbounded_Reclaim_Pool.
>
> Checking on Google indicates that this is a GNAT specific package.
>
> Are there any special reasons that the RM doesn't include a collection
> of "useful" storage pool types like this one?
There is even more important thing about pools missing: there is no name
for the default storage pool.
Let I have some specialized pools, which serves the purpose of allocation
policy rather than actually providing storage. Such a pool sits on top of
another pool. E.g. a mark and release pool uses another pool to allocate
its blocks there. Other examples are stack pool, doubly-linked lists of
non-tagged elements, graphs etc.
The problem is that there is no way specify this default pool, because it
has no name. In order to get at it I need to use tricks like:
type Integer_Access is access Integer;
-- Hopefully it is the default pool
and then use Integer_Access'Storage_Pool. There is no any guaranty that
this would work.
P.S. It is clear that some compiler might use several default storage
pools, e.g. special pools for allocating objects of according to their
sizes. But that is no reason for not providing a default storage pool in
addition to them, which would work for any object.
P.P.S. There also should be a pragma or for-clause to re-route all other
default pools to some pool. E.g. to a debugging pool. GNAT has a debugging
pool but it is very difficult to use, because you have to rewrite all
declarations of specific access types. That should rather be:
for System.Default_Pool use My_Debugging_Pool;
Robert A Duff
> The problem is that there is no way specify this default pool, because it
> has no name. In order to get at it I need to use tricks like:
See AI-190:
Provide a means to force the use of user-defined pools, and a means to
specify a particular pool to be used by default.
which will give you at least part of what you want in Ada 2012.
- Bob
J-P. Rosen
> On Mon, 30 Aug 2010 15:01:56 +0200, Jacob Sparre Andersen wrote:
>> Are there any special reasons that the RM doesn't include a collection
>> of "useful" storage pool types like this one?
>
> There is even more important thing about pools missing: there is no name
> for the default storage pool.
>
Did you have a look at AI05-0190-1 (Global storage pool controls) ?
This should respond to a least some of your concerns.
--
---------------------------------------------------------
J-P. Rosen (ro...@adalog.fr)
Visit Adalog's web site at http://www.adalog.fr
Simon Wright
> Now if you either declare an access type in a scope or else use an
> anonymous type then any object allocated by new and assigned to this
> pointer will be automatically destroyed when the scope is left:
>
> declare
> type Pointer is access all Object;
> X1 : Pointer := new Object;
> X2 : access Object := new Object;
> begin
> ... -- Using X1 and X2
> end; -- Targets of X1 and X2 are finalized and freed here
I'm pretty sure that GNAT does this by using the Reclaim pool mentioned
above to implement the storage pool.
> For a reference-counted GC there are several implementations of. For
> example this:
>
> http://www.dmitry-kazakov.de/ada/components.htm#Objects_etc
And the Booch Components, http://sourceforge.net/projects/booch95/files/
Simon Wright
> Are there some facilities like C++'s auto_ptr (i.e. containers that
> behaves like pointers but with a reference counter, so that when the
> last container is finalized the pointed data is release automatially)?
> I haven't seen any, have I missed it?
Of course Wikipedia may be wrong (pace Yannick) but
http://en.wikipedia.org/wiki/Auto_ptr#Semantics indicates that auto_ptr
doesn't in fact have reference counting?
Robert A Duff
> "Dmitry A. Kazakov" <mai...@dmitry-kazakov.de> writes:
>
>> Now if you either declare an access type in a scope or else use an
>> anonymous type then any object allocated by new and assigned to this
>> pointer will be automatically destroyed when the scope is left:
>>
>> declare
>> type Pointer is access all Object;
>> X1 : Pointer := new Object;
>> X2 : access Object := new Object;
>> begin
>> ... -- Using X1 and X2
>> end; -- Targets of X1 and X2 are finalized and freed here
>
> I'm pretty sure that GNAT does this by using the Reclaim pool mentioned
> above to implement the storage pool.
I don't think so. I think those objects are finalized,
but never freed.
And I think that's desirable behavior, because things like
"Reclaim pool" have a cost, so should be enabled only upon
explicit request.
- Bob
Jeffrey Carter
And PragmARC.Safe_Pointers:
http://pragmada.x10hosting.com/
--
Jeff Carter
"How'd you like to hide the egg and gurgitate
a few saucers of mocha java?"
Never Give a Sucker an Even Break
101
--- news://freenews.netfront.net/ - complaints: ne...@netfront.net ---
Simon Wright
>> I'm pretty sure that GNAT does this by using the Reclaim pool mentioned
>> above to implement the storage pool.
>
> I don't think so. I think those objects are finalized, but never
> freed.
Oh, I misinterpreted the header comment in s-pooloc.ads
-- Storage pool for use with local objects with automatic reclaim
But I see that a couple of lines below there is
-- Needed to ensure that library routines can execute allocators
which leaves me confused. I guess I don't need to know ...
Natasha Kerensikova
Well, I guess this shows how lacking I am in C++. I genuinely thought
auto_ptr was a kind of memory manager, less thin that what is described
in Wikipedia.
Anyway, I still see the use of the thing I thought was auto_ptr, and
others pointing at implementations seems to confirm this usefulness.
However I still don't really understand why it's not part of the
standard, but I guess I will be more able to judge how useful it really
is in Ada after having much more Ada practise.
Thanks for correcting me,
Natasha