Too much memory used

[Greg K]

Hi all.

I know that this issue has been raised here few times already, but it
is still not clear to me how much memory is required by looking at the
numbers.
I am using AMPL+Couenne on NEOS server, submitting huge problem (still
within NEOS limits) using NEOS Submission Tool and I am getting the
following AMPL specific error:

Error: Too much memory used -- 389980144 bytes; couldn't get 32800
more.
processing commands.
Error: Highest address used = 46914137485327.
Error (512) in /opt/ampl/ampl -R amplin

As I understood from other similar topics in this group, the problem
is in shortage of RAM on my PC from which I am trying to submit the
problem. Can someone help me to understand the numbers above, so I can
figure out what kind of computer and how much memory is required to
run this problem?

Thanks.

[Robert Fourer]

If you are using the NEOS Submission Tool (rather than "Kestrel") then AMPL
is being run by NEOS rather than by you. The reference to "/opt/ampl/ampl
-R amplin" supports this, as it is typical of the sort of AMPL invocation
that is used by NEOS.

"Too much memory used -- 389980144 bytes" is typical of the message returned
by AMPL when it runs out of memory. It says that that AMPL process was able
to allocate only about 390M bytes on the NEOS computer on which it was
running. Since computers now typically have many gigabytes of memory, this
number is surprisingly small, but then I don't know anything about the
computer that NEOS is using for these runs. It you want to send me the
files I can trying running them on a computer that I know has a lot of
memory, to see how much memory NEOS might actually need to make available.

"Highest address used = 46914137485327" seems like an error in the message,
as this is a far higher address than one could expect AMPL to ever try to
address.

Bob Fourer
4...@ampl.com

[Greg K]

Thanks for your reply Robert.

I had a chance to run the same problem on a 4GB RAM 64-bit machine and
I got the following numbers

Too much memory used -- 14034966144 bytes; couldn't get 38400 more.
Highest address used = 18446744073709551616.

And again, I am having difficulties understanding these numbers.
14034966144 bytes is approximately 13 Gb. That had confused me even
more, since I've though that the memory it is running out of is a RAM,
but 13 Gb is much more than available 4 Gb of RAM. Does it mean that
we are talking about disk memory? But there is much more than 13 gigs
of memory available.

And the most confusing number is that of "Highest address used", that
equals to 18446744073709551616 which seems like some enormous number.
What does this number means and indicates?

Thanks for your help

Greg

[Robert Fourer]

When the 4GB of physical memory is used up, the operating system can
simulate additional memory by using a paging file on disk to store some of
the information that would otherwise be in RAM. The size of the paging file
is limited by the operating system, however, generally to a size much
smaller than the total available disk space. That could explain why memory
ran out after the use of 13GB.

The "Highest address used" = 18446744073709551616 = 2^64 is actually the
highest address possible on a 64-bit system. Obviously not all addresses up
to this one were used. I'll investigate why AMPL might be giving this
number, but for diagnostic purposes the "memory used" number is the better
one to look at.

When solving a problem that has integer variables, it is possible that the
search tree is growing exponentially with the number of variables or
constraints, and that consequently past a certain size the problem is not
solvable with the technology you're using. Then it is necessary to change
the algorithmic settings, solution strategy, or model formulation to get
useful results.

Bob Fourer
4...@ampl.com

> -----Original Message-----
> From: am...@googlegroups.com [mailto:am...@googlegroups.com]
> On Behalf Of Greg K
> Sent: Friday, June 24, 2011 12:35 PM
> To: AMPL Modeling Language

[Greg K]

Thanks Bob for your prompt response.

>> That could explain why memory ran out after the use of 13GB.

Does it mean that it won't probably work on 8 Gb machine?

>>The size of the paging file is limited by the operating system

Is there a way to increase this limit? Will Linux have larger limit
than Windows?

[Robert Fourer]

It's hard to predict how much memory might be needed. You can get some
clues by looking at the listing produced by setting

option cplex_options 'mipdisplay 2 mipinterval 1000';

(or adding these directives to your current cplex_options string). In
particular the persistence of a large Gap and increasing numbers of Nodes
Left indicate that the size of the search tree may continue to increase for
some time.

In Windows the size of the paging file can be set in the System control
panel, under Advanced system settings, and then Performance, and then the
Advanced tab. There must be something analogous under Linux but I'm not
familiar with it.

Bob Fourer
4...@ampl.com

> -----Original Message-----
> From: am...@googlegroups.com [mailto:am...@googlegroups.com]
> On Behalf Of Greg K
> Sent: Friday, June 24, 2011 7:51 PM
> To: AMPL Modeling Language

[Robert Fourer]

A further follow-up on this issue:

The "Highest address used" is only (marginally) useful when all memory is
contiguous and starts at a low address. This was only true of a few
operating systems, however, so to avoid confusing we are removing this
message from subsequent releases.

Bob Fourer
4...@ampl.com

> -----Original Message-----
> From: am...@googlegroups.com [mailto:am...@googlegroups.com]
> On Behalf Of Greg K
> Sent: Friday, June 24, 2011 12:35 PM
> To: AMPL Modeling Language

[mahootchi]

Hello

I am getting the following error when I run my program through NEOS.
I just wonder whether there is any chance to fix it.

Thanks


ob 2401819 sent to e031.chtc.wisc.edu
password: udUFKhnm
---------- Begin Solver Output -----------
Executing /opt/neos/Drivers/minos-ampl/minos-driver.py
File exists
You are using the solver minos.
Executing AMPL.
processing data.
Error: Error at _cmdno 66442 executing "solve" command
Error: (file amplin, line 804, offset 38787):
Error:
processing commands.
Error: Too much memory used -- 390084568 bytes; couldn't get 32800
more.

Error (512) in /opt/ampl/ampl -R amplin

Error: Highest address used = 670046431.
Error:

[Saber Jalilzadeh-Galaeh]

"Too much memory used -- 390084568 bytes" is typical of the message returned 
by AMPL when it runs out of "memory".  It says that that AMPL process was able 

to allocate only about 390M bytes on the NEOS computer on which it was 
running.

For more details take a look at the discussion:

http://groups.google.com/group/ampl/browse_thread/thread/bf5a0e252485a27e/378f653bb060d81c?lnk=gst&q=Too+much+memory+used#378f653bb060d81c

Saber

---

Von: mahootchi <mmaho...@gmail.com>
An: AMPL Modeling Language <am...@googlegroups.com>
Gesendet: 21:48 Donnerstag, 11.August 2011
Betreff: [AMPL 4911] Re: Too much memory used

--
You received this message because you are subscribed to the Google Groups "AMPL Modeling Language" group.
To post to this group, send email to am...@googlegroups.com.
To unsubscribe from this group, send email to ampl+unsub...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/ampl?hl=en.

[Robert Fourer]

For best results I suggest sending questions about out-of-memory conditions
on NEOS computers to NEOS Server support, sup...@neos-server.org.