Account Options

  1. Sign in
The old Google Groups will be going away soon, but your browser is incompatible with the new version.
Google Groups Home
« Groups Home
Garbage hunting
There are currently too many topics in this group that display first. To make this topic appear first, remove this option from another topic.
There was an error processing your request. Please try again.
flag
  Messages 1 - 25 of 41 - Collapse all  -  Translate all to Translated (View all originals)   Newer >
The group you are posting to is a Usenet group. Messages posted to this group will make your email address visible to anyone on the Internet.
Your reply message has not been sent.
Your post was successful
 
From:
To:
Cc:
Followup To:
Add Cc | Add Followup-to | Edit Subject
Subject:
Validation:
For verification purposes please type the characters you see in the picture below or the numbers you hear by clicking the accessibility icon. Listen and type the numbers you hear
 
Drew McDermott  
View profile  
 More options Jan 4 1999, 3:00 am
Newsgroups: comp.lang.lisp
From: Drew McDermott <drew.mcderm...@yale.edu>
Date: 1999/01/04
Subject: Garbage hunting
I often find myself wanting a tool that would allow me to find all the
nongarbage that Lisp is refusing to collect.  I have been using
Harlequin Common Lisp lately, and I find that on big problems it often
mushrooms to about 50M even though I'm pretty sure I'm eliminating all
pointers to the large structures my program generates for temporary
use.  What I'd like is a tool that will tell me what the 50 MBytes
consists of and why it can't be collected.  It seems to me this could be
written in a portable way (starting at all symbols accessible
anywhere).  A portable tool wouldn't find the storage that Harlequin has
lost in some special way all its own, but it would allow me to convince
them that it exists (or exonerate them by showing a trail to something I
thought I deleted all the references to).

Does anyone know of any such package?

     -- Drew McDermott


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Lyman S. Taylor  
View profile  
 More options Jan 4 1999, 3:00 am
Newsgroups: comp.lang.lisp
From: ly...@cc.gatech.edu (Lyman S. Taylor)
Date: 1999/01/04
Subject: Re: Garbage hunting
In article <369139EF.2A62...@yale.edu>,
Drew McDermott  <drew.mcderm...@yale.edu> wrote:

>I often find myself wanting a tool that would allow me to find all the
>nongarbage that Lisp is refusing to collect.

   Pardon me if I'm oversimplying things but....

   Is it uncollected garbage or does the heap space not reduced/resized after
   expansion?

   I can't recall how informative the invocation of

        (room t )

   is in Lispworks but it presumambly will tell you where this allocation
   (if any) is in the "generations". A categorization of the type of stuff may
   not be present.  There is no standard requirement for the utility of
   this function's output.

>use.  What I'd like is a tool that will tell me what the 50 MBytes
>consists of and why it can't be collected.

   Not sure if applicable but sometimes forgotten....
   If the result of some expression is huge and is printed to the
   Listener, then the Listener "history" symbols ( * , ** , etc. ) will hold
   onto it (directly or indirectly ) for a while  even though nothing in
   your "program" is refering to the data.  Four or five NILs subsequently
   entered at the Listener solves that problem, eventually.

> It seems to me this could be
>written in a portable way (starting at all symbols accessible
>anywhere).

   I'm not so sure this can be portable...  I would imagine that
   most garbage collectors are written in terms of implementation
   specific primatives. Doing this without stepping on the "toes"
   of the garbage collector, implementation internal allocation data
   structures, and being portable?  That seems like a tall order.

   Wrapping a common "interface" to implementation specific code seems
   more tractable.  Seems like this is something ROOM should/could
   optionally report though.

   It seems as though "magic" would be required to do a measurement
   without preturbing what is being observed.  The implementation
   specific primatives would provide this.

--

Lyman S. Taylor            "I'm a Doctor! Not a commando."
(ly...@cc.gatech.edu)         The enhanced EMH Doctor in a ST:Voyager epidsode.


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Kelly Murray  
View profile  
 More options Jan 4 1999, 3:00 am
Newsgroups: comp.lang.lisp
From: Kelly Murray <k...@IntelliMarket.Com>
Date: 1999/01/04
Subject: Re: Garbage hunting

Drew McDermott wrote:

> I often find myself wanting a tool that would allow me to find all the
> nongarbage that Lisp is refusing to collect.  I have been using

I'm not aware of such a tool, but something related which will tell
you if indeed there is some unclaimed "garbage" is to dump out all
your objects into a file, and then in another fresh lisp restore
from the dump and then compare the memory useage.
For ACL, see excl:fasl-write and excl:fasl-read, Harlequin may
have something similiar.  the SAVE-OBJECT ascii representation
code is a more portable solution.  

-Kelly Murray  k...@intellimarket.com
  still offering $500 for an unrestricted CL JPEG encode/decode util.


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Duane Rettig  
View profile  
 More options Jan 4 1999, 3:00 am
Newsgroups: comp.lang.lisp
From: Duane Rettig <du...@franz.com>
Date: 1999/01/04
Subject: Re: Garbage hunting

Drew McDermott <drew.mcderm...@yale.edu> writes:
> I often find myself wanting a tool that would allow me to find all the
> nongarbage that Lisp is refusing to collect.  I have been using
> Harlequin Common Lisp lately, and I find that on big problems it often
> mushrooms to about 50M even though I'm pretty sure I'm eliminating all
> pointers to the large structures my program generates for temporary
> use.  What I'd like is a tool that will tell me what the 50 MBytes
> consists of and why it can't be collected.  It seems to me this could be
> written in a portable way (starting at all symbols accessible
> anywhere).  A portable tool wouldn't find the storage that Harlequin has
> lost in some special way all its own, but it would allow me to convince
> them that it exists (or exonerate them by showing a trail to something I
> thought I deleted all the references to).

> Does anyone know of any such package?

I'll risk a blatant plug by saying that Allegro CL has such a function,
though unexported and undocumented, called excl::get-references, which
accepts a lisp object and returns a vector holding all of the objects in
the heap which point to the object in question.  The reason why it is
not exported or documented is because it gets messy fairly quickly; if
you realize that every returned vector is itself a new pointer to the
object in question, a chain of calls to excl::get-references can get
fairly cluttered with vector objects that were not part of the original
heap.  Nevertheless, it has been a useful function, if you are willing
to put up with its droppings.

Perhaps Harlequin also has such a function.

--
Duane Rettig          Franz Inc.            http://www.franz.com/ (www)
1995 University Ave Suite 275  Berkeley, CA 94704
Phone: (510) 548-3600; FAX: (510) 548-8253   du...@Franz.COM (internet)


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
R. Toy  
View profile  
 More options Jan 4 1999, 3:00 am
Newsgroups: comp.lang.lisp
From: "R. Toy" <r...@mindspring.com>
Date: 1999/01/04
Subject: Re: Garbage hunting

Christopher R. Barry wrote:
> In general, once an application allocates RAM from the system, there
> is no way for the OS to reclaim that memory until the application's
> process is terminated. There are "relocating allocators" that some
[snip]

> Neither ACL5 nor CMUCL have such a capability, nor does (most
> unfortunately) Netscape. I would never, ever restart any of these
> applications if they were capable of letting the OS reclaim unused
> memory, because Netscape can swell to 140MB after a few days and then

I think you are mistaken about CMUCL.  I've seen CMUCL grow and shrink
in size as an application runs, at least ps and top say so.  Both the
total size and the RSS change.

Ray

--
---------------------------------------------------------------------------
----> Raymond Toy    r...@mindspring.com
                        http://www.mindspring.com/~rtoy


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Duane Rettig  
View profile  
 More options Jan 4 1999, 3:00 am
Newsgroups: comp.lang.lisp
From: Duane Rettig <du...@franz.com>
Date: 1999/01/04
Subject: Re: Garbage hunting
cba...@2xtreme.net (Christopher R. Barry) writes:

I think that the original requestor is more concerned about individual
objects than about blocks of memory, but at any rate your statement is
not necessarily true; it depends on how the memory was allocated
originally.  It also depends on the operating system as to how
mmap/munmap (and VirtualAlloc/VirtualFree on Windows) will respond to
such requests, and if there are no bugs in these allocation functions,
as the paragraph you supplied below implies:

ACL5 certainly does have this capability.  The sys:resize-areas function
can reduce the virtual memory utilization, as long as there are no
"Gaps" in the heap (sections of memory not owned by the lisp allocator).

I did notice when I ran an experiment on ACL5 on NT that even after the
memory was relinquished by the lisp, the operating system didn't actually
take it back until it needed the physical pages (i.e. I started some other
programs that would require virtual memory allocation).  At that point,
Task manager did record the shedding of the pages, and even a global gc
did not bring them back, though it touches most of the allocated pages.

--
Duane Rettig          Franz Inc.            http://www.franz.com/ (www)
1995 University Ave Suite 275  Berkeley, CA 94704
Phone: (510) 548-3600; FAX: (510) 548-8253   du...@Franz.COM (internet)


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
cdm  
View profile  
 More options Jan 4 1999, 3:00 am
Newsgroups: comp.lang.lisp
From: "cdm" <c...@pacific.net>
Date: 1999/01/04
Subject: Re: Garbage hunting

Drew McDermott <drew.mcderm...@yale.edu> wrote in message

news:369139EF.2A623EF@yale.edu...

>I often find myself wanting a tool that would allow me to find all the
>nongarbage that Lisp is refusing to collect.

I wrote one of these a few years ago at Lucid.  Here is how it worked:

First, it treats the collectable heap as a directed graph.  Then it finds
all the `bridges' in the graph, i.e. those pointers that hold a unique
reference to an object.  Then, it weighs those pointers by determining
how much total storage would be freed if the pointer didn't exist.  Finally,
it produced a report that noted which symbols contained such pointers,
and which datatypes such pointers pointed at.  The former would tell
you where the large amounts of data lived, the latter would tell you what
the
large structures were.

It turned out to be quite a useful tool, but I think it has since been lost.
The algorithm for finding bridges in the heap was pretty bad, and I'm
sure someone more versed in graph theory could do better.

~jrm


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Christopher R. Barry  
View profile  
 More options Jan 5 1999, 3:00 am
Newsgroups: comp.lang.lisp
From: cba...@2xtreme.net (Christopher R. Barry)
Date: 1999/01/05
Subject: Re: Garbage hunting

In general, once an application allocates RAM from the system, there
is no way for the OS to reclaim that memory until the application's
process is terminated. There are "relocating allocators" that some
applications use that allow them to return unused memory to the
OS, but usage of these is far from pervasive as of yet. For example,
XEmacs allows this compile-time option:

  The `--rel-alloc' option can be used to either enable or disable use
  of the relocating allocator.  Turning on --rel-alloc will allow XEmacs
  to return unused memory to the operating system, thereby reducing its
  memory footprint.  However, it may make XEmacs runs more slowly,
  especially if your system's `mmap' implemntation is missing or
  inefficient.

Neither ACL5 nor CMUCL have such a capability, nor does (most
unfortunately) Netscape. I would never, ever restart any of these
applications if they were capable of letting the OS reclaim unused
memory, because Netscape can swell to 140MB after a few days and then
it's back down to 25MB after a restart.

Christopher


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Christopher R. Barry  
View profile  
 More options Jan 5 1999, 3:00 am
Newsgroups: comp.lang.lisp
From: cba...@2xtreme.net (Christopher R. Barry)
Date: 1999/01/05
Subject: Re: Garbage hunting

"R. Toy" <r...@mindspring.com> writes:
> Christopher R. Barry wrote:
> > In general, once an application allocates RAM from the system, there
> > is no way for the OS to reclaim that memory until the application's
> > process is terminated. There are "relocating allocators" that some
> [snip]

> > Neither ACL5 nor CMUCL have such a capability, nor does (most
> > unfortunately) Netscape. I would never, ever restart any of these
> > applications if they were capable of letting the OS reclaim unused
> > memory, because Netscape can swell to 140MB after a few days and then

> I think you are mistaken about CMUCL.  I've seen CMUCL grow and shrink
> in size as an application runs, at least ps and top say so.  Both the
> total size and the RSS change.

After playing around a little, I can see that it does this a little bit,
but I guess I had never noticed it before because it could be a lot
better. Here are the numbers:

Fresh launch of CMUCL:

                        SIZE  RSS SHARE
                        ____  ___ _____
 9928 cbarry     0   0  8108 8108  4360 S    1104  0.0 12.7   0:00 lisp

After doing a (fact 15000):

 9928 cbarry     0   0 90988  42M  2616 S     14M  0.0 69.3   0:23 lisp

A global GC after that:

 9928 cbarry     0   0 75868  27M  2876 S       0  0.0 44.1   0:23 lisp

And basically, CMUCL will never use less memory than this for the
duration of it's usage. I have a WindowMaker dock app that shows me my
real-time memory and swap usage and after doing that (fact 15000)
there is nothing in the world that is going to make my swap usage back
down to 0 until I shut down and restart CMUCL. CMUCL does grow and
shrink somewhat, but it grows unstoppably over time, as does ACL5 and
most other apps out there that at times use a lot of memory.

Christopher


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Robert Virding  
View profile  
 More options Jan 5 1999, 3:00 am
Newsgroups: comp.lang.lisp
From: r...@erix.ericsson.se (Robert Virding)
Date: 1999/01/05
Subject: Re: Garbage hunting

In article <369139EF.2A62...@yale.edu>, Drew McDermott <drew.mcderm...@yale.edu> writes:
>I often find myself wanting a tool that would allow me to find all the
>nongarbage that Lisp is refusing to collect.  I have been using
>Harlequin Common Lisp lately, and I find that on big problems it often
>mushrooms to about 50M even though I'm pretty sure I'm eliminating all
>pointers to the large structures my program generates for temporary
>use.  What I'd like is a tool that will tell me what the 50 MBytes
>consists of and why it can't be collected.  It seems to me this could be
>written in a portable way (starting at all symbols accessible
>anywhere).  A portable tool wouldn't find the storage that Harlequin has
>lost in some special way all its own, but it would allow me to convince
>them that it exists (or exonerate them by showing a trail to something I
>thought I deleted all the references to).

>Does anyone know of any such package?

I think that there is a slight semantic problem here.  If there is any
nongarbage that HCL is refusing to collect so it can be reused then
there is definitely a bug in the collector.  This does not mean that
HCL returns all free memory, from its point of view, to the OS.
Depending on what type of collector it is the free memory might be in
chunks which it cannot return to the OS, or it might feel that it
might as well hang on to it seeing it will probably want to use it
again.

However a tool which shows how the memory is being used can be very
useful, especially when debugging the collector. :-)

--
Robert Virding                          Tel: +46 (0)8 719 95 28
Computer Science Laboratory             Email: r...@erix.ericsson.se
Ericsson Telecom AB                     WWW: http://www.ericsson.se/cslab/~rv
S-126 25 ÄLVSJÖ, SWEDEN                 OSE: http://www.erlang.org/
"Folk säger att jag inte bryr mig om någonting, men det skiter jag i".


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Erik Naggum  
View profile  
 More options Jan 5 1999, 3:00 am
Newsgroups: comp.lang.lisp
From: Erik Naggum <e...@naggum.no>
Date: 1999/01/05
Subject: Re: Garbage hunting
* "R. Toy" <r...@mindspring.com>
| I think you are mistaken about CMUCL.  I've seen CMUCL grow and shrink in
| size as an application runs, at least ps and top say so.  Both the total
| size and the RSS change.

  this may be misleading.  under Linux, the VM size of the Allegro CL
  process does not decrease according to /proc/self/status even when the
  working set is greatly reduced by using SYSTEM:RESIZE-AREAS.  I was a
  little surprised by this, since ps and top and other tools report a
  decrease.  I guess it's time to trust the kernel and not various "smart"
  tools that try to provide user-friendly information.

#:Erik


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Drew McDermott  
View profile  
 More options Jan 5 1999, 3:00 am
Newsgroups: comp.lang.lisp
From: Drew McDermott <drew.mcderm...@yale.edu>
Date: 1999/01/05
Subject: Re: Garbage hunting

> In response to my query about a tool for finding Lisp garbage,
> Christopher R. Barry wrote:
> In general, once an application allocates RAM from the system, there
> is no way for the OS to reclaim that memory until the application's
> process is terminated. ...

Perhaps I should be a bit clearer.  I am willing to allow Lisp to keep the
memory indefinitely.  However, what I observe is that it seems to expand and
then *lose* the memory in question.  In other words, suppose I run program 1,
kill Lisp, and then run program 2, and observe that in each case Lisp expands
to 50 Mb.  Now if I run program 1 and then program 2 without killing Lisp, it
grows to 50Mb after program 1, and then *keeps growing* to accommodate program
2.  I have 128Mb of primary memory on this computer, so the system runs just
fine during program 1, but then starts thrashing during program 2, runs slower
and slower, and eventually the Lisp dies completely, occasionally bringing
down the entire computer, which has to be restarted.

So: I would really like to know what's hanging around after program 1
finishes, especially since I go out of my way to destroy every global
reference I can find to the huge data structures created during its execution.

   -- Drew McDermott


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
R. Toy  
View profile  
 More options Jan 5 1999, 3:00 am
Newsgroups: comp.lang.lisp
From: "R. Toy" <r...@mindspring.com>
Date: 1999/01/05
Subject: Re: Garbage hunting

Erik Naggum wrote:

> * "R. Toy" <r...@mindspring.com>
> | I think you are mistaken about CMUCL.  I've seen CMUCL grow and shrink in
> | size as an application runs, at least ps and top say so.  Both the total
> | size and the RSS change.

>   this may be misleading.  under Linux, the VM size of the Allegro CL
>   process does not decrease according to /proc/self/status even when the
>   working set is greatly reduced by using SYSTEM:RESIZE-AREAS.  I was a
>   little surprised by this, since ps and top and other tools report a
>   decrease.  I guess it's time to trust the kernel and not various "smart"
>   tools that try to provide user-friendly information.

Hmm, I wasn't aware of this "feature" of Linux.  However, I think the
same (decreased memory usage according to ps and top) occurs on Solaris
for CMUCL.
Maybe Solaris version of ps and top has the same "feature"?

Ray

--
---------------------------------------------------------------------------
----> Raymond Toy    r...@mindspring.com
                        http://www.mindspring.com/~rtoy


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Sunil Mishra  
View profile  
 More options Jan 5 1999, 3:00 am
Newsgroups: comp.lang.lisp
From: Sunil Mishra <smis...@whizzy.cc.gatech.edu>
Date: 1999/01/05
Subject: Re: Garbage hunting

AFAIK, things that will hang around:

1. Code. This turned out to be a fairly non-trivial problem in an old
rule-based system I was working with. Each time I compiled the rule file,
each rule would expand into I think four new functions.

2. Symbols.

3. Constants?

One thing worth trying might be to load up program 1, check it's usage, run
program 1, and check it's usage again, just to get a feel for how much of
the heap program 1 has eaten up. If after releasing all your allocated
storage and manually invoking a GC you end up with higher usage, you really
do have a problem. You might consider getting in touch with Harlequin
support, they have been very helpful to me.

Sunil


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
David Gadbois  
View profile  
 More options Jan 5 1999, 3:00 am
Newsgroups: comp.lang.lisp
From: David Gadbois <gadb...@lagavulin.cyc.com>
Date: 1999/01/05
Subject: Re: Garbage hunting

Erik Naggum <e...@naggum.no> writes:
>   this may be misleading.  under Linux, the VM size of the Allegro CL
>   process does not decrease according to /proc/self/status even when the
>   working set is greatly reduced by using SYSTEM:RESIZE-AREAS.

Note that the Linux VmSize measures all mapped address space
regardless of whether a page exists to back it.  One trick in Linux to
avoid races with other pieces of code that are allocating address
space is to pre-map a VM arena and allocate out of that.  Deallocation
consists of creating a new mapping on top of the address space you
want to free up.  That way, other code doing non-fixed mappings can't
get its mitts on your address space in between figuring out where to
put it and actually grabbing it.  The unused parts of the arena jack
up VmSize but doesn't use any resources other than page table entries.
I don't know if Allegro does that or something like it, but it shows
that VmSize is not a good measure of "memory" usage.

--David Gadbois


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Christopher R. Barry  
View profile  
 More options Jan 5 1999, 3:00 am
Newsgroups: comp.lang.lisp
From: cba...@2xtreme.net (Christopher R. Barry)
Date: 1999/01/05
Subject: Re: Garbage hunting

Heh, well, I do remember hearing some figure from Microsoft that
Windows NT 5.0 will fix something like 160 memory leaks and 200
crashing bugs in Windows NT 4.0. :-)

Christopher


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Erik Naggum  
View profile  
 More options Jan 5 1999, 3:00 am
Newsgroups: comp.lang.lisp
From: Erik Naggum <e...@naggum.no>
Date: 1999/01/05
Subject: Re: Garbage hunting
* David Gadbois <gadb...@lagavulin.cyc.com>
| I don't know if Allegro does that or something like it, but it shows
| that VmSize is not a good measure of "memory" usage.

  thanks.  now I don't know what would be a good measure, anymore.  do you
  have any suggestions?

#:Erik


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Duane Rettig  
View profile  
 More options Jan 5 1999, 3:00 am
Newsgroups: comp.lang.lisp
From: Duane Rettig <du...@franz.com>
Date: 1999/01/05
Subject: Re: Garbage hunting

Erik Naggum <e...@naggum.no> writes:
> * David Gadbois <gadb...@lagavulin.cyc.com>
> | I don't know if Allegro does that or something like it, but it shows
> | that VmSize is not a good measure of "memory" usage.

>   thanks.  now I don't know what would be a good measure, anymore.  do you
>   have any suggestions?

Q: How do you tell if a toadstool is poisonous?

A: Eat it; if you died, it was poisonous.

Sorry for the tasteless joke, but it does illustrate one way to
measure the measuring tools.  Some of our customers that need
multi-hundred-megabyte processes and which do not trust the tools
that measure memory usage check these tools out by pushing the
limits of their systems.  Which tools to trust starts to become
clear as more experience with the failures (actually the
 _exact_ _point_ of the failures) is seen.  Sometimes the point-of-failure
experiments involve only swap space, so the experiments are relatively
easy.  Other times, the experiments involve virtual address availability,
which is sometimes much harder unless your system is beefed up with
ram and/or swap on the order of gigabytes.

--
Duane Rettig          Franz Inc.            http://www.franz.com/ (www)
1995 University Ave Suite 275  Berkeley, CA 94704
Phone: (510) 548-3600; FAX: (510) 548-8253   du...@Franz.COM (internet)


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
David Gadbois  
View profile  
 More options Jan 5 1999, 3:00 am
Newsgroups: comp.lang.lisp
From: David Gadbois <gadb...@lagavulin.cyc.com>
Date: 1999/01/05
Subject: Re: Garbage hunting

Erik Naggum <e...@naggum.no> writes:
> * David Gadbois <gadb...@lagavulin.cyc.com>
> | I don't know if Allegro does that or something like it, but it
> | shows that VmSize is not a good measure of "memory" usage.

>   thanks.  now I don't know what would be a good measure, anymore.
>   do you have any suggestions?

Alas, there does not seem to be a general way of doing this in Linux.
I use ROOM and Lisp implementation-specific elaborations thereof.

One kernel hacking project I have been thinking about is adding some
way of getting information on what pages have been dirtied at some
time since their initialization.  Such a facility, with a suitable
procfs interface, would be just the ticket for get a better handle on
memory usage.  Even better, it would make for a really cheap write
barrier for a generational garbage collector:  The VM hardware already
sets dirty bits, and the kernel just has to save the info somewhere
when it clears the bits.  It would be much superior to other write
barrier implementations like trapping SIGSEGVs, checking address
bounds, or blinding marking a card.

Anyone have any advice on how to go about such a project?

--David Gadbois


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Harley Davis  
View profile  
 More options Jan 5 1999, 3:00 am
Newsgroups: comp.lang.lisp
From: "Harley Davis" <spamless_davis@spamless_ilog.com>
Date: 1999/01/05
Subject: Re: Garbage hunting

>* David Gadbois <gadb...@lagavulin.cyc.com>
>| I don't know if Allegro does that or something like it, but it shows
>| that VmSize is not a good measure of "memory" usage.

>  thanks.  now I don't know what would be a good measure, anymore.  do you
>  have any suggestions?

When you start hearing the disk drive whir, you've allocated too much.

-- Harley

PS Just another reason why thin clients are a bad idea!


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Matthias Buelow  
View profile  
 More options Jan 6 1999, 3:00 am
Newsgroups: comp.lang.lisp
From: to...@cip.informatik.uni-wuerzburg.de (Matthias Buelow)
Date: 1999/01/06
Subject: Re: Garbage hunting
In article <87n23y4dv6....@2xtreme.net>
cba...@2xtreme.net (Christopher R. Barry) writes:

>Neither ACL5 nor CMUCL have such a capability, nor does (most
>unfortunately) Netscape. I would never, ever restart any of these
>applications if they were capable of letting the OS reclaim unused
>memory, because Netscape can swell to 140MB after a few days and then
>it's back down to 25MB after a restart.

The problem with Netscape in particular is not that it cannot give
back memory that was once allocated due and is no more due to the
system's design but because it is horribly buggy and infested with
memory leaks.  Garbage collection would certainly pay here. :)

On many systems, the malloc(3) memory allocation facility, a C library
function which is usually used by applications, X Window and X toolkits
(XtAlloc), has the ability to return memory to the system by lowering
the brk (traditionally the top of a process' data segment under Unix,
without mmapped stuff), if a contiguous block just under the brk is free,
and indeed they do so often (examples are the malloc implementations of
Digital Unix, Free/OpenBSD, Gnu/Linux and probably others).
It is true however, that traditional Unix malloc implementations have
not been able to do that (both for speed and pseudo-philosophical reasons
like "a program should always be able to get memory it once had
allocated" etc.)

With ACL5 or CMUCL I would suspect that they do not use malloc at the
lower levels but have their own brk-pusher under the hood, which may
or may not implement lowering the brk if there's enough room free at
the top of the heap.

--
  Matthias K. Buelow      * Long live the Thing King! *


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Erik Naggum  
View profile  
 More options Jan 6 1999, 3:00 am
Newsgroups: comp.lang.lisp
From: Erik Naggum <e...@naggum.no>
Date: 1999/01/06
Subject: Re: Garbage hunting
* "Harley Davis" <spamless_davis@spamless_ilog.com>
| When you start hearing the disk drive whir, you've allocated too much.

  well, with 512M RAM, no swap space, and the machine in a separate room,
  if I start hearing the disks, I may well be in serious trouble, but it
  won't be because of memory allocation.

#:Erik


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Peter Van Eynde  
View profile  
 More options Jan 6 1999, 3:00 am
Newsgroups: comp.lang.lisp
From: pvane...@mail.inthan.be (Peter Van Eynde)
Date: 1999/01/06
Subject: Re: Garbage hunting
On 06 Jan 1999 05:55:23 +0100, Matthias Buelow wrote:

>With ACL5 or CMUCL I would suspect that they do not use malloc at the
>lower levels but have their own brk-pusher under the hood, which may
>or may not implement lowering the brk if there's enough room free at
>the top of the heap.

CMUCL at least uses mmap to get memory and to load the core file, and we
use munmap to release it, for example after doing a GC we free the
oldspace. This is clearly visible if you use xosview to watch the memory
consumption.

Groetjes, Peter

--
It's logic Jim, but not as we know it. | pvane...@debian.org for pleasure,
"God, root, what is difference?",Pitr  | pvane...@inthan.be for more pleasure!


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Barry Margolin  
View profile  
 More options Jan 6 1999, 3:00 am
Newsgroups: comp.lang.lisp
From: Barry Margolin <bar...@bbnplanet.com>
Date: 1999/01/06
Subject: Re: Garbage hunting
In article <ye067alvw3o....@utopia.informatik.uni-wuerzburg.de>,

Matthias Buelow <to...@cip.informatik.uni-wuerzburg.de> wrote:
>It is true however, that traditional Unix malloc implementations have
>not been able to do that (both for speed and pseudo-philosophical reasons
>like "a program should always be able to get memory it once had
>allocated" etc.)

In fact, I believe that early free() man pages specifically said that you
could safely access the freed memory until the next call to malloc() or
realloc().

>With ACL5 or CMUCL I would suspect that they do not use malloc at the
>lower levels but have their own brk-pusher under the hood, which may
>or may not implement lowering the brk if there's enough room free at
>the top of the heap.

Actually, I suspect that Unix Lisp implementations that are able to return
memory don't use brk() or malloc() at all to implement Lisp heap.  They
probably use mmap() with /dev/zero or shmget(), as these allow random
chunks of address space to be allocated and freed.  They work especially
well with generational copying GC's -- when you move everything from
oldspace to copyspace, you can then release the memory segment that
contained oldspace.  The brk() mechanism is especially *unsuited* for this,
since it will be pretty rare that the space you've just emptied happened to
be the one adjacent to the break.

--
Barry Margolin, bar...@bbnplanet.com
GTE Internetworking, Powered by BBN, Burlington, MA
*** DON'T SEND TECHNICAL QUESTIONS DIRECTLY TO ME, post them to newsgroups.
Don't bother cc'ing followups to me.


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Raymond Toy  
View profile  
 More options Jan 6 1999, 3:00 am
Newsgroups: comp.lang.lisp
From: Raymond Toy <t...@rtp.ericsson.se>
Date: 1999/01/06
Subject: Re: Garbage hunting

>>>>> "Christopher" == Christopher R Barry <cba...@2xtreme.net> writes:

    Christopher> After playing around a little, I can see that it does this a little bit,
    Christopher> but I guess I had never noticed it before because it could be a lot
    Christopher> better. Here are the numbers:

    Christopher> Fresh launch of CMUCL:

    Christopher>                         SIZE  RSS SHARE
    Christopher>                         ____  ___ _____
    Christopher>  9928 cbarry     0   0  8108 8108  4360 S    1104  0.0 12.7   0:00 lisp

    Christopher> After doing a (fact 15000):

    Christopher>  9928 cbarry     0   0 90988  42M  2616 S     14M  0.0 69.3   0:23 lisp

    Christopher> A global GC after that:

    Christopher>  9928 cbarry     0   0 75868  27M  2876 S       0  0.0 44.1   0:23 lisp

    Christopher> And basically, CMUCL will never use less memory than this for the
    Christopher> duration of it's usage. I have a WindowMaker dock app that shows me my

This is interesting.  Is this Linux running 18a?

On my sparc, top shows a size and RSS of about 23M and 8.9M at start
up.  After running (fact 15000) for a while (I stopped it after
sucking about 100M of memory), the size and RSS had grown to about
100M each.  After (gc), the size and RES are 27M and 7480K.  This is a
post 18b binary on Sparc.  Perhaps the conservative generational GC on
Linux is leaking some memory?

Ray


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Messages 1 - 25 of 41   Newer >
« Back to Discussions « Newer topic     Older topic »