[Sbcl-devel] question about mystery vectors using memory

[JTK]

Apologies in advance if this is a hopeless or pointless question. If the answer is "stop using an old version of SBCL" that's useful too.

I have a long running 1.0.42 64 bit SBCL gengc process on Linux (I'm using old SBCL because
I haven't upgraded my code for a newer version; there are matters of clsql compatibility, etc).

It runs for a long time, months, with several threads. At the end, there is just one thread "initial thread".

(ROOM) shows about 2.9 gigabytes of usage:

1,546,423,296 bytes for 1,688,136 simple-array-unsigned-byte-64 objects
1,027,264,928 bytes for 1,230,615 simple-vector-objects
…
2,928,858,816 bytes for 8,634,596 dynamic objects


I can't figure out what these vectors are. It's mostly unsigned-64 and generic vectors of length about 100.

So I deleted package after package, each time running (PROGN (GC :FULL T) (ROOM)). I'm left with nothing but
ASDF, the SB-, and GRAY-STREAM packages, and the dummy package I'm working in; CL-USER is gone. Yet the mystery
objects are still there, consuming 2.9 gigabytes of memory. I would have thought that at some point their top level
references would be gone along with the package, and they'd be deleted.

Does anyone have an idea what these could be?

Some detritus left after foreign function calls? Internal references to CLOS objects I don't know about? Just a leak in the GC?

Thanks for any tips,

John




------------------------------------------------------------------------------
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
Download AppDynamics Lite for free today:
http://p.sf.net/sfu/appdyn_sfd2d_oct
_______________________________________________
Sbcl-devel mailing list
Sbcl-...@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/sbcl-devel

[Nikodemus Siivola]

On 30 October 2012 09:37, JTK <jet...@gmail.com> wrote:

> So I deleted package after package, each time running (PROGN (GC :FULL T) (ROOM)). I'm left with nothing but
> ASDF, the SB-, and GRAY-STREAM packages, and the dummy package I'm working in; CL-USER is gone. Yet the mystery
> objects are still there, consuming 2.9 gigabytes of memory. I would have thought that at some point their top level
> references would be gone along with the package, and they'd be deleted.

Deleting a package is no sort of guarantee that eg. functions
accessible through symbols in that package can be collected, even if
there are no other user-visible references to them.

(It would be nice, yes.)

> Some detritus left after foreign function calls? Internal references to CLOS objects I don't know about? Just a leak in the GC?

My first guess would be CLOS instance vectors. Each CLOS object looks
roughly like this:

<pointer-to-class-wrapper><pointer-to-instance-vector>

where instance-vector is a SIMPLE-VECTOR that holds the slot-contents
of that object.

My second guess would be hash-tables (which also use SIMPLE-VECTORs
internally) or SBCL's info database, which holds those pesky
references to global functions and variables I alluded to earlier.

Method dispatch caches also uses SIMPLE_VECTORs internally.

It /is/ possibly to find out what they are. See eg.
SB-VM:MAP-ALLOCATED-OBJECTS. Both SB-VM (internal, unsupported) and
SB-INTROSPECT (supported) hold some tools for figuring out what refers
to what.

Cheers,

-- Nikodemus

[JTK]

Thank you for the tips, Nikodemus

To recap, I have an SBCL 1.0.42 process that is allocating about 2 million generic and unsigned-64 vectors of length
100, for about 2GB of mystery allocation. I have no idea what they are.

Following your tips, I looked through room.lisp for more undocumented functions that could be of use in probing the memory usage.

I managed to lobotomize my SBCL process so much trying to wipe the root object responsible for all my uncollectable vectors
(like deleting the info database) that I can't define any crawlers that use SB-VM:MAP-ALLOCATED-OBJECTS.

I also found that I have to be pretty careful using SB-VM:MAP-ALLOCATED-OBJECTS because it suspends GC and then
consing can crash SBCL.

Using (SB-VM::INSTANCE-USAGE :DYNAMIC), it seems that I have about 500K hash tables and 500K spinlocks and 500K
MYSQL-DATABASE objects. That's a number sort of close to the 2.5M unexplained vectors that also exist.

I never make spinlocks, just mutexes. Spinlocks are apparently deprecated after 1.0.42, so this might all be some quirk
or misuse of an outdated SBCL. CLSQL just uses SB-THREAD:MAKE-MUTEX

It's evident that something weird happened with threading and open databases, and I don't know where the uncollectable root is.
I'd be inclined to guess that a bunch of threads died, and didn't release some spinlocks,
which are someone pinning down some objects against GC. Is this a silly hypothesis?

For future reference, if this happens again, is the following a complete enumeration of roots where a mass of un-GC'able objects might reside:

1. conservative roots in the stack

2. user toplevel variables including *,**,etc, and functions.

3. another active thread (none, if no other threads)

4. objects referenced in a closure

5. sb-c::*info-environment* - can point to functions that are in deleted packages.

6. NOT global vars in PCL/CLOS because CLOS doesn't store pointers to instances

7. ??? Something weird with terminated threads not releasing locks …. the mechanics of which I don't get.

I realize that these are vague questions, so I can't expect precise answers, but if any blatant misconceptions jump out,
please let me know.

Thanks,
John Klein

[Attila Lendvai]

for the record, we have some hacks here:

http://dwim.hu/darcsweb/darcsweb.cgi?r=HEAD%20hu.dwim.debug;a=headblob;f=/source/path-to-root.lisp

it's a PATH-TO-ROOT function that tries to give you paths that hold
back a given object from getting GC'd.

not a super stable tool, and requires a good understanding of the heap
and the GC, but it serves us well when something is leaking memory...

it may be out of date, though. it was a couple of years ago when we
ran it the last time...

hth,

--
attila

Notice the erosion of your (digital) freedom, and do something about it!

PGP: 2FA1 A9DC 9C1E BA25 A59C 963F 5D5F 45C7 DFCD 0A39
OTR XMPP: 8647EEAC EA30FEEF E1B55146 573E52EE 21B1FF06