Tracing Open References To Dested Objects Preventing GC

[Josh Lang]

We appear to have an issue when we have longer reboots that some items that are destructed aren't being fully released, even by a GC, so our total object count goes sky high after ~10-14 days and we start to bog down the CPU.

We're on an older (ancient?) driver, 3.2.15 (in the process of migrating to 3.6.8 on a test port however).  Looking at /DEST_OBJ_DUMP tells me there's items with open references, but given that find_object() can no longer find any of them since they're already destructed, is there anyway to glean what those open references preventing GC from happening actually are? That's presuming I'm understanding the DEST_OBJ_DUMP correctly...

For instance:
players/necs/items/undead_spirit#14396849 ref  5

I believe that would be the file_name() and then indicating there's 5 current references to the object, even though its destructed and find_object(" players/necs/items/undead_spirit#14396849") returns 0, but I have no clue how to figure out what those 5 references are/could be?

It doesn't appear to be an issue on the 3.6.8 port as I can't replicate it there, but since our time table for migration isn't fully known yet, figured I'd ask if there's a good way for me to troubleshoot this on our outdated live version...

[Zesstra]

Hej Josh,

I don't remember any in-game way of accessing and investigating destructed
objects at least not in 3.2.15. Although you could attach a debugger and
investate data structures with it, it is tedious work if you are not
experienced with the internals of the driver.
Your reading of the destructed object dump is correct BTW (file_name() being
now object_name()).

Such issues could be caused by errors in the ref-counting, the
clean-up/garbage collector or the destructed objects may even be indeed not
free'able (no error involved).

You may check the changelogs between your version and the 3.6.8 to identify
possible fixes for your problem. With some luck a fix earlier in the
development history (or a work-around) might be found. However, I assume there
many candidates.

But another approach is to investigate which objects are cloned so often, how
they are produced and which data structures hold references to them. I suggest
to count those objects and classify per blueprint, see if you find a
similarity in usage.
Maybe also create a reduced test case for the issue to help isolate the cause.
It seems you create a lot of transient clones? Could they (in addition to your
observation with destructed objects) leak? Maybe you can give us some info
about these objects.

The driver replaces references to destructed objects (with time) with zeroes
thus removing the reference. Latest during the GC most destructed objects
should get freeable. However, there are exceptions. E.g. if you have a
destructed blueprint whose program is used by one or more clones it can only
be finally removed once all clones have beed removed. That means if you use an
object as blueprint, clone it, destruct the blueprint, load and clone it
again, destruct the blueprint etc. you end up with non-freeable blueprints as
long as the clones exist.
Also, the driver does not deal well with circular data structures (a structure
referencing itself (indirectly).

Besides some bugfixes, we also had a change how often the driver performs the
cleanup of data structures (e.g. removing references to destructed objects
from them) between your version and the 3.6.8. These may help in general, but
not with references still existing after GC runs. (see DC_DATA_CLEAN_TIME in
configure_driver() nowadays). It was not configurable in 3.2.15 and AFAIR the
driver only processed a certain percentage of the existing objects which could
lead to long times for data clean-up. (again, unlikely decisive in your case)

Best wishes,
Zesstra
--
MorgenGrauen -
1 Welt, mehr als 200 Programmierer , mehr als 16000 Raeume,
viel mehr als 7000 unterschiedliche Figuren, 90 Quests, 13 Gilden,
ueber 5000 Waffen und Ruestungen, keine Umlaute und ein Haufen Verrueckter.
Existenz: mehr als 25 Jahre
http://mg.mud.de/