How to keep track of objects?
Josef Wolf
I am trying to keep track of all created objects of a set of CLOS classes. I
thought of a mixin like that:
(defparameter *tracked-objects* ())
(defclass track-mixin ()
())
(defmethod initialize-instance :after ((tm track-mixin))
(push tm *tracked-objects*))
This works fine so far. The only problem is: the objects are no longer GCed,
since they are referenced from *TRACKED-OBJECTS*. But I'd rather keep the
semantic of objects disappearing when they get out of scope.
Any suggestions?
Pascal J. Bourguignon
Use a weak data structure instead.
Unfortunately, weak data structures are not standardized.
Happily, there's a portability library out there:
(Well, only for clisp, sbcl and cmucl so far).
--
__Pascal Bourguignon__ http://www.informatimago.com/
A bad day in () is better than a good day in {}.
Josef Wolf
> Josef Wolf <j...@raven.inka.de> writes:
>> I am trying to keep track of all created objects of a set of CLOS classes. I
>> thought of a mixin like that:
>>
>> (defparameter *tracked-objects* ())
>>
>> (defclass track-mixin ()
>> ())
>>
>> (defmethod initialize-instance :after ((tm track-mixin))
>> (push tm *tracked-objects*))
>>
>> This works fine so far. The only problem is: the objects are no longer GCed,
>> since they are referenced from *TRACKED-OBJECTS*. But I'd rather keep the
>> semantic of objects disappearing when they get out of scope.
>>
>> Any suggestions?
>
> Use a weak data structure instead.
>
> Unfortunately, weak data structures are not standardized.
>
> Happily, there's a portability library out there:
>
> http://git.informatimago.com/viewgit/index.php?a=tree&p=public/lisp&h=5ce8bc5d2d914fb0f0888de13f10ad8e16c8cbef&hb=7c3d4719bbaa534b42c8882a17c7dc983acfb7ab&f=clext
Thanks! Really impressing, that!
But now that I've browsed through (and learned about) weak data structures, I
realized that my description of the problem was wrong.
What I _really_ need is that the object gets removed from *TRACKED-OBJECTS*
list as soon as it looses all references from the user.
I guess I need to give a more verbose explanation here. The required
functionality is two-fold here.
1. As long as the object exists in the world of the user, other parts of the
program should be able to query and do operations on the object.
2. As soon as there are no more references left in the world of the user,
the query should no longer allow the user to bring the object back to
live.
The problem is with item 2. While weak data structures _allow_ the object to
get GCed, there don't seem to exist a mechanism to _force_ removal from the
weak data structure (whether GCed or not) when it is the only reference left.
If you think of the tagOrId concept in Tk/canvas, that comes pretty close to
what I am trying to do.
Raffael Cavallaro
> 1. As long as the object exists in the world of the user, other parts of the
> program should be able to query and do operations on the object.
> 2. As soon as there are no more references left in the world of the user,
> the query should no longer allow the user to bring the object back to
> live.
See if your implementaton has some sort of object finalization hook -
most (maybe all?) do. This is usually called when the object is about
to be collected by the gc, though in some implementations it is called
*after* the object has already been collected. In your finalizer you
would remove the object from *tracked-objects*.
ccl - ccl::terminate-when-unreachable, ccl::terminate
lispworks - hcl::flag-special-free-action, hcl::add-special-free-action, etc.
sbcl - sb-ext::finalize
etc.
For an example of a cross-platform library that uses the finalization
facilities of a number of different implementations see trivial garbage:
<http://common-lisp.net/~loliveira/darcs/trivial-garbage/trivial-garbage.lisp>
warmest regards,
Ralph
--
Raffael Cavallaro
Raffael Cavallaro
> In your finalizer you would remove the object from *tracked-objects*.
Sorry - ignore this - it makes no sense at all - the objects would
obviously never be collected as long as they're in *tracked-objects*.
You want a weak hash table as Pascal B. suggests. BTW, the
trivial-garbage library I mentioned is a largely portable
weak-hash-table library.
Teach me to post when I have the flu...
Pascal J. Bourguignon
> On 2011-04-13, Pascal J. Bourguignon <p...@informatimago.com> wrote:
>> Josef Wolf <j...@raven.inka.de> writes:
>>> I am trying to keep track of all created objects of a set of CLOS classes. I
>>> thought of a mixin like that:
> [...]
> But now that I've browsed through (and learned about) weak data structures, I
> realized that my description of the problem was wrong.
>
> What I _really_ need is that the object gets removed from *TRACKED-OBJECTS*
> list as soon as it looses all references from the user.
>
> I guess I need to give a more verbose explanation here. The required
> functionality is two-fold here.
>
> 1. As long as the object exists in the world of the user, other parts of the
> program should be able to query and do operations on the object.
> 2. As soon as there are no more references left in the world of the user,
> the query should no longer allow the user to bring the object back to
> live.
>
> The problem is with item 2. While weak data structures _allow_ the object to
> get GCed, there don't seem to exist a mechanism to _force_ removal from the
> weak data structure (whether GCed or not) when it is the only reference left.
But this is not what you require. You require that the object
disappear. It does. It doesn't matter if a weak-pointer or something
else is still there. There won't be any reference to the object, even
from the weak-pointer.
So the only thing you have to do is to clean the list from time to time.
(defvar *object* '() "List of weak pointers.")
(defun clean-list-of-objects ()
(setf *objects* (remove-if-not (lambda (wp)
(nth-value (weak-pointer-value wp) 1))
*objects*)))
Notice that:
1- there are not two functions, to get the value and whether the weak
pointer is still valid, because between a call to test the validity
and a call to get the value, the validity could change and no value
may be available.
2- therefore weak-pointer-value returns the two values at once.
3- for the same reason, you must always check the validity, when you
need to process the objects, and you cannot rely on the removal by
clean-list-of-objects.
4- therefore an alternative is even not to use such a
clean-list-of-object function, but to remove the invalid weak pointer
when you process them:
(defun map-objects (fun)
(setf *objects*
(let ((wps (cons nil *objects*))
(*objects* '())) ; if fun allocates new objects, they'll be
; collected here.
(loop
:for wps-cell = wps :then (cdr wps-cell)
:for wp = (cadr wps-cell)
:while (cdr wps-cell)
:do (multiple-value-bind (object valid) (weak-pointer-value wp)
(if valid
(funcall fun object)
(setf (cdr wps-cell) (cddr wps-cell)))))
(append *objects* (cdr wps)))))
Tim Bradshaw
> 1. As long as the object exists in the world of the user, other parts of the
> program should be able to query and do operations on the object.
> 2. As soon as there are no more references left in the world of the user,
> the query should no longer allow the user to bring the object back to
> live.
If you want the object to *promptly* disappear, then almost certainly
you need a mechanism for saying, explicitly, that the object is gone.
GC does not do that: it just discovers at some later point that there
are no references to the object. "Some later point" might even be be
"essentially never" (for instance for a generational GC if the object
moved out of the ephemeral generations and a full GC is never run).
I'm presuming you have such a mechanism, since otherwise how could you
say whether the object still exists in the world of the user.
So the trick is then to arrange life so that when that mechanism is
invoked - when you tell the object it is no longer in use - it in turn
informs the various things that have a hold on it to release it.
Tim Bradshaw
> 1. As long as the object exists in the world of the user, other parts of the
> program should be able to query and do operations on the object.
> 2. As soon as there are no more references left in the world of the user,
> the query should no longer allow the user to bring the object back to
> live.
If you want the object to *promptly* disappear, then almost certainly
Josef Wolf
> On 2011-04-13 20:57:26 +0100, Josef Wolf said:
>> 1. As long as the object exists in the world of the user, other parts of the
>> program should be able to query and do operations on the object.
>> 2. As soon as there are no more references left in the world of the user,
>> the query should no longer allow the user to bring the object back to
>> live.
>
> If you want the object to *promptly* disappear, then almost certainly
> you need a mechanism for saying, explicitly, that the object is gone.
> GC does not do that: it just discovers at some later point that there
> are no references to the object. "Some later point" might even be be
> "essentially never" (for instance for a generational GC if the object
> moved out of the ephemeral generations and a full GC is never run).
Yeah, that's exactly the problem.
> I'm presuming you have such a mechanism, since otherwise how could you
> say whether the object still exists in the world of the user.
No, I don't have such a mechanism. I just noticed the problem when I was
trying to implement it. I attached a more verbose example below of how I
thought it should work. I thought there might be some kind of reference
count which could be queried to find out whether there's only one reference
left. Or maybe classes could be queried for "living" objects, or something.
Here's the example:
(defparameter *tagged-objects* ())
(defclass tags-mixin ()
((tags :initarg :tags :accessor tags :initform ())))
(defmethod initialize-instance :after ((tm tags-mixin) &key tags)
(when tags
(setf (tags tm) tags)
(push tm *tagged-objects*)))
(defun find-objects-with-tag (tag)
(remove-if-not #'(lambda (item)
(member tag (tags item)))
*tagged-objects*))
(defclass foo (tags-mixin)
((name :accessor name :initarg :name)))
(defclass bar (tags-mixin)
((name :accessor name :initarg :name)))
(defun make-foo (name &key tags)
(declare (optimize (debug 3) (safety 3)))
(make-instance 'foo :name name :tags tags))
(defun make-bar (name &key tags)
(make-instance 'bar :name name :tags tags))
(defun find-object-names (tag)
(mapcar #'(lambda (o) (name o))
(find-objects-with-tag tag)))
(defparameter *some-foo* (make-foo 'a :tags '(hoho hehe haha)))
(defparameter *some-bar* (make-bar 'b :tags '( hehe haha hihi)))
(let ((other-foo (make-foo 'c :tags '(hoho hehe)))
(other-bar (make-bar 'd :tags '(hoho haha))))
(find-object-names 'hoho)) ; this should return '(D C A)
(find-objects-names 'hoho) ; this should return '(A)
(setf *some-foo* nil)
(find-objects-names 'hoho) ; this should return '()
The FIND-OBJECTS-WITH-TAG within the LET form should find the objects named
A, C and D. But the second FIND-OBJECTS-WITH-TAG should no longer find C
and D. They should disappear from the *TAGGED-OBJECTS* list at the moment
when the last (user visible) reference to them is lost. And the third call
should not find anything, since the only living object (B) doesn't have
the HOHO tag.
Tim Bradshaw
> No, I don't have such a mechanism. I just noticed the problem when I was
> trying to implement it. I attached a more verbose example below of how I
> thought it should work. I thought there might be some kind of reference
> count which could be queried to find out whether there's only one reference
> left. Or maybe classes could be queried for "living" objects, or something.
No, none of that happens in GCd systems. Reference counted systems can
do this but have problems of their own.
Luís Oliveira
> Happily, there's a portability library out there:
>
> http://git.informatimago.com/viewgit/index.php?a=tree&p=public/lisp&h=5ce8bc5d2d914fb0f0888de13f10ad8e16c8cbef&hb=7c3d4719bbaa534b42c8882a17c7dc983acfb7ab&f=clext
>
> (Well, only for clisp, sbcl and cmucl so far).
Trivial-garbage[1] is another such library. It's been ported to SBCL,
CMUCL, SCL, CLISP, ECL, Allegro, OpenMCL, Corman Lisp, and Lispworks.
Cheers,
[1] http://www.cliki.net/trivial-garbage
--
Luís Oliveira
http://r42.eu/~luis
Pascal J. Bourguignon
> (defparameter *some-foo* (make-foo 'a :tags '(hoho hehe haha)))
> (defparameter *some-bar* (make-bar 'b :tags '( hehe haha hihi)))
> (let ((other-foo (make-foo 'c :tags '(hoho hehe)))
> (other-bar (make-bar 'd :tags '(hoho haha))))
> (find-object-names 'hoho)) ; this should return '(D C A)
> (find-objects-names 'hoho) ; this should return '(A)
There's no reason why it should return just A, because when we reach
here, we have proof there is enough memory to hold both D, C, and A.
Why should we lose time trying to reclaim D and C just right now???
> (setf *some-foo* nil)
> (find-objects-names 'hoho) ; this should return '()
Same as above, nothing changed.
The lisp way to do it is as follow:
;;;; The tagged objects mechanism:
(defgeneric tags (object)
(:documentation "Accessor of tags, a list of symbols."))
;; Reloading a file doesn't remove the instances, so we keep the old
;; tagged objects if there is any. --> defvar
(defvar *tagged-objects* '())
(defun find-objects-with-tag (tag)
(remove-if-not (lambda (item) (member tag (tags item)))
*tagged-objects*))
(defun find-object-names (tag)
(mapcar (function name) (find-objects-with-tag tag)))
(defmacro with-tagged-objects (bindings &body body)
(assert (lambda (binding)
(and (= 2 (length binding))
(symbolp (first binding)))))
`(let ,bindings
(setf *tagged-objects* (list* ,@(mapcar (function first) bindings)
*tagged-objects*))
(unwind-protect
(progn ,@body)
(setf *tagged-objects*
,(loop ;; we don't assume much about *tagged-objects*,
;; new objects may have been prepended, the order may have changed,
;; the same objects may have been added twice, etc. Hence this form.
:for form = '*tagged-objects* :then `(delete ,var ,form :count 1)
:for (var nil) :in bindings
:finally (return form))))))
;;;; Let's make some tagged objects:
(defclass tags-mixin ()
((tags :initarg :tags :accessor tags :initform ())))
(defmethod initialize-instance :after ((tm tags-mixin) &key tags)
(when tags
(setf (tags tm) tags)))
(defclass foo (tags-mixin)
((name :accessor name :initarg :name)))
(defclass bar (tags-mixin)
((name :accessor name :initarg :name)))
(defun make-foo (name &key tags)
(make-instance 'foo :name name :tags tags))
(defun make-bar (name &key tags)
(make-instance 'bar :name name :tags tags))
;;;; How it's used:
(defparameter *some-foo* (make-foo 'a :tags '(hoho hehe haha)))
(defparameter *some-bar* (make-bar 'b :tags '( hehe haha hihi)))
(with-tagged-objects ((*some-foo* *some-foo*))
(with-tagged-objects ((other-foo (make-foo 'c :tags '(hoho hehe)))
(other-bar (make-bar 'd :tags '(hoho haha))))
(print (find-object-names 'hoho))) ; this should return '(D C A)
(print (find-object-names 'hoho))) ; this should return '(A)
(print (find-object-names 'hoho)) ; this should return '()
;; prints:
(C D A)
(A)
NIL
Josef Wolf
> Josef Wolf <j...@raven.inka.de> writes:
>
>> (defparameter *some-foo* (make-foo 'a :tags '(hoho hehe haha)))
>> (defparameter *some-bar* (make-bar 'b :tags '( hehe haha hihi)))
>> (let ((other-foo (make-foo 'c :tags '(hoho hehe)))
>> (other-bar (make-bar 'd :tags '(hoho haha))))
>> (find-object-names 'hoho)) ; this should return '(D C A)
>> (find-objects-names 'hoho) ; this should return '(A)
>
> There's no reason why it should return just A, because when we reach
> here, we have proof there is enough memory to hold both D, C, and A.
>
> Why should we lose time trying to reclaim D and C just right now???
>
>> (setf *some-foo* nil)
>> (find-objects-names 'hoho) ; this should return '()
>
> Same as above, nothing changed.
>
>
> The lisp way to do it is as follow:
[ ... ]
> (defmacro with-tagged-objects (bindings &body body)
> (assert (lambda (binding)
> (and (= 2 (length binding))
> (symbolp (first binding)))))
Hmm, I don't understand this ASSERT. Maybe you've forgotten a MAPCAR around
it or something?
Anyway, after studying your with-tagged-objects solution and thinking once
more about what I am trying to achieve, I am now convinced that Tim Bradshaw
was pretty much right: I want to explicitly state the end-of-life of such an
object.
The whole point of this tags concept is to "fire and forget". In other words:
instantiate all the (various) objects at startup time (e.g. while reading the
configuration) and forget about them. Later, in any part of the program, the
objects can be referenced by their tags. That would of course not work if the
object would be destroyed automatically when it gets out of scope.
Pascal J. Bourguignon
> On 2011-04-14, Pascal J. Bourguignon <p...@informatimago.com> wrote:
>> Josef Wolf <j...@raven.inka.de> writes:
>>
>>> (defparameter *some-foo* (make-foo 'a :tags '(hoho hehe haha)))
>>> (defparameter *some-bar* (make-bar 'b :tags '( hehe haha hihi)))
>>> (let ((other-foo (make-foo 'c :tags '(hoho hehe)))
>>> (other-bar (make-bar 'd :tags '(hoho haha))))
>>> (find-object-names 'hoho)) ; this should return '(D C A)
>>> (find-objects-names 'hoho) ; this should return '(A)
>>
>> There's no reason why it should return just A, because when we reach
>> here, we have proof there is enough memory to hold both D, C, and A.
>>
>> Why should we lose time trying to reclaim D and C just right now???
>>
>>> (setf *some-foo* nil)
>>> (find-objects-names 'hoho) ; this should return '()
>>
>> Same as above, nothing changed.
>>
>>
>> The lisp way to do it is as follow:
>
> [ ... ]
>
>> (defmacro with-tagged-objects (bindings &body body)
>> (assert (lambda (binding)
>> (and (= 2 (length binding))
>> (symbolp (first binding)))))
>
> Hmm, I don't understand this ASSERT. Maybe you've forgotten a MAPCAR around
> it or something?
Definitely! :-)
(assert (mapcar (lambda (binding)
(and (= 2 (length binding))
(symbolp (first binding))))
bindings))
was intended.
> Anyway, after studying your with-tagged-objects solution and thinking once
> more about what I am trying to achieve, I am now convinced that Tim Bradshaw
> was pretty much right: I want to explicitly state the end-of-life of such an
> object.
>
> The whole point of this tags concept is to "fire and forget". In other words:
> instantiate all the (various) objects at startup time (e.g. while reading the
> configuration) and forget about them. Later, in any part of the program, the
> objects can be referenced by their tags. That would of course not work if the
> object would be destroyed automatically when it gets out of scope.
Ok.
Joshua Taylor
> Josef Wolf <j...@raven.inka.de> writes:
>> Hmm, I don't understand this ASSERT. Maybe you've forgotten a MAPCAR around
>> it or something?
>
> Definitely! :-)
>
>
> (assert (mapcar (lambda (binding)
> (and (= 2 (length binding))
> (symbolp (first binding))))
> bindings))
>
> was intended.
I think EVERY would be more useful than MAPCAR. The present assertion
just ensures that bindings isn't the empty list.
//JT
Pascal J. Bourguignon
Of course. I guess that's what you get to post last thing before sleep,
and to answer first thing after it.