Customization of evaluation for a specific custom type of object
Vassilis Radis
Suppose that I have:
(defclass reference () ((host) (port) (file)))
and also I have:
(defun fetch-remote-value (reference) ...) which fetches and deserializes a lisp object.
How could I intervene in the evaluation process so as whenever a reference object is being evaluated, the remote value gets fetched and re-evaluated again to produce the final result?
A more elaborate description of what I want to accomplish:
Using cl-store I serialize lisp objects and send them to a remote file(or db or anything) to be saved. Upon successful storage I keep the host,port and file in a reference object. I would like, whenever eval gets called on a reference object, to first retrieve the object, and then call eval on the retrieved value. Since a reference can be also serialized in other (parent) objects or aggregate types, I can get free on-demand recursive remote reference resolution by modyfing eval so i dont have to traverse and resolve the loaded object's child references myself which I suppose is a real PITA. And Of course there is the option of whether , after resolution, we should replace the value or not, thus having to resolve it once or every time respectively, but if we made it an option in the reference, it would be sweeet.
Pascal J. Bourguignon
For CL:EVAL all the lisp objects are self evaluating, apart from conses
(which are interpreted according to their car), and symbols (which are
either symbol-macroexpanded, or interpreted as variable references).
Thus: (eval '42) --> 42
(eval '"Hello?") --> "Hello?"
(let ((o (make-instance reference)))
(eval o)) --> #<REFERENCE>
(eval (function sin))> --> #<FUNCTION SIN>
There is no way to have CL:EVAL do anything else, this is what is is
specified to do.
In general, CL:EVAL is not called on lisp objects, because it is silly,
since it merely return the lisp object itself. On anything thats' not a
cons cell or a symbol, eval is identity.
So it would be useless to do anything when a reference object is
evaluated, because in normal programs, objects ARE NEVER EVALUATED!
On the other hand, other functions are often applied to lisp objects; in
particular, generic functions are often applied to CLOS objects.
So assuming you have a generic function M, with a method specialized on
a class C, and assuming you have a REFERENCE object R such as
(typep (fetch-remote-value r) 'C)
then you could write a method for the generic function M, specialized on
REFERENCE, such:
(defmethod m ((ref reference))
(m (fetch-remote-value r)))
and then when calling:
(m r)
the method of the generic function M, specialized on the class C would
be called with the object fetched.
If you accorded very little value to your reference objects, you could
even change their class on the fly:
(defmethod fetch ((ref reference))
(multiple-value-bind (class init-args) (fetch-remote-data r)
(apply (function change-class) ref class init-args)))
and implement the method M as:
(defmethod m ((ref reference))
(fetch ref)
(m ref))
so that next time you call (m r), this method is bypassed and the method
specialized on C is directly called.
On the other hand, notice how this means that you have either to call
FETCH on a reference before calling another function on it, or you have
to provide a reference fetching method for all the appliable generic
functions.
Since methods are not attached to classes in CLOS, but to generic
functions, it's not easy to do more.
One could play trick with metaclasses of generic functions, but
eg. AFAIUI, point 19 of 11.1.2.1.2 prevents you to redefine the method
specialized on T of the NO-APPLICABLE-METHOD generic function. You can
only add new methods specialized on non-standardized classes.
GENERIC-FUNCTION is a standardized class... That means that you will
have to add a (:generic-function-class ...) option to all the
concerned defgeneric forms (you do write defgeneric forms for all your
generic functions don't you?).
Without using the MOP, you could define a different defmethod macro that
you would use to define your methods specialized on classes like C, that
would automatically define a corresponding method specialized on
REFERENCE.
Oh, and of course, we didn't speak about normal functions at all... For
them, the only solution is to fetch explicitely.
--
__Pascal Bourguignon__ http://www.informatimago.com/
A bad day in () is better than a good day in {}.
Vassilis Radis
Can I intercept the evaluation of symbols so that when their value is an object of type XXX then instead of returning the object as the result of the symbol eval, to return the result of (fetch-remote-value object) ?
Paul Wallich
I think that this version still runs into the problem that Pascal was
alluding to: a Lisp newcomer's idea of "evaluation" does not correspond
to what CL:EVAL does. What you're actually asking is something more like
"can I write programs so that when I refer to something that has a
remote value, that remote value will get fetched/calculated and used?"
-- note no reference to "eval" or "evaluation".
The answer to that is yes along several axes. You can take a look at
"persistence" libraries for one axis, or at various ways of using
"accessors" to fetch/calculate/generate components of objects or
structures as needed.
paul
Vassilis Radis
Why is it conceptually wrong to think of intercepting symbol evaluation and just before returning the resulting value, examine it, and if it is of type REFERENCE, replace the returning value by the result of evaluating (fetch)?
Where is the conceptual mistake?
Also I took a look in the source code of eval in sbcl (src/code/eval.lisp) and basically
it is a big case that essentialy is structured like this:
(typecase exp
(symbol
(ecase (info :variable :kind exp)
((:special :global :constant :unknown)
(symbol-value exp))
(:alien
(%simple-eval original-exp lexenv))))
(list
(... ALOT OF CODE...))
(t
exp))
which means that there is point in trying to go after my second version question. Conceptually isnt it like injecting in the above code a conditional statement on the results of (symbol-value exp) or (%simple-eval original-exp lexenv) that will check for the type and call fetch?
Thanks a lot.
Pascal J. Bourguignon
> Thanks for such a quick answer. I will definitely try to understand
> what you wrote since I am new, but let me ask a this so as to
> understand better: Apart from the fact that we cant change cl:eval
> because that is how it is, why wouldnt it be reasonable or acceptable
> to be able to override (eval XXX) when XXX inherits from some type or
> more generally when XXX satisfies a predicate and make it do our
> stuff? Wouldnt it be like a kind of specializing eval on our type? Why
> doesnt it make sense? At least theoretically.
Theorically, and in practice, what you can do is to define your own
evaluation routine. Actually a lot of the functions defined in a
program can be seen as a special kind of eval. Any function that takes
some kind of data, and processes it in a special way.
CL:EVAL is not often used either in lisp programs, but it's important on
a definitionnal plane. CL:EVAL defines the semantics of a lisp
program. But the function that really implement those semantics is
COMPILE-FILE (and COMPILE, which is less often used, like EVAL).
So the point is that we don't modify the semantics of the base CL
language, but we routinely write little "interpreters" (functions), or
even little "compilers" (macros), allowing us to process specific data
in specific ways, up to the point we can consider some of these specific
data as code (domain specific code).
But then, we have to more or less explicitely invoke those interpreters
or comcpilers.
So, in practice, you can write:
(defun db-eval (expression)
(cond
((typep expression 'reference) (fetch-remote-value expression))
((symbolp expression) ...)
((atom expression) expression)
(t ...)))
re-implementing CL:EVAL (it takes on the order of one page of lisp (more
for CL:EVAL, since we have to implement 27 special operators, lexical
scope, exit-points, etc). At one time, this should be available as a
reusable library (eg. from sicl).
(db-eval '(let ((r (make-instance 'reference :host "h1" :port 4242 :file "f1")))
(do-something r)))
but notice that you cannot get out of the same problems as EVAL, notably
that you only have access in your expression to the NIL environment,
because the other environments are not provided by CL at run-time.
Or alternatively, you can write macros, which are hooks into the CL
compiler, to generate the code you want:
(let ((h1 "h")
(p1 4242)
(f1 "file"))
(with-db-references
(let ((r (make-instance 'reference :host h1 :port p1 :file f1)))
(do-something r))))
which could expand to something like:
(let ((h1 "h")
(p1 4242)
(f1 "file"))
(progn
(let ((r (make-instance 'reference :host h1 :port p1 :file f1)))
(do-something (fetch-remote-value r)))))
But this would require that you implement in with-db-references a code
walker (or use one code walker library, there are serveral). The
problem here is to identify when a variable holds a reference value.
You could add a type inference engine to your macro, but what will you
do with:
(let ((h1 "h")
(p1 4242)
(f1 "file"))
(progn
(let ((r (if (zerop (random 2))
(make-instance 'reference :host h1 :port p1 :file f1)
"Hi Mima!")))
(do-something r))))
The problem is that in lisp, variables don't matter. What matter is the
lisp objects. Type is an attribute of lisp objects. So program
decision depending on type, are defered most of the time till run-time.
Therefore in practice it's a lot of work for not much.
> 2nd question: You said that object evaluate to themselves, but what
> about bindings ? Can we intercept the evaluation of variables and
> change their value before evluating them? Like specializing
> symbol-value and returning the fetched object but what about lexical
> bindings... Comments?
But that's the point. Again, in lisp, variables are of little
importance. The objects exist and can be processed with no variable,
using the functional style.
For example, in:
(mapcar '1+ '(1 2 3)) --> (2 3 4)
there's no variable! With operators such as alexandria:compose,
alexandria:curry, etc, you can write whole programs without a single
variable.
On the other hand, if your question concerns variables, you may define a
symbol macro, which will look like a variable, and which will allow you
to implement the processing you want on any reference:
(macrolet ((my-object (reference :host h1 :port p1 :file f1))
(other-object (reference :host h2 :port p2 :file f2)))
(do-something my-object)
(setf my-object other-object)
(do-something my-object))
this will expand to something like:
(progn
(do-something (reference :host h1 :port p1 :file f1))
(setf (reference :host h1 :port p1 :file f1)
(reference :host h2 :port p2 :file f2))
(do-something (reference :host h1 :port p1 :file f1)))
so with a wise implementation of reference and (setf reference) you can
do what you want. Of course, you can hide these macrolets in your own
macro:
(with-db-objects ((my-object ...)
(other-object ...))
...)
But here, we don't process the objects, we hide function calls behind
something that looks like variable references.
Pascal J. Bourguignon
With symbol macrolets, you could write something like:
(defmacro with-db-objects ((&rest vars) &body body)
"VARS is a list of variable or binding, a binding is a list with a
variable name and an expression."
(let ((lets '())
(macrolets '()))
(loop :for binding :in vars
:collect (if (consp binding)
(let ((temp (gensym)))
(push (list temp (second binding)) lets)
(push (list (first binding) temp) macrolets))
(progn
(push (list binding binding) lets)
(push (list binding binding) macrolets))))
`(let ,(reverse lets)
(symbol-macrolet ,(mapcar (lambda (binding)
`(,(first binding) (deref ,(second binding))))
(reverse macrolets))
,@body))))
(macroexpand '(with-db-objects (o1
o2
(o3 (reference :host h3 :port p3 :file f3))
(o4 (reference :host h4 :port p4 :file f4)))
(do-something o1 o2 o3 o4)
(setf o1 o2)
(do-something o1 o3)))
-->
(LET ((O1 O1)
(O2 O2)
(#:G15846 (REFERENCE :HOST H3 :PORT P3 :FILE F3))
(#:G15847 (REFERENCE :HOST H4 :PORT P4 :FILE F4)))
(SYMBOL-MACROLET ((O1 (DEREF O1))
(O2 (DEREF O2))
(O3 (DEREF #:G15846))
(O4 (DEREF #:G15847)))
(DO-SOMETHING O1 O2 O3 O4)
(SETF O1 O2)
(DO-SOMETHING O1 O3)))
So all you have to do, is to say what it means to deref an object, both
in the case of reading it, and in the case of setf'ing it:
(defgeneric deref (object)
(:method ((ref reference))
(fetch-remote-value ref))
(:method ((object t))
object))
(defgeneric (setf deref) (new-value object)
(:method (new-value (ref reference))
???)
;; eg. when setting a reference to a reference, we may copy it:
(:method ((new-value reference) (ref reference))
(copy-reference ref new-value))
(:method (new-value (object t))
???))
Thomas A. Russ
> You are correct that persistence libraries might do what I need in a
> generic sense. But in order to actually gain something from this
> attempt of mine I would like to insist:
>
> Why is it conceptually wrong to think of intercepting symbol
> evaluation and just before returning the resulting value, examine it,
> and if it is of type REFERENCE, replace the returning value by the
> result of evaluating (fetch)?
>
> Where is the conceptual mistake?
The conceptual mistake is that there is no user-accessible way to
intercept that evaluation. Unlike generic functions, which can be
extended in standard ways by introducing new methods and :before, :after
and :around methods, you can't do that with ordinary functions. At
least not in any portable way.
Now, some lisp systems have an ADVICE feature, where you can in effect
add additional behavior to ordinary functions. So in those systems, you
could, perhaps, advise the CL:EVAL function (if they allow advice on CL
internal functions, that is) and intercept the return value.
But you don't actually have any guarantee that eval is actually called
at runtime by code. The compiler can, in effect, in-line the code for
evaluation into the compiled code, and then there would be no way to
intercept the function EVAL, because it is never invoked. And the
standard explicitly allows the compiler to assume that the Common Lisp
standard functions are not changed. They are what the standard says
they are, and so can be in-lined without restriction.
> Also I took a look in the source code of eval in sbcl (src/code/eval.lisp) and basically
> it is a big case that essentialy is structured like this:
>
> (typecase exp
> (symbol
> (ecase (info :variable :kind exp)
> ((:special :global :constant :unknown)
> (symbol-value exp))
> (:alien
> (%simple-eval original-exp lexenv))))
> (list
> (... ALOT OF CODE...))
> (t
> exp))
Yes, but with a compliant program you don't get to modify that code,
since that is part of the Common Lisp system. And even if you did
modify it, there isn't any guarantee that your modification will affect
compiled code unless you also modified the compiler.
The bottom line is that there is no provision for modifying the
evaluation semantics of Common Lisp, and so there is no portable way to
do so.
--
Thomas A. Russ, USC/Information Sciences Institute
Thomas A. Russ
> I want to implement a kind of reference system like this:
> Suppose that I have:
>
> (defclass reference () ((host) (port) (file)))
>
> and also I have:
>
> (defun fetch-remote-value (reference) ...) which fetches and
> deserializes a lisp object.
This should probably be a method rather than a regular function.
> How could I intervene in the evaluation process so as whenever a
> reference object is being evaluated, the remote value gets fetched and
> re-evaluated again to produce the final result?
You can't intervene in the evaulation process of CL:EVAL. There are no
hooks for doing so, as Pascal Bourguignon points out.
You would have to implement your own evaluator and use that if you want
to have different evaluation semantics. So that would be one option.
But perhaps you don't need to do that at all. Instead, what you have is
an object, and you normally don't care about the object itself, but
rather you care about accessing particular fields in the object. So
instead of looking at reconstituting the remote object when it is
encountered, you could do that instead at the time you need to get
something from the object.
So you would then need to crate the objects as being remotable, with
machinery that intercepts method calls and creates the real object. I'm
not sure if that is something that can be done with the MOP or not.
For starters, I would look at what some of the persistent lisp object
code does to handle the issue.
Pascal J. Bourguignon
> You are correct that persistence libraries might do what I need in a
> generic sense. But in order to actually gain something from this
> attempt of mine I would like to insist:
>
> Why is it conceptually wrong to think of intercepting symbol
> evaluation and just before returning the resulting value, examine it,
> and if it is of type REFERENCE, replace the returning value by the
> result of evaluating (fetch)?
>
> Where is the conceptual mistake?
You changed the target.
First it was the object evaluation. I explained that it's not a
worthwhile goal, because lisp objects are not usually evaluated.
The only occurences where a (non symbol and non cons) lisp object is
evaluated, are:
- numerical literals, eg.: 42
- vector or array literals, eg.: #(1 2 3) #2A((1 2) (3 4))
- string literal, eg.: "abc"
- very rarely, structure literals: #S(point :x 2 :y 3)
other literals are usually quoted, and therefore not evaluated, or are
not present in lisp sources.
Actually, lisp objects of non standard types may be inserted in lisp
sources, either by using #., or thru a macro generating the source, but
then you must provide a load-form (by defining a make-load-form method
on the class of the object) to allow this code to be compiled and saved
in a fasl file. So there's some added complexity, that is not often
worth it; better just avoid random objects in source code.
Now it's symbol evaluation. I explained the difficulties and provided
by the standard solution: use of symbol macros. So you don't really
intercept the evalution of existing variables, but you define a symbol
macrolet that shadows the existing variables, and that the standard
mechanism specified and implemented into EVAL, COMPILE and COMPILE-FILE,
substitute by the given expansion.
> Also I took a look in the source code of eval in sbcl
> (src/code/eval.lisp) and basically it is a big case that essentialy
> is structured like this:
>
> (typecase exp
> (symbol
> (ecase (info :variable :kind exp)
> ((:special :global :constant :unknown)
> (symbol-value exp))
> (:alien
> (%simple-eval original-exp lexenv))))
> (list
> (... ALOT OF CODE...))
> (t
> exp))
>
> which means that there is point in trying to go after my second
> version question. Conceptually isnt it like injecting in the above
> code a conditional statement on the results of (symbol-value exp) or
> (%simple-eval original-exp lexenv) that will check for the type and
> call fetch?
The standard hooks provided by Common Lisp to change the behavior of
read, load, eval, compile and compile-file are:
- reader macros,
- macros,
- symbol macros,
- compiler macros.
If you want to change those functions, just write your own. See chapter
4 of SICL.
SICP = Structure and Interpretation of Computer Programs
http://mitpress.mit.edu/sicp/full-text/book/book-Z-H-4.html
[Note: the above code is missing the processing of symbol macros. I
guess that's because sbcl does it before calling the function you gave
the extract, notably given that sbcl compiles everything. ccl eval goes
to cheap-eval which calls cheap-eval-in-environment which contains:
(cond ((symbolp form)
(multiple-value-bind (expansion win) (cheap-eval-macroexpand-1 form env)
(if win
(cheap-eval-in-environment expansion env)
(let* ((defenv (definition-environment env))
(constant (if defenv (assq form (defenv.constants defenv))))
(constval (%cdr constant)))
(if constant
(if (neq (%unbound-marker-8) constval)
constval
(error "Can't determine value of constant symbol ~s" form))
(if (constant-symbol-p form)
(%sym-global-value form)
(symbol-value form)))))))
...)
where you can see that the form (the symbol) is first macro expanded,
and if it was a symbol macro, the macro expansion succeeds, and the
expansion is evaluated instead.
If you modified an eval to allow for the optional, user specified
processing of a symbol reference, you would get the same code as the
processing of symbol macros. It is already there, jus use it!]
Pascal J. Bourguignon
> Vassilis Radis <rad...@gmail.com> writes:
>
>> How could I intervene in the evaluation process so as whenever a
>> reference object is being evaluated, the remote value gets fetched and
>> re-evaluated again to produce the final result?
>
> You can't intervene in the evaulation process of CL:EVAL. There are no
> hooks for doing so, as Pascal Bourguignon points out.
The hooks are symbol macros and macros. But granted, there's no hook to
change the self evaluation of non symbol atoms.
> You would have to implement your own evaluator and use that if you want
> to have different evaluation semantics. So that would be one option.
--
Vassilis Radis
> But perhaps you don't need to do that at all. Instead, what you have is
> an object, and you normally don't care about the object itself, but
> rather you care about accessing particular fields in the object. So
> instead of looking at reconstituting the remote object when it is
> encountered, you could do that instead at the time you need to get
> something from the object.
For object access only it is true. E.g. elephant does that. But I would like to offer the retrieval functionality in a level that, as I understand now, is lower than the level of the language semantics, so it cant be easily done.
In a nutshell I would like to decouple the "reference" from the mechanism of the retrieval of the underlying value of the "reference" so that when the "reference" is e.g. a hex memory address I would aquire its value from that memory point , but when the "reference" is something else I would do something else to fetch the value. The best illustration I can think of is that in C pseufo code:
Suppose that we have a pointer to something : void *a , and we give it a value:
a = 0xXXXXXXXX;
The contents of a is a memory address, 0xXXXXXXXX, and when we write *a the computer goes to address 0xXXXXXXX and fetches the value.
My idea is to have on some higher level something like either:
1. A symbol, e.g '+', so that I could
do this: void +a; a = "scheme://somewhere/...";
and then when we write +a the computer whould go fetch it.
2. Derive the fetching mechanism based on the contents of the reference, that is, keep only '*', but when we write *a the computer should perform a check and if a is a hex then fetch from memory else fetch from somewhere else.
3. Get the value normaly but change the implementation so that just before it returnes the fetched value from memory, it does a little checking and if the value if of a specific type, then go fetch another second value based on the type of the first and return that instead.
I though I could achieve the third option within lisp by means of overriding or messing NOT with the way evaluation is done, but by intercepting the result of the evaluation before it is returned and triggering a second evaluation based on a predicate on the first result.
Btw, I ve thought of the whole issue in a more abstract and philosophical way: Treating every data as a reference (name) only and no underlying real value at all. Any direct request for the value will return the dummy unique reference(e.g. a UUID), and literally any operation on the data will be implemented as a method specializing on the name of the operation and the dummy reference or as an access of a very large hash table with key: (concatenate uuid oper-name) and value whatever we want. In general any data can be defined in terms of operation-name and a dummy unique name denoting the specific data instance, without having to provide a value tied to the unique name.
Vassilis Radis
>
> You changed the target.
>
> First it was the object evaluation. I explained that it's not a
> worthwhile goal, because lisp objects are not usually evaluated.
>
I am sorry but you misunderstand me. I know I changed the target and that is because I understood your explanation and I just search for alternatives to explore if what i am asking really makes sense, because i dont consider myself that much experienced for this kind of question.
That is also why I post alternative questions without having delved into the answers you and others already have given in the mean time: They are full of useful information that I will now have to study. I just try to keep up with the basic idea of what you write while keeping the discussion open so that I have material to look into and implement. Are my questions so boring?