strange (for me) problem with "No matching ctor found for class ..."

[Johannes]

Hi!

I have a function f- defined using a local function f which uses a local var v:

user> (def f- (let [v 1

              f (fn [x] v)] f))

;; => #'user/f-

Putting f- in a map

user> {:d f-}

;; => {:d #function[user/fn--13335/f--13336]}

all is as expected. But giving the map as argument to eval

user> (eval {:d f-})

this error message appear:

IllegalArgumentException No matching ctor found for class user$fn__13335$f__13336  clojure.lang.Reflector.invokeConstructor (Reflector.java:163)

This problem doesn't arise if the local function f doesn't  use v:

user> (def f- (let [v 1

              f (fn [x] x)] f))

;; => #'user/f-

user> (eval {:d f-})

;; => {:d #function[user/fn--13345/f--13346]}

Is there any explanation for this behavior?

Johannes

[Ben Kovitz]

On May 4, 2016, at 2:32 PM, Johannes <bra...@nordakademie.de> wrote:

> Is there any explanation for this behavior?

My understanding is that you can't eval function objects. Whatever you give to eval needs to be code that you could enter into the REPL. You couldn't type #function[user/fn--13335/f--13336] into the REPL and have it evaluate to that function. More to the point, the reader doesn't handle the function object represented by that string. For the same reason, you can't put function objects directly into code generated by macros.

That said, I've inadvertently given a function object to eval a few times, and *sometimes* it worked. I don't have an explanation for that. It actually caused me some confusion, because, since it worked the first time I tried it, I concluded with surprise and glee that you *can* eval function objects. But it's not true, at least not reliably. If you want to (reliably) make eval return something containing a function object, you need to give eval an expression containing a symbol or expression that evaluates to that function object—not the function object itself.

--
Ben Kovitz
http://mypage.iu.edu/~bkovitz/

[Johannes]

Maybe you are right. But I've deliberately given function objects to eval often without any problems until I used functions of some special kind. I do not understand why the behavior for the 2 versions of the function f- I've shown is different.

Johannes

[Ben Kovitz]

On May 4, 2016, at 5:28 PM, Johannes <bra...@nordakademie.de> wrote:

Maybe you are right. But I've deliberately given function objects to eval often without any problems until I used functions of some special kind. I do not understand why the behavior for the 2 versions of the function f- I've shown is different.

Oops, I think I missed the main problem: you were passing a pre-evaluated map object directly to eval. What eval really wants is the kind of thing that the reader returns after it reads a string. The reader doesn’t evaluate symbols, like f-. That’s eval’s job. When the reader sees the string f-, it returns the symbol f-. You passed eval the actual function object, not the symbol f-. Eval wants to see the symbol; it will replace it with the object that it evaluates to.

In other words, I think you just forgot to quote the expressions that you were passing to eval. Adding the quote, here’s what I get:

user=> (def f- (let [v 1
  #_=>               f (fn [x] v)] f))
#'user/f-

user=> (eval '{:d f-})
{:d #function[user/fn--10235/f--10236]}

user=> (def f- (let [v 1
  #_=>               f (fn [x] x)] f))
#'user/f-

user=> (eval '{:d f-})
{:d #function[user/fn--10243/f--10244]}

[Johannes]

okay, but why does the following work without quotation?

user> (def f- (let [v 1

              f (fn [x] x)] f))

;; => #'user/f-

user> (eval {:d f-})

;; => {:d #function[user/fn--14679/f--14680]}

user>  

The problem is that the place in my program where I give the map containing the function object to eval, the map cannot be quoted.

Johannes

[Ben Kovitz]

On May 4, 2016, at 6:08 PM, Johannes <bra...@nordakademie.de> wrote:

okay, but why does the following work without quotation?
user> (def f- (let [v 1
              f (fn [x] x)] f))
;; => #'user/f-
user> (eval {:d f-})
;; => {:d #function[user/fn--14679/f--14680]}
user>  

The problem is that the place in my program where I give the map containing the function object to eval, the map cannot be quoted.

Hmm, I thought I had read “eval doesn’t accept function objects” in a couple places on the Internet, but when I checked this page:

http://clojure.org/reference/evaluation

I found the sentence "Any object other than those discussed above will evaluate to itself.” So, it appears to be a bug, either in the documentation or in eval.

I’m only a Clojure noob, having started four weeks ago. so we’ll both have to wait for a more authoritative explanation from someone more familiar with Clojure internals. The reason I know about this (or thought I did) is because I recently struggled with a similar problem: dynamically supplying a method map to extend a protocol to a record. My solution was to define a top-level dynamically rebindable symbol and refer to it inside the expression passed to eval, something like this (untested code):

(def ^:dynamic method-map-holder)

(def dynamic-def-record [record-sym fields-vector method-map]
  (binding [method-map-holder method-map]
    (eval `(defrecord ~record-sym ~fields-vector))
    (eval `(extend ~record-sym
                   ~'my-project.core/MyProtocol
                   method-map-holder))))

The trick is to call binding to put the method-map (containing function objects) into a place where eval can see it. That place is: bound to a top-level symbol, which also appears in the object passed to eval. Note that inside the ` form passed to eval, method-map-holder is not preceded by a tilde. eval will see that exact symbol (qualified by the enclosing namespace). The point of all this is to avoid evaluating the map before passing it to eval.

Be aware that the above is an ugly hack, and probably not the best way to do it. I showed my actual code to someone with years of Clojure experience, and he immediately said, “Calling eval is a code smell: there’s almost always a better way to do what you’re trying to do,” and we quickly found a likely way to do what I needed with a macro. However, we did not find a way around temporarily binding the method-map to a dynamically rebindable symbol.

[Mike Rodriguez]

I won't speak directly to your use-case other than saying that `extend` is already a function, so there is no reason to call it with a macro or via strange evaluation orders via quotes and eval.

You can define the record first, as normal, then call `extend` after dynamically and without messing with the evaluation order of functions etc.

I may be missing something more that you were trying to do, but from your example I didn't see any need for the complexity.

On the topic of if eval allows arbitrary objects to be embedded in the "code" it is evaluating:

I can see how the docs @ http://clojure.org/reference/evaluation could mislead someone to thinking that it is perfectly fine and acceptable to embed any objects into a call to `eval`, but I really don't think that is what the docs are discussing in this case.  Calling `eval` is a way to dig into the evaluator of Clojure to use it "ad hoc" yourself at a different time than "normal".

So at this point, you've circumvented Clojure's standard evaluation and you are attempting to take matters into your own hand.  When you do that, there starts to become a lot of subtle issue, like the one of whether or not you can get away with embedding already evaluated objects into a new evaluation.  Since, in your example, you functions are already evaluated to their function objects, you are effectively "double evaluating" them.  When you do that, expect issue.

This is a subtle and confusing topic because there are indeed times when you can get away with embedding objects into the code you are calling eval on.  However, I'd argue that is not the normal case and you shouldn't rely on that because it doesn't take much change to break it and no longer be able to do it.  For example, functions that have closures, I believe, can never successfully be embedded into a call to eval.

The real point here is in understanding evaluation semantics.  The reader reads characters/text into data structures.  However these data structure are not unbounded.  They are the data structures that are used to describe the syntax of the Clojure language, e.g. [], {}, (), :keywords, symbols, "strings", 123, etc.

If you try to print a function object, you'll see it has no clear representation as a valid data structures in the Clojure syntax subset of data structures.  

It'll be something like: <clojure.lang.AFnblah@123>

This is your warning sign that it is going to be trouble to call eval on it.  Eval is intended to evaluate _code_ represented as data.  This function object is not in a form amendable for interpretation as code.

If you do not try to control evaluation yourself, when evaluating functions typically looks like this (fn [x] (inc x)).  That is not a function object to the evaluator.  It is a list, with 3 elements.  The first is a symbol, the next is a vector, the last is another list, and we can describe its subforms similarly.

Clojure does provide a facility to give a meaningful reader/data syntax representation to arbitrary objects to allow evaluation later for things like serialization.  This is via the 

clojure.core/print-dup family of functionality.  Functions themselves in Clojure do not out-of-the-box come with a useful implementation of print-dup, but it is a multimethod and can be dynamically extended to new types of data.

I think you should try to solve your problem in a simpler way than going to this level of depth and/or trying to call eval yourself on already-pre-evaluated forms, but for the sake of learning, you may be interested in seeing how to make a macro for creating functions that do have a print-dup implementation that would allow them to be printed as readable, then "evaluatable" code data structures later on.  https://github.com/sorenmacbeth/serializable-fn is an example of this.  I have used it for inspiration before where I did want to be able to serialize pre-eval'ed functions, but I had some tweaks to it of my own.  The basic idea holds.  However, I only have needed to go to this level of complexity when I was trying to serialize this data so that I could store it away and use it in another separate process later, which I think is where it can become useful.

Hopefully you find this useful.

[Jason Wolfe]

Hi Johannes, 

I've run into this before.  Here is a ticket [1] and some context and a potential workaround [2]:

[1] http://dev.clojure.org/jira/browse/CLJ-1206

[2] https://groups.google.com/d/msg/clojure/BZwinR2zNgU/8HGOgzOxzosJ

-Jason

[Johannes]

Hi Mike!

Am Samstag, 7. Mai 2016 19:00:11 UTC+2 schrieb Mike Rodriguez:

...

I can see how the docs @ http://clojure.org/reference/evaluation could mislead someone to thinking that it is perfectly fine and acceptable to embed any objects into a call to `eval`, but I really don't think that is what the docs are discussing in this case.  Calling `eval` is a way to dig into the evaluator of Clojure to use it "ad hoc" yourself at a different time than "normal".

indeed, the docs say that function objects should evaluate to themselves. 

So at this point, you've circumvented Clojure's standard evaluation and you are attempting to take matters into your own hand.  When you do that, there starts to become a lot of subtle issue, like the one of whether or not you can get away with embedding already evaluated objects into a new evaluation.  Since, in your example, you functions are already evaluated to their function objects, you are effectively "double evaluating" them.  When you do that, expect issue.

This is a subtle and confusing topic because there are indeed times when you can get away with embedding objects into the code you are calling eval on.  However, I'd argue that is not the normal case and you shouldn't rely on that because it doesn't take much change to break it and no longer be able to do it.  For example, functions that have closures, I believe, can never successfully be embedded into a call to eval.

The real point here is in understanding evaluation semantics.  The reader reads characters/text into data structures.  However these data structure are not unbounded.  They are the data structures that are used to describe the syntax of the Clojure language, e.g. [], {}, (), :keywords, symbols, "strings", 123, etc.

If you try to print a function object, you'll see it has no clear representation as a valid data structures in the Clojure syntax subset of data structures.  

It'll be something like: <clojure.lang.AFnblah@123>

I think, I understand, but nevertheless I cannot understand why

{:f `(~(let [v (fn [] 1)

                   f (fn [x] (v))] f))} 

evaluates as expected, but

(eval {:f `(~(let [v (fn [] 1)

                   f (fn [x] (v))] f))})

does not. That looks like a bug, for me.

This is your warning sign that it is going to be trouble to call eval on it.  Eval is intended to evaluate _code_ represented as data.  This function object is not in a form amendable for interpretation as code.

If you do not try to control evaluation yourself, when evaluating functions typically looks like this (fn [x] (inc x)).  That is not a function object to the evaluator.  It is a list, with 3 elements.  The first is a symbol, the next is a vector, the last is another list, and we can describe its subforms similarly.

Clojure does provide a facility to give a meaningful reader/data syntax representation to arbitrary objects to allow evaluation later for things like serialization.  This is via the 

clojure.core/print-dup family of functionality.  Functions themselves in Clojure do not out-of-the-box come with a useful implementation of print-dup, but it is a multimethod and can be dynamically extended to new types of data.

I think you should try to solve your problem in a simpler way than going to this level of depth and/or trying to call eval yourself on already-pre-evaluated forms, but for the sake of learning, you may be interested in seeing how to make a macro for creating functions that do have a print-dup implementation that would allow them to be printed as readable, then "evaluatable" code data structures later on.  https://github.com/sorenmacbeth/serializable-fn is an example of this.  I have used it for inspiration before where I did want to be able to serialize pre-eval'ed functions, but I had some tweaks to it of my own.  The basic idea holds.  However, I only have needed to go to this level of complexity when I was trying to serialize this data so that I could store it away and use it in another separate process later, which I think is where it can become useful.

I will take a look at your hint. 

Hopefully you find this useful.

thank you, Mike

Johannes 

[Johannes]

Hi Jason,

thank for your hints, perhaps I can use your workaround.

Johannes