eval/macros with functions with metadata

[Ben Wolfson]

or, the dreaded "no matching ctor found" exception.

Is there a way to write the function

(defn eval-at-one [f] (eval `(~f 1)))

such that it works when invoked like this:

(eval-at-one (fn [x] x))
 ;; --> 1

and like this

(eval-at-one (with-meta (fn [x] x) {}))
 ;; -> IllegalArgumentException No matching ctor found for class clojure.lang.AFunction$1  clojure.lang.Reflector.invokeConstructor (Reflector.java:163)

?

I thought that the object returned by with-meta might be hanging onto the original, metadata-less object, which could then be retrieved and used without difficulty, but this appears not to be the case.

--
Ben Wolfson
"Human kind has used its intelligence to vary the flavour of drinks, which may be sweet, aromatic, fermented or spirit-based. ... Family and social life also offer numerous other occasions to consume drinks for pleasure." [Larousse, "Drink" entry]

[Gary Fredericks]

what's the use case for evaling a function object?

--
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Gary Fredericks
(803)-295-0195
frederi...@gmail.com
www.gfredericks.com

[Ben Wolfson]

On Wed, Aug 28, 2013 at 2:46 AM, Gary Fredericks <frederi...@gmail.com> wrote:

what's the use case for evaling a function object?

This is just a minimal example that illustrates the difficulty, in the actual code where I came across it, a new function with relatively complex internals is created at runtime.

 

[Jamie Brandon]

Clojure struggles with staged programming because of JVM limitations.

"The deeper issue here is that JVM bytecode is fundamentally static:
it's not allowed to contain references to objects that only exist at
runtime. In languages that have a built-in concept of a "runtime
image," like Smalltalk or some Lisps, the line between static code and
runtime data is blurred."

https://groups.google.com/forum/#!searchin/clojure/eval$20closure/clojure/scAjN7_Xig0/bfWuWnzX7z4J

https://groups.google.com/forum/#!searchin/clojure/eval$20closure/clojure/eQpeivDn-Vg/Jv29pOd0BgwJ

https://groups.google.com/d/topic/clojure/jPGu5MUsGBs/discussion

If you aren't worried about leaking memory, a possible workaround is
to intern the fn in some namespace and then put the symbol in the
macro output.

[Ben Wolfson]

On Wed, Aug 28, 2013 at 9:27 AM, Jamie Brandon <ja...@scattered-thoughts.net> wrote:

If you aren't worried about leaking memory, a possible workaround is
to intern the fn in some namespace and then put the symbol in the
macro output.

Hrm, I hope it doesn't come to that---as long as I'm creating the functions with their metadata, I can make things work, because I can add extra information to the with-meta call or do extra things when making the call. What I'd like, though, is for it to be possible to copy the metadata over to a new function by client code, just using the regular (with-meta (fn [blah] blah) (meta old-fn)) mechanism. But then the sneaky symbol, or whatever extra thing, in the metadata, that's used in the form to be eval-ed is pointing to the original function, not the one passed in. Maybe just providing a utility for fixing up the metadata is the right move, though it's kidn of dissatisfying.

[Jamie Brandon]

You will also run into problems if your functions close over any data:

user> (let [f (fn [] 1)] (eval `(~f)))
1
nil
user> (let [x 1 f (fn [] x)] (eval `(~f)))

IllegalArgumentException No matching ctor found for class

user$eval38616$f__38617 clojure.lang.Reflector.invokeConstructor
(Reflector.java:163)

I'm struggling with this as well, I'm definitely interested in finding
a non-hacky way to pass closures through eval without leaking memory.

Perhaps something like:

(let [x 1
f (fn [] x)
f-sym (gensym)]
(intern *ns* gensym (weak-ref f))
(with-meta (eval `(fn [] (do-stuff-with @~f))) {::strong-ref f}))

So that way f will be freed up when the resulting fn is gced.

[Aaron Cohen]

I'm not sure if you'll consider this hacky or not.

 (def ^:dynamic *fn-helper*)

(defn eval-at-one [f] (binding [*fn-helper* f] (eval '(*fn-helper* 1))))

[Jamie Brandon]

That sort of works for my use case.

What I want to do is define a grammar...

(def num-graph
(graph
num ~(or ~succ ~zero)
succ (succ ^x ~num)
zero zero))

... attach actions ....

(def num-out
(output-in num-graph
'zero (fnk [] 0)
'succ (fnk [x] (inc x))))

... and compile the result ...

strucjure.regression.tests> (graph->view 'num num-out)
(clojure.core/letfn
[(num ...)
(succ ...)
(zero ...)]
(clojure.core/fn
[input__2288__auto__]
(num input__2288__auto__)))

This works fine as long as the fnks dont close over anything, but
that's very limiting.

If I eval that code and *then* wrap it in (binding ...) I can have
closures but that now means that I can't do call-site compilation.
I'll have to poke around a bit more...

[Aaron Cohen]

How about this alternative?

(deftype Wrapper [f]

  clojure.lang.IFn

  (invoke [this] (f))

  (invoke [this a] (f a))

  ; repeat until you get bored

)

(defn e [f]

  (let [wr (Wrapper. f)]

    (eval `(~wr 1))))

(e inc)

(e (with-meta (fn [x] (+ 4 x)) {}))

[Aaron Cohen]

Sorry, that one doesn't work. It worked in my repl, but I must have been using old code.

[Jamie Brandon]

Yeah, as far as I can tell eval can't emit pointers so we can't pass
closures or references at all except by indirection through an
existing singleton (like a named var).

[Aaron Cohen]

This is a little hard for me to follow because I'm not sure what graph or output-in are doing.

If the letfn is being generated by your compilation step, why can't that go inside of your generated fn rather than outside it?

(clojure.core/fn

  [input__2288__auto__]

    (clojure.core/letfn
     [(num ...)
      (succ ...)
      (zero ...)]

        (num input__2288__auto__)))

[Jamie Brandon]

Sorry, I guess that wasn't very clear. Let's put a really simple
example with just the pattern-matching subset:

strucjure.regression.sandbox=> (def p (pattern [1 ^x _ 3]))
#'strucjure.regression.sandbox/p

strucjure.regression.sandbox=> p
[1 #strucjure.pattern.Bind{:symbol x, :pattern #strucjure.pattern.Any{}} 3]

strucjure.regression.sandbox=> (def o (output ~p (fnk [x] x)))
#'strucjure.regression.sandbox/o

strucjure.regression.sandbox=> o
#strucjure.pattern.Output{:pattern [1 #strucjure.pattern.Bind{:symbol
x, :pattern #strucjure.pattern.Any{}} 3], :fnk #<
clojure.lang.AFunction$1@18ab1e1c>}

strucjure.regression.sandbox=> ((view ~p) [1 2 3])
[[1 2 3] nil]

strucjure.regression.sandbox=> ((view ~o) [1 2 3])
[2 nil]

strucjure.regression.sandbox=> (pprint (macroexpand-1 '(view ~o)))
(fn
[input3910]
(clojure.core/when
(clojure.core/vector? input3910)
(clojure.core/when
(clojure.core/>= (clojure.core/count input3910) 3)
(clojure.core/let
[input-sym3911 (clojure.core/nth input3910 0)]
(clojure.core/when
(clojure.core/= input-sym3911 '1)
(clojure.core/let
[input-sym3912 (clojure.core/nth input3910 1)]
(clojure.core/let
[x input-sym3912]
(clojure.core/let
[input-sym3913 (clojure.core/nth input3910 2)]
(clojure.core/when
(clojure.core/= input-sym3913 '3)
[(#<sandbox$fn__3090$pos_fn__1649__auto____3091
strucjure.regression.sandbox$fn__3090$pos_fn__1649__auto____3091@62b1f96f>
x)
(clojure.core/seq (clojure.core/subvec input3910 3))])))))))))

You can see towards the end of the generated code where it calls the
fnk directly as #<sandbox$fn__3090...>

The motivation here is that you can define the grammar for a language
once and then make different transformations by attaching output fnks.
It would be very useful if those output fnks could be closures.

It's not obvious in this small example but the compiler produces some
important optimisations, otherwise I could just interpret the grammar
and there would be no problem.

In general, for staged programming it's useful to be able to pass
arbitrary data between stages eg metaocaml
(http://okmij.org/ftp/ML/MetaOCaml.html) often passes refs between
stages. I think some of the terra (http://terralang.org/) examples
pass closures too.

[Ben Wolfson]

Weak references do work, and it looks like I can just stuff values into a single java.util.WeakHashMap under gensym-generated keys and not worry about polluting namespaces. Still kind of hacky, but seems to work.

[Jason Wolfe]

Sounds interesting, I'd love to hear more about the use case. 

 

We ran into this issue as well when implementing the new positional compilation for Graph.  We filed a ticket [1], and worked around it with a funny trick:

Suppose you want to eval a complicated thing, and the crux of the issue is that this fails: 

(let [y 1 f (fn [x] (+ x y))] (eval `(~f 1)))

A solution is to rework what you eval, so that it has a 'hole' for the function and you slot the function into this hole afterwards (which Hotspot seems to be able to optimize just fine):

(let [y 1 f (fn [x] (+ x y))] ((eval `(fn [g#] (g# 1))) f))

Happy to elaborate if you like.  

Cheers,

Jason

[1] http://dev.clojure.org/jira/browse/CLJ-1206

[Jamie Brandon]

I certainly like that better than the binding solution, especially if
hotspot can potentially inline the functions. It still tricky to do
that in a macro, especially when the code to be evaled depends on the
current lexical scope, but I can maybe work around that. Thanks.