Adding primitive type hints to anonymous functions

[Ben Mabey]

Does the new primitive support added in 1.3 extend to anonymous functions?

If so, I am doing something wrong because I can't get them to work:

(definterface IPrimitiveTester
(getType [^int x])
(getType [^long x])
(getType [^float x])
(getType [^double x])
(getType [^Object x]))

(deftype PrimitiveTester []
IPrimitiveTester
(getType [this ^int x] :int)
(getType [this ^long x] :long)
(getType [this ^float x] :float)
(getType [this ^double x] :double)
(getType [this ^Object x] :object))

(defmacro pt [x]
`(.getType (PrimitiveTester.) ~x))

(defn with-defn ^double [^double x]
(+ x 0.5))

(def with-def (fn ^double [^double x]
(+ x 0.5)))

(pt (with-defn 1.0)) ; => :double

(pt (with-def 1.0)) ; => :object

So.. is the above user error, known limitation, bug, or new issue?

Thanks,
Ben

[Tassilo Horn]

Ben Mabey <b...@benmabey.com> writes:

Hi Ben,

> Does the new primitive support added in 1.3 extend to anonymous
> functions?

I guess, it should. At least they are not disregarded...

--8<---------------cut here---------------start------------->8---
user> (def with-def (fn ^float [^float x]
(+ x 0.5)))
;Only long and double primitives are supported
; [Thrown class java.lang.IllegalArgumentException]
; Evaluation aborted.
--8<---------------cut here---------------end--------------->8---

> If so, I am doing something wrong because I can't get them to work:
>
> (definterface IPrimitiveTester
> (getType [^int x])
> (getType [^long x])
> (getType [^float x])
> (getType [^double x])
> (getType [^Object x]))
>
> (deftype PrimitiveTester []
> IPrimitiveTester
> (getType [this ^int x] :int)
> (getType [this ^long x] :long)
> (getType [this ^float x] :float)
> (getType [this ^double x] :double)
> (getType [this ^Object x] :object))
>
> (defmacro pt [x]
> `(.getType (PrimitiveTester.) ~x))
>
> (defn with-defn ^double [^double x]
> (+ x 0.5))
>
> (def with-def (fn ^double [^double x]
> (+ x 0.5)))
>
> (pt (with-defn 1.0)) ; => :double
> (pt (with-def 1.0)) ; => :object
>
> So.. is the above user error, known limitation, bug, or new issue?

Looks like a bug to me. And it's pretty strange, because defn expands
into a (def foo (fn ...)) anyway...

Oh, you can add the return type metadata to the def-ed var like so:

--8<---------------cut here---------------start------------->8---
user> (def ^{:tag Double/TYPE}
with-def (fn [^double x]
(+ x 0.5)))
#'user/with-def
user> (pt (with-def 3))
:double
--8<---------------cut here---------------end--------------->8---

But OTOH, the metadata of `with-defn' does not have :tag. And basically
it shouldn't, because with overloaded arities, the return type may
differ between versions...

Bye,
Tassilo

[Tassilo Horn]

Hi again,

I think, I got it. I wrote a little helper function to print the
metadata of a form:

--8<---------------cut here---------------start------------->8---
(use 'clojure.walk)
(defn print-meta
([form level]
(prewalk
(fn [x]
(when-let [m (meta x)]
(println "Level" level ":" x "=>" m)
(print-meta m (inc level)))
x)
form))
([form]
(print-meta form 0)))
--8<---------------cut here---------------end--------------->8---

Now see what happens:

--8<---------------cut here---------------start------------->8---
user> (print-meta (macroexpand-all '(def b (fn ^foo [^bar x] ^baz x))))
Level 0 : x => {:tag bar}
Level 0 : x => {:tag baz}
(def b (fn* ([x] x)))
user> (print-meta (macroexpand-all '(defn b ^foo [^bar x] ^baz x)))
Level 0 : b => {:arglists (quote ([x]))}
Level 1 : [x] => {:tag foo}
Level 1 : x => {:tag bar}
Level 0 : x => {:tag bar}
Level 0 : x => {:tag baz}
(def b (fn* ([x] x)))
--8<---------------cut here---------------end--------------->8---

So the return type metadata is actually added to the :arglist metadata
contents, so it's actually meta-metadata. What you seem to have to do
is to make sure the :arglists metadata is there and reflects the type
hints on the actual function:

--8<---------------cut here---------------start------------->8---
user> (def ^{:arglists '(^double [^double x])}

with-def (fn ^double [^double x] (+ x 0.5)))

#'user/with-def
user> (pt (with-def 1))

:double
--8<---------------cut here---------------end--------------->8---

Well, that's not really obvious. Maybe `def' could do a better job here
and build up a correct :arglists metadata value somehow...

Bye,
Tassilo

[Ben Mabey]

Hi Tassilo,
Thanks for doing the detective work! I still have some questions
though... In my example I used def because I was able to reproduce the
bug with def and it was the easiest way to show it. However, in practice
these anonymous functions won't be bound to a root var (hence the
anonymous part :) ). I would like to be able to write code like this:

(map-doubles (fn ^double [^double x] (+ x 0.5)) array)

I don't seem to be able to do that since the type hints are not put into
effect without a var (it would seem):

(pt ((fn ^double [^double x] (+ x 0.5)) 1.0)) ; => :object

Not just put into effect, but not even stored on the function (as far as
I can tell):

(meta (fn ^double [^double x] (+ x 0.5))) ; => nil

I've tried adding the appropriate metadata onto the function (I'm
stealing the metadata from our earlier with-defn):

(meta (with-meta (fn ^double [^double x] (+ x 0.5)) (select-keys (meta
#'with-defn) [:arglists]))) ; => {:arglists ([x])}

I can verify that the arglist is tagged with the primitive just like it
is with the var #'with-defn:

(defn arglist-tag [var]
(-> var meta :arglists first meta :tag))

(arglist-tag #'with-defn) ; => double
(arglist-tag (with-meta (fn ^double [^double x] (+ x 0.5)) (select-keys
(meta #'with-defn) [:arglists])))A ; => double

Sadly, even with the arglist meta-metadata boxing is still occurring:

(pt ((with-meta (fn ^double [^double x] (+ x 0.5)) (select-keys (meta
#'with-defn) [:arglists])) 1.0)) ; => object

It is probably obvious that this would not work to someone who
understands clojure's implementation, but to me this was a surprise.
From this I gather that type hints on vars are what clojure uses and
type hints on the actual functions are ignored.

Am I missing something, or does the new primitive support only apply to
bound functions?

-Ben

[Tassilo Horn]

Ben Mabey <b...@benmabey.com> writes:

> Am I missing something, or does the new primitive support only apply
> to bound functions?

On the one hand, the docs don't mention such a restriction, but on the
other hand, all examples given are always defn-defined. I tried to look
into the compiler to find out, but that's not a component you understand
directly. So hopefully someone with a better knowledge of the compiler's
internals jumps in the discussion.

Bye,
Tassilo

[Cedric Greevey]

It's even stranger than that. It won't work with local Vars:

user=> (with-local-vars [a (fn [^double x] (+ x 0.5))] (.setMeta a
{:tag Double/TYPE}) (pt (a 0.1)))
:object

And yes, there's a {:tag double} on a after the .setMeta call. In
fact, that degree of hackery is needed. Neither ^double nor ^{:tag
Double/TYPE} on the a in the binding form will attach the metadata to
the Var, unlike with def, where the latter works (and the former seems
to, at first, but fails when the fn is called with a *very strange*
error message about a class named clojure.core$double@1eb1db2 not
being resolved).

Attaching ^double or ^{:tag Double/TYPE} to the (fn ...) form itself
also does not work, there or with a normal let-bound local.

Either PrimitiveTester doesn't work the way it seems like it should,
or else something's strange/wonky with how Clojure compiles function
calls that don't use a namespaced global Var.

Actually, on further testing, I'm not sure that (def ^{Double/TYPE}
...) is really working either, rather than boxing and then unboxing on
each call:

user=> (def ^{:tag Double/TYPE} b (fn [^double x] (+ x 0.5)))
#'user/b
user=> (defn c ^double [^double x] (+ x 0.5))
#'user/c
user=> (pt (b 0.1))
:double
user=> (pt (c 0.1))
:double

So far, so good? But:

user=> (ancestors (.getClass b))
#{java.util.concurrent.Callable clojure.lang.AFn clojure.lang.IObj
clojure.lang.IFn clojure.lang.IFn$DO clojure.lang.IMeta
java.lang.Object java.io.Serializable java.lang.Runnable
java.util.Comparator clojure.lang.AFunction clojure.lang.Fn}

IFn$DO?

user=> (ancestors (.getClass c))
#{java.util.concurrent.Callable clojure.lang.AFn clojure.lang.IObj
clojure.lang.IFn$DD clojure.lang.IFn clojure.lang.IMeta
java.lang.Object java.io.Serializable java.lang.Runnable
java.util.Comparator clojure.lang.AFunction clojure.lang.Fn}

IFn$DD?

user=> (def d (fn ^double [^double x] (+ x 0.5)))
#'user/d
user=> (ancestors (.getClass d))
#{java.util.concurrent.Callable clojure.lang.AFn clojure.lang.IObj
clojure.lang.IFn$DD clojure.lang.IFn clojure.lang.IMeta
java.lang.Object java.io.Serializable java.lang.Runnable
java.util.Comparator clojure.lang.AFunction clojure.lang.Fn}

IFn$DD again here.

This suggests:

user=> (let [a ^clojure.lang.IFn$DD (fn ^double [^double x] (+ x
0.5))] (pt (a 0.1)))
:object

Nope, that doesn't work either!

This really *should* be simple. Therefore, there's a bug somewhere.
But I'm not sure where. Compiler.java, most likely.

[Tassilo Horn]

Cedric Greevey <cgre...@gmail.com> writes:

Hi Cedric,

> Actually, on further testing, I'm not sure that (def ^{Double/TYPE}
> ...) is really working either, rather than boxing and then unboxing on
> each call:
>
> user=> (def ^{:tag Double/TYPE} b (fn [^double x] (+ x 0.5)))
> #'user/b
> user=> (defn c ^double [^double x] (+ x 0.5))
> #'user/c
> user=> (pt (b 0.1))
> :double
> user=> (pt (c 0.1))
> :double
>
> So far, so good? But:
>
> user=> (ancestors (.getClass b))
> #{java.util.concurrent.Callable clojure.lang.AFn clojure.lang.IObj
> clojure.lang.IFn clojure.lang.IFn$DO clojure.lang.IMeta
> java.lang.Object java.io.Serializable java.lang.Runnable
> java.util.Comparator clojure.lang.AFunction clojure.lang.Fn}
>
> IFn$DO?

A Function that gets a double and returns an object, I think. That
would support your claim that b doesn't return a primitive double,
although the Var is tagged.

> user=> (ancestors (.getClass c))
> #{java.util.concurrent.Callable clojure.lang.AFn clojure.lang.IObj
> clojure.lang.IFn$DD clojure.lang.IFn clojure.lang.IMeta
> java.lang.Object java.io.Serializable java.lang.Runnable
> java.util.Comparator clojure.lang.AFunction clojure.lang.Fn}
>
> IFn$DD?

Gets and returns a primitive double.

> user=> (def d (fn ^double [^double x] (+ x 0.5)))
> #'user/d
> user=> (ancestors (.getClass d))
> #{java.util.concurrent.Callable clojure.lang.AFn clojure.lang.IObj
> clojure.lang.IFn$DD clojure.lang.IFn clojure.lang.IMeta
> java.lang.Object java.io.Serializable java.lang.Runnable
> java.util.Comparator clojure.lang.AFunction clojure.lang.Fn}
>
> IFn$DD again here.

Hm, strange. That means (to my best understanding), that the function
is correctly compiled to have a primitive version. So maybe it's just
the call to anonymous functions that requires :arglists metadata to
correctly resolve the primitive version...

> This really *should* be simple. Therefore, there's a bug somewhere.
> But I'm not sure where. Compiler.java, most likely.

At least, it seems that (fn ^double [^double x] (+ x 0.5)) compiles to a
class supporting primitive, unboxed calls.

Bye,
Tassilo

[Cedric Greevey]

On Thu, Jan 26, 2012 at 2:28 AM, Tassilo Horn <tas...@member.fsf.org> wrote:
> At least, it seems that (fn ^double [^double x] (+ x 0.5)) compiles to a
> class supporting primitive, unboxed calls.

Yes. That was my conclusion also. But apparently the only way to get
the compiler to emit such a call is by binding the fn to a global Var
with the appropriate hints and calling that.

Even a local var with both tag and arglists metadata doesn't work:

user=> (with-local-vars [a (fn ^double [^double x] (+ 0.5 x))]
(.setMeta a {:tag Double/TYPE :arglists [^double[^double 'x]]}) (pt (a
0.1)))
:object

Changing the call to (@a 0.1) has no effect.

[Tassilo Horn]

Cedric Greevey <cgre...@gmail.com> writes:

> On Thu, Jan 26, 2012 at 2:28 AM, Tassilo Horn <tas...@member.fsf.org> wrote:
>> At least, it seems that (fn ^double [^double x] (+ x 0.5)) compiles to a
>> class supporting primitive, unboxed calls.
>
> Yes. That was my conclusion also. But apparently the only way to get
> the compiler to emit such a call is by binding the fn to a global Var
> with the appropriate hints and calling that.

Yeah, I know. And now I can also see it in the Compiler's InvokeExpr
class. In its constructor, there's some extra work in case of a VarExpr
where the signature :tag metadata is pulled out from the :arglists
metadata.

Why is the :arglists metadata added to the Var a function's bound to
instead of the function itself?

Bye,
Tassilo

[Cedric Greevey]

AFAICT, the compiler will need to know the call is definitely
primitive and the fn is definitely primitive-compatible at compile
time if it is to safely emit a primitive call. This is why I tried a
^Ifn$DD hint on the variable at one point.

When the fn comes from a non-dynamic Var the compiler can easily be
sure what primitive signatures it supports.

When the fn is potentially runtime-variable, it seems to me that a
hint on the local *should* suffice, and type inference from
assignments *should* be possible, as it is for locals assigned
primitive values. (If the hint was wrong, you'd get runtime
ClassCastExceptions such as my_ns.my_defn$fn_3072 cannot be cast to
clojure.lang.IFn$DD, much as you can get various CCEs with incorrect
(reference) type hints elsewhere.) However, this seems to be either
broken or not implemented yet.

[Ben Mabey]

Should a ticket be created for this then?

[Tassilo Horn]

Ben Mabey <b...@benmabey.com> writes:

> Should a ticket be created for this then?

Yes, please. And add a link to this discussion.

Bye,
Tassilo