Surprising behaviour related to records, protocols and AOT

[Ragnar Dahlén]

Hi,

Today I encountered a, to me, slightly surprising behaviour seemingly

related clojure records. 

The setup is as follows:

1. One namespace defines a record type:

    (ns defrecordissue.arecord)

    (defrecord ARecord [])

2. Another namespace defines a protocol, and extends it to the record

   type defined in 1:

    (ns defrecordissue.aprotocol

      (:require [defrecordissue.arecord])

      (:import [defrecordissue.arecord ARecord]))

     

    (defprotocol AProtocol

      (afn [this]))

     

    (extend-protocol AProtocol

      ARecord

      (afn [this] 42))

3. A third namespace constructs an instance of the record and invokes

   the protocol function on the record:

    (ns defrecordissue.aot1

      (:require [defrecordissue.aprotocol]

                [defrecordissue.arecord]))

     

    (defrecordissue.aprotocol/afn (defrecordissue.arecord/->ARecord))

When the defrecordissue.aot1 namespace is compiled, in my case using

`lein compile defrecordissue.aot1`, compilation fails with the

following exception:

Exception in thread "main" java.lang.IllegalArgumentException: No implementation of method: :afn of protocol: #'defrecordissue.aprotocol/AProtocol found for class: defrecordissue.arecord.ARecord, compiling:(aot1.clj:5:1)

at clojure.lang.Compiler$InvokeExpr.eval(Compiler.java:3463)

at clojure.lang.Compiler.compile1(Compiler.java:7153)

at clojure.lang.Compiler.compile(Compiler.java:7219)

at clojure.lang.RT.compile(RT.java:398)

at clojure.lang.RT.load(RT.java:438)

at clojure.lang.RT.load(RT.java:411)

at clojure.core$load$fn__5018.invoke(core.clj:5530)

at clojure.core$load.doInvoke(core.clj:5529)

at clojure.lang.RestFn.invoke(RestFn.java:408)

at clojure.core$load_one.invoke(core.clj:5336)

at clojure.core$compile$fn__5023.invoke(core.clj:5541)

at clojure.core$compile.invoke(core.clj:5540)

at user$eval7.invoke(NO_SOURCE_FILE:1)

at clojure.lang.Compiler.eval(Compiler.java:6619)

at clojure.lang.Compiler.eval(Compiler.java:6609)

at clojure.lang.Compiler.eval(Compiler.java:6582)

at clojure.core$eval.invoke(core.clj:2852)

at clojure.main$eval_opt.invoke(main.clj:308)

at clojure.main$initialize.invoke(main.clj:327)

at clojure.main$null_opt.invoke(main.clj:362)

at clojure.main$main.doInvoke(main.clj:440)

at clojure.lang.RestFn.invoke(RestFn.java:421)

at clojure.lang.Var.invoke(Var.java:419)

at clojure.lang.AFn.applyToHelper(AFn.java:163)

at clojure.lang.Var.applyTo(Var.java:532)

at clojure.main.main(main.java:37)

Caused by: java.lang.IllegalArgumentException: No implementation of method: :afn of protocol: #'defrecordissue.aprotocol/AProtocol found for class: defrecordissue.arecord.ARecord

at clojure.core$_cache_protocol_fn.invoke(core_deftype.clj:541)

at defrecordissue.aprotocol$fn__40$G__35__45.invoke(aprotocol.clj:5)

at clojure.lang.AFn.applyToHelper(AFn.java:161)

at clojure.lang.AFn.applyTo(AFn.java:151)

at clojure.lang.Compiler$InvokeExpr.eval(Compiler.java:3458)

... 25 more 

If I change 3) to construct the record class directly, like so:

   (ns defrecordissue.aot2

     (:require [defrecordissue.aprotocol]

               [defrecordissue.arecord]))

    

   (defrecordissue.aprotocol/afn (defrecordissue.arecord.ARecord.))

Compilation succeeds.

My suspicion is that this is somehow related to

http://dev.clojure.org/jira/browse/CLJ-371, but I don't understant

exactly what is happening. 

I should also add that without the `lein clean`, compilation succeeds

the second time, since a class for the record is now available on the

classpath. Therefore, I can get around this problem by AOT-compiling

the namespace defining the record type.

I created a simple leiningen project on GitHub that illustrates the

issue, see README for usage:

https://github.com/ragnard/defrecordissue

Any enlightenment is much appreciated.

Cheers,

Ragnar

[Andrew Sernyak]

I guess you have to import defrecord generated class before you want to use it, check the sample from clojuredocs about defrecord

; If you define a defrecord in one namespace and want to use it
; from another, first require the namespace and then import
; the record as a regular class.
; The require+import order makes sense if you consider that first
; the namespace has to be compiled--which generates a class for
; the record--and then the generated class must be imported.
; (Thanks to raek in #clojure for the explanations!)

; Namespace 1 in "my/data.clj", where a defrecord is declared
(ns my.data)

(defrecord Employee [name surname])


; Namescape 2 in "my/queries.clj", where a defrecord is used
(ns my.queries
  (:require my.data)
  (:import [my.data Employee]))

(println
  "Employees named Albert:"
  (filter #(= "Albert" (.name %))
    [(Employee. "Albert" "Smith")
     (Employee. "John" "Maynard")
     (Employee. "Albert" "Cheng")]))
  

[Ragnar Dahlén]

Changing 3) to also import the record class:

    (ns defrecordissue.aot1

      (:require [defrecordissue.aprotocol]

                [defrecordissue.arecord])

      (:import [defrecordissue.arecord ARecord]))

makes no difference. Compilation still fails with the same exception.

This is obviously a contrived example. When I encountered this in the

wild, the consumer of the equivalent of the defrecord.aprotocol

namespace did not construct a record instance directly. It has no

awareness of the record type, and should not need to have so.

[Andrew Sernyak]

I guess extend-type does changes only to generated java class and the var defrecordissue.arecord->ARecord contains the 'old' version of ARecord constructor. Obviously it would be weird for defprotocol to change the variable in another namespace. Especially when you can extend a record from anywhere.

So If you want to create a record that implements your protocol via var from record namespace, you should do some hackery to update that variable. I've done a pull-request for you, but using direct constructor will be more idiomatic

;
; this won't work unless you update manualy a variable ->ARecord in the namespace
;
;(defrecordissue.aprotocol/afn (defrecordissue.arecord/->ARecord))
; 
; like
(defmacro from-ns[nmsps & body] 
  "launches body from namespace"
  `(binding 
     [*ns* (find-ns ~nmsps)] 
       (eval
          (quote (do ~@body)))))
(from-ns 'defrecordissue.arecord 
         (import '(defrecordissue.arecord.ARecord))
         (alter-var-root 
           ('->ARecord (ns-publics 'defrecordissue.arecord)) 
           (fn[x] (fn[] (new ARecord)))))
(println  (defrecordissue.aprotocol/afn (defrecordissue.arecord/->ARecord)))
; 42

ndrw 

[Ragnar Dahlén]

Thank you for your explanation. I also suspect there is some subtle

issue with the class file being used by the different constructors.

However, I would be surprised if this behaviour is intended, and that 

the 'hackery' you proposed is the only, and prefered way of solving this.

To better illustrate the core issue, I updated the example slightly

as follows:

Premise: 

defrecordissue.arecord and defrecordissue.protocol constitute some

library.

1. defrecordissue.arecord defines a record type, and a function that

   will return an instance of the record:

    (ns defrecordissue.arecord)

    (defrecord ARecord [])

    (defn make-record

      []

      (->ARecord))

2. defrecordissue.protocol defines a protocol, and extends it to the

   record type defined in 1. It also defines a public function

   intended to be used by libraries:

    (ns defrecordissue.aprotocol

      (:require [defrecordissue.arecord])

      (:import [defrecordissue.arecord ARecord]))

     

    (defprotocol AProtocol

      (afn [this]))

     

    (extend-protocol AProtocol

      ARecord

      (afn [this] 42))

    (defn some-public-fn

      []

      (afn (defrecordissue.arecord/make-record)))

3. defrecordissue.consumer is a consumer of the library, knows

   nothing of any protocols or records, but only wants to call a

   function thats part of the public api:

    (ns defrecordissue.consumer

      (:require [defrecordissue.aprotocol]))

     

    (defrecordissue.aprotocol/some-public-fn)

This fails with the same root cause.

I've created a new branch for this in the GitHub repo.

https://github.com/ragnard/defrecordissue/tree/more-realistic

/Ragge

[Aysylu Greenberg]

I was wondering if anybody has found a solution to this problem. I'm experiencing the same issue in this project. If you disable aot (this line), the tests start failing with a similar error message.

Thanks,

Aysylu

[Kevin Downey]

I don't know about the rest of this thread, but loom seems to suffer
from what I've outlined in
http://dev.clojure.org/jira/browse/CLJ-322?focusedCommentId=32246&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-32246
pulling in the interface that the protocol generates instead of the
protocol.

On 10/26/13, 5:42 PM, Aysylu Greenberg wrote:
> I was wondering if anybody has found a solution to this problem. I'm

> experiencing the same issue in this project <https://github.com/aysylu/loom>.
> If you disable aot (this line<https://github.com/aysylu/loom/blob/master/project.clj#L9>),

--
And what is good, Phaedrus,
And what is not good—
Need we ask anyone to tell us these things?

[Aysylu Greenberg]

Thank you for the link, but I don't think what you outlined there is the same as what I have. I'm referring to records by class and protocols defined in the namespace, so I think my situation is different.

[Aysylu Greenberg]

As discovered by Kevin Downey, (use [loom.graph] :reload) in tests was causing the namespace to be reloaded, which redefined the records in loom.graph, but didn't reload loom.attrs , so the extends didn't get run again for the new record type breaking all kinds of things. Changing the use to require fixed this issue.