protocols and namespaces confusion (with an example from core.matrix)

[Paul Gowder]

Hi folks, 

I have a kinda basic question (I think?) that I should really know the answer to after writing Clojure for like 8 months, but don't.  It's about requiring namespaces with protocol definitions in them but not the actual implementations of those protocols, and how it works.  

tl/dr: if a protocol is defined in namespace a and implemented (extended to the type at issue) in namespace b, can you call functions from it in namespace c where namespace c only requires namespace a, not namespace b?

Backstory: I was poking around core.matrix recently in pursuit of restarting work on a months ago issue I promised to work on, and it looks like at least one namespace calls a function it shouldn't be able to call.

So: clojure.core.matrix.impl.pprint requires only one namespace within core.matrix, clojure.core.matrix.protocols. (I'm going to use ... hereafter to stand for clojure.core.matrix.) ...protocols defines a protocol PSliceSeq2, which gives the name, docstring, etc. for a function called get-slice-seq. PSliceSeq2 is actually implemented for the generic object case in ...impl.defaults and for ISeq in ...impl.sequence. 

But ...impl.pprint calls ...protocols/get-slice-seq. And it seems to work.  Even though I can't find either ...impl.defaults or ...impl.sequence anywhere in the dependency tree for ...impl.pprint.

This is a puzzle to me.  I mean, the ...impl.pprint namespace will typically be required from some other namespace, and presumably that higher-level namespace will require everything that one needs. But still, I hadn't thought that one would be able to call get-slice-seq in a function in ...impl.pprint without requiring a namespace that actually defines the function, as opposed to just defining the protocol.

So can someone explain how this magic works?  Is the compiler just way smarter than I thought it was, and capable of figuring this stuff out on its own?

Thanks!

-Paul

[Gregg Reynolds]

Hi Paul,

can't give you specifics but (going from memory) I might be able to point you in the right direction.  take a look at the initialization code of core.matrix.  you have to feed it an implementation, and it dynamically arranges bindings.  sorry I can't give you more detail but I suspect you'll find your answer somewhere in that vicinity.

Gregg

[Alex Miller]

On Wednesday, October 19, 2016 at 7:23:21 AM UTC-5, Paul Gowder wrote:

Hi folks, 

I have a kinda basic question (I think?) that I should really know the answer to after writing Clojure for like 8 months, but don't.  It's about requiring namespaces with protocol definitions in them but not the actual implementations of those protocols, and how it works.  

tl/dr: if a protocol is defined in namespace a and implemented (extended to the type at issue) in namespace b, can you call functions from it in namespace c where namespace c only requires namespace a, not namespace b?

Yes. You are invoking the protocol function in a and as a caller, that's all you need to know about. HOWEVER, the implementation in b must be loaded before you can do so. Your namespace doesn't necessarily have to do it, but someone does. Each protocol is essentially a little piece of hidden state in the runtime. Each time you extend the protocol to an implementation, that state is modified. If you don't load the code that does the extension, the state will not be updated.

All of this also applies to multimethods.

 

Backstory: I was poking around core.matrix recently in pursuit of restarting work on a months ago issue I promised to work on, and it looks like at least one namespace calls a function it shouldn't be able to call.

So: clojure.core.matrix.impl.pprint requires only one namespace within core.matrix, clojure.core.matrix.protocols. (I'm going to use ... hereafter to stand for clojure.core.matrix.) ...protocols defines a protocol PSliceSeq2, which gives the name, docstring, etc. for a function called get-slice-seq. PSliceSeq2 is actually implemented for the generic object case in ...impl.defaults and for ISeq in ...impl.sequence. 

But ...impl.pprint calls ...protocols/get-slice-seq. And it seems to work.  Even though I can't find either ...impl.defaults or ...impl.sequence anywhere in the dependency tree for ...impl.pprint.

I haven't looked at the code, but something must be loading the extension before you invoke it.

[Paul Gowder]

Thanks Gregg and Alex! I didn't realize that protocols (or the black magic interfacing of core.matrix) were that fancy. Definitely going into my "code to really dig into" list.

Cheers,

-Paul

[Gregg Reynolds]

On Wed, Oct 19, 2016 at 5:00 PM, Paul Gowder <paul....@gmail.com> wrote:

Thanks Gregg and Alex!  I didn't realize that protocols (or the black magic interfacing of core.matrix) were that fancy.  Definitely going into my "code to really dig into" list.

It's remotely possible that you might find the following helpful:  lab.clj.polymorphism.  It was in part inspired by the technique used in core.matrix.  See the section "Implementation techniques" in the readme.  At the time, it made sense to me. ;}

-gregg