Bizarre issue when calling protocol-fn wrapper from different namespace

[Dimitrios Jim Piliouras]

Hi folks,

This has me completely stumped - I would massively appreciate a helping hand! Suppose the following simple directory structure:


— someProject.impl.{foo.clj, bar.clj,baz.clj}
— someProject.proto.clj
— someProject.api.clj

`proto.clj` contains a single protocol with two methods - let’s call them X & Y. Each implementation namespace (foo, bar, baz), requires `[someProject.proto :as proto]`, extends it to 3 types (bytes/chars/String), and defines two public fns x & y which delegate to `proto/X` & `proto/Y` respectively. Everything is good so far. I can fire up a REPL, load any of the impl namespaces (foo, bar, baz), call the corresponding x or y fn and get the right result.

Now, I want to provide a unified API so that the caller doesn’t need to (potentially) require 3 namespaces. Hence the `someProject.api` ns, which contains require clauses for all impl namespaces + two multi-methods `X-with` & `Y-with` with 3 implementations (`defmethod`) each. Each implementation delegates to the x or y fn in the right impl namespace. In other words, `X-with :foo` calls `(foo/x)`, `X-with :bar` calls `(bar/x)` etc etc. Remember, that calling x or y inside any impl namespace works correctly, so all I’m doing here is providing a multi-method wrapper. However, things don’t work as I was expecting in this namespace…Loading `someProject.api` in a fresh REPL and calling `X-with :foo` bottoms out at the protocol extension for X in the baz ns, which is the last require clause in the api namespace.

So basically, the protocol extensions in each impl namespace work fine when called from their wrapper fn in the namespace they were defined, but don’t quite work when the same wrapper fn is called from some other namespace! What am I missing? :(

Many thanks in advance…
Dimitris

[Justin Smith]

if you define proto method x, it belongs to the protocol namespace no
matter where it is called, and calling it as if it belonged to the
namespace defining the object extending the protocol will and should
fail

> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clo...@googlegroups.com
> Note that posts from new members are moderated - please be patient with your first post.
> To unsubscribe from this group, send email to
> clojure+u...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/clojure/A549FB15-0B12-4E20-9D98-4F5A56330DC4%40gmail.com.

[Justin Smith]

it might be helpful to consider that in the jvm methods are not data,
and the proto function makes the method into concrete data belongs to
the namespace that owns the protocol

[Dimitrios Jim Piliouras]

But the call-chain is api/x-with-foo => foo/x => proto/X so it does bottom out in the ns the protocol was defined in. It's just that the middle step could come from 3 different namespaces all containing protocol extensions.

To view this discussion on the web visit https://groups.google.com/d/msgid/clojure/CAGokn9%2B1ZiFHBzuS_bwSyrx0DsM_ocu7dk844DnU-h5D3b9fzA%40mail.gmail.com.

[Justin Smith]

there is no foo/x unless you defined one - the protocol function is
created by defprotocol and is not owned by the object implementing the
protocol

On Thu, Nov 21, 2019 at 3:29 PM Dimitrios Jim Piliouras

> To view this discussion on the web visit https://groups.google.com/d/msgid/clojure/CAE3Kzw%2B6BqfX3di1xgE8UM2ngpBBhdzCi4GpCEtAeVX56oyGEg%40mail.gmail.com.

[Justin Smith]

on rereading I've clearly misunderstood you, I think we need to see
actual code reproducing this error in order to know what failed here

[Dimitrios Jim Piliouras]

Ok, here is the protocol definition:

https://github.com/jimpil/cryptohash-clj/blob/master/src/cryptohash_clj/proto.clj

Here are the implementations (notice the two public fns at the end of each file):

https://github.com/jimpil/cryptohash-clj/tree/master/src/cryptohash_clj/impl

And here is the api with the two multi-methods:

https://github.com/jimpil/cryptohash-clj/blob/master/src/cryptohash_clj/api.clj

Everything works as expected inside the impl namespaces, but not in the api namespace, even though all it does is to delegate to some impl ns.

Thanks again...

Kind regards,

Dimitris

ps: this repo is WIP

To view this discussion on the web visit https://groups.google.com/d/msgid/clojure/CAGokn9%2BPgbiZxq-n7L%3DgLhqNB2z-OdCaTOx7uJA5zG2NavJLiQ%40mail.gmail.com.

[Justin Smith]

if I'm not mistaken, the usage of apply in the api ns ensures that the
method will only be found if "opts" is exactly one opt, otherwise the
arity won't be found and that will be reported as no such method

On Fri, Nov 22, 2019 at 12:37 AM Dimitrios Jim Piliouras

> To view this discussion on the web visit https://groups.google.com/d/msgid/clojure/767A3F45-4F93-42ED-93B3-36B962FEC918%40gmail.com.

[dimitris]

I don't fully follow, I'm afraid...

I am applying a regular fn (e.g. `b/chash`) to a list of args which can be 1-2 or 2-3 for (depending on which multi-method we're talking about). The regular functions being applied, all provide two arities for the respective invocations, so everything is covered. I can literally copy/paste the apply expression into the corresponding impl namespace and it works fine on the REPL.

As I understand it, protocol extensions are namespace-specific. Otherwise, you couldn't extend the same protocol over the same type in two different namespaces. So the only way to have protocol extensions be useful outside the namespace they were defined in, is to provide wrapper functions that delegate to the protocol invocations in that ns, and call those instead. Well, that's what I've done...it shouldn't matter where the protocol is defined, right?

I've never encountered a "no such method" error so I'm not sure how this could be playing a role here...

[Carlo Zancanaro]

Hey Dimitris,

I think what you're running into here is a fundamental
misunderstanding of protocols. Specifically, this is not true:

> As I understand it, protocol extensions are namespace-specific.

In Clojure, protocol extensions are global. If you extend a
protocol for a type then it doesn't matter which namespace calls
the protocol function, it only matters what type the function is
called on. Protocols are approximately interfaces in Java, except
they can have implementations added to existing types at runtime.

With your current code, I am a little bit confused why you want to
use a protocol. You have a multimethod which performs dispatch
based on a key, but which then calls the same protocol function
for each dispatch value - why not just call the appropriate
function in the right namespace? The protocol seems to be
attempting to add a second level of indirection, but it's not
clear to me how or why that is useful.

Carlo

[dimitris]

Hi Carlo thanks for you reply...

Quoted from clojure.org/reference/protocols:

Avoid the 'expression problem' by allowing independent extension of the set of types, protocols, and implementations of protocols on types, by different parties.

What you said about protocol extensions being global directly contradicts the above. You can definitely extend the same protocol on the same type in multiple namespaces (i.e. independent extension by different parties). If extensions were global that would be no different than monkey-patching, right?

Now, you ask why I need the protocol in the first place. That's to provide polymorphism for bytes/chars/String input. The multi-method is there to provide polymorphism for all the different hashing implementations (bcrypt/scrypt/pbkdf2/argon2). These two are completely separate and shouldn't interfere.

[dimitris]

Ok, you might be right after all :(...

I called :impls on the protocol and it returns a map from Class => impls-map. So my understanding was not not quite right...Yes, you can extend the same protocol in multiple namespaces, but for distinct types only! So basically in my case, i need to copy/paste the protocol in each impl namespace separately.

Many thanks for your insight Carlo :)