reacting to an article on clojure protocols, ruby monkey patching, and safety
Laurent PETIT
The following article:
http://kirindave.tumblr.com/post/658770511/monkey-patching-gorilla-engineering-protocols-in
claims that, as far as code safety is concerned, clojure's solution is far better than e.g. ruby monkey patching.
If I understand things well, one problem with ruby monkey patching is that if a library I use opens a class C and adds a method whose signature is M to it, and if in my own code I also open the same class C and add a method whose signature is M to it, then the library may break because it will call my implementation of M, not the librarie's original one.
Now to clojure. I can see the same problem occur, while the article's author claims that in clojure there's (almost) no problem anymore.
If several libraries, including my program, redefine blindlessly a protocol implementation for the same type, then the "last to speak" wins.
So isn't the problem basically the same ?
Meikel Brandmeyer
On Thu, Jun 03, 2010 at 01:56:36PM +0200, Laurent PETIT wrote:
> Now to clojure. I can see the same problem occur, while the article's author
> claims that in clojure there's (almost) no problem anymore.
> If several libraries, including my program, redefine blindlessly a protocol
> implementation for the same type, then the "last to speak" wins.
> So isn't the problem basically the same ?
No. A protocol lives in a namespace. So its methods have to be
unique only in the same namespace. Types and reify implement
protocols (and their Java interface) at definition time. Later
on only the extend version via a map works. So the object itself
is not modified. Only the protocol functions know about these
extensions. And due to they belonging to a namespace it is clear
what has to be called by the protocol method you call.
And redefining things in foreign namespaces is rather not a
technique we should support...
Eh? Does that make sense?
Sincerely
Meikel
Christophe Grand
> If I understand things well, one problem with ruby monkey patching is that
> if a library I use opens a class C and adds a method whose signature is M to
> it, and if in my own code I also open the same class C and add a method
> whose signature is M to it, then the library may break because it will call
> my implementation of M, not the librarie's original one.
>
> Now to clojure. I can see the same problem occur, while the article's author
> claims that in clojure there's (almost) no problem anymore.
> If several libraries, including my program, redefine blindlessly a protocol
> implementation for the same type, then the "last to speak" wins.
> So isn't the problem basically the same ?
my 2 cents:
The surface of the problem is reduced because of namespacing: two
different "fold" methods (with different semantics) from two protocols
won't clash. Plus if there are two extensions of the same protocol to
the same type, they should be rather equivalent since they satisfies
the same semantics.
I think one must only extend a protocol to a type if he owns either
the type or the protocol.
Christophe
--
European Clojure Training Session: Brussels, 23-25/6 http://conj-labs.eu/
Professional: http://cgrand.net/ (fr)
On Clojure: http://clj-me.cgrand.net/ (en)
Konrad Hinsen
> If I understand things well, one problem with ruby monkey patching
> is that if a library I use opens a class C and adds a method whose
> signature is M to it, and if in my own code I also open the same
> class C and add a method whose signature is M to it, then the
> library may break because it will call my implementation of M, not
> the librarie's original one.
>
> Now to clojure. I can see the same problem occur, while the
> article's author claims that in clojure there's (almost) no problem
> anymore.
> If several libraries, including my program, redefine blindlessly a
> protocol implementation for the same type, then the "last to speak"
> wins.
> So isn't the problem basically the same ?
The difference is that with the Clojure approach, the type definition
and the protocol definition are separate and can well live in
different namespaces. Extending a protocol to a type is safe if your
code "owns" at least one of the two. With the Ruby or Python approach
of dynamically modifiable classes, the class definition defines both
the data and the protocol, all in one, so there is only one "owner"
and everyone else messing around with it will be bothered by sleepless
nights ;-)
So it is fair to say that Clojure has a safe solution for a much wider
range of situations, even though, as you noted, there is still the
possibility of interfering protocol implementations.
Konrad.
Richard Newman
> different "fold" methods (with different semantics) from two protocols
> won't clash. Plus if there are two extensions of the same protocol to
> the same type, they should be rather equivalent since they satisfies
> the same semantics.
Incidentally, this is the difference between pre-web and post-web
tools. Before the web, databases and identifiers would be "age", or
"name"; after the web we'd have "http://foo.com/age", or in Clojure
com.foo/age.
The former behaves as if it owns the entire world, and it's very
likely to result in collisions; the latter is very unlikely. This
issue of ambiguous identification is one of the problems that semantic
web tools try to solve.
The former, of course, is how Ruby works -- if you open the string
class and add a 'frob' method, then it will clash with someone else's
'frob' method. The latter is how Clojure works, because protocols are
namespaced, and good namespaces are reverse domains or otherwise
controlled.
If you use com.foo.bar/frob, you're claiming to obey the contract of
that particular frob operation, so even in the situation where
collision occurs, both implementations should at least share a
conception of what 'frobbing' means.
> I think one must only extend a protocol to a type if he owns either
> the type or the protocol.
That sounds like a reasonable rule of thumb, though I'd change it to
"publicly extend", or add "if you want to be safe" :)
Laurent PETIT
On Thu, Jun 3, 2010 at 1:56 PM, Laurent PETIT <lauren...@gmail.com> wrote:my 2 cents:
> If I understand things well, one problem with ruby monkey patching is that
> if a library I use opens a class C and adds a method whose signature is M to
> it, and if in my own code I also open the same class C and add a method
> whose signature is M to it, then the library may break because it will call
> my implementation of M, not the librarie's original one.
>
> Now to clojure. I can see the same problem occur, while the article's author
> claims that in clojure there's (almost) no problem anymore.
> If several libraries, including my program, redefine blindlessly a protocol
> implementation for the same type, then the "last to speak" wins.
> So isn't the problem basically the same ?
The surface of the problem is reduced because of namespacing: two
different "fold" methods (with different semantics) from two protocols
won't clash.
Of course. I was not thinking about that. I was talking about working on the same protocol.
Here is the scenario I have in mind: clojure 1.3 ships in december with loooots of protocols for the whole set of abstractions currently implemented as interfaces.
A bunch of developers all over the place start adding implementations for these protocols for types not covered by clojure itself ....
Plus if there are two extensions of the same protocol to
the same type, they should be rather equivalent since they satisfies
the same semantics.
... granted, the more specific the semantic will be, the less different the implementations should behave.
So back to my scenario: lots of people adding protocol impls for lots of "common java types" : new implementations each time, copied implementations some other time. Of course some implementations will be buggy the first time they are delivered. Depending on N libraries each providing their own implem. in a particular version could lead, depending on the order the libraries are loaded in the environment, in having sometimes a buggy version, sometimes an "almost correct version".
I think one must only extend a protocol to a type if he owns either
the type or the protocol.
But I understand the rule : "extend with great care if you neither own the protocol nor the type".
Laurent PETIT
Hi,
No. A protocol lives in a namespace. So its methods have to be
On Thu, Jun 03, 2010 at 01:56:36PM +0200, Laurent PETIT wrote:
> Now to clojure. I can see the same problem occur, while the article's author
> claims that in clojure there's (almost) no problem anymore.
> If several libraries, including my program, redefine blindlessly a protocol
> implementation for the same type, then the "last to speak" wins.
> So isn't the problem basically the same ?
unique only in the same namespace. Types and reify implement
protocols (and their Java interface) at definition time. Later
on only the extend version via a map works. So the object itself
is not modified. Only the protocol functions know about these
extensions. And due to they belonging to a namespace it is clear
what has to be called by the protocol method you call.
The above point has been answered to Christophe. I was not talking about the way that namespacing will reduce the problem by allowing different function names (with probably different semantics) to live in different namespaces. This is no different than distinguishing 2 ordinary functions with the same name, living in different namespaces, from the caller's point of vue.
And redefining things in foreign namespaces is rather not a
technique we should support...
Eh? Does that make sense?
How can the expression problem be claimed to be (almost) solved, if only the owner of a protocol and of a namespace can extend the protocol on types ?
Stuart Halloway
No, I don't understand your last point.The above point has been answered to Christophe. I was not talking about the way that namespacing will reduce the problem by allowing different function names (with probably different semantics) to live in different namespaces. This is no different than distinguishing 2 ordinary functions with the same name, living in different namespaces, from the caller's point of vue.
And redefining things in foreign namespaces is rather not a
technique we should support...
Eh? Does that make sense?
How can the expression problem be claimed to be (almost) solved, if only the owner of a protocol and of a namespace can extend the protocol on types ?
Laurent PETIT
I think I clearly understand the benefits of namespaces in this case. I was reacting to Meikel's sentence:
"And redefining things in foreign namespaces is rather not a technique we should support..."
Rich Hickey
On Jun 3, 9:28 am, Laurent PETIT <laurent.pe...@gmail.com> wrote:
> Hi,
>
> 2010/6/3 Christophe Grand <christo...@cgrand.net>
>
>
>
>
>
> > On Thu, Jun 3, 2010 at 1:56 PM, Laurent PETIT <laurent.pe...@gmail.com>
provide a definition.
Actually, I think the biggest issue will be people extend protocols to
types for which they don't make sense, e.g. for which the protocol
authored considered but rejected an implementation due to a semantic
mismatch. No extension will be there (by design), and people without
sufficient understanding/skills might fill the void with broken ideas.
I don't have a means to prevent this at present, but I'd like to
suggest this policy moving forward:
If a protocol comes with Clojure itself, avoid extending it to types
you don't own, especially e.g. java.lang.String and other core Java
interfaces. Rest assured if a protocol should extend to it, it will,
else lobby for it.
Rich
Meikel Brandmeyer
Am 03.06.2010 um 16:16 schrieb Laurent PETIT:
> I think I clearly understand the benefits of namespaces in this case. I was reacting to Meikel's sentence:
>
> "And redefining things in foreign namespaces is rather not a technique we should support..."
I think what I meant was mentioned also by the others: don't extend a protocol to a type if you don't own either one. Doing so if you don't own one of them, this is basically equivalent to
(in-ns 'some.other.namespace)
(let [orig-x some-x]
(defn some-x
[foo]
(if (my-foo? foo)
(do-stuff foo)
(orig-x foo))))
At least IMHO.
Sincerely
Meikel
Laurent PETIT
Sorry Meikel, but I'm having trouble following you today.
Does the above example stand for "pseudo-code" for explaining what happens when one reimplements a protocol (in which case I'm pretty sure you're wrong - redefining a protocol extension on a type redefines it for all following calls, from any thread), or does the above example stand for pseudo-code for how one would "try" to "break the rule" (mentioned by Christophe, Rich, and others) in a (hopefully) non-intrusive way ?
Laurent PETIT
Something like being able to declare a "black list" of symbols (e.g. #{ #"java\..*", #"clojure\.core\..*"} ) for which no implementation may be provided without e.g. raising a warning ?
I don't have a means to prevent this at present, but I'd like to
suggest this policy moving forward:
If a protocol comes with Clojure itself, avoid extending it to types
you don't own, especially e.g. java.lang.String and other core Java
interfaces. Rest assured if a protocol should extend to it, it will,
else lobby for it.
May I add this policy concerning Clojure protocols (as well as the rule "one must only extend a protocol to a type if he owns either the type or the protocol. If one breaks the rule, one should be prepared to withdraw should the implementor of either provide a definition") to the assembla Wiki page on best practices / conventions ?
Stuart Halloway
Yes please.
Laurent PETIT
I don't have a means to prevent this at present, but I'd like to
suggest this policy moving forward:
If a protocol comes with Clojure itself, avoid extending it to types
you don't own, especially e.g. java.lang.String and other core Java
interfaces. Rest assured if a protocol should extend to it, it will,
else lobby for it.
Another concern I have is when using multiple libraries from separate "vendors", with potentially long chains of transitive dependencies. I can force myself into respecting the above mentioned rules and laws. Much harder to be sure that every library I'll use does that too. I would like to have a way to be "warned" that something wrong may happen. It doesn't bother me if this happens at runtime (worst case if all the libraries I use are already compiled and I don't own the source code) : when the libraries are "referred", I would like to be able to spot warning messages if a redefinition of a protocol on a type is done in more than one namespace ("in more than one namespace", so that working interactively in the REPL would not trigger the warnings).
Meikel Brandmeyer
Am 03.06.2010 um 21:51 schrieb Laurent PETIT:
>> (in-ns 'some.other.namespace)
>>
>> (let [orig-x some-x]
>> (defn some-x
>> [foo]
>> (if (my-foo? foo)
>> (do-stuff foo)
>> (orig-x foo))))
>
> Sorry Meikel, but I'm having trouble following you today.
> Does the above example stand for "pseudo-code" for explaining what happens when one reimplements a protocol (in which case I'm pretty sure you're wrong - redefining a protocol extension on a type redefines it for all following calls, from any thread), or does the above example stand for pseudo-code for how one would "try" to "break the rule" (mentioned by Christophe, Rich, and others) in a (hopefully) non-intrusive way ?
The above is working code (modulo foo related implementations). You can at any moment change namespaces. Change some function there and go back to your original namespace. This is effectively monkey-patching. This works now.
So now suppose seq does not support strings. Someone goes in to "fix" this in his library. Say, his fix returns (\f \o \o). Someone else also "fixes" seq: ("f" "o" "o"). You load both libraries and are in trouble. The same happens if both extend the Sequable protocol for j.l.String.
So extending a protocol you don't own to a type you don't own is like going into someone else's namespace and "fixing" things there. Not doing so is a question of good manners, which are in general lacking nowadays… So let's hope that we are all well-behaved and don't adopt such techniques.
Sincerely
Meikel
Laurent PETIT
> May I add this policy concerning Clojure protocols (as well as the rule "one must only extend a protocol to a type if he owns either the type or the protocol. If one breaks the rule, one should be prepared to withdraw should the implementor of either provide a definition") to the assembla Wiki page on best practices / conventions ?Yes please.
Laurent PETIT
I'm fearing that too, and that's the whole point of me creating this thread to start with. The spirit of clojure will not (probably) be to prevent this (this would remove power from the user, though there is a precedent with the non introduction of reader macros), but I would like at least to know how I could "audit" my project (once all its dependencies have been set up) to spot where the problems (conflicting protocols redefinitions) are.