Datatypes and Protocols - early experience program

2344 views
Skip to first unread message

Rich Hickey

unread,
Nov 12, 2009, 7:10:37 AM11/12/09
to Clojure
An early version of the code for a few important new language
features, datatypes[1] and protocols[2] is now available in the 'new'
branch[3]. Note also that the build system[4] has builds of the new
branch, and that the new branch works with current contrib.

If you have the time and inclination, please try them out. Feedback is
particularly welcome as they are being refined.

Thanks,

Rich

[1] http://www.assembla.com/wiki/show/clojure/Datatypes
[2] http://www.assembla.com/wiki/show/clojure/Protocols
[3] http://github.com/richhickey/clojure/tree/new
[4] http://build.clojure.org/

Sean Devlin

unread,
Nov 12, 2009, 8:29:34 AM11/12/09
to Clojure
Rich,
Just read the section on reify. I'm not quite sure what this new
mechanism lets me do. Could you provide an example of the problem it
solves? I personally would benefit from seeing the "Old, painful way"
contrasted to the "New, awesome way". This would probably help with
the other features too.

Thanks,
Sean

Rich Hickey

unread,
Nov 12, 2009, 9:39:05 AM11/12/09
to Clojure


On Nov 12, 8:29 am, Sean Devlin <francoisdev...@gmail.com> wrote:
> Rich,
> Just read the section on reify.  I'm not quite sure what this new
> mechanism lets me do.  Could you provide an example of the problem it
> solves?  I personally would benefit from seeing the "Old, painful way"
> contrasted to the "New, awesome way".  This would probably help with
> the other features too.
>

reify is the most subtle, as it is a subset of proxy, limited to
implementing interfaces only, and less dynamic (no equivalent to
update-proxy). What you get in return is a construct with fewer host
implications, and much better performance, as stated in the wiki doc:

"The result is better performance than proxy, both in construction
(proxy creates the instance and a fn instance for each method), and
invocation. reify is preferable to proxy in all cases where its
limitations are not prohibitive."

Rich

Michael Jaaka

unread,
Nov 12, 2009, 5:17:51 PM11/12/09
to clo...@googlegroups.com
Oh its looks like Google Go (http://golang.org) and Nice Interfaces (http://nice.sourceforge.net/).
Good! It sounds better than overrated polyphormism and class hierarchy.

Wiadomość napisana przez Rich Hickey w dniu 2009-11-12, o godz. 15:39:
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clo...@googlegroups.com
> Note that posts from new members are moderated - please be patient with your first post.
> To unsubscribe from this group, send email to
> clojure+u...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en

Meikel Brandmeyer

unread,
Nov 12, 2009, 6:54:09 PM11/12/09
to clo...@googlegroups.com
Hi,

Am 12.11.2009 um 13:10 schrieb Rich Hickey:

> An early version of the code for a few important new language
> features, datatypes[1] and protocols[2] is now available in the 'new'
> branch[3]. Note also that the build system[4] has builds of the new
> branch, and that the new branch works with current contrib.
>
> If you have the time and inclination, please try them out. Feedback is
> particularly welcome as they are being refined.

I implemented my lazymap library in terms reify where gen-class was
required before. Seems to work smoothly, but it is also only a simple
lib. Should ISeqs still extend ASeq? Is this a case where we still
need gen-class?

Otherwise I hadn't much chance to test... http://bitbucket.org/kotarak/lazymap/

Sincerely
Meikel

Chouser

unread,
Nov 12, 2009, 7:59:08 PM11/12/09
to clo...@googlegroups.com
On Thu, Nov 12, 2009 at 7:10 AM, Rich Hickey <richh...@gmail.com> wrote:
>
> If you have the time and inclination, please try them out. Feedback is
> particularly welcome as they are being refined.

For what it's worth, here are 2-3 finger trees implemented using
defprotocol and deftype.

http://tinyurl.com/yeh5fgg/finger_tree.clj

Here's an earlier version that's almost idectical except its
implemented using def-interface and reify instead:

http://tinyurl.com/y9jned5/finger_tree.clj

--Chouser

Chouser

unread,
Nov 12, 2009, 8:22:35 PM11/12/09
to clo...@googlegroups.com
On Thu, Nov 12, 2009 at 7:59 PM, Chouser <cho...@gmail.com> wrote:
> On Thu, Nov 12, 2009 at 7:10 AM, Rich Hickey <richh...@gmail.com> wrote:
>>
>> If you have the time and inclination, please try them out. Feedback is
>> particularly welcome as they are being refined.
>
> For what it's worth, here are 2-3 finger trees implemented using
> defprotocol and deftype.
>
> http://tinyurl.com/yeh5fgg/finger_tree.clj

I should have noted that this is a very early version and doesn't
yet take advantage of some features now available like reusing
the same protocol function in multiple protocols, or using 'case'
instead of 'cond' or 'condp'.

--Chouser

Krukow

unread,
Nov 13, 2009, 2:13:15 AM11/13/09
to Clojure


On Nov 12, 1:10 pm, Rich Hickey <richhic...@gmail.com> wrote:
> An early version of the code for a few important new language
> features, datatypes[1] and protocols[2] is now available in the 'new'
> branch[3]. Note also that the build system[4] has builds of the new
> branch, and that the new branch works with current contrib.
>
> If you have the time and inclination, please try them out. Feedback is
> particularly welcome as they are being refined.

I really like the semantics of your constructs. I have a comment about
regularity of syntax:

The way to specify method names in reify and deftype vs. function
names defprotocol and extend are different. It looks like when dealing
with interface-method implementations one uses .methodName (i.e., with
the dot), but when dealing with protocol functions one uses no dot.
Further, extend uses maps (the docs says why this is the case).

I was thinking this may make syntax irregular. I suspect this is a
deliberate design choice to distinguish clojure protocols from java
interfaces? Is this the case?

A stupid example:

;;uses dot
(deftype Sometype [x]
[java.lang.Comparable]
(.compareTo [o] ...))

::uses no dot
(defprotocol RSeqable :on clojure.lang.Seqable
"Seqable and reverse seqable"
(rseq [s] "reverse seq"))

;;do I mix dot and not?
(extend ::Sometype
:RSeqable
{:rseq (fn [a]...))
:.seq (fn [a] ...)} ;; do I write :.seq here or :seq?

I guess one can reintroduce the regularity using the :on feature of
protocol functions.

Any thoughts?
/Karl

Mark Engelberg

unread,
Nov 13, 2009, 2:13:57 AM11/13/09
to clo...@googlegroups.com
I'm still trying to get my head around the new features. Seeing more
code examples will definitely help. In the meantime, here is some
stream-of-consciousness thoughts and questions.

Datatypes:

I'm a little worried about the strong overlap between reify/proxy,
deftype/defstruct, and defclass/gen-class. I can just imagine the
questions a year from now when people join the Clojure community and
want to understand how they differ. So I think that eventually, there
needs to be a very clear "story" as to why you'd choose one over the
other. Or better yet, maybe some of the older constructs can be
phased out completely.

Is there a way to customize the way that types defined by deftype
print in the REPL?

While these datatype and protocol constructs are taking shape, maybe
now is the time to discuss what kind of "privacy" settings are
worthwhile in a language like Clojure. I think Java's system of
private/public/protected is probably overkill for Clojure. But do
people feel that some degree of data hiding is worthwhile? For
example, might you want to hide some deftype fields from keyword
lookup?

Protocols:

I don't understand whether there's any way to provide a partial
implementation or default implementation of a given
protocol/interface, and I believe this to be an important issue.

For example, a protocol for < and > that provides a default
implementation of > in terms of < and a default implementation of < in
terms of >, so that you only need to implement one and you get the
other for free.

I'm also thinking about the relationship in Clojure's source between
ISeq and ASeq. ASeq provides the partial, default implementation of
more in terms of next, for example. How does this kind of thing look
with the new protocol system?

Konrad Hinsen

unread,
Nov 13, 2009, 3:58:19 AM11/13/09
to clo...@googlegroups.com
On 13 Nov 2009, at 08:13, Mark Engelberg wrote:

> Is there a way to customize the way that types defined by deftype
> print in the REPL?

Implement the multimethod clojure.core/print-method for the associated
type tag:

(deftype Foo ...)

(defmethod clojure.core/print-method ::Foo [x] ...)

> While these datatype and protocol constructs are taking shape, maybe
> now is the time to discuss what kind of "privacy" settings are
> worthwhile in a language like Clojure. I think Java's system of
> private/public/protected is probably overkill for Clojure. But do
> people feel that some degree of data hiding is worthwhile? For
> example, might you want to hide some deftype fields from keyword
> lookup?

Coming from a Python background, I don't think access restrictions are
necessary. However, flagging fields as "not meant for use by
outsiders" could be of interest for documentation tools, to make it
clear what client code can safely rely on.

Note also that you can always make the deftype private (it's a var
like any other) and restrict all use of it to public functions defined
in the same namespace. That doesn't exclude defining protocols and
multimethods on this type elsewhere, because its type tag is a
namespace-qualified symbol that can be used anywhere.

> Protocols:
>
> I don't understand whether there's any way to provide a partial
> implementation or default implementation of a given
> protocol/interface, and I believe this to be an important issue.

I don't think that partial implementations are possible at the moment,
but I agree that it would be useful. A default implementation can be
provided as an implementation of Object. It's not quite the same as a
default implementation for a multimethod, as it doesn't apply to types
identified by a metadata type tag, but in practice it can be good
enough or even better.

> For example, a protocol for < and > that provides a default
> implementation of > in terms of < and a default implementation of < in
> terms of >, so that you only need to implement one and you get the
> other for free.

Right. I have such a case in my test implementation for multiarrays
(soon to be put on Google Code...), where I'd want to define a default
implementation for "rank" as "length of the shape vector".

Konrad.

Alex Osborne

unread,
Nov 13, 2009, 4:55:12 AM11/13/09
to clo...@googlegroups.com
Mark Engelberg wrote:
> Protocols:
>
> I don't understand whether there's any way to provide a partial
> implementation or default implementation of a given
> protocol/interface, and I believe this to be an important issue.
>
> For example, a protocol for < and > that provides a default
> implementation of > in terms of < and a default implementation of < in
> terms of >, so that you only need to implement one and you get the
> other for free.
>

How about this?

(defprotocol MyComparable
(comparinate [x y] "Returns a negative integer if x < y, zero if they
are equal and a positive integer if x > y.")
(equal-to [x y] "True if x is equal to y.")
(less-than [x y] "True if x is smaller than y.")
(greater-than [x y] "True if x is greater than y."))

(defn mixin-comparable [type compare-fn]
(extend type
MyComparable
{:comparinate compare-fn
:equal-to (fn [x y] (zero? (comparinate x y)))
:less-than (fn [x y] (neg? (comparinate x y)))
:greater-than (fn [x y] (neg? (comparinate x y)))}))

(mixin-comparable Integer -)
(mixin-comparable String #(- (count %1) (count %2)))

(less-than 8 2) ; => false
(less-than "x" "xxxx") ; => true

> I'm also thinking about the relationship in Clojure's source between
> ISeq and ASeq. ASeq provides the partial, default implementation of
> more in terms of next, for example. How does this kind of thing look
> with the new protocol system?

See above. But another way would just be to define ASeq as a map and
then merge with it:

(def aseq-impl {:more (fn [obj] (if-let [s (next obj)] s '() ))
:count (fn [obj] ...)})

(extend MyList
ISeq
(merge
aseq-impl
{:next (fn [lst] ...)
:count (fn [lst] ...) ; overrides the count from aseq-impl
:cons (fn [lst x] ...)}))

Code is data. :-)

I don't know whether doing things this way is a good idea or not, but
protocols are new: lets experiment and find out what works and what doesn't.

Jarkko Oranen

unread,
Nov 13, 2009, 5:26:38 AM11/13/09
to Clojure
On Nov 13, 9:13 am, Krukow <karl.kru...@gmail.com> wrote:
> I was thinking this may make syntax irregular. I suspect this is a
> deliberate design choice to distinguish clojure protocols from java
> interfaces? Is this the case?
>

As far as I understand it, in defprotocol's case, I suspect there is
no dot because the specified operations will be available as normal
Clojure functions, whereas in deftype's case you'll need to use Java
interop or keywords. For example, after

(defprotocol Someproto
(foo [x] "do stuff"))

you will be able to call (foo something-implementing-someproto), but
with deftype, you need to use (.field instance) or, if the type uses
the default ILookup implementation, (:field instance).

The extend example should just use :seq, as defprotocol will create a
function "seq" matching the .seq method in the Seqable interface.
(Because no explicit mapping is provided)

I hope I got my details right here. I haven't actually tried these
things yet. :)

Chris Kent

unread,
Nov 13, 2009, 5:27:22 AM11/13/09
to clo...@googlegroups.com
Mark Engelberg <mark.engelberg <at> gmail.com> writes:

> I'm a little worried about the strong overlap between reify/proxy,
> deftype/defstruct, and defclass/gen-class. I can just imagine the
> questions a year from now when people join the Clojure community and
> want to understand how they differ. So I think that eventually, there
> needs to be a very clear "story" as to why you'd choose one over the
> other. Or better yet, maybe some of the older constructs can be
> phased out completely.

What are the plans for the future of proxy? I assume it won't go away because
reify's inability to extend an existing class is a show-stopper for some Java
interop scenarios. Will the syntax be brought in line with reify so dots will
be needed in front of method names? As things stand it's a potential source of
confusion to have two such similar features with subtly different syntax.

Chris


Meikel Brandmeyer

unread,
Nov 13, 2009, 3:17:58 AM11/13/09
to Clojure
Hi,

On Nov 13, 8:13 am, Mark Engelberg <mark.engelb...@gmail.com> wrote:

> Is there a way to customize the way that types defined by deftype
> print in the REPL?

One can add a method to print-method for the type.

> While these datatype and protocol constructs are taking shape, maybe
> now is the time to discuss what kind of "privacy" settings are
> worthwhile in a language like Clojure.  I think Java's system of
> private/public/protected is probably overkill for Clojure.  But do
> people feel that some degree of data hiding is worthwhile?  For
> example, might you want to hide some deftype fields from keyword
> lookup?

I for now don't care for privacy settings. Everything is public. The
docstrings explain the contract. Period.

> I'm also thinking about the relationship in Clojure's source between
> ISeq and ASeq.  ASeq provides the partial, default implementation of
> more in terms of next, for example.  How does this kind of thing look
> with the new protocol system?

A pretty simple solution is (.more [] (lazy-seq (next this))), no? But
I stumbled over this, too. APersistentMap does a lot like implementing
the IFn invokes for key lookup or the IPersistentCollection equiv.
Giving up on this will create a lot of more work.

Sincerely
Meikel

Rich Hickey

unread,
Nov 13, 2009, 7:56:56 AM11/13/09
to clo...@googlegroups.com
On Fri, Nov 13, 2009 at 2:13 AM, Krukow <karl....@gmail.com> wrote:
>
>
> On Nov 12, 1:10 pm, Rich Hickey <richhic...@gmail.com> wrote:
>> An early version of the code for a few important new language
>> features, datatypes[1] and protocols[2] is now available in the 'new'
>> branch[3]. Note also that the build system[4] has builds of the new
>> branch, and that the new branch works with current contrib.
>>
>> If you have the time and inclination, please try them out. Feedback is
>> particularly welcome as they are being refined.
>
> I really like the semantics of your constructs. I have a comment about
> regularity of syntax:
>
> The way to specify method names in reify and deftype vs. function
> names defprotocol and extend are different.

That's because methods and functions *are* different. I think making
them look the same only makes it more confusing since:

- methods can only be defined by the definer of a type, protocol
extension fns can be defined by anyone anywhere

- methods have class scope - direct use of fields as locals,
functions must use (:field self)

- functions are first class values and can be put in maps etc, methods can't

- methods have implicit this, functions don't

- functions can be closures, methods, other than in reify, can't

> It looks like when dealing
> with interface-method implementations one uses .methodName (i.e., with
> the dot), but when dealing with protocol functions one uses no dot.
> Further, extend uses maps (the docs says why this is the case).
>
> I was thinking this may make syntax irregular. I suspect this is a
> deliberate design choice to distinguish clojure protocols from java
> interfaces? Is this the case?

Yes.

>
> A stupid example:
>
> ;;uses dot
> (deftype Sometype [x]
>   [java.lang.Comparable]
>   (.compareTo [o] ...))
>
> ::uses no dot
> (defprotocol RSeqable :on clojure.lang.Seqable
>  "Seqable and reverse seqable"
>  (rseq [s] "reverse seq"))
>
> ;;do I mix dot and not?
> (extend ::Sometype
>  :RSeqable
>   {:rseq (fn [a]...))
>    :.seq (fn [a] ...)} ;; do I write :.seq here or :seq?
>

You don't mix methods and protocol functions, so once that is clear I
don't think this will be a question.

In a sense, deftypes and protocols are bridging two polymorphism
systems. As long as one doesn't conflate the two, it becomes clearer.

For instance, you could use deftype and protocols in complete
ignorance/avoidance of Java and interfaces:

(deftype Foo [a b c])

(defprotocol P (bar [x] "bar docs"))

(extend ::Foo P {:bar (fn [afoo] :foo-thing)})

(bar (Foo 1 2 3))
:foo-thing

This is a simple, powerful, flexible and dynamic system, leveraging
one's understanding of Clojure functions.

If and only if there is some requirement that instances of Foo
implement some Java interfaces, then you will need to understand Java
interfaces and methods. And there will be a clear mechanism and place
to put them - in your deftype, just like methods have to be put inside
class definitions in Java. You have similar class scope for fields and
access to this, etc. One thing you do not have is implicit scope for
methods, the leading dot helps remind you that in order to call
someMethod, even in the body of another method in the same deftype,
you will have to use (.someMethod this ...). People have argued
against implicit this for similar reasons, and I am starting to come
around :)

The documentation is comprehensive in mentioning everything you can
do. But one doesn't need to use everything.

Rich

MikeM

unread,
Nov 13, 2009, 9:03:11 AM11/13/09
to Clojure

> (deftype Foo [a b c])
>
> (defprotocol P (bar [x] "bar docs"))
>
> (extend ::Foo P {:bar (fn [afoo] :foo-thing)})
>

A common error may be to:

(extend Foo P {:bar (fn [afoo] :foo-thing)})

when (extend ::Foo ... is intended. I notice that (extend Foo...
doesn't throw - should extend check that it is supplied a class,
intfc, or keyword and throw if something else is supplied?
Alternately, could extend be changed to allow (extend Foo ... and do
the right thing ? ie determine that Foo is the constructor function
for a type and do the extension for the type Foo.

James Reeves

unread,
Nov 13, 2009, 9:24:12 AM11/13/09
to Clojure
Are there any plans to use protocols to define polymorphic functions
like conj and get? Perhaps with an "untype" function to remove type
metadata so one could always get at the datastructures hidden by the
protocol. e.g.

(defn sql-get [table key]
(sql-query
(str "select * from " table " where " (get (untype table) :primary-
key) " = ?")
key))

(extend ::sql-table Gettable
{:get sql-get})

Then 'get' could be used in a more generic fashion:

user=> (def accounts (sql-table accounts))

user=> (get accounts 10)
{:id 10, :login "jsmith", :password "1234"}

- James

Rich Hickey

unread,
Nov 13, 2009, 9:30:20 AM11/13/09
to Clojure


On Nov 13, 2:13 am, Mark Engelberg <mark.engelb...@gmail.com> wrote:
> I'm still trying to get my head around the new features. Seeing more
> code examples will definitely help. In the meantime, here is some
> stream-of-consciousness thoughts and questions.
>
> Datatypes:
>
> I'm a little worried about the strong overlap between reify/proxy,
> deftype/defstruct, and defclass/gen-class. I can just imagine the
> questions a year from now when people join the Clojure community and
> want to understand how they differ. So I think that eventually, there
> needs to be a very clear "story" as to why you'd choose one over the
> other. Or better yet, maybe some of the older constructs can be
> phased out completely.
>

Yes, but there will be a transition period. I certainly tried to
explain the decision points on the wiki.

A big part of the design thinking behind these features went like
this:

Clojure is built on a set of abstractions, and leverages/requires that
the host platform provide some sort of high-performance polymorphism
construct in order to make that viable. That said, Clojure was
bootstrapped on the host language and didn't really provide similar
constructs itself (multimethods are more powerful but slower), leaving
people that wanted to do things similar to what I did, in order to
write Clojure and its data structures, to either write Java or use
Clojure interop to, effectively, write Java in Clojure clothing.

So I took a step back and said, what part of Java did I *need* in
order to implement Clojure and its data structures, what could I do
without, and what semantics was I willing to support - for Clojure -
i.e. not in terms of interop. What I ended up with was - a high-
performance way to define and implement interfaces. What I explicitly
left out was - concrete derivation and implementation inheritance.

reify is Clojure semantics and proxy is Java/host semantics. Why
doesn't it replace proxy? Because proxy can derive from concrete
classes with constructors that take arguments. Supporting that
actually brings in a ton of semantics from Java, things I don't want
in Clojure's semantics. reify should be possible and portable in any
port of Clojure, proxy may not. Will the performance improvements of
reify make it into proxy? Probably at some point, not a priority now.

*** Prefer reify to proxy unless some interop API forces you to use
proxy. You shouldn't be creating things in Clojure that would require
you to use proxy. ***

defstruct is likely to be completely replaced by deftype, and at some
point could be deprecated/removed.

*** Prefer deftype to defstruct, unconditionally. ***

AOT deftype vs gen-class touches on the same Clojure semantics vs Java/
host semantics, with the objectives from before - support implementing
interfaces but not concrete derivation. So, no concrete base classes,
no super calls, self-ctor calls, statics, methods not implementing
interface methods etc. Will the performance improvements of deftype
make it into gen-class? Probably at some point, not a priority now.

Like proxy, gen-class will remain as an interop feature.

*** Prefer deftype to gen-class unless some interop API forces you to
use gen-class. ***

There will be a definterface similar to and probably replacing gen-
interface, with an API to match deftype.

So, with definterface, deftype, and reify you have a very clean way to
specify and implement a subset of the Java/C# polymorphism model, that
subset which I find clean and reasonable, with an expectation of
portability, and performance exactly equivalent to the same features
on the host.

I could have stopped there, and almost did. But there are three
aspects of that polymorphism model that aren't sufficient for Clojure:

- It is insufficiently dynamic. There is a static component - named
interfaces, that must be AOT compiled.

- Client code must use the interop style (.method x), and type hints,
in order to tap into the performance

- It is 'closed' polymorphism, i.e. the set of things a type can do
is fixed at the definition time of the type. This results in the
'expression problem', in this case the inability to extend types with
new capabilities/functions.

We've all experienced the expression problem - sometimes you simply
can't request/require that some type implement YourInterface in order
to play nicely with your design. You can see this in Clojure's
implementation as well - RT.count/seq/get etc all try to use Clojure's
abstraction interface first, but then have hand-written clauses for
types (e.g. String) that couldn't be retrofitted with the interface.

Multimethods, OTOH, don't suffer from this problem. But it is
difficult to get something as generic as Clojure's multimethods to
compete with interface dispatch in Java. Also, multimethods are kind
of atomic, often you need a set of them to completely specify an
abstraction. Finally, multimethods are a good story for the Clojure
side of an abstraction, but should you define a valuable abstraction
and useful code in Clojure and want to enable extension or
interoperation from Java or other JVM langs, what's the recipe?

Protocols take a subset of multimethod power, open extension, combine
it with a fixed, but extremely common, dispatch mechanism (single
dispatch on 'type' of first arg), allow a set of functions
constituting an abstraction to be named, specified, and implemented as
group, and provide a clear way to extend the protocol using ordinary
capabilities of the host (:on interface).

*** Prefer using protocols to specify your abstractions, vs
interfaces. ***

This will give you open extension and a dynamic system. You can always
make your protocol reach any type, and, you can always make your
protocol extensible through an interface using :on interface. In
particular note, calls to a protocol fn to an instance of the :on
interface go straight through, and are as fast as calls using (.method
#^AnInterface x), so there is no up-front performance compromise in
choosing protocols.


> While these datatype and protocol constructs are taking shape, maybe
> now is the time to discuss what kind of "privacy" settings are
> worthwhile in a language like Clojure. I think Java's system of
> private/public/protected is probably overkill for Clojure. But do
> people feel that some degree of data hiding is worthwhile?

I don't.

> Protocols:
>
> I don't understand whether there's any way to provide a partial
> implementation or default implementation of a given
> protocol/interface, and I believe this to be an important issue.
>
> For example, a protocol for < and > that provides a default
> implementation of > in terms of < and a default implementation of < in
> terms of >, so that you only need to implement one and you get the
> other for free.
>
> I'm also thinking about the relationship in Clojure's source between
> ISeq and ASeq. ASeq provides the partial, default implementation of
> more in terms of next, for example. How does this kind of thing look
> with the new protocol system?

This was an important consideration in the deftype/protocol design.
One reasonable argument for concrete implementation is abstract
superclasses, especially when used correctly. And Clojure's
implementation does use them, as you note. Some of the problems with
abstract classes are:

- they create a hierarchical type relationship (for no good reason).

- unless you are going to open that huge can of worms that is
multiple concrete inheritance, you only get a single inheritable
implementation.

- they, too, are closed. If you are going to allow open extension,
but implementation reuse requires derivation, there is an open/closed
mismatch.

Protocols are designed to support hierarchy-free, open, multiple,
mechanical mixins. This is enabled by the fact that extend is an
ordinary function, and the mappings of names to implementation
functions are ordinary maps. One can create mixins by simply making
maps of names to functions. And one can use mixins in an ad hoc
manner, merging and replacing functions using ordinary map
manipulation:

(extend ::MyType AProtocol (assoc a-mixin-map :a-fn-to-replace a-
replacement-fn))

I think people will find this quite powerful and programmable.

Rich

AlexK

unread,
Nov 13, 2009, 9:26:54 AM11/13/09
to Clojure
Hi everybody,

after playing around with protocols & datatypes, I found them very fun
to use.
Some questions:
Performance
I don't see (with my limited benchmarking) any significant difference
between multifns and protocolfns:

user=> (defprotocol Test (protocol-fn [it] "Protocol-fn"))
Test
user=> (extend Object Test {:protocol-fn (fn [it] nil)})
nil
user=> (defmulti multi-fn type)
#'user/multi-fn
user=> (defmethod multi-fn Object [it] nil)
#<MultiFn clojure.lang.MultiFn@c8769b>
user=> (defn simple-fn [it] nil)
#'user/simple-fn

user=> (dotimes [_ 10] (time (dotimes [_ 100000] (protocol-fn :it))))
"Elapsed time: 105.532562 msecs"
"Elapsed time: 57.0031 msecs"
"Elapsed time: 33.210602 msecs"
"Elapsed time: 30.47827 msecs"
"Elapsed time: 26.326202 msecs"
"Elapsed time: 27.764654 msecs"
"Elapsed time: 28.381284 msecs"
"Elapsed time: 28.741735 msecs"
"Elapsed time: 28.697525 msecs"
"Elapsed time: 25.894514 msecs"

user=> (dotimes [_ 10] (time (dotimes [_ 100000] (multi-fn :it))))
"Elapsed time: 372.338313 msecs"
"Elapsed time: 73.104641 msecs"
"Elapsed time: 58.832009 msecs"
"Elapsed time: 60.312924 msecs"
"Elapsed time: 58.626328 msecs"
"Elapsed time: 57.005242 msecs"
"Elapsed time: 54.493328 msecs"
"Elapsed time: 56.283221 msecs"
"Elapsed time: 54.575182 msecs"
"Elapsed time: 54.939474 msecs"

user=> (dotimes [_ 10] (time (dotimes [_ 100000] (simple-fn :it))))
"Elapsed time: 28.504607 msecs"
"Elapsed time: 17.564177 msecs"
"Elapsed time: 1.877194 msecs"
"Elapsed time: 2.340661 msecs"
"Elapsed time: 1.581906 msecs"
"Elapsed time: 1.792407 msecs"
"Elapsed time: 1.878591 msecs"
"Elapsed time: 1.919937 msecs"
"Elapsed time: 2.367759 msecs"
"Elapsed time: 1.90555 msecs"

This is what I have been expecting of course (Fn < Protocolfn <
Multifn), but i was thinking that with Protocols dispatching would be
significantly faster than with Multimethods. Am I missing something?
Because they don't seem to provide the speed for implementing the core
abstractions (like (seq <coll>)).

Syntax
With protocols you define the protocols using symbols and extend types
by using the keywordized name
eg.

(defprotocol Test (protocol-fn [it] "Protocol-fn"))
(extend Object Test {:protocol-fn (fn [it] nil)})

i understand that extend is a function and evaluates its arguments,
and that {:protocol-fn (fn [it] nil)} is a real map, but wouldn't it
be possible just to use a {<generic-fn> <new-method-fn>} map instead?
The protocol should know its generic functions, so that wouldn't be
ambiguous.
{protocol-fn (fn [it] nil)}) ; seems a lot clearer to me


Extensibility
I've noticed that extending a protocol-fn redefines it:

(def old-fn protocol-fn) ; from above
(extend Number Test {:protocol-fn (fn [it] :a-number)})
(def new-fn protocol-fn)

(= old-fn new-fn)
false

this worries me, because the semantics are differing from MultiFns and
are less dynamic. Especially coupled with dynamic development this
could lead to some gotchas.

Clojure 1.1.0-alpha-SNAPSHOT
user=> (defprotocol Test (prtcfn [it]))
Test
user=> (extend Object Test {:prtcfn (fn [it] :object)})
nil
user=> (prtcfn (Object.))
:object
user=> (def old-fn prtcfn)
#'user/old-fn
user=> (old-fn (Object.))
:object
user=> (extend Number Test {:prtcfn (fn [it] :number)})
nil
user=> (prtcfn (Object.))
:object
user=> (prtcfn 1)
:number
user=> (def new-fn prtcfn)
#'user/new-fn
user=> (new-fn 1)
:number
user=> (old-fn 1)
:object

When some protocol-fns get bound in a closure this could hurt a lot.


Sorry if this seems like nitpicking, but this is just what I noticed
while experimenting

Rich Hickey

unread,
Nov 13, 2009, 9:43:13 AM11/13/09
to Clojure


On Nov 13, 3:58 am, Konrad Hinsen <konrad.hin...@fastmail.net> wrote:
> On 13 Nov 2009, at 08:13, Mark Engelberg wrote:
>

> > Protocols:
>
> > I don't understand whether there's any way to provide a partial
> > implementation or default implementation of a given
> > protocol/interface, and I believe this to be an important issue.
>
> I don't think that partial implementations are possible at the moment,
> but I agree that it would be useful.

Yes, just create mixin maps and use them in your extends.

> A default implementation can be
> provided as an implementation of Object. It's not quite the same as a
> default implementation for a multimethod, as it doesn't apply to types
> identified by a metadata type tag, but in practice it can be good
> enough or even better.
>

That's not true for protocols. Make sure to leave any preconceptions
from multimethods and type tags behind. In particular, protocols do
not, and will not, utilize the isa/hierarchy system. Right now, the
dispatch code routes through (type x), but the intention is to support
through type only classes and deftype types. The use of type metadata
should be deprecated once this in place. At that point, Object does
serve as the default for everything other than nil.

***But*** one should generally avoid using hierarchy for
implementation inheritance! You may encounter it in interop
situations, but otherwise use mixins.

Rich

Rich Hickey

unread,
Nov 13, 2009, 9:50:42 AM11/13/09
to Clojure


On Nov 13, 9:03 am, MikeM <michael.messini...@invista.com> wrote:
> > (deftype Foo [a b c])
>
> > (defprotocol P (bar [x] "bar docs"))
>
> > (extend ::Foo P {:bar (fn [afoo] :foo-thing)})
>
> A common error may be to:
>
> (extend Foo P {:bar (fn [afoo] :foo-thing)})
>
> when (extend ::Foo ... is intended. I notice that (extend Foo...
> doesn't throw - should extend check that it is supplied a class,
> intfc, or keyword and throw if something else is supplied?

Yes it could.

> Alternately, could extend be changed to allow (extend Foo ... and do
> the right thing ? ie determine that Foo is the constructor function
> for a type and do the extension for the type Foo.

Dunno yet - there isn't a path from the factory fn value to its name
or deftype.

Rich

Rich Hickey

unread,
Nov 13, 2009, 9:47:35 AM11/13/09
to Clojure
Yes. Right now the priority is to get these new features out, without
breaking anyone's code.

Rich

Rich Hickey

unread,
Nov 13, 2009, 9:46:18 AM11/13/09
to Clojure
Yes, the latter (mixin maps) is preferred.

Rich

Sean Devlin

unread,
Nov 13, 2009, 10:42:58 AM11/13/09
to Clojure
Rich,
I was wondering something about defprotocol.

Here's your example:

(defprotocol AProtocol :on AnInterface
"A doc string for AProtocol abstraction"
(bar [a b] "bar docs" :on barMethod)
(baz ([a] [a b] [a b &amp; c]) "baz docs"))

In this case, you provide the docs for each method after parameters.
Would the following be possible:

(defprotocol AProtocol :on AnInterface
"A doc string for AProtocol abstraction"
(bar "bar docs" [a b] :on barMethod)
(baz "baz docs" ([a] [a b] [a b &amp; c])))

This matches the rhythm of the rest of the language.

Sean

On Nov 12, 7:10 am, Rich Hickey <richhic...@gmail.com> wrote:

Stuart Halloway

unread,
Nov 13, 2009, 10:48:37 AM11/13/09
to clo...@googlegroups.com
>> But do
>> people feel that some degree of data hiding is worthwhile?
>
> I don't.


Hooray for benevolent dictators!

Rich Hickey

unread,
Nov 13, 2009, 11:11:25 AM11/13/09
to Clojure


On Nov 13, 10:42 am, Sean Devlin <francoisdev...@gmail.com> wrote:
> Rich,
> I was wondering something about defprotocol.
>
> Here's your example:
>
> (defprotocol AProtocol :on AnInterface
>   "A doc string for AProtocol abstraction"
>   (bar [a b] "bar docs" :on barMethod)
>   (baz ([a] [a b] [a b &amp; c]) "baz docs"))
>
> In this case, you provide the docs for each method after parameters.
> Would the following be possible:
>
> (defprotocol AProtocol :on AnInterface
>   "A doc string for AProtocol abstraction"
>   (bar "bar docs" [a b] :on barMethod)
>   (baz "baz docs" ([a] [a b] [a b &amp; c])))
>
> This matches the rhythm of the rest of the language.
>

It does. I'm still on the fence about it. What do others think?

Rich

Rich Hickey

unread,
Nov 13, 2009, 11:07:08 AM11/13/09
to Clojure


On Nov 13, 9:26 am, AlexK <alexander.konstanti...@informatik.haw-
hamburg.de> wrote:
> Hi everybody,
>
> after playing around with protocols & datatypes, I found them very fun
> to use.
> Some questions:
> Performance
> I don't see (with my limited benchmarking) any significant difference
> between multifns and protocolfns:
>
...
> This is what I have been expecting of course (Fn < Protocolfn <
> Multifn), but i was thinking that with Protocols dispatching would be
> significantly faster than with Multimethods. Am I missing something?
> Because they don't seem to provide the speed for implementing the core
> abstractions (like (seq <coll>)).
>

This kind of do-nothing microbenchmarking demonstrates nothing.
Protocol dispatching is significantly faster than multimethod
dispatching, and I haven't even looked into call-site optimization.
Protocol dispatch is not as fast as interface dispatch, and may or may
not become so with call-site caching. But, protocols :on interfaces
are precisely as fast when called with instances of the :on interface
as calls to the :on interface.

> Syntax
> With protocols you define the protocols using symbols and extend types
> by using the keywordized name
> eg.
>
> (defprotocol Test (protocol-fn [it] "Protocol-fn"))
> (extend Object Test {:protocol-fn (fn [it] nil)})
>
> i understand that extend is a function and evaluates its arguments,
> and that {:protocol-fn (fn [it] nil)} is a real map, but wouldn't it
> be possible just to use a {<generic-fn> <new-method-fn>} map instead?
> The protocol should know its generic functions, so that wouldn't be
> ambiguous.
> {protocol-fn (fn [it] nil)}) ; seems a lot clearer to me

I wouldn't want to key anything off of fn identity. It would make
mixins quite difficult.

>
> Extensibility
> I've noticed that extending a protocol-fn redefines it:
>
> (def old-fn protocol-fn) ; from above
> (extend Number Test {:protocol-fn (fn [it] :a-number)})
> (def new-fn protocol-fn)
>
> (= old-fn new-fn)
> false
>
> this worries me, because the semantics are differing from MultiFns and
> are less dynamic.

They are no less dynamic, they just work differently.
This is no different from ordinary functions:

#'user/foo
user=> (def old-foo foo)
#'user/old-foo
user=> (def foov #'foo)
#'user/foov
user=> (old-foo)
:old-foo
user=> (foov)
:old-foo
user=> (defn foo [] :new-foo)
#'user/foo
user=> (foo)
:new-foo
user=> (old-foo)
:old-foo
user=> (foov)
:new-foo

> When some protocol-fns get bound in a closure this could hurt a lot.
>

That claim is not supported by your example. In general, when making a
long-term relationship with a function, you do so with its var, as
above. A closure referencing a var closes over the var itself, not its
current value:

user=> (def f (future (dotimes [_ 10] (prn (foo)) (Thread/sleep
1000))))
#'user/f
user=> :new-foo
:new-foo
:new-foo
:new-foo
:new-foo
:new-foo
:new-foo
;;;;;;;;;;;;;;;while that is going
(defn foo [] :newer-foo)
#'user/foo
:newer-foo
:newer-foo
:newer-foo

Admittedly, it is a difference from multimethods. With protocols, both
protocols and their functions/methods are immutable. Redefining or
extending a protocol modifies only the protocol and fn vars. I prefer
that, and don't consider the above behavior a problem. What do others
think?

Rich

Rich Hickey

unread,
Nov 13, 2009, 11:10:08 AM11/13/09
to Clojure


On Nov 13, 10:48 am, Stuart Halloway <stuart.hallo...@gmail.com>
wrote:
I was just putting in my vote :)

Rich

David Nolen

unread,
Nov 13, 2009, 11:22:57 AM11/13/09
to clo...@googlegroups.com
I place my vote for no data hiding.

David Nolen

unread,
Nov 13, 2009, 11:24:28 AM11/13/09
to clo...@googlegroups.com
Is there an argument for putting it after the argument list? :)

Rich Hickey

unread,
Nov 13, 2009, 11:40:14 AM11/13/09
to Clojure


On Nov 13, 9:24 am, James Reeves <weavejes...@googlemail.com> wrote:
> Are there any plans to use protocols to define polymorphic functions
> like conj and get? Perhaps with an "untype" function to remove type
> metadata so one could always get at the datastructures hidden by the
> protocol. e.g.
>
> (defn sql-get [table key]
>   (sql-query
>     (str "select * from " table " where " (get (untype table) :primary-
> key) " = ?")
>     key))
>
> (extend ::sql-table Gettable
>   {:get sql-get})
>
> Then 'get' could be used in a more generic fashion:
>
> user=> (def accounts (sql-table accounts))
>
> user=> (get accounts 10)
> {:id 10, :login "jsmith", :password "1234"}
>

I'll have to think about that. Your 'untype' above is really just a
specialization of type aliasing:

(as-type nil table)

where (as-type ::Foo x) would create a view of x with dynamic type
Foo.

I'm still thinking about such aliasing and its implications.

Rich

Konrad Hinsen

unread,
Nov 13, 2009, 12:17:49 PM11/13/09
to clo...@googlegroups.com
On 13.11.2009, at 17:07, Rich Hickey wrote:

> Admittedly, it is a difference from multimethods. With protocols, both
> protocols and their functions/methods are immutable. Redefining or
> extending a protocol modifies only the protocol and fn vars. I prefer
> that, and don't consider the above behavior a problem. What do others
> think?

For most applications the difference doesn't matter. Having protocols
as immutable values bound to vars that change with every extend could
lead to both interesting use cases and undesirable surprises when used
with threads. Threads can have thread-local implementations of
protocols, intentionally or by mistake.

What makes this behaviour a bit disturbing is the fact that a var in a
namespace that refers to a protocol or a method is changed from code
in another namespace. This is of course possible otherwise as well,
but highly unusual. Consider this example:

(defprotocol Foo
(bar [x]))

(def bar nil)

Now in some other namespace, at some later time:

(extend Object
Foo
{:bar (fn [x] x)})

This will redefine bar in the first namespace to a method of Foo.
While I don't expect such a situation to be frequent, it is certainly
highly unexpected. All the more so since the code in the second
namespace doesn't even mention the symbol bar that it changes.

Konrad.

Constantine Vetoshev

unread,
Nov 13, 2009, 1:01:32 PM11/13/09
to Clojure
On Nov 12, 7:10 am, Rich Hickey <richhic...@gmail.com> wrote:
> [1]http://www.assembla.com/wiki/show/clojure/Datatypes

Could you please elaborate on why you chose to make IPersistentMap an
optional interface for deftype'd types, rather than making it
automatic?

I'm asking because I found the automatic defstruct-map equivalence
convenient in writing the Cupboard database library (http://github.com/
gcv/cupboard). It guaranteed that any reading Clojure code could read
back any map or any struct written by any other Clojure program
(unless the map contains closures, of course), without any other
knowledge of the writing program. It allows for data-centric designs
when thinking about storing objects to a database, i.e., the data can
be used without the type definitions which originally produced it.

I can, of course, require that any deftype'd types saved in Cupboard
databases implement IPersistentMap, but I'm curious about the
reasoning for not making maps part of the default nature of deftype.
Making IPersistentMap the default could also make deftype a nearly
drop-in replacement for defstruct.

Thanks,
Constantine Vetoshev

Konrad Hinsen

unread,
Nov 13, 2009, 1:09:25 PM11/13/09
to clo...@googlegroups.com
I'd prefer the doc string right after the function name, as in other
situations, but it's not important enough that I'd argue for it at any
length.

Konrad.

Sean Devlin

unread,
Nov 13, 2009, 1:10:23 PM11/13/09
to Clojure
I agree w/ Constantine. This would be very, very useful.

Sean

AlexK

unread,
Nov 13, 2009, 12:58:20 PM11/13/09
to Clojure


On 13 Nov., 17:07, Rich Hickey <richhic...@gmail.com> wrote:
> This kind of do-nothing microbenchmarking demonstrates nothing.
> Protocol dispatching is significantly faster than multimethod
> dispatching, and I haven't even looked into call-site optimization.
> Protocol dispatch is not as fast as interface dispatch, and may or may
> not become so with call-site caching. But, protocols :on interfaces
> are precisely as fast when called with instances of the :on interface
> as calls to the :on interface.

Forget what I said about Performance, I didn't realize that the
functions could be mapped to interface-methods. When mapped to an
interface they get JITed to awesome speeds :-)

> I wouldn't want to key anything off of fn identity. It would make
> mixins quite difficult.

This was just a minor thought of mine, if it impedes with this awesome
literal mixin concept then I gladly take those mixins.

(def composed-mixin (merge mixin-a mixin-b mixin-c))
seems absolutely logical and natural

>
> .......
>
> Admittedly, it is a difference from multimethods. With protocols, both
> protocols and their functions/methods are immutable. Redefining or
> extending a protocol modifies only the protocol and fn vars. I prefer
> that, and don't consider the above behavior a problem. What do others
> think?
>
> Rich

that was a bad Example probably...
hypothetical example for impementing the type function as a
Protocolfn:


(defprotocol Typed (my-type [obj] "Doc.."))
Typed
(extend Object Typed {:my-type (fn [obj] (.getClass obj))})
nil
(defmulti foo my-type)
#'user/foo
(extend clojure.lang.IObj Typed {:my-type (fn [obj] (if (:type ^obj)
(:type ^obj) (.getClass obj)))})

now our foo multifn woudn't work as expected (by me at least).
I just didn't expect that (extend <sth> <Type> <fn-map>) would
redefine some Vars.
extend has to mutate something, but I think that redefining some Vars
is wierder (and thus should be explicit via a visible (def ...))
than mutating the "generic function" (I may be biased by CLOS of
course)

Alex


Stuart Sierra

unread,
Nov 13, 2009, 3:19:09 PM11/13/09
to Clojure
On Nov 12, 7:10 am, Rich Hickey <richhic...@gmail.com> wrote:
> An early version of the code for a few important new language
> features, datatypes[1] and protocols[2]

Very simple example here: http://paste.lisp.org/display/90329

This shows how to do arithmetic with complex numbers using deftype and
defprotocol. It does not support mixing complex and real numbers; not
sure how that would work.

-SS

Laurent PETIT

unread,
Nov 13, 2009, 3:42:30 PM11/13/09
to clo...@googlegroups.com
Hi,

2009/11/13 Stuart Sierra <the.stua...@gmail.com>:
Why not have used add/sub/... instead of +/-/... in the extension of
Arithmetic for ::Complex ? :-p

This example has maybe a problem : doesn't the symmetry of these
Arithmetic operators seems to be crying for type multiple dispatch in
this particular case ? :-)

>
> -SS

Rich Hickey

unread,
Nov 13, 2009, 3:46:17 PM11/13/09
to Clojure


On Nov 13, 1:01 pm, Constantine Vetoshev <gepar...@gmail.com> wrote:
> On Nov 12, 7:10 am, Rich Hickey <richhic...@gmail.com> wrote:
>
> > [1]http://www.assembla.com/wiki/show/clojure/Datatypes
>
> Could you please elaborate on why you chose to make IPersistentMap an
> optional interface for deftype'd types, rather than making it
> automatic?
>

Because we want to be able to use deftype to implement
PersistentHashMap and datatypes like it.

> I'm asking because I found the automatic defstruct-map equivalence
> convenient in writing the Cupboard database library (http://github.com/
> gcv/cupboard). It guaranteed that any reading Clojure code could read
> back any map or any struct written by any other Clojure program
> (unless the map contains closures, of course), without any other
> knowledge of the writing program. It allows for data-centric designs
> when thinking about storing objects to a database, i.e., the data can
> be used without the type definitions which originally produced it.
>
> I can, of course, require that any deftype'd types saved in Cupboard
> databases implement IPersistentMap, but I'm curious about the
> reasoning for not making maps part of the default nature of deftype.
> Making IPersistentMap the default could also make deftype a nearly
> drop-in replacement for defstruct.
>

Making it default, and having some way to opt out, is another
possibility. This is still an open question, as well as how to make
these default implementations more extensible.

Rich

Stuart Halloway

unread,
Nov 13, 2009, 4:26:28 PM11/13/09
to clo...@googlegroups.com
Initial gut reaction: would like a shortcut syntax for opting in(or
out), as this will be a very common use case. Suspect that most
classes would rather have it than not, but that may be the Ruby
programmer in me.

Stu

Stuart Sierra

unread,
Nov 13, 2009, 5:03:17 PM11/13/09
to Clojure
On Nov 13, 3:42 pm, Laurent PETIT <laurent.pe...@gmail.com> wrote:
> > Very simple example here:http://paste.lisp.org/display/90329
>
> Why not have used add/sub/... instead of +/-/... in the extension of
> Arithmetic for ::Complex ? :-p

Just a little simpler, that's all.

> This example has maybe a problem : doesn't the symmetry of these
> Arithmetic operators seems to be crying for type multiple dispatch in
> this particular case ? :-)

Yes, in the general case, arithmetic requires multimethods. But
multimethods -- or any dynamic type dispatch -- are too slow for math-
heavy code anyway. So, for that reason, it's not a great example.
But it was the first thing I thought of.

-SS

Mark Engelberg

unread,
Nov 13, 2009, 8:50:58 PM11/13/09
to clo...@googlegroups.com
Rich, thanks for the extended explanation of the overlap between the
old and new constructs; I found this explanation much clearer than
what is currently on the wiki. Basically, the key for me was
realizing that these new constructs are all you're likely to need as
long as you are just programming "in Clojure" and not worrying about
Java interop.

A related question I have is to better understand the
overlap/relationship between multimethods and the new protocols. At
first glance, they seem fairly orthogonal. If you have something that
dispatches only on type, or corresponds to a Java interface, use a
protocol, otherwise use multimethods. However, I notice that one
thing protocols give you is the ability to batch function requirements
and say, "these functions need to be implemented together". It seems
to me like there is just as much value to being able to include
multimethods as part of such a requirement of things to implement
together. So would it make sense for multimethods to be included as
part of protocols, or should there be some similar grouping system for
multimethods?

Regarding partial implementation, Rich and a couple others have
suggested that the ability to merge together maps of implementations
basically gives you partial implementation. But it seems to me that a
lot of times, several interface functions will share some sort of
local state, using closures. If I'm visualizing this correctly, these
sorts of partial implementations couldn't possibly be mixed-in,
because there would be no way to share that state. Off the top of my
head, I don't yet have a concrete example of this -- has anyone
encountered this yet in their experiments with protocols?

--Mark

John Harrop

unread,
Nov 13, 2009, 10:37:27 PM11/13/09
to clo...@googlegroups.com
On Fri, Nov 13, 2009 at 8:50 PM, Mark Engelberg <mark.en...@gmail.com> wrote:
Rich, thanks for the extended explanation of the overlap between the
old and new constructs; I found this explanation much clearer than
what is currently on the wiki.  Basically, the key for me was
realizing that these new constructs are all you're likely to need as
long as you are just programming "in Clojure" and not worrying about
Java interop.

A related question I have is to better understand the
overlap/relationship between multimethods and the new protocols.  At
first glance, they seem fairly orthogonal.  If you have something that
dispatches only on type, or corresponds to a Java interface, use a
protocol, otherwise use multimethods.  However, I notice that one
thing protocols give you is the ability to batch function requirements
and say, "these functions need to be implemented together".  It seems
to me like there is just as much value to being able to include
multimethods as part of such a requirement of things to implement
together.  So would it make sense for multimethods to be included as
part of protocols, or should there be some similar grouping system for
multimethods?

Since a multimethod maybe used where any Clojure function may be used, and a Clojure function may be used to implement a protocol, can't multimethods be used to implement (part or all of) a protocol?

cody koeninger

unread,
Nov 14, 2009, 1:32:14 AM11/14/09
to Clojure

On Nov 13, 9:42 am, Sean Devlin <francoisdev...@gmail.com> wrote:
> In this case, you provide the docs for each method after parameters.
> Would the following be possible:
>
> (defprotocol AProtocol :on AnInterface
>   "A doc string for AProtocol abstraction"
>   (bar "bar docs" [a b] :on barMethod)
>   (baz "baz docs" ([a] [a b] [a b &amp; c])))
>
> This matches the rhythm of the rest of the language.
>
> Sean


+1 for docstring position that matches the rest of the language. I
actually think this is a big deal.

Other issue regarding docstrings:

user=> (doc AProtocol)
-------------------------
user/AProtocol
nil
protocol doc
nil
user=> (doc bar)
-------------------------
user/bar
([a b])
bar doc
nil


doc gives no indication of any relationship between bar and AProtocol,
other than the shared namespace. Is the idea to use a convention of
max 1 protocol per namespace?

Mark Engelberg

unread,
Nov 14, 2009, 3:45:04 AM11/14/09
to clo...@googlegroups.com
On Fri, Nov 13, 2009 at 12:58 AM, Konrad Hinsen
<konrad...@fastmail.net> wrote:
> Coming from a Python background, I don't think access restrictions are
> necessary. However, flagging fields as "not meant for use by
> outsiders" could be of interest for documentation tools, to make it
> clear what client code can safely rely on.

Even Python has the ability to mangle names so that they are
effectively private. They aren't truly hidden, but there's no way
you'll misunderstand and call a private thing by accident.

As a longtime Python programmer, I guess that's the level of privacy I like.

In general, I really hate it when it's difficult to tell what parts of
an API are the things that the end-user is really supposed to call,
and what part of the data you're supposed to access directly versus an
accessor method. It bugs me when I call (:real complex) to extract
the real part out of a complex number, only to discover later that I
was supposed to call (get-real complex) because the accessor is the
only "safe way" to get the value I want.

I think anything in datatypes or protocols that help library designers
separate the data that's safe to access directly from the data that's
not, or the functions that are safe to call versus those that are not
-- I think that's a good thing.

I don't really care how strictly the language *enforces* that
separation, but I think the ability to specify that separation is a
good thing.

John Harrop

unread,
Nov 14, 2009, 3:52:20 AM11/14/09
to clo...@googlegroups.com
+1 for ability to specify, especially if it's recognized by tools such as IDEs and they filter private stuff from other namespaces from what they use as possible completions, etc.

Particularly, I'm in favor of removing the exception throw when a private var is dereferenced from outside its namespace, particularly as it will solve the macro-expansion-uses-private-implementation-function issue (making the expansion generate the function as a local function or anonymous function seems messy to me when the function's own content does not depend on the macro arguments). Let :private true in a var's metadata influence tools like IDEs and documentation generators, so they don't show as part of the API, and let IDEs give warnings if they see a symbol that resolves to an external private var in your code (though not in the expansions of non-private macros you call). That should suffice to prevent accidental self-inflicted gunshot wounds to the foot, while preserving maximum flexibility.

Konrad Hinsen

unread,
Nov 14, 2009, 8:09:27 AM11/14/09
to clo...@googlegroups.com
On 13 Nov 2009, at 23:03, Stuart Sierra wrote:

>> This example has maybe a problem : doesn't the symmetry of these
>> Arithmetic operators seems to be crying for type multiple dispatch in
>> this particular case ? :-)
>
> Yes, in the general case, arithmetic requires multimethods. But
> multimethods -- or any dynamic type dispatch -- are too slow for math-
> heavy code anyway.

That depends on how it is used and on what types.
clojure.contrib.complex has a multimethod-based implementation which
moreover does a multimethod dispatch on the real/imaginary parts,
making it possible to use any numeric type, built-in or not, inside
the complex type. That's about as inefficient as it can get, but also
very flexible. You can do symbolic computing with that kind of
framework, and then the overhead is tolerable, but for number
crunching it is not.

I think arithmetic is a bad "first example" for just about any
implementation technique. Doing it well is very hard, and as far as I
know no one has yet come up with any technique that is fully
satisfying for all applications.

Konrad.

Konrad Hinsen

unread,
Nov 14, 2009, 8:17:23 AM11/14/09
to clo...@googlegroups.com
On 14 Nov 2009, at 02:50, Mark Engelberg wrote:

> together. So would it make sense for multimethods to be included as
> part of protocols, or should there be some similar grouping system for
> multimethods?

The "old ideas/scratchpad" section of

http://www.assembla.com/wiki/show/clojure/Protocols

mentions multiprotocols, which are about what you describe.

> basically gives you partial implementation. But it seems to me that a
> lot of times, several interface functions will share some sort of
> local state, using closures.


The shared state would normally be stored in the object that the
dispatching acts on.

> If I'm visualizing this correctly, these sorts of partial
> implementations couldn't possibly be mixed-in,
> because there would be no way to share that state.

They could still all reference a common var, just like any set of
functions can. But...

> Off the top of my head, I don't yet have a concrete example of this
> -- has anyone
> encountered this yet in their experiments with protocols?

... me neither, so it seems premature to discuss the details.

Konrad.

Jonas Enlund

unread,
Nov 14, 2009, 8:28:49 AM11/14/09
to clo...@googlegroups.com
Hi there!

I have built a simple Matrix datatype with defprotocol and deftype.
You can take a look at it at http://gist.github.com/234535
(constructive criticism welcome!). Some simple examples are provided
at the end of the file.

I have a few questions.

- Why must i write (matrix/Matrix ...) instead of (Matrix ...) inside
the deftype body? Is this a bug? I didn't have to write it that way in
earlier versions of the new-branch.
- Am I supposed to have all the interface implementations inside the
deftype body? I didn't figure out how to move the implementation of
the clojure.lang.* interfaces to (extends ...).

/Jonas

Konrad Hinsen

unread,
Nov 14, 2009, 8:34:48 AM11/14/09
to clo...@googlegroups.com
On 14 Nov 2009, at 09:45, Mark Engelberg wrote:

> In general, I really hate it when it's difficult to tell what parts of
> an API are the things that the end-user is really supposed to call,
> and what part of the data you're supposed to access directly versus an
> accessor method. It bugs me when I call (:real complex) to extract

I agree. The distinction should be clear, even though not enforced by
the language.

A privacy indication that would fit well with the rest of Clojure is
to tag the field name in the deftype with the "private" metadata
field. The question is just how to make this information accessible to
IDEs and documentation tools. In fact, the first question is how an
IDE can get any information about the fields of a type. There doesn't
seem to be a documented way to do it. One source would be the doc
string of the constructor function.

Konrad.

Rich Hickey

unread,
Nov 14, 2009, 9:15:08 AM11/14/09
to Clojure


On Nov 14, 8:28 am, Jonas Enlund <jonas.enl...@gmail.com> wrote:
> Hi there!
>
> I have built a simple Matrix datatype with defprotocol and deftype.
> You can take a look at it athttp://gist.github.com/234535
> (constructive criticism welcome!). Some simple examples are provided
> at the end of the file.
>
> I have a few questions.
>
> - Why must i write (matrix/Matrix ...) instead of (Matrix ...) inside
> the deftype body? Is this a bug? I didn't have to write it that way in
> earlier versions of the new-branch.

Inside the method bodies, the type name (Matrix) now designates the
class, so you can use it to call the ctor instead of the factory fn:

;change all (Matrix a b c) to (Matrix. a b c)

(Matrix. 1 (count v) (into [] v))


> - Am I supposed to have all the interface implementations inside the
> deftype body?

Yes.

> I didn't figure out how to move the implementation of
> the clojure.lang.* interfaces to (extends ...).
>

Interface methods have nothing to do with protocol fns. They go in
separate places.

Rich

Rich Hickey

unread,
Nov 14, 2009, 9:19:37 AM11/14/09
to Clojure
Making the doc refer to the protocol is on the todo list.

> Is the idea to use a convention of
> max 1 protocol per namespace?

No. But all functions of protocols in the same ns live in the ns, so
you must have unique function names across protocols on the same ns.

Rich

Richard Newman

unread,
Nov 14, 2009, 1:42:32 PM11/14/09
to clo...@googlegroups.com
> I don't really care how strictly the language *enforces* that
> separation, but I think the ability to specify that separation is a
> good thing.

I'd go so far as to request that it does not enforce separation. I'm
sure anyone who's spent enough time using Other People's Libraries has
hit instances where the only way to get things to work is to use the
(lower-level, undocumented) API because the high-level API doesn't
allow you to pass some crucial parameter, or where you need to reuse
some internal function or spend the time to rewrite it yourself.

In an ideal world (read: fantasy land) you can just fix the library
and send a push request on GitHub, but we rarely live in that place.

Similarly, I'm sure lots of people have had tons of 'fun' with final
classes, private and protected members, and so on in Java... "if I
could just call that method...". I recall several experiences where if
I could have called a protected method I could have worked around a
bug in an application server. Instead, I had to patch their runtime
libraries, and wait a few weeks for the change to roll upstream.

> As a longtime Python programmer, I guess that's the level of privacy
> I like.

As a Common Lisper and Python programmer, I'm quite fond of this
too... "there but hidden".

I like CL's package support for this kind of situation, where
unexported symbols can still be reached via foo::bar, at the cost of
an obvious "code smell".

To *really* enforce hiding, you can actually unintern symbols,
rendering the things they named unreachable. Unusual, but possible.

John Harrop

unread,
Nov 14, 2009, 3:19:49 PM11/14/09
to clo...@googlegroups.com
On Sat, Nov 14, 2009 at 1:42 PM, Richard Newman <holy...@gmail.com> wrote:
I like CL's package support for this kind of situation, where
unexported symbols can still be reached via foo::bar, at the cost of
an obvious "code smell".

This suggests an alternate fix for the private functions in macros problem:

1. Keep the throw when dereferencing another ns's private vars.
2. Add a way to override that throw when dereferencing -- a
   "deref-private" that always works.
3. Add a reader macro, say #!, with the property that #!foo expands
   to (deref-private #'foo).
4. Macros can use #!priv-fn in their expansions to produce an
   expansion that calls a private function without issues.
5. #! outside of this circumstance would be a code smell.
6. Optionally, syntax-quote, when resolving symbols into fully
   qualified ones, might auto-#! all that reference private vars IN
   THE CURRENT NAMESPACE. Then no explicit #! is needed on the
   hypothetical priv-fn in the macro: `(priv-fn foo ~bar) instead of
   `(#!priv-fn foo ~bar). At the same time, referencing a *foreign*
   private function in a macro still needs the #!.Then #! is ALWAYS a
   smell.

Michael Wood

unread,
Nov 15, 2009, 8:45:25 AM11/15/09
to clo...@googlegroups.com
2009/11/14 John Harrop <jharr...@gmail.com>:
> On Sat, Nov 14, 2009 at 1:42 PM, Richard Newman <holy...@gmail.com> wrote:
>>
>> I like CL's package support for this kind of situation, where
>> unexported symbols can still be reached via foo::bar, at the cost of
>> an obvious "code smell".
>
> This suggests an alternate fix for the private functions in macros problem:
> 1. Keep the throw when dereferencing another ns's private vars.
> 2. Add a way to override that throw when dereferencing -- a
>    "deref-private" that always works.
> 3. Add a reader macro, say #!, with the property that #!foo expands
>    to (deref-private #'foo).

Except you'll have to find another reader macro. #! is already taken.
It's a comment-to-end-of-line reader macro to support Unix scripts.

A quick search of clojure.org didn't find it, but it's mentioned in
the Clojure Programming wikibook:

http://en.wikibooks.org/wiki/Clojure_Programming/Tutorials_and_Tips#Shebang_Scripting_in_Clojure

--
Michael Wood <esio...@gmail.com>

John Harrop

unread,
Nov 15, 2009, 4:20:19 PM11/15/09
to clo...@googlegroups.com
On Sun, Nov 15, 2009 at 8:45 AM, Michael Wood <esio...@gmail.com> wrote:
2009/11/14 John Harrop <jharr...@gmail.com>:
> On Sat, Nov 14, 2009 at 1:42 PM, Richard Newman <holy...@gmail.com> wrote:
>>
>> I like CL's package support for this kind of situation, where
>> unexported symbols can still be reached via foo::bar, at the cost of
>> an obvious "code smell".
>
> This suggests an alternate fix for the private functions in macros problem:
> 1. Keep the throw when dereferencing another ns's private vars.
> 2. Add a way to override that throw when dereferencing -- a
>    "deref-private" that always works.
> 3. Add a reader macro, say #!, with the property that #!foo expands
>    to (deref-private #'foo).

Except you'll have to find another reader macro.  #! is already taken.
 It's a comment-to-end-of-line reader macro to support Unix scripts.

That's weird. It's not documented anywhere on the site. And it seems to hang the REPL:

user=> nil #!foo

and nothing. Enter doesn't print "nil" and a fresh user=> prompt as it should and nothing else apparently works either. The REPL is either wedged or interpreting everything as comment from then on.

Weirdly enough, ; hangs the REPL likewise.

Maybe use @!? That's closer to @ and the ! is still there to suggest something perilous, as it does with the mutable-state fns.

It would prevent using normal deref-@ with symbols that start with !, but does anyone actually use symbols that start with ! at this time? Even not= is not= instead of !=.

David Brown

unread,
Nov 15, 2009, 8:17:18 PM11/15/09
to clo...@googlegroups.com
On Sun, Nov 15, 2009 at 04:20:19PM -0500, John Harrop wrote:

>That's weird. It's not documented anywhere on the site. And it seems to hang
>the REPL:
>
>user=> nil #!foo
>
>and nothing. Enter doesn't print "nil" and a fresh user=> prompt as it
>should and nothing else apparently works either. The REPL is either wedged
>or interpreting everything as comment from then on.
>
>Weirdly enough, ; hangs the REPL likewise.

How's the data getting to the REPL? jline, or ide? I'm using rlwrap
at the console, and netierh seems to cause problems. However, #! eats
everything, and doesn't prompt again, but does after the next line.
The ; prints a new prompt, so they are definitely treated differently.

David

John Harrop

unread,
Nov 16, 2009, 12:12:12 AM11/16/09
to clo...@googlegroups.com
On Sun, Nov 15, 2009 at 8:17 PM, David Brown <clo...@davidb.org> wrote:
On Sun, Nov 15, 2009 at 04:20:19PM -0500, John Harrop wrote:

>That's weird. It's not documented anywhere on the site. And it seems to hang
>the REPL:
>
>user=> nil #!foo
>
>and nothing. Enter doesn't print "nil" and a fresh user=> prompt as it
>should and nothing else apparently works either. The REPL is either wedged
>or interpreting everything as comment from then on.
>
>Weirdly enough, ; hangs the REPL likewise.

How's the data getting to the REPL?  jline, or ide?

Enclojure IDE. 

Michael Wood

unread,
Nov 16, 2009, 2:16:04 AM11/16/09