Interfaces and ABCs

[Antony Lee]

Currently the project implements Java interfaces as "standard" Python classes whose methods unconditionally raise AbstractMethodException when called.  This doesn't seem to be very useful to me as the interface does not enforce the presence of an actual implementation of the abstract methods.
If we still want to have "interfaces", it seems better to rely on the abc module, which provides an ABCMeta metaclass and an abstractmethod decorator, which does more or less what we'd expect (i.e., a class inheriting from an interface cannot be instantiated if there is an abstract method that is not defined.  I did this in the abc branch of my repo (https://github.com/anntzer/clojure-py/tree/abc), and of course it appeared that some of the interfaces were not fully implemented (to the extent that the REPL cannot be started).
So perhaps we should start filling the holes in the implementation of the interfaces (throwing in a bunch of "def method(*args): pass" doesn't count :-)) (-- or get rid of the interfaces altogether...)
Antony

[Timothy Baldridge]

I started down that road as well, but I ran into a few issues. One of
them is that this doesn't really fit the Clojure-jvm semantics. For
instance, this is valid Clojure code.

(deftype Foo [x]
ISeq
(next [this] x))

So here we can actually implement only one method (we skipped .seq and
.first). However, ABCs demand that we must implement all interfaces.

Any thoughts?

Timothy (halgari)

[Konrad Hinsen]

--On 22 mai 2012 06:09:42 -0500 Timothy Baldridge <tbald...@gmail.com>
wrote:

> I started down that road as well, but I ran into a few issues. One of
> them is that this doesn't really fit the Clojure-jvm semantics. For
> instance, this is valid Clojure code.
>
> (deftype Foo [x]
> ISeq
> (next [this] x))

What does this do if one of the non-implemented methods is called? Fail, I
suppose? We could do the same in Python, by providing default
implementations that raise an exception.

Konrad.

[Antony Lee]

This raises an AbstractMethodError at runtime in clj-jvm, and this is also the case in the current implementation of clj-py (the one without ABCs).  So I guess that we have to stick to that... even though I don't really understand the point of having interfaces at all in that case.  Oh well.
Antony

2012/5/22 Konrad Hinsen <google...@khinsen.fastmail.net>

[Timothy Baldridge]

I agree that it doesn't make sense to have the interfaces. There is
one other option, however:

In ClojureScript, everything is protocol based. I would like to figure
out some way of switching over to protocols more. Right now we only
take advantage of protocols for ISeq and INamed.

This would be a fairly major change, but it would allow us to extend
the code much easier later on. For instance, we could extend the
Counted protocol to support numpy's arrays.

Right now, the only reason we have these interfaces is so that we can
do type checks on them. For instance, the compiler treats
PersistentList, Cons, and LazySeq all the same since they are all ISeq
implementors. However, we could do this exact thing with protocols
just as easily.

Thoughts?

Timothy

--
“One of the main causes of the fall of the Roman Empire was
that–lacking zero–they had no way to indicate successful termination
of their C programs.”
(Robert Firth)

[Antony Lee]

I guess I don't have enough experience in "classical OOP" to make a proper call (most of my experience comes from Python and C), but getting rid of interfaces (which seem to be an artefact of the JVM when compared to protocols) seems reasonable to me.  Probably I should first have a look at how protocols are actually implemented in clj-py though :-)
Antony

2012/5/22 Timothy Baldridge <tbald...@gmail.com>

[Timothy Baldridge]

The one I have in clojure-py is pretty basic. So here's the issue

Let's say we want to define a polymorphic (protocol) function called
"foo". And then we want to extend it to support Cons, ints and tuples.

For Cons we can simply do this:

Cons.foo = myfunc

After that calling Cons().foo() will call myfunc. However, if we try
this with int and tuple we get an error, since we're not allowed to
modify python internal classes. So what we do currently do is we try
to set the attribute directly first, and if that fails, then we store
it all in a dict for later use. Using a dict is slower than doing a
simple attribute lookup, so that's the reason for that.

We could extend all this with decorators. Perhaps that's the better
route. For instance:

@extends("clojure.protocols.ISeq")
@extends("clojure.protocols.Counted")
class Cons(object):

@overrides("clojure.protocols.Counted", "counted")
def count(self):
return self._count

@overrides("clojure.protocols.ISeq", "next")
def renamedNext(self):
return null

This is perhaps a bit more verbose than what we have now, but it would
get rid of the need for interfaces, and still allow for "extends?",
"satisfies?", and renaming actual method names.

Anyway, most of the mechanics for this are at the end of RT.py.

Timothy

[Antony Lee]

If we choose to use decorators it seems better to leverage the first-class-ness of classes and methods in Python:

@extends(clojure.protocols.ISeq, clojure.protocols.Counted) # this is hardly more verbose than normal inheritance

class Cons(object):

    @overrides(clojure.protocols.Counted.counted)

    def count(self): return self._count

Possibly @overrides("counted") could also be allowed (resolving to the first protocol that we are extending that defines that method) for succintness.

[Timothy Baldridge]

> If we choose to use decorators it seems better to leverage the
> first-class-ness of classes and methods in Python:

Even better!

I'm liking this solution more and more. We should try this for a small
interface first (perhaps ISeq since it already kindof works this way).
Once that is done we would just need to work through all the
/lang/*.py files and make these changes.


Timothy

[Antony Lee]

I tried to start implementing this, however because ISeq is used all over the place this is a bit tricky as any mistake makes the whole thing blow up.  Probably I should have started with a more obscure interface :-)
Still, the more I think about it, the more I believe that ProtocolFns (i.e., a form of generic functions) are more fundamental than protocols (at least in python-land) -- especially as all ProtocolFns live in a single namespace (for protocols that are in the same namespace, I mean).  Protocols are just a way, a posteriori, to group ProtocolFns semantically.
If incomplete implementations are allowed, I don't see any real "safe" use of "extends?".  After all, (extends? ISeq type) gives no guarantee whatsoever about the validity of (seq <instance-of-type>), so it seems to be more a form of metadata than anything else to me.
Antony

2012/5/22 Timothy Baldridge <tbald...@gmail.com>

> If we choose to use decorators it seems better to leverage the

[Timothy Baldridge]

That is correct. Protocols are basically a dict of all the types that
implements at least one of the ProtocolFns defined in the Protocol.
extends? basically is just a dict lookup in that case.

satisfies? However is what you are probably looking for. satisfies?
would loop over all the fns in a Protocol and make sure each and every
fn is extended for the given type. However, since this is much more
expensive operation extends? is probably what we'd want to use in most
cases.

To be honest, I don't really care for these semantics, but it's the
way Clojure works. You can actually extend a protocol without ever
extending a single one of its fns. I guess that's the difference
between a dynamic language and a more static one.

Timothy

[Antony Lee]

I started to more towards a protocol-based implementation, which is
actually not *too* hard to do. Please try the "protocols" branch of my
repo (g...@github.com:anntzer/clojure-py.git).

So a protocol is just a set (or a list) of method names. At the very
beginning of the startup, I populate the clojure.protocols namespace
(actually, a Python module, as there is no support for namespaces yet)
with core protocols and protocolfns. For some reason I decided to
implement each protocol as a class with a custom metaclass that 1/
forbids instantiation (like a Java interface) and 2/ overrides
is{subclass,instance} (so that they have the meaning of extends? /
satisfies?). Perhaps that was an overkill and I'll go back to the old
way of doing it but we'll see.

At that point I am able to define all base types and use them to extend
the relevant protocols using the @protocol.extends(some-protocol) class
decorator. This is more work than expected (and far from finished yet)
though because there is no notion of protocol inheritance (a protocol
cannot inherit from another) so there are a ton of protocols to be
registered. Also, at that point RT is not loaded yet (it requires some
of these types to be defined), but protocolfns are already available in
the clojure.protocols namespace.

Once these base types are defined, I am able to import RT and off we
go...

So the current state is: all the relevant protocols are implemented, but
most are not extended with the relevant classes yet. Only ISeq and
Seqable have been taken care of, and incompletely (basically, only in
the places where that was needed to start a REPL -- not even to pass all
tests).

On Wed, May 23, 2012 at 08:51:54PM -0500, Timothy Baldridge wrote:
> That is correct. Protocols are basically a dict of all the types that
> implements at least one of the ProtocolFns defined in the Protocol.
> extends? basically is just a dict lookup in that case.
>
> satisfies? However is what you are probably looking for. satisfies?
> would loop over all the fns in a Protocol and make sure each and every
> fn is extended for the given type. However, since this is much more
> expensive operation extends? is probably what we'd want to use in most
> cases.

Actually this is not the case for clj-jvm:
Clojure 1.4.0
user=> (defprotocol Foo (foo [this]))
Foo
user=> (extend java.lang.Number Foo {})
nil
user=> (and (extends? Foo java.lang.Number) (satisfies? Foo 1))
true

The only difference I can see is that one applies to a type and the
other to a value. On the other hand the two functions have somewhat
different implementations in clj-jvm so I may have missed something.

>
> To be honest, I don't really care for these semantics, but it's the
> way Clojure works. You can actually extend a protocol without ever
> extending a single one of its fns. I guess that's the difference
> between a dynamic language and a more static one.

Sure, I'll live with that (and that's how issubclass and isinstance
currently work). Should we provide a function that checks for full
implementation though?