ISeq documentation and mutual deftypes.

[Phillip Lord]

I've been playing with some code recently. I was wondering how hard
would it be to implement, for example ISeq in clojure. The plan was to
use deftype and a few bits of other code.

The ISeq interface looks like this:

public interface ISeq extends IPersistentCollection {

Object first();

ISeq next();

ISeq more();

ISeq cons(Object o);

}


And here is the problem -- it looks exactly like this! There is no
documentation. While first and cons are guessable, I am still not sure
of the difference between next and more. Likewise, IPersistentCollection
has "equiv" and "empty" -- guessable again.

The other question: is it possible to have two mutually refering deftype
definitions.

So

(deftype Alice ()
(other [] (Brian.)))

(deftype Brian ()
(other [] (Alice.)))

The closest I have got it:

(declare create-alice)
(declare create-brian)

(deftype Alice ()
(other [] (create-brian)))
(deftype Brian ()
(other [] (create-alice)))

The last could be

(deftype Brian ()
(other [] (Alice.)))

but the symmetry seemed good.

[Alex Miller]

next() should return either the remaining seq or null (think Clojure function next)

more() should return either the remaining seq or empty list (like Clojure function rest)

Inside Clojure, most seqs extend ASeq, which implements more() on top of the abstract next():

public ISeq more(){

    ISeq s = next();

    if(s == null)

        return PersistentList.EMPTY;

    return s;

}

equiv() is like equals() but equals() follows Java semantics and equiv() follows Clojure semantics. For your custom types, it's likely they mean the same thing.  

empty() returns a collection of the same type without any elements.

Not sure on mutually recurring deftypes - sounds tricky. It is entirely possible this has come up on clojure or clojure-dev mailing lists in the past so I would start by searching there.

Alex

[Phillip Lord]

Alex Miller <al...@puredanger.com> writes:

> next() should return either the remaining seq or null (think Clojure
> function next)
> more() should return either the remaining seq or empty list (like Clojure
> function rest)
>
> Inside Clojure, most seqs extend ASeq, which implements more() on top of
> the abstract next():
>
> public ISeq more(){
> ISeq s = next();
> if(s == null)
> return PersistentList.EMPTY;
> return s;
> }
>
> equiv() is like equals() but equals() follows Java semantics and equiv()
> follows Clojure semantics. For your custom types, it's likely they mean the
> same thing.
>
> empty() returns a collection of the same type without any elements.

Is the only place this is written down is by working reading the
implementing classes?

> Not sure on mutually recurring deftypes - sounds tricky. It is entirely
> possible this has come up on clojure or clojure-dev mailing lists in the
> past so I would start by searching there.

Well, I have a solution of forms for this.

Phil

[John D. Hume]

On Feb 11, 2014 4:41 AM, "Phillip Lord" <philli...@newcastle.ac.uk> wrote:
>
> Is the only place this is written down is by working reading the
> implementing classes?

I believe so. Other than the new (and quite minimal) Java API for calling Clojure code[1], the details of Clojure's underlying Java classes are considered implementation details and could change from release to release without being considered a breaking change.

[1]: https://github.com/clojure/clojure/blob/master/changes.md#21-java-api

[Phillip Lord]

"John D. Hume" <duelin....@gmail.com> writes:

> On Feb 11, 2014 4:41 AM, "Phillip Lord" <philli...@newcastle.ac.uk>
> wrote:
>>
>> Is the only place this is written down is by working reading the
>> implementing classes?
>
> I believe so. Other than the new (and quite minimal) Java API for calling
> Clojure code[1], the details of Clojure's underlying Java classes are
> considered implementation details and could change from release to release
> without being considered a breaking change.

And the interfaces? I mean, ISeq is defines the sequence abstraction.

Phil

[John D. Hume]

When I said "underlying java classes" I meant "underlying Java classes and interfaces." 

Although there are contrib and third party libraries that depend on these implementation details, I think the message to the community at large has always been not to depend on them, and as far as I know, the maintainers have not committed to keeping anything other than the newly documented Java API and the existing Clojure API in Clojure stable.

So if I'm not mistaken, in spite of the fact that you can find many examples of people doing it, there is no blessed way to create custom Clojure data structures (unless you count defrecord) and no official documentation to encourage or assist you. 

Hopefully I am mistaken, or this will change someday soon. :)

[Alex Miller]

I'd say there is a range of "public"-ness to the internals of Clojure.

- The new Clojure API (clojure.java.api.Clojure) is an official public API for external callers of Clojure. This API basically consists of ways to resolve vars and invoke functions.

- For Clojure users in Clojure, pretty much any var that's public and has a docstring, and shows up in the api docs can be considered public API.

- Clojure vars that are private or have no docstring (such that the var is omitted from public api docs) are likely places to tread very carefully.

- The Clojure internal Java interfaces are certainly intended to allow library builders to create useful stuff that plays in the Clojure world. I do not know that anyone has ever said that they are "public", but I certainly think that any change to a core interface likely to break external users would be considered very carefully. 

- The Clojure internal Java classes should in most cases be considered private and subject to change without notice. There are grey areas even there.

In general, we do not place a high value on encapsulation or hiding internals. In most cases, the internals are left available if they might be useful to an advanced user doing interesting things, with the caveat that the weirder things you do, the more likely you are to be accidentally broken in a future release.

Re documentation, I personally would like to have more around the Java interfaces, but I don't know whether that will happen. I think 80% of the interfaces are pretty obvious but admittedly that last 20% can take a while to ferret out. I'm happy to answer questions as I see them on the list.

[Mikera]

This is useful information - thanks Alex! 

Might be worth putting some of this on a Clojure.org page somewhere, perhaps linked to the Java interop section?

As someone who quite regularly interfaces to Clojure from Java, it would be useful to make *some* of the interfaces into an official public Java API. The ones I'm thinking of in particular are:

IFn

ISeq

ILookup

IPersistentMap

IPersistentVector

IPersistentList

IPersistentCollection

IMeta 

IObj (for metadata updates)

Together these give just enough API surface area to efficiently navigate Clojure data structures, which is pretty crucial for interop......

It might be a good implementation option to put the "public" parts of these interfaces in the clojure.java.api package, and have the internal Clojure interfaces / classes in clojure.lang inherit from these.

[Alex Miller]

IFn (along with the new clojure.java.api.Clojure class) are the totality of the official Java API for Clojure.

I would consider some of the other interfaces you mention to effectively be "SPI" (along with "trait" interfaces like Counted, Seqable, Indexed, etc). If something were going to be added to clojure.org, I think it would be along the lines of how to write new impls (not just collections but sequences, functions, whatever) that can conform to the same expectations as built-in objects. 

[Brandon Bloom]

The other question: is it possible to have two mutually refering deftype
definitions.

Although recursive types makes sense on the JVM, it doesn't necessarily make sense on hosts without an explicit type system in which to resolve the mutual recursion; consider ClojureScript. You don't need recursive type definitions in order to have mutually recursive instances. You can simply indirect through mutation.

For you particular case...

> The closest I have got it: 

> (declare create-alice) 
> (declare create-brian) 

Yup, that's A-OK. You can also just write (declare create-alice create-brian).

The tricky bit comes in when you actually need to refer to the types by name directly. Say, if you wanted to create mutually recursive deftypes with type hints (maybe for host interop reasons). In that case, you can indirect through interfaces/protocols:

(defprotocol IFoo ...)

(defprotocol IBar ...)

(deftype Foo [^IBar bar] ...)

(deftype Bar [^IFoo foo] ...)

If you genuinely need mutually recursive types and can't indirect through protocols (or interfaces), then you probably have a particular JVM interop use case and should just write Java for that purpose.

[Mikera]

Clearly they are useful as SPI.

But I'd argue they are also API-relevant: if you get hold of a Clojure var through the Java API and invoke it via IFn, then these interfaces are pretty useful to help you construct parameters and deal with return values. They are not completely essential (since you can also do this through the core API) but still very handy.

Not a big deal I guess, but I do like the ability to use the Clojure data structure interfaces from Java.

[Phillip Lord]

Mikera <mike.r.an...@gmail.com> writes:

> This is useful information - thanks Alex!
>
> Might be worth putting some of this on a Clojure.org page somewhere,
> perhaps linked to the Java interop section?
>
> As someone who quite regularly interfaces to Clojure from Java, it would be
> useful to make *some* of the interfaces into an official public Java API.
> The ones I'm thinking of in particular are:
> IFn
> ISeq
> ILookup
> IPersistentMap
> IPersistentVector
> IPersistentList
> IPersistentCollection
> IMeta
> IObj (for metadata updates)
>
> Together these give just enough API surface area to efficiently navigate
> Clojure data structures, which is pretty crucial for interop......

I'm not sure this is just an interop issue. As the website says:

"Clojure is written in terms of abstractions. There are abstractions for
sequences, collections, callability, etc. In addition, Clojure supplies
many implementations of these abstractions. The abstractions are
specified by host interfaces, and the implementations by host classes."

For these abstractions to be useful, they must be usable in "client"
code. Otherwise, they are an implementation detail.

I guess, in the abstract (oops), the problem is that ISeq is a Java
interface, when really it should be a protocol, i.e. defined in Clojure.
In the meantime, documentation would be good!

Phil

[Phillip Lord]

Brandon Bloom <brandon...@gmail.com> writes:
>
>> The closest I have got it:
>> (declare create-alice)
>> (declare create-brian)
>
> Yup, that's A-OK. You can also just write (declare create-alice
> create-brian).
>
> The tricky bit comes in when you actually need to refer to the types by
> name directly. Say, if you wanted to create mutually recursive deftypes
> with type hints (maybe for host interop reasons). In that case, you can
> indirect through interfaces/protocols:
>
> (defprotocol IFoo ...)
> (defprotocol IBar ...)
>
> (deftype Foo [^IBar bar] ...)
> (deftype Bar [^IFoo foo] ...)
>
> If you genuinely need mutually recursive types and can't indirect through
> protocols (or interfaces), then you probably have a particular JVM interop
> use case and should just write Java for that purpose.

Thank you, this is a very good answer!

Phil