Fully lazy sequences are coming - feedback wanted!

[Rich Hickey]

I'm pretty much finished with the fully-lazy implementation and am
happy so far with the results. I think this will be an important
addition to Clojure and am planning to add it.

Now comes the hard part - names and change. The essence of the fully
lazy seqs is that they will not consume any resources, perform any
computation or trigger any side effects until they are consumed. This
is a change to the way the sequence functions work now, in that, in
order to determine whether they should return a seq or nil, they need
to touch at least one item. So, there will be an additional function
on seqs, one that returns the items other than the first as a logical,
non-nil, possibly empty collection. Calling seq on this collection
will give you what rest currently gives you - the next seq object or
nil if none. So the core operations on seqs will be:

;item
(first x)

;collection of remaining items, possibly empty
(possibly-empty-collection-of-the-remaining-items x)

;seq on next item, or nil if none
(seq-on-the-next-item-if-any-else-nil x)

(first x) is uncontroversial and won't change. The second is a new
function. The third is currently called 'rest'.

I have some ideas for names, and there are definitely tradeoffs
between short-term pain and long-term goodness in some of the options.
The first option is to leave rest alone, and give the new function a
new name, like more.

;item
(first x)

;collection of remaining items, possibly empty
(more x)

;seq on next item, or nil if none
(rest x)

Note that (rest x) === (seq (more x))

This is implemented in the lazy branch, SVN rev 1281. It has the
attribute of requiring the fewest changes to existing code, and the
drawback of leaving us with less-than-ideal names, especially insofar
as more (or whatever you choose to call it) will in some way seem
synonymous with rest. This naming scheme, and the changes it implies,
is documented here:

http://clojure.org/lazier1

The second option is to choose the best possible names, and deal with
some short term pain in porting and confusion. I think the best names
are:

;item
(first x)

;collection of remaining items, possibly empty
(rest x)

;seq on next item, or nil if none
(next x)

This is implemented in the lazy branch, SVN rev 1282. Note that this
changes the meaning of rest, and gives the current rest operation a
new name, next. It has the attributes of using the most appropriate
names (IMO) and the drawback of changing the semantics of a frequently
used function name, but still offering that functionality under a
different name. It would also break the compatibility of rest with
Common Lisp's. As with the previous model, the third function can be
defined in terms of the second - (next x) === (seq (rest x)). This
naming scheme, and the changes it implies, is documented here:

http://clojure.org/lazier

A third option would be to retire rest and use only new names:

;item
(first x)

;collection of remaining items, possibly empty
(more x)

;seq on next item, or nil if none
(next x)

I haven't implemented this.

I prefer first/rest/next. I think rest is the best complement to
first, and it should mean the logical collection once things are fully
lazy. I think next implies the next seq, as well as the eager nature
of the operation.

I am looking for feedback from people willing to read and understand
the linked-to documentation and the fully lazy model, and especially
from those trying the lazy branch code and porting some of your own.
Questions on the model welcome as well. Chouser has also blogged a bit
about this, with some useful descriptions of nil punning:

http://blog.n01se.net/?p=39

I've been working on this for a few months, in lieu of more
interesting things, because I knew it would be a breaking change and
we're trying to get the biggest of those behind us. I appreciate any
effort you spend in trying to provide informed input.

Thanks,

Rich

[Howard Lewis Ship]

I would vote that you change to the optimum names, but work with Stu
Halloway to ensure that either his book gets updated before it is
printed, or there is a "cheat sheet" on the Clojure site to translate
from his book to any new names.

--
Howard M. Lewis Ship

Creator Apache Tapestry and Apache HiveMind

[CuppoJava]

I'm also in support of the optimal names. Clojure is not too widely
used in production code yet, and it would be a shame to start
compromising design decisions for backwards compatibility already.

This is actually one of my (and many other people's) favorite parts
about Clojure, the beauty of Lisp without the baggage. I wouldn't like
for Clojure to start carrying baggage of its own.

[CuppoJava]

One thing I did find confusing though was in regards to the doc.
Is there a way to more clearly differentiate between a seq and a
sequence? Up until now, I've always thought of "seq" as just being
shorthand for "sequence" which isn't the case apparently.

[jim]

I also favor the optimal names. We're the pioneers in using clojure,
so we should expect a few arrows. Hopefully, the number of clojure
users in the future will be an order of magnitude greater than where
we are now. For us to take short term hit, we can save a large number
of people a lot of cycles later on. I also prefer it on aesthetic
grounds. Breaking compatibility doesn't bother me much.

Jim

[Vincent Foley]

Hello Rich,

I'll play around with the lazy branch this week, and this is just a
name suggestion: what do you think of first/tail/rest where (rest s)
== (seq (tail s))? tail is already used in other functional languages
such as Haskell and OCaml to represent all-but-the-first elements, so
it wouldn't be completely foreign.

Vincent.

[Aaron Scott]

I agree with the op. While the language is still relatively young
please break things so they sit better in the long term. Accurate and
descriptive names are totally valueable, and I'm pretty handy with
find/replace on the editor anyway :p

Aaron

On Feb 15, 2009, at 10:43 AM, CuppoJava <patrick...@hotmail.com>
wrote:

[Konrad Hinsen]

On 15.02.2009, at 20:48, Vincent Foley wrote:

> I'll play around with the lazy branch this week, and this is just a
> name suggestion: what do you think of first/tail/rest where (rest s)
> == (seq (tail s))? tail is already used in other functional languages
> such as Haskell and OCaml to represent all-but-the-first elements, so
> it wouldn't be completely foreign.

I like that choice as well.

Otherwise, I'll join the crowd who thinks that it's better to get
everything in order now without making compromises for backwards
compatibility.

Konrad.

[Rich Hickey]

On Feb 15, 2:48 pm, Vincent Foley <vfo...@gmail.com> wrote:
> Hello Rich,
>
> I'll play around with the lazy branch this week, and this is just a
> name suggestion: what do you think of first/tail/rest where (rest s)
> == (seq (tail s))? tail is already used in other functional languages
> such as Haskell and OCaml to represent all-but-the-first elements, so
> it wouldn't be completely foreign.
>

That falls into the synonym category - tail/rest mean roughly the same
thing. tail complements head, which we aren't using. Coming from
outside, I'd have no idea which did what or why.

Rich

[kyle smith]

"It would also break the compatibility of rest with Common Lisp's"

This is of mild concern to me, but I think if there was a prominent
warning on clojure.org, I could get over it.

[Stephen C. Gilardi]

On Feb 15, 2009, at 12:18 PM, Rich Hickey wrote:

> I am looking for feedback from people willing to read and understand
> the linked-to documentation and the fully lazy model, and especially
> from those trying the lazy branch code and porting some of your own.

I'm trying svn rev 1282 with the following test (which depends on
javadb, (derby)):

user=> (use 'clojure.contrib.sql.test)
nil
user=> (db-write)

It hangs there. This works on the trunk.

I looked for uses of rest in the following libs and didn't find any:

clojure.contrib.sql
clojure.contrib.sql.internal
clojure.contrib.sql.test

I tried using Chouser's "-Dclojure.assert-if-lazy-seq=please"
facility. While I was able to trigger an exception from it using
sample code, it wasn't triggered during the hang.

I'd like to figure this out.

- Has anyone gotten past this already?
- Does anyone see the problem by inspecting the lib code?
- This seems like an opportunity for me to use a Java debugger with
Clojure for the first time. Has anyone written about using JSwat or
another debugger with Clojure?

I would appreciate hearing any tips for getting to the cause of this.

--Steve

[Stephen C. Gilardi]

On Feb 15, 2009, at 4:30 PM, Stephen C. Gilardi wrote:

- This seems like an opportunity for me to use a Java debugger with Clojure for the first time. Has anyone written about using JSwat or another debugger with Clojure?

:-) clojure.org Getting Started page.

--Steve

[Rich Hickey]

On Feb 15, 4:30 pm, "Stephen C. Gilardi" <squee...@mac.com> wrote:
> On Feb 15, 2009, at 12:18 PM, Rich Hickey wrote:
>
> > I am looking for feedback from people willing to read and understand
> > the linked-to documentation and the fully lazy model, and especially
> > from those trying the lazy branch code and porting some of your own.
>
> I'm trying svn rev 1282 with the following test (which depends on
> javadb, (derby)):
>
> user=> (use 'clojure.contrib.sql.test)
> nil
> user=> (db-write)
>
> It hangs there. This works on the trunk.
>

Are you burning cycles while hung, or just blocked?

Rich

[Chouser]

Here's an example of what I think will be the worst kind of breakage
resulting from changing the meaning of rest from
seq-on-the-next-item-if-any-else-nil to
possibly-empty-collection-of-the-remaining-items:

(defn my-interpose [x & coll]
(loop [v [x] coll coll]
(if coll
(recur (-> v (conj (first coll)) (conj x)) (rest coll))
v)))

This is a bit like the builtin interpose, except it takes multiple
args instead of a collection, and it returns a vector with the
interposed value surrounding all the others:

(my-interpose 'x 'a 'b 'c)
-> [x a x b x c x]

At least that's what it does in svn 1282 trunk. In 1282 lazy branch,
it's an infinite loop. Can you spot the problem?

When discussing this yesterday in IRC, I was pretty firmly against
Rich's preferred names, for exactly this reason. And worse than
trying to fix my own code would be the potential confusion over which
versions of examples, libs, etc. work with which versions of Clojure.

...but my position has softened, as I tried to construct an example
for this post that actually broke in a bad way. My first several
attempts produced code that worked in both versions.

For example, my-interpose above takes multiple args so that I could
safely assume that 'coll' is a seq. My first (unposted) version took
a collection as a second argument, but in that case a simple
"(if coll" is probably already an error, in case a user passed in an
empty vector, or some other collection. The solution would be to test
the seq of the coll:

(defn my-interpose [x coll]
(loop [v [x] coll coll]
(if (seq coll) ; Don't assume coll is a seq-or-nil
(recur (-> v (conj (first coll)) (conj x)) (rest coll))
v)))

That also happens to solve the lazy-branch infinite-loop problem --
what's more correct in trunk is more correct in lazy, in this case.
So I kept refining my-interpose, trying to get a version that was
correct in trunk but caused a non-exception error in lazy. After
several iterations I finally got the one at the top of this post.

...but even that one can be caught easily by turning on
clojure.assert-if-lazy-seq, in which case you get an exception
pointing directly to the line that needs to be changed:

java.lang.Exception: LazySeq used in 'if'

So the same changes that will already have to be made to nil puns for
the other seq functions would now have to be made for uses of the new
'rest' function.

Sorry if this has been a bit long-winded, but I wanted to explain why
I've changed my mind a bit -- changing the meaning of 'rest' may not
be as bad as I had been thinking.

--Chouser

[Chouser]

On Sun, Feb 15, 2009 at 4:30 PM, Stephen C. Gilardi <sque...@mac.com> wrote:
>
> I'm trying svn rev 1282 with the following test (which depends on javadb,
> (derby)):
>
> user=> (use 'clojure.contrib.sql.test)
> nil
> user=> (db-write)
>
> It hangs there. This works on the trunk.

I just tried this on 1282 lazy branch with assert-if-lazy-seq, and I
get no exception and no hang:

user=> (time (db-write))
"Elapsed time: 802.020886 msecs"

I wonder what's different?

I seem to have version 1.6.0_07 of javadb, java, and javac, running on
Ubuntu.

--Chouser

[Stephen C. Gilardi]

On Feb 15, 2009, at 4:40 PM, Rich Hickey wrote:

Are you burning cycles while hung, or just blocked?

One core is pinned.

--Steve

[Konrad Hinsen]

On 15.02.2009, at 18:18, Rich Hickey wrote:

> I've been working on this for a few months, in lieu of more
> interesting things, because I knew it would be a breaking change and
> we're trying to get the biggest of those behind us. I appreciate any
> effort you spend in trying to provide informed input.

For those who want to play with this without keeping two versions of
their source code files, I have added a new macro lazy-and-standard-
branch to clojure.contrib.macros. Here is an example of how to use it:

(lazy-and-standard-branch

(defn value-seq [f seed]
(lazy-seq
(let [[value next] (f seed)]
(cons value (value-seq f next)))))

(defn value-seq [f seed]
(let [[value next] (f seed)]
(lazy-cons value (value-seq f next))))

)

Konrad.

[Mark Engelberg]

On Sun, Feb 15, 2009 at 1:44 PM, Chouser <cho...@gmail.com> wrote:
> (defn my-interpose [x coll]
> (loop [v [x] coll coll]
> (if (seq coll) ; Don't assume coll is a seq-or-nil
> (recur (-> v (conj (first coll)) (conj x)) (rest coll))
> v)))

You know, there is an empty? predicate. Why not write it as:

(defn my-interpose [x coll]
(loop [v [x] coll coll]

(if (empty? coll) v ; Don't assume coll is a seq-or-nil
(recur (-> v (conj (first coll)) (conj x)) (rest coll)))))

I know that your first version is viewed as more idiomatic in Clojure,
but I've never understood why Rich and others prefer that style. It
assumes that converting something to a seq is guaranteed to be a
computationally cheap operation, and I see no reason to assume that
will always be the case. I can certainly imagine seq-able
collections that take some time seq-ify, so converting to a seq to
test for empty, and then just throwing it away causing it to be
recomputed in rest doesn't seem as future-proof as just using empty?.

[Stephen C. Gilardi]

On Feb 15, 2009, at 4:52 PM, Chouser wrote:

> I just tried this on 1282 lazy branch with assert-if-lazy-seq, and I
> get no exception and no hang:
>
> user=> (time (db-write))
> "Elapsed time: 802.020886 msecs"
>
> I wonder what's different?

Based on it working for you, the current theory I'm working to verify
is that this was caused by a clojure-contrib.jar compiled with trunk
interacting with a clojure.jar from lazy 1282.

Should we branch contrib and do the fixups on a lazy branch? Chouser,
have you already fixed it enough to compile with clojure contrib's
build.xml?

--Steve

[Konrad Hinsen]

On 15.02.2009, at 23:00, Konrad Hinsen wrote:

> For those who want to play with this without keeping two versions of
> their source code files, I have added a new macro lazy-and-standard-
> branch to clojure.contrib.macros. Here is an example of how to use it:

BTW, my library modules in clojure.contrib (accumulators, monads,
probabilities) now work with the lazy branch as well as with the
standard one. The changes were minor and quick to do. The nil-punning
compiler flag was quite helpful.

Konrad.

[Stephen C. Gilardi]

On Feb 15, 2009, at 5:03 PM, Stephen C. Gilardi wrote:

> Based on it working for you, the current theory I'm working to
> verify is that this was caused by a clojure-contrib.jar compiled
> with trunk interacting with a clojure.jar from lazy 1282.

I've confirmed this. Thanks for the help. The test I wrote about is
now working for me with lazy 1282.

--Steve

[Rich Hickey]

On Feb 15, 2009, at 4:44 PM, Chouser wrote:

>
> Here's an example of what I think will be the worst kind of breakage
> resulting from changing the meaning of rest from
> seq-on-the-next-item-if-any-else-nil to
> possibly-empty-collection-of-the-remaining-items:
>
> (defn my-interpose [x & coll]
> (loop [v [x] coll coll]
> (if coll
> (recur (-> v (conj (first coll)) (conj x)) (rest coll))
> v)))
>
> This is a bit like the builtin interpose, except it takes multiple
> args instead of a collection, and it returns a vector with the
> interposed value surrounding all the others:
>
> (my-interpose 'x 'a 'b 'c)
> -> [x a x b x c x]
>
> At least that's what it does in svn 1282 trunk. In 1282 lazy branch,
> it's an infinite loop. Can you spot the problem?
>
> When discussing this yesterday in IRC, I was pretty firmly against
> Rich's preferred names, for exactly this reason. And worse than
> trying to fix my own code would be the potential confusion over which
> versions of examples, libs, etc. work with which versions of Clojure.

While not knowing if sample code has been ported will still be an
issue, anyone following the porting recipe:

http://clojure.org/lazier#toc7

will avoid this one as well, as the call will be to next, not rest.

I would just clarify that to say that the best route is *not* to
structurally change code that uses rest, just have it call next
instead (unless you are writing a lazy-seq body). Using next is going
to let you preserve your code structure and yields the simplest idioms
- since next (still) nil puns!

core.clj e.g. is full of code that presumes it is walking a seq chain,
and so contains lots of next calls:

http://code.google.com/p/clojure/source/diff?spec=svn1282&r=1282&format=side&path=/branches/lazy/src/clj/clojure/core.clj

There's nothing wrong with that idiom. I do not recommend that people
leave their rest calls and 'fix' the nil puns - instead, change your
rest calls to next, then deal with your own lazy-cons calls (possibly
restoring some rest calls in lazy-seq bodies), then try the
clojure.assert-if-lazy-seq flag to find any conditional use of lazy
sequences.

Rich

[Rich Hickey]

When walking a chain of seqs, empty? made no sense as there is no such
thing as an empty seq. Now that rest returns a collection, this makes
more sense (although still not my preference), but to each his own.
Let's please not get bogged down in a style discussion now. You should
be quite happy for this:

(rest [1])
-> () ;an empty sequence, note - not a canonic/sentinel value!

Also note empty? is still defined like this:

(not (seq coll))

Rich

[Chouser]

On Sun, Feb 15, 2009 at 5:03 PM, Stephen C. Gilardi <sque...@mac.com> wrote:
>
> Should we branch contrib and do the fixups on a lazy branch? Chouser, have
> you already fixed it enough to compile with clojure contrib's build.xml?

I don't ever compile clojure-contrib, I just put its src dir in my
classpath. I've fixed a couple functions here and there using a macro
something like Konrads, but I think that's going to make the code
cluttered pretty quickly. A branch of clojure-contrig is probably
quite sensible at this
point.

--Chouser

[Rich Hickey]

Great! Thanks for the report.

Rich

[Stuart Halloway]

I prefer first/rest/next.

Because of where the book is in the production cycle, it will be
difficult for me to change the prose. But if this gets decided (and
clojure-contrib updated) in the next week or two I think I can get the
book printed with the changes incorporated throughout.

Cheers,
Stu

[James G. Sack (jim)]

Rich Hickey wrote:
>..

> The second option is to choose the best possible names, and deal with
> some short term pain in porting and confusion. I think the best names
> are:
>
> ;item
> (first x)
>
> ;collection of remaining items, possibly empty
> (rest x)
>
> ;seq on next item, or nil if none
> (next x)

(I would say "seq-on-remainder-of-collection")

I really like the first/rest decomposition concept. first (if exists) is
an item, and rest is the remainder-of-whatever following the first.

To me next connotes another item like the first, and that may be
misleading. So I do not think that next is a good name.

Please allow me as an inexpert, relatively uninvolved reader to raise an
emperor's new clothes type question: why is there a need for next
anyway. Are there that many idioms or code internals that justify a
shortcut for (seq rest)?

Regards,
..jim

[Howard Lewis Ship]

At the risk of over-complicating things, perhaps there should be a
macro/function to "require" a specific version of Clojure? In this
way, a script written for the "new" naming could prevent itself from
executing incorrectly using the old naming. Something like Python's
"from future" concept.

--
Howard M. Lewis Ship

Creator Apache Tapestry and Apache HiveMind

[Howard Lewis Ship]

How about next-seq or rest-seq?

[samppi]

While it's not the most important issue, I agree with CuppoJava about
"Sequence" vs "Seq", while we're talking about names. This pair of
terms seems sort of arbitrary, and will probably cause a little
semantic pain and confusion to newcomers in the future. Is there a
better term than "Sequence" and "Seq" for either of the concepts of
"lazy sequence that can be empty" or "sequence that is nil if it's
empty"? Overloading the word "sequence" more probably will sometimes
get confusing.

On Feb 15, 11:58 am, CuppoJava <patrickli_2...@hotmail.com> wrote:
> One thing I did find confusing though was in regards to the doc.
> Is there a way to more clearly differentiate between a seq and a
> sequence? Up until now, I've always thought of "seq" as just being
> shorthand for "sequence" which isn't the case apparently.

[Rich Hickey]

next exists right now - it's called rest, and there are plenty of
calls to it.

Rich

[Mark Engelberg]

My thoughts so far:

1. It always troubled me that filter, when written in the most
natural way, had a "hang on to the head" problem when skipping over
large numbers of items. I think this is something worth solving, and
I'm glad that while developing the lazier branch, you came up with a
compiler enhancement to address this. In my mind, this may be the
most valuable aspect of the new changes. The new version of filter is
definitely more complex than the old version, but it's not too bad.
If the compiler enhancement could be backported to make the old (most
natural) version work as well, I think that would be even better.

2. I definitely prefer that you go with the best names, rather than
worrying about backward compatibility at this point. So I like the
idea of changing the meaning of rest. I'm not particularly keen on
the name "next", but I don't care that much. I feel fairly certain
that my own personal programming style will be to stick with
first/rest, and I doubt I'll use "next", so to me the name choice only
matters to the extent that it uses up a name that might be natural in
another context.

3. As I've noted here previously, I never cared much for nil punning,
and I always try to write my own code in a way that doesn't rely on
it. So I don't care if it goes away.

4. The new model is definitely more complicated to understand than
the previous model. There was already a certain degree of mental
overlap between collections and the seq interface. Now, there is also
the subtle distinction between a seq and a sequence. rest and next
are very similar, but one can return something empty, and one can
return nil. Making the right choice, and interfacing with other code
is now a bit more complicated (although people can always call seq to
convert it into the seq/nil paradigm with certainty, which is not much
different than before). I think the additional complexity is worth it
to solve things like the filter problem, but I think it's definitely
more confusing than before.

5. At first glance, it seems like sticking with the original
lazy-cons model, but removing nil punning and adding an empty sequence
sentinel, along with your compiler enhancement, would accomplish
everything the new "lazier" branch accomplishes with much less mental
complexity and subtle overlap, and resulting in the most intuitive
version of filter working as expected. I know you like the seq/nil
model, and the nil punning, but since you're already moving in the
direction of reducing reliance on this approach, I hope you've
considered going "all the way", to see if it would solve the problem
more elegantly. If you have considered this already, I'd be curious
to know whether it didn't solve the problem, or whether it just
resulted in too much breakage with existing code, or whether it's just
a style you don't like as a matter of taste...

[Rich Hickey]

On Feb 15, 2009, at 8:22 PM, Mark Engelberg wrote:

>
> My thoughts so far:

>
>
> 4. The new model is definitely more complicated to understand than
> the previous model. There was already a certain degree of mental
> overlap between collections and the seq interface. Now, there is also
> the subtle distinction between a seq and a sequence.

There will need to be good descriptions of these, but the similarity
is more in names than anything else - seqs are what they always were -
cursors, and sequences are just collections.

> rest and next
> are very similar, but one can return something empty, and one can
> return nil. Making the right choice, and interfacing with other code
> is now a bit more complicated (although people can always call seq to
> convert it into the seq/nil paradigm with certainty, which is not much
> different than before).

Code that returns sequences should use rest. next is just a
convenience function for terminal/consumer code. If you look through
the ported core code, most recurs use next, most conses inside lazy-
seqs use rest. But as you noted, if you want to ignore next, that's
fine.

> I think the additional complexity is worth it
> to solve things like the filter problem, but I think it's definitely
> more confusing than before.
>
> 5. At first glance, it seems like sticking with the original
> lazy-cons model, but removing nil punning and adding an empty sequence
> sentinel, along with your compiler enhancement, would accomplish
> everything the new "lazier" branch accomplishes with much less mental
> complexity and subtle overlap, and resulting in the most intuitive
> version of filter working as expected.

I realize you are focused on filter, but that point of the fully lazy
branch is full laziness, which would not fall out of what you
describe. lazy-cons requires the lazy sequence function do all the
work that precedes the call to lazy-cons, making functions like drop
and filter not fully lazy.

> I know you like the seq/nil
> model, and the nil punning, but since you're already moving in the
> direction of reducing reliance on this approach, I hope you've
> considered going "all the way", to see if it would solve the problem
> more elegantly. If you have considered this already, I'd be curious
> to know whether it didn't solve the problem, or whether it just
> resulted in too much breakage with existing code, or whether it's just
> a style you don't like as a matter of taste...

I'm not sure what you mean by "all the way". If you mean removing seqs
entirely, here are some issues:

(seq x) acts as a single interface to the sequence system, if it were
to return a seq[uence] collection instead of seq/nil, then all
sequence function bodies would have to return () explicitly, adding
another branch to all implementations that could otherwise use when:

(defn map [f coll]
(lazy-seq
(if (empty? coll)
()
(cons (f (first coll)) (map f (rest coll))))))

instead of:

(defn map [f coll]
(lazy-seq
(when (seq coll)
(cons (f (first coll)) (map f (rest coll))))))

But the actual definition in the lazy branch pulls the seq out of the
lazy sequence like so:

(defn map [f coll]
(lazy-seq
(when-let [s (seq coll)]
(cons (f (first s)) (map f (rest s))))))

Doing so yields a significant (> 60%) speed improvement, without it,
all lazy calls (empty?/first/rest) have a double indirection, and no
way to get rid of it.

Rich

[Feng]

I tried to migrate swank-clojure to lazy branch rev1282. Here are
steps I did.

1) search and replace rest to next
2) search and replace seq? to sequence?
3) change lazy-cons to lazy-seq following the recipe
4) fixed if LazySeq exceptions

see attached diff at the end.

Code loads fine. However, I got below error as soon as emacs slime-
connect to swank server.
The failing code occurred in eval form read from emacs connection. In
jswat debugger, I can see form.toString() in
clojure.lang.Compiler.eval frame as "(#'swank.commands.contrib/swank-
require (quote (:swank-repl ...)))". I even cut & pasted it to plain
repl and it ran fine. However, browsing object shows form.more() is a
clojure.core$map_xxx$fn_xxx, which looks like a lazy-seq closure.

What could be wrong here?

clojure.lang.PersistentList
[Thrown class java.lang.ClassCastException]

Restarts:
0: [ABORT] Return to SLIME's top level.

Backtrace:
0: swank.commands.contrib$eval__1128.<clinit>(Unknown Source)
1: sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
2: sun.reflect.NativeConstructorAccessorImpl.newInstance
(NativeConstructorAccessorImpl.java:39)
3: sun.reflect.DelegatingConstructorAccessorImpl.newInstance
(DelegatingConstructorAccessorImpl.java:27)
4: java.lang.reflect.Constructor.newInstance(Constructor.java:494)
5: java.lang.Class.newInstance0(Class.java:350)
6: java.lang.Class.newInstance(Class.java:303)
7: clojure.lang.Compiler$FnExpr.eval(Compiler.java:3263)
8: clojure.lang.Compiler.eval(Compiler.java:4209)
9: clojure.core$eval__3811.invoke(core.clj:1695)
10: swank.core$eval_in_emacs_package__272.invoke(core.clj:55)
11: swank.core$eval_for_emacs__346.invoke(core.clj:123)
12: clojure.lang.Var.invoke(Var.java:344)
13: clojure.lang.AFn.applyToHelper(AFn.java:179)
14: clojure.lang.Var.applyTo(Var.java:453)
15: clojure.core$apply__3084.doInvoke(core.clj:410)
16: clojure.lang.RestFn.invoke(RestFn.java:428)
17: swank.core$eval_from_control__275.invoke(core.clj:62)
18: swank.core$spawn_worker_thread__369$fn__397$fn__399.invoke
(core.clj:157)
19: clojure.lang.AFn.applyToHelper(AFn.java:171)
20: clojure.lang.AFn.applyTo(AFn.java:164)
21: clojure.core$apply__3084.doInvoke(core.clj:410)
22: clojure.lang.RestFn.invoke(RestFn.java:428)
23: swank.core$spawn_worker_thread__369$fn__397.doInvoke(core.clj:
153)
24: clojure.lang.RestFn.invoke(RestFn.java:402)
25: clojure.lang.AFn.run(AFn.java:37)
26: java.lang.Thread.run(Thread.java:613)

thanks,
- Feng

diff --git a/swank/commands/basic.clj b/swank/commands/basic.clj
index 79d4354..8072d5b 100644
--- a/swank/commands/basic.clj
+++ b/swank/commands/basic.clj
@@ -98,8 +98,9 @@
:short-message ~(.toString t)))

(defn- exception-causes [#^Throwable t]
- (lazy-cons t (when-let [cause (.getCause t)]
- (exception-causes cause))))
+ (lazy-seq
+ (cons t (when-let [cause (.getCause t)]
+ (exception-causes cause)))))

(defn- compile-file-for-emacs*
"Compiles a file for emacs. Because clojure doesn't compile, this
is
diff --git a/swank/commands/contrib.clj b/swank/commands/contrib.clj
index 6c0ed07..2185efd 100644
--- a/swank/commands/contrib.clj
+++ b/swank/commands/contrib.clj
@@ -3,7 +3,7 @@

(defslimefn swank-require [keys]
(binding [*ns* (find-ns 'swank.commands.contrib)]
- (doseq [k (if (seq? keys) keys (list keys))]
+ (doseq [k (if (sequence? keys) keys (list keys))]
(try
(require (symbol (str "swank.commands.contrib." (name k))))
- (catch java.io.FileNotFoundException fne nil)))))
\ No newline at end of file
+ (catch java.io.FileNotFoundException fne nil)))))
diff --git a/swank/commands/contrib/swank_arglists.clj b/swank/
commands/contrib/swank_arglists.clj
index 4a87d89..e7a6cd9 100644
--- a/swank/commands/contrib/swank_arglists.clj
+++ b/swank/commands/contrib/swank_arglists.clj
@@ -9,8 +9,8 @@
print-lines]} (apply hash-map options)]
;; Yeah, I'm lazy -- I'll flesh this out later
(if (and raw-specs
- (seq? raw-specs)
- (seq? (first raw-specs)))
+ (sequence? raw-specs)
+ (sequence? (first raw-specs)))
((slime-fn 'operator-arglist) (ffirst raw-specs) *current-
package*)
nil)))

diff --git a/swank/commands/indent.clj b/swank/commands/indent.clj
index 6248399..af58ce3 100644
--- a/swank/commands/indent.clj
+++ b/swank/commands/indent.clj
@@ -33,10 +33,7 @@
(filter (comp var? val) (mapcat ns-map nss)))))

(defn- every-other [coll]
- (when coll
- (lazy-cons
- (first coll)
- (every-other (drop 2 coll)))))
+ (take-nth 2 coll))

(defn- update-indentation-delta
"Update the cache and return the changes in a (symbol '. indent)
list.
@@ -53,15 +50,15 @@
(let [vars (filter (comp var? val) (mapcat ns-map
nss))]
(mapcat in-cache? vars)))]
(if force
- (when-let [updates (considerations-for (all-ns))]
+ (when-let [updates (seq (considerations-for (all-ns)))]
(dosync (apply alter cache assoc updates))
- (every-other (rest updates)))
+ (every-other (next updates)))
(let [ns (maybe-ns *current-package*)
in-ns? (fn [[sym var]] (and (var? var) (= ns ((meta
var) :ns))))]
(when ns
- (when-let [updates (filter identity (considerations-
for (list ns)))]
+ (when-let [updates (seq (filter identity
(considerations-for (list ns))))]
(dosync (apply alter cache assoc updates))
- (every-other (rest updates))))))))))
+ (every-other (next updates))))))))))

(defn- perform-indentation-update
"Update the indentation cache in connection and update emacs.
@@ -71,7 +68,7 @@
(let [delta (update-indentation-delta cache force)]
(dosync
(ref-set (conn :indent-cache-pkg) (hash (all-ns)))
- (when delta
+ (when (seq delta)
(send-to-emacs `(:indentation-update ~delta))))))))

(defn- sync-indentation-to-emacs
diff --git a/swank/commands/inspector.clj b/swank/commands/
inspector.clj
index a91f724..80f66f8 100644
--- a/swank/commands/inspector.clj
+++ b/swank/commands/inspector.clj
@@ -72,7 +72,7 @@
(vector? obj) :vector
(var? obj) :var
(string? obj) :string
- (seq? obj) :seq
+ (sequence? obj) :seq
(instance? Class obj) :class
(instance? clojure.lang.Namespace obj) :namespace)))

@@ -189,7 +189,7 @@
(fn spec-value [val]
(cond
(string? val) val
- (seq? val) (spec-seq val)))]
+ (sequence? val) (spec-seq val)))]
(map spec-value specs)))

;; Works for infinite sequences, but it lies about length. Luckily,
emacs doesn't
@@ -258,7 +258,7 @@
(defslimefn inspector-pop []
(with-emacs-package
(cond
- (rest @*inspector-stack*)
+ (next @*inspector-stack*)
(inspect-object
(dosync
(ref-pop *inspector-stack*)
diff --git a/swank/core.clj b/swank/core.clj
index 260cdd7..87c6623 100644
--- a/swank/core.clj
+++ b/swank/core.clj
@@ -59,7 +59,7 @@
"Blocks for a mbox message from the control thread and executes it
when received. The mbox message is expected to be a slime-fn."
([] (let [form (mb/receive (current-thread))]
- (apply (ns-resolve *ns* (first form)) (rest form)))))
+ (apply (ns-resolve *ns* (first form)) (next form)))))

(defn eval-loop
"A loop which continuosly reads actions from the control thread and
@@ -67,8 +67,9 @@
([] (continuously (eval-from-control))))

(defn- exception-causes [#^Throwable t]
- (lazy-cons t (when-let [cause (.getCause t)]
- (exception-causes cause))))
+ (lazy-seq
+ (cons t (when-let [cause (.getCause t)]
+ (exception-causes cause)))))

(defn- debug-quit-exception? [t]
(some #(identical? *debug-quit-exception* %) (exception-causes t)))
@@ -110,7 +111,7 @@
(send-to-emacs (list :debug-return (current-thread) level
nil))))))

(defn doall-seq [coll]
- (if (seq? coll)
+ (if (sequence? coll)
(doall coll)
coll))

@@ -118,7 +119,7 @@
(try
(binding [*current-package* buffer-package]
(if-let [f (slime-fn (first form))]
- (let [form (cons f (rest form))
+ (let [form (cons f (next form))
result (doall-seq (eval-in-emacs-package form))]
(run-hook *pre-reply-hook*)
(send-to-emacs `(:return ~(thread-name (current-thread))
(:ok ~result) ~id)))
diff --git a/swank/core/protocol.clj b/swank/core/protocol.clj
index 35dfa43..d8bf4c4 100644
--- a/swank/core/protocol.clj
+++ b/swank/core/protocol.clj
@@ -45,7 +45,7 @@
(let [len (hex->num (read-chars 6 rdr))
msg (read-chars len rdr)
form (read-from-string (fix-namespace msg))]
- (if (seq? form)
+ (if (sequence? form)
(deep-replace {'t true} form)
form))))

diff --git a/swank/util.clj b/swank/util.clj
index 3ba2ecc..fcb9c22 100644
--- a/swank/util.clj
+++ b/swank/util.clj
@@ -31,10 +31,10 @@
([item coll] (position item coll 0))
([item coll start]
(loop [coll (drop start coll), i start]
- (when coll
+ (when (seq coll)
(if (= (first coll) item)
i
- (recur (rest coll) (inc i))))))
+ (recur (next coll) (inc i))))))
{:tag Integer})

(defn categorize-by
@@ -54,7 +54,7 @@


(defn deep-replace [smap coll]
- (map #(if (or (seq? %) (vector? %))
+ (map #(if (or (sequence? %) (vector? %))
(deep-replace smap %)
%)
(replace smap coll)))

[Feng]

On Feb 15, 11:46 pm, Feng <hou...@gmail.com> wrote:
> I tried to migrate swank-clojure to lazy branch rev1282. Here are
> steps I did.
>
> 1) search and replace rest to next
> 2) search and replace seq? to sequence?
> 3) change lazy-cons to lazy-seq following the recipe
> 4) fixed if LazySeq exceptions
>
> see attached diff at the end.
>
> Code loads fine. However, I got below error as soon as emacs slime-
> connect to swank server.
> The failing code occurred in eval form read from emacs connection. In
> jswat debugger, I can see form.toString() in
> clojure.lang.Compiler.eval frame as "(#'swank.commands.contrib/swank-
> require (quote (:swank-repl ...)))". I even cut & pasted it to plain
> repl and it ran fine.

However, browsing object shows form.more() is a clojure.core$map_xxx
$fn_xxx, which looks like a lazy-seq closure.

Correction, form.more() is a clojure.lang.Cons object. Its _first and
_more are two objects of class clojure.core$map__3637$fn__3639.

[Mark Engelberg]

On Sun, Feb 15, 2009 at 6:44 PM, Rich Hickey <richh...@gmail.com> wrote:
> I realize you are focused on filter, but that point of the fully lazy
> branch is full laziness, which would not fall out of what you
> describe. lazy-cons requires the lazy sequence function do all the
> work that precedes the call to lazy-cons, making functions like drop
> and filter not fully lazy.

I was thinking that with an empty sentinel, lazy-cons does not have to
do work ahead of time to see whether you're at the end. If you have a
lazy-cons, you know you're not empty without evaluating the first or
rest, and if you have the empty object, you're empty. So I think you
could make lazy-cons fully lazy. I haven't fully thought this through
though, so maybe I'm missing something.

> Doing so yields a significant (> 60%) speed improvement, without it,
> all lazy calls (empty?/first/rest) have a double indirection, and no
> way to get rid of it.

The 60% speed improvement is compelling, but I was thinking that
(especially with an empty sentinel) empty? could be implemented more
efficiently than (not (seq coll)), so the empty?/first/rest style
wouldn't have such a performance hit.

[Stefan Rusek]

At first I found this kind of confusing, but after reading Chouser's
article and the help. It makes a lot of sense. I found it easiest to
understand when I thought about it as two pairs of related names. The
first/rest pair and the seq/more pair.

[Konrad Hinsen]

On Feb 16, 2009, at 3:44, Rich Hickey wrote:

> There will need to be good descriptions of these, but the similarity
> is more in names than anything else - seqs are what they always were -
> cursors, and sequences are just collections.

That distinction is quite clear, the problem is indeed just in the
names, in my opinion. What's the difference between a sequence and
what the rest of the Lisp world calls a list? Would it be reasonable
to call sequences lists?

Konrad.

[James Reeves]

On Feb 15, 5:18 pm, Rich Hickey <richhic...@gmail.com> wrote:
> The second option is to choose the best possible names, and deal with
> some short term pain in porting and confusion. I think the best names
> are:
>
> ;item
> (first x)
>
> ;collection of remaining items, possibly empty
> (rest x)
>
> ;seq on next item, or nil if none
> (next x)

After reading though all the information on the new, lazier sequences,
I like this option best. 'next' is a very appropriate name, as it
implies an eager load of the next item in the same way that first is
an eager load of the first item. It also implies iteration in a way
that, say, 'second' does not, so you'd also expect 'next' to have a
tail. I also like way rest is now lazier.

I don't think you should compromise Clojure's design at this point, so
I definitely think you should go with first/rest/next. I don't like
first/more/rest, as 'more' doesn't seem the best word to describe the
new function, and for the same reason I also dislike first/more/next.
Finally, I agree that tail/rest is also inappropriate, as tail and
rest do not mean different things.

So I'm all for the optimal first/rest/next solution.

- James

[Christophe Grand]

Hi,

Rich Hickey a écrit :

> I am looking for feedback from people willing to read and understand
> the linked-to documentation and the fully lazy model, and especially
> from those trying the lazy branch code and porting some of your own.
>

I just ported Enlive
(http://github.com/cgrand/enlive/commit/3245678e6ae0a82152dbf4a6fb8916d2514b60dd):
* found/replaced rest by next and rrest by nnext,
* no broken nil punnings (!) but several calls to seq? that could be
rewritten in a less brittle way,
* no metadata on sequence, is this an oversight? or is this related to
the lack of metadata on closures? (I'm willing to work on this.)

I'll quickly get over my imperative interpretation of 'next and I trust
in your naming skills: if first/next/fine is the best option, go with it.

Christophe

--
Professional: http://cgrand.net/ (fr)
On Clojure: http://clj-me.blogspot.com/ (en)

[GS]

On Feb 16, 1:44 pm, Rich Hickey <richhic...@gmail.com> wrote:

> > My thoughts so far:
>
> > 4.  The new model is definitely more complicated to understand than
> > the previous model.  There was already a certain degree of mental
> > overlap between collections and the seq interface.  Now, there is also
> > the subtle distinction between a seq and a sequence.
>
> There will need to be good descriptions of these, but the similarity  
> is more in names than anything else - seqs are what they always
> were -  cursors, and sequences are just collections.

Would it therefore make sense to call them cursors and collections
instead of seqs and sequences?

Gavin

[Mibu]

I'm all for breaking bad habits and names and I love it that you give
good design considerations precedence over heritage, but here I think
using the first/rest/next combo is confusing, and will continue to be
confusing in the long-term.

rest is expected to be a sequence by Lispers, and next is expected to
be an item by Java-ers. Both are universally recognized as such and
are frequently used. The semantics given to them by their common use
in computing (and especially in CS education) supersedes their
semantics in daily English or the semantics which should have been
most appropriate to them if their semantics were defined today first.

Synonyms are also not a problem as they receive their unique semantics
soon enough if they are regularly used, and if they're not then there
is no need to waste of good short word on them. I'm against more
because it evokes the semantics of the more? predicate but maybe first/
remaining/rest or instead of remaining: rest-collection, rest-coll, or
restc.

I seem to be in the minority about this...

On Feb 15, 7:18 pm, Rich Hickey <richhic...@gmail.com> wrote:
> I'm pretty much finished with the fully-lazy implementation and am
> happy so far with the results. I think this will be an important
> addition to Clojure and am planning to add it.
>
> Now comes the hard part - names and change. The essence of the fully
> lazy seqs is that they will not consume any resources, perform any
> computation or trigger any side effects until they are consumed. This
> is a change to the way the sequence functions work now, in that, in
> order to determine whether they should return a seq or nil, they need
> to touch at least one item. So, there will be an additional function
> on seqs, one that returns the items other than the first as a logical,
> non-nil, possibly empty collection. Calling seq on this collection
> will give you what rest currently gives you - the next seq object or
> nil if none. So the core operations on seqs will be:
>

> ;item
> (first x)
>
> ;collection of remaining items, possibly empty

> (possibly-empty-collection-of-the-remaining-items x)

>
> ;seq on next item, or nil if none

> (seq-on-the-next-item-if-any-else-nil x)
>
> (first x) is uncontroversial and won't change. The second is a new
> function. The third is currently called 'rest'.
>
> I have some ideas for names, and there are definitely tradeoffs
> between short-term pain and long-term goodness in some of the options.
> The first option is to leave rest alone, and give the new function a
> new name, like more.
>

> ;item
> (first x)
>
> ;collection of remaining items, possibly empty

> (more x)

>
> ;seq on next item, or nil if none

> (rest x)
>
> Note that (rest x) === (seq (more x))
>
> This is implemented in the lazy branch, SVN rev 1281. It has the
> attribute of requiring the fewest changes to existing code, and the
> drawback of leaving us with less-than-ideal names, especially insofar
> as more (or whatever you choose to call it) will in some way seem
> synonymous with rest. This naming scheme, and the changes it implies,
> is documented here:
>
> http://clojure.org/lazier1
>

> The second option is to choose the best possible names, and deal with
> some short term pain in porting and confusion. I think the best names
> are:
>
> ;item
> (first x)
>
> ;collection of remaining items, possibly empty
> (rest x)
>
> ;seq on next item, or nil if none
> (next x)
>

> This is implemented in the lazy branch, SVN rev 1282. Note that this
> changes the meaning of rest, and gives the current rest operation a
> new name, next. It has the attributes of using the most appropriate
> names (IMO) and the drawback of changing the semantics of a frequently
> used function name, but still offering that functionality under a
> different name. It would also break the compatibility of rest with
> Common Lisp's. As with the previous model, the third function can be
> defined in terms of the second - (next x) === (seq (rest x)). This
> naming scheme, and the changes it implies, is documented here:
>
> http://clojure.org/lazier
>
> A third option would be to retire rest and use only new names:
>

> ;item
> (first x)
>
> ;collection of remaining items, possibly empty

> (more x)

>
> ;seq on next item, or nil if none
> (next x)
>

> I haven't implemented this.
>
> I prefer first/rest/next. I think rest is the best complement to
> first, and it should mean the logical collection once things are fully
> lazy. I think next implies the next seq, as well as the eager nature
> of the operation.
>

> I am looking for feedback from people willing to read and understand
> the linked-to documentation and the fully lazy model, and especially
> from those trying the lazy branch code and porting some of your own.

[Jeffrey Straszheim]

I'd vote for the breaking changes.  We don't have so much code written that it cannot be fixed.

However, this depends on the book in production.  Having _Programming Clojure_ come out with incompatible code would be a big blow, I think.

[jwhitlark]

While I'm fairly new to clojure, and with apologies to Stewart
Halloway for complicating his job on the book, (which is excellent so
far, btw) I think it would be worth while to chose the optimum naming
convention, if it can be done fast enough to update the book.
Consider how long some warts had been around before Python 3 removed
them, we're going to have to deal with these things for a long time...

That said, having the book fall out of compliance with clojure would
be REALLY bad. I had that experience with "The Definitive Guide to
Django", which wasn't, and it really turned me off. (The fact that
people made snarky comments on IRC when I asked why things didn't work
didn't help either; after a change like this you can't just tell
people to RTFM.)

~Jason

[Mark Volkmann]

If we're going to be making name changes that break code anyway, I'll
make another appeal to make the function naming convention more
consistent. Most multi-word function names have a hyphen between the
words, but the following do not.

butlast, doall, dorun, doseq, dosync, dotimes, doto, fnseq, gensym,
macroexpand, macroexpand-1, mapcat, nthrest

If we want to keep these names as-is then why do we have hyphens in so
many of the other multi-word function names?

--
R. Mark Volkmann
Object Computing, Inc.

[Dan]

+1

And ideally real soon so Stuart can change that at the same time as adding the fully lazy changes.

[David Nolen]

butlast, doall, dorun, doseq, dosync, dotimes, doto, fnseq, gensym,
macroexpand, macroexpand-1, mapcat, nthrest

-1

Because they are similar to other Lisps I assume.  The same reason for println vs print-line. Changing these are a bad idea in IMHO.

Breaking the meaning of rest with Common Lisp is a consideration to take seriously.  However as long as the documentation clearly states (highlights in big bold colors ;) this difference (and the fact the operation is support by next) then I think this is a change people can live with.

[Rich Hickey]

Changing these names is not on the table.

Rich

[wlr]

Regarding Programming Clojure:

I think that placing the burden of "book vs actual" incompatibility
upon Rich is misplaced. If anything, pressure from the Clojure
community should be placed on the Pragmatic Programmers to allow
Stuart to "do the right thing" regarding when the book is released,
viz., when Clojure has stabilized.

Realize who is making the open contribution and who is profiting from
that contribution and keep the priorities straight.

My 2 cents.

Walt

[Perry Trolard]

I agree with the majority of posters that the breaking changes in the
service of optimal names is the right way to go.

I found the explanation & recipe for porting at clojure.org/lazier
clear & easy to follow. I didn't do full ports of any projects, but I
did some selective porting & found it to be straightforward.

That said, the only problem I see is with the names. Like Mibu, I
think "next" isn't ideal -- it connotes an item in an iteration to me.
If you think "next *seq*" when you see "next" (how Rich explains it
at /lazier), it helps, but it's still not exactly right; or it
requires a different mental model from the non-lazy-branch "rest": the
cursor moving to the "next" item rather than the abstracted "rest of
the coll" (where you think about a cursor).

I think the issue is that "rest" is the right name for both rest &
next. The only difference between them, from the perspective of users,
is how empty rests are represented ('() or nil), & that's a hard
distinction to make manifest in a short name.

If it's the case that rest will almost exclusively appear in the
context of constructing lazy-seqs

(lazy-seq
(cons [something] (rest [something]))

& next will appear all over, it makes sense to me to sacrifice brevity
in the case of rest, & give next the right name: "rest" (that's
tortuous, I know).

rest* isn't quite right, but you get the idea: make the fully-lazy
rest the special-kind-of-rest, & the consumer-code rest the
transparent one. This way, people's concepts about recursing through
seqs of colls & testing for the end won't have to change; they'll only
have to change their understanding of how to make lazy sequences. I
know in my code I do a lot more of the former.

Anyone who's on board with this line of thought have ideas for the
right name of fully-lazy rest?

Best,
Perry

[Jeffrey Straszheim]

You're right, of course, but in life compromises must happen.  If Rich proceeds *with no regard* for Pragmatic's needs, they have a recourse which is simply no Clojure book.  Or a Clojure book that has broken examples.

[Perry Trolard]

> cursor moving to the "next" item rather than the abstracted "rest of
> the coll" (where you think about a cursor).

Correction: where you *don't* think about a cursor...

Perry

[wlr]

On Feb 16, 12:06 pm, Jeffrey Straszheim <straszheimjeff...@gmail.com>
wrote:

> You're right, of course, but in life compromises must happen.  If Rich
> proceeds *with no regard* for Pragmatic's needs, they have a recourse which
> is simply no Clojure book.  Or a Clojure book that has broken examples.
>

Agreed. I'm afraid my original came out sounding more militant than
what I intended. It was really a reaction to some posts which
correctly suggested that Clojure would suffer by being touted by an
out-of-synch Programming Clojure but also failed to suggest compromise
on Pragmatic's side of the equation. The tone seemed to be permissive
of Pragmatic proceeding *with no regard* for Clojure's current and
soon-to-be reality.

I don't believe that folks who have already forked over money for the
book would be out of line in suggesting to Pragmatic that they adjust
the publication schedule. Neither would inquiries from prospective
buyers as to Pragmatic's intentions regarding book vs actual
faithfulness. I didn't mean to suggest an all-out, mean-spirited
attack but can see how that inference could easily have been drawn.

Walt

[Stuart Halloway]

Thanks Rich! :-)

[Stuart Halloway]

I agree with Walt, and there is no need to pressure the Prags, we are
on it! :-)

That said, it would be *very* helpful to me if we could get the
lazyness thing settled this week...

Stuart

[Rich Hickey]

On Feb 15, 12:18 pm, Rich Hickey <richhic...@gmail.com> wrote:
> I'm pretty much finished with the fully-lazy implementation and am
> happy so far with the results. I think this will be an important
> addition to Clojure and am planning to add it.

Thanks all for the feedback!

It seems the Sequence/ISeq dichotomy was a sticking point for many.
After some tweaking, I've been able to get rid of Sequence entirely,
SVN 1284+ in lazy branch. This is source compatible with 1282 (first/
rest/next), except that sequence? no longer exists - go back to seq?.

New docs here:

http://clojure.org/lazy

Let me know if that is simpler.

Rich

[James Reeves]

On Feb 16, 2:22 pm, Mibu <mibu.cloj...@gmail.com> wrote:
> rest is expected to be a sequence by Lispers, and next is expected to
> be an item by Java-ers.

I actually think next is pretty close to the next method on Java
iterators. In java.util.Iterator, the next method evaluates the next
item, increments state the iterator, then returns the evaluated item.
In Clojure, the next function evaluates the next time, then returns a
cons containing the evaluated item and the Clojure equivalent of an
incremented iterator. It's essentially the functional equivalent of
the Java method.

Regarding Lispers, Clojure already differs from common Lisp in that
nil != (). I think the proposed lazier changes make rest more
consistant; instead of returning a collection or nil, it just returns
a collection. Philosophically, I think this is closer to the idea of a
cons cell than it was previously.

Of course, this is all just my opinion :)

- James

[Luc Prefontaine]

We are in production and we fully agree, this thing should be settled now.
In fact if it's done within 10 days, that would fit our current plans.

For reasons out of our control we have been postponing an update to prod., we still
have a window to get this change out.

It's feasible to do the code changes and run a test suite against the new runtime
within two weeks, maybe less.

Luc

[Stephen C. Gilardi]

On Feb 16, 2009, at 2:23 PM, Rich Hickey wrote:

> New docs here:
>
> http://clojure.org/lazy

In the html doc:

rest... "returns a possibly empty seq, never nil"

then later

"never returns nil
- currently not enforced on 3rd party seqs"

In "(doc rest)"

"may return nil"

What's the cleaned up version of all that? Is it worth guaranteeing
that rest never returns nil or should "never returns nil" be removed
from the html docs?

--Steve

[Rich Hickey]

I've fixed the doc string. I'm not going to add code for 3rd party
seqs, enforcing this contract is up to them. If they derive from ASeq
and define next, ASeq will do the right thing for them.

Rich

[Aaron Scott]

How about e-rest, for the empty set returning version?

[Mark Engelberg]

Browsing the source code for LazySeq, I noticed that isEmpty is
implemented as follows:
public boolean isEmpty() {
return count() == 0;
}

Since count realizes the whole list, this seems like a bad way to test
for empty on a lazy sequence.

[Rich Hickey]

Fixed in (lazy) 1286 - thanks for the report.

Rich

[kotor]

I definitely support your second option; first / rest / next. In my
mind, rest means "collection of remaining items" and should return a
collection, and next will also do exactly what I would expect it to
do. Clojure is sufficiently different from Common Lisp already that
breaking the compatibilty of rest is a non-issue for me. As others
have mentioned, a big part of the appeal of Clojure is being involved
in the early stages of a language that has a real opportunity to grow
"cleanly", with developer input and without the inertia of layers of
past compromises. I really like Clojure so far, and thank Rich and the
other main contributors for their work.

[Konrad Hinsen]

On Feb 16, 2009, at 20:23, Rich Hickey wrote:

> It seems the Sequence/ISeq dichotomy was a sticking point for many.
> After some tweaking, I've been able to get rid of Sequence entirely,
> SVN 1284+ in lazy branch. This is source compatible with 1282 (first/
> rest/next), except that sequence? no longer exists - go back to seq?.
>
> New docs here:
>
> http://clojure.org/lazy
>
> Let me know if that is simpler.

I'd say yes.

The remaining weird feature is the seq function and its use. The name
suggests that it converts to a seq, which is in fact what it used to
do. Now it converts to a seq unless the resulting seq would be empty.
For an empty seq, it actually converts a seq to a non-seq!

Would it be possible to make an empty seq test as false? One could
then do away with the conversion to seq in tests completely, and seq
could always return a seq, including an empty one. Of course, this
would imply that a logical test on a seq evaluates its first element,
but that doesn't look unreasonable to me.

Konrad.

[Rich Hickey]

On Feb 17, 2009, at 4:40 AM, Konrad Hinsen wrote:

>
> On Feb 16, 2009, at 20:23, Rich Hickey wrote:
>
>> It seems the Sequence/ISeq dichotomy was a sticking point for many.
>> After some tweaking, I've been able to get rid of Sequence entirely,
>> SVN 1284+ in lazy branch. This is source compatible with 1282 (first/
>> rest/next), except that sequence? no longer exists - go back to seq?.
>>
>> New docs here:
>>
>> http://clojure.org/lazy
>>
>> Let me know if that is simpler.
>
> I'd say yes.
>
> The remaining weird feature is the seq function and its use. The name
> suggests that it converts to a seq, which is in fact what it used to
> do. Now it converts to a seq unless the resulting seq would be empty.
> For an empty seq, it actually converts a seq to a non-seq!
>

There will always be a tension between treating the first node in a
list as a node vs as the entire list. The seq function is firmly in
the former camp, essentially returning the node containing the first
item. seq also has an important role regarding lazy seqs - when given
one it forces it and returns the inner seq. This is the big reason why
empty? is not a replacement for seq. One way to look at (seq x) is as
a version of (not (empty? x)), where the truth value is more useful
than 'true'.

I will be adding a sequence function that will act as a constructor/
coercion, so:

(seq []) -> nil
(sequence []) -> ()

In addition, sequence, when given a seq, will not force it, if it is
lazy.

It will work like this:

(defn sequence [x]
(if (seq? x)
x
(or (seq x) ())))

People that don't like nil punning need never use seq/next.

> Would it be possible to make an empty seq test as false?

No - then you could no longer distinguish between an empty collection
and nothing. Additionally, that would be a big performance hit for 'if'.

Rich

[Michael Reid]

I am fully on board with this idea. 'next' seems unfit to me exactly
as a few people have pointed out, it connotes an item rather than the
rest of a seq.

That said, I really have no grounds to be confident either way that

rest will almost exclusively appear in the context of constructing

lazy-seqs. Can any other more experienced folks can offer some
thoughts on this conjecture?

Someone suggested 'next-seq' earlier. I would also suggest rest-seq,
which is almost a literal translation of the invariant:

(rest-seq x) === (seq (rest x))

I like this, but as you point out Perry, there is a typing penalty to
be paid here; however, I think it is a small one and completely
justified.

Jim asked above why seq-on-the-next-item-if-any-else-nil is needed at
all, and clearly as Rich stated, there is a large number of uses of
rest right now proving its usefulness.

For me, this highlights the fact that in the new fully lazy seq model,
the function 'seq-on-the-next-item-if-any-else-nil" is not core to the
abstraction. As a result, having a "second class' hyphenated name such
as rest-seq (or next-seq), almost confers the "exists for convenience"
nature of this function that we are trying to name.

/mike.

[Mark Engelberg]

Since there is no canonical empty sequence, this makes me wonder
whether one particular empty sequence might have some kind of
performance benefit over another.

For example, if I were going to give a name to one empty sequence to
reuse within my code, would one of these be preferable?:
(def empty '())
(def empty (sequence []))
or some other variation?

[Rich Hickey]

There cannot be a canonical empty sequence since lazy-seqs don't know
if they are empty until forced.

This should suffice, and seems better than any parochial name like
empty (which is also names a core function BTW):

()

I recommend people not use '() in Clojure, it's ugly and unnecessary.

Rich