I'm pretty much finished with the fully-lazy implementation and am
happy so far with the results. I think this will be an important
addition to Clojure and am planning to add it.
Now comes the hard part - names and change. The essence of the fully
lazy seqs is that they will not consume any resources, perform any
computation or trigger any side effects until they are consumed. This
is a change to the way the sequence functions work now, in that, in
order to determine whether they should return a seq or nil, they need
to touch at least one item. So, there will be an additional function
on seqs, one that returns the items other than the first as a logical,
non-nil, possibly empty collection. Calling seq on this collection
will give you what rest currently gives you - the next seq object or
nil if none. So the core operations on seqs will be:
;item
(first x)
;collection of remaining items, possibly empty
(possibly-empty-collection-of-the-remaining-items x)
;seq on next item, or nil if none
(seq-on-the-next-item-if-any-else-nil x)
(first x) is uncontroversial and won't change. The second is a new
function. The third is currently called 'rest'.
I have some ideas for names, and there are definitely tradeoffs
between short-term pain and long-term goodness in some of the options.
The first option is to leave rest alone, and give the new function a
new name, like more.
;item
(first x)
;collection of remaining items, possibly empty
(more x)
;seq on next item, or nil if none
(rest x)
Note that (rest x) === (seq (more x))
This is implemented in the lazy branch, SVN rev 1281. It has the
attribute of requiring the fewest changes to existing code, and the
drawback of leaving us with less-than-ideal names, especially insofar
as more (or whatever you choose to call it) will in some way seem
synonymous with rest. This naming scheme, and the changes it implies,
is documented here:
The second option is to choose the best possible names, and deal with
some short term pain in porting and confusion. I think the best names
are:
;item
(first x)
;collection of remaining items, possibly empty
(rest x)
;seq on next item, or nil if none
(next x)
This is implemented in the lazy branch, SVN rev 1282. Note that this
changes the meaning of rest, and gives the current rest operation a
new name, next. It has the attributes of using the most appropriate
names (IMO) and the drawback of changing the semantics of a frequently
used function name, but still offering that functionality under a
different name. It would also break the compatibility of rest with
Common Lisp's. As with the previous model, the third function can be
defined in terms of the second - (next x) === (seq (rest x)). This
naming scheme, and the changes it implies, is documented here:
A third option would be to retire rest and use only new names:
;item
(first x)
;collection of remaining items, possibly empty
(more x)
;seq on next item, or nil if none
(next x)
I haven't implemented this.
I prefer first/rest/next. I think rest is the best complement to
first, and it should mean the logical collection once things are fully
lazy. I think next implies the next seq, as well as the eager nature
of the operation.
I am looking for feedback from people willing to read and understand
the linked-to documentation and the fully lazy model, and especially
from those trying the lazy branch code and porting some of your own.
Questions on the model welcome as well. Chouser has also blogged a bit
about this, with some useful descriptions of nil punning:
I've been working on this for a few months, in lieu of more
interesting things, because I knew it would be a breaking change and
we're trying to get the biggest of those behind us. I appreciate any
effort you spend in trying to provide informed input.
I would vote that you change to the optimum names, but work with Stu
Halloway to ensure that either his book gets updated before it is
printed, or there is a "cheat sheet" on the Clojure site to translate
from his book to any new names.
On Sun, Feb 15, 2009 at 9:18 AM, Rich Hickey <richhic...@gmail.com> wrote:
> I'm pretty much finished with the fully-lazy implementation and am
> happy so far with the results. I think this will be an important
> addition to Clojure and am planning to add it.
> Now comes the hard part - names and change. The essence of the fully
> lazy seqs is that they will not consume any resources, perform any
> computation or trigger any side effects until they are consumed. This
> is a change to the way the sequence functions work now, in that, in
> order to determine whether they should return a seq or nil, they need
> to touch at least one item. So, there will be an additional function
> on seqs, one that returns the items other than the first as a logical,
> non-nil, possibly empty collection. Calling seq on this collection
> will give you what rest currently gives you - the next seq object or
> nil if none. So the core operations on seqs will be:
> ;item
> (first x)
> ;collection of remaining items, possibly empty
> (possibly-empty-collection-of-the-remaining-items x)
> ;seq on next item, or nil if none
> (seq-on-the-next-item-if-any-else-nil x)
> (first x) is uncontroversial and won't change. The second is a new
> function. The third is currently called 'rest'.
> I have some ideas for names, and there are definitely tradeoffs
> between short-term pain and long-term goodness in some of the options.
> The first option is to leave rest alone, and give the new function a
> new name, like more.
> ;item
> (first x)
> ;collection of remaining items, possibly empty
> (more x)
> ;seq on next item, or nil if none
> (rest x)
> Note that (rest x) === (seq (more x))
> This is implemented in the lazy branch, SVN rev 1281. It has the
> attribute of requiring the fewest changes to existing code, and the
> drawback of leaving us with less-than-ideal names, especially insofar
> as more (or whatever you choose to call it) will in some way seem
> synonymous with rest. This naming scheme, and the changes it implies,
> is documented here:
> The second option is to choose the best possible names, and deal with
> some short term pain in porting and confusion. I think the best names
> are:
> ;item
> (first x)
> ;collection of remaining items, possibly empty
> (rest x)
> ;seq on next item, or nil if none
> (next x)
> This is implemented in the lazy branch, SVN rev 1282. Note that this
> changes the meaning of rest, and gives the current rest operation a
> new name, next. It has the attributes of using the most appropriate
> names (IMO) and the drawback of changing the semantics of a frequently
> used function name, but still offering that functionality under a
> different name. It would also break the compatibility of rest with
> Common Lisp's. As with the previous model, the third function can be
> defined in terms of the second - (next x) === (seq (rest x)). This
> naming scheme, and the changes it implies, is documented here:
> A third option would be to retire rest and use only new names:
> ;item
> (first x)
> ;collection of remaining items, possibly empty
> (more x)
> ;seq on next item, or nil if none
> (next x)
> I haven't implemented this.
> I prefer first/rest/next. I think rest is the best complement to
> first, and it should mean the logical collection once things are fully
> lazy. I think next implies the next seq, as well as the eager nature
> of the operation.
> I am looking for feedback from people willing to read and understand
> the linked-to documentation and the fully lazy model, and especially
> from those trying the lazy branch code and porting some of your own.
> Questions on the model welcome as well. Chouser has also blogged a bit
> about this, with some useful descriptions of nil punning:
> I've been working on this for a few months, in lieu of more
> interesting things, because I knew it would be a breaking change and
> we're trying to get the biggest of those behind us. I appreciate any
> effort you spend in trying to provide informed input.
I'm also in support of the optimal names. Clojure is not too widely
used in production code yet, and it would be a shame to start
compromising design decisions for backwards compatibility already.
This is actually one of my (and many other people's) favorite parts
about Clojure, the beauty of Lisp without the baggage. I wouldn't like
for Clojure to start carrying baggage of its own.
One thing I did find confusing though was in regards to the doc.
Is there a way to more clearly differentiate between a seq and a
sequence? Up until now, I've always thought of "seq" as just being
shorthand for "sequence" which isn't the case apparently.
I also favor the optimal names. We're the pioneers in using clojure,
so we should expect a few arrows. Hopefully, the number of clojure
users in the future will be an order of magnitude greater than where
we are now. For us to take short term hit, we can save a large number
of people a lot of cycles later on. I also prefer it on aesthetic
grounds. Breaking compatibility doesn't bother me much.
I'll play around with the lazy branch this week, and this is just a
name suggestion: what do you think of first/tail/rest where (rest s)
== (seq (tail s))? tail is already used in other functional languages
such as Haskell and OCaml to represent all-but-the-first elements, so
it wouldn't be completely foreign.
Vincent.
On Feb 15, 12:18 pm, Rich Hickey <richhic...@gmail.com> wrote:
> I'm pretty much finished with the fully-lazy implementation and am
> happy so far with the results. I think this will be an important
> addition to Clojure and am planning to add it.
> Now comes the hard part - names and change. The essence of the fully
> lazy seqs is that they will not consume any resources, perform any
> computation or trigger any side effects until they are consumed. This
> is a change to the way the sequence functions work now, in that, in
> order to determine whether they should return a seq or nil, they need
> to touch at least one item. So, there will be an additional function
> on seqs, one that returns the items other than the first as a logical,
> non-nil, possibly empty collection. Calling seq on this collection
> will give you what rest currently gives you - the next seq object or
> nil if none. So the core operations on seqs will be:
> ;item
> (first x)
> ;collection of remaining items, possibly empty
> (possibly-empty-collection-of-the-remaining-items x)
> ;seq on next item, or nil if none
> (seq-on-the-next-item-if-any-else-nil x)
> (first x) is uncontroversial and won't change. The second is a new
> function. The third is currently called 'rest'.
> I have some ideas for names, and there are definitely tradeoffs
> between short-term pain and long-term goodness in some of the options.
> The first option is to leave rest alone, and give the new function a
> new name, like more.
> ;item
> (first x)
> ;collection of remaining items, possibly empty
> (more x)
> ;seq on next item, or nil if none
> (rest x)
> Note that (rest x) === (seq (more x))
> This is implemented in the lazy branch, SVN rev 1281. It has the
> attribute of requiring the fewest changes to existing code, and the
> drawback of leaving us with less-than-ideal names, especially insofar
> as more (or whatever you choose to call it) will in some way seem
> synonymous with rest. This naming scheme, and the changes it implies,
> is documented here:
> The second option is to choose the best possible names, and deal with
> some short term pain in porting and confusion. I think the best names
> are:
> ;item
> (first x)
> ;collection of remaining items, possibly empty
> (rest x)
> ;seq on next item, or nil if none
> (next x)
> This is implemented in the lazy branch, SVN rev 1282. Note that this
> changes the meaning of rest, and gives the current rest operation a
> new name, next. It has the attributes of using the most appropriate
> names (IMO) and the drawback of changing the semantics of a frequently
> used function name, but still offering that functionality under a
> different name. It would also break the compatibility of rest with
> Common Lisp's. As with the previous model, the third function can be
> defined in terms of the second - (next x) === (seq (rest x)). This
> naming scheme, and the changes it implies, is documented here:
> A third option would be to retire rest and use only new names:
> ;item
> (first x)
> ;collection of remaining items, possibly empty
> (more x)
> ;seq on next item, or nil if none
> (next x)
> I haven't implemented this.
> I prefer first/rest/next. I think rest is the best complement to
> first, and it should mean the logical collection once things are fully
> lazy. I think next implies the next seq, as well as the eager nature
> of the operation.
> I am looking for feedback from people willing to read and understand
> the linked-to documentation and the fully lazy model, and especially
> from those trying the lazy branch code and porting some of your own.
> Questions on the model welcome as well. Chouser has also blogged a bit
> about this, with some useful descriptions of nil punning:
> I've been working on this for a few months, in lieu of more
> interesting things, because I knew it would be a breaking change and
> we're trying to get the biggest of those behind us. I appreciate any
> effort you spend in trying to provide informed input.
I agree with the op. While the language is still relatively young
please break things so they sit better in the long term. Accurate and
descriptive names are totally valueable, and I'm pretty handy with
find/replace on the editor anyway :p
Aaron
On Feb 15, 2009, at 10:43 AM, CuppoJava <patrickli_2...@hotmail.com>
wrote:
> I'm also in support of the optimal names. Clojure is not too widely
> used in production code yet, and it would be a shame to start
> compromising design decisions for backwards compatibility already.
> This is actually one of my (and many other people's) favorite parts
> about Clojure, the beauty of Lisp without the baggage. I wouldn't like
> for Clojure to start carrying baggage of its own.
> I'll play around with the lazy branch this week, and this is just a > name suggestion: what do you think of first/tail/rest where (rest s) > == (seq (tail s))? tail is already used in other functional languages > such as Haskell and OCaml to represent all-but-the-first elements, so > it wouldn't be completely foreign.
I like that choice as well.
Otherwise, I'll join the crowd who thinks that it's better to get everything in order now without making compromises for backwards compatibility.
On Feb 15, 2:48 pm, Vincent Foley <vfo...@gmail.com> wrote:
> Hello Rich,
> I'll play around with the lazy branch this week, and this is just a
> name suggestion: what do you think of first/tail/rest where (rest s)
> == (seq (tail s))? tail is already used in other functional languages
> such as Haskell and OCaml to represent all-but-the-first elements, so
> it wouldn't be completely foreign.
That falls into the synonym category - tail/rest mean roughly the same
thing. tail complements head, which we aren't using. Coming from
outside, I'd have no idea which did what or why.
"It would also break the compatibility of rest with Common Lisp's"
This is of mild concern to me, but I think if there was a prominent
warning on clojure.org, I could get over it.
> I am looking for feedback from people willing to read and understand > the linked-to documentation and the fully lazy model, and especially > from those trying the lazy branch code and porting some of your own.
I'm trying svn rev 1282 with the following test (which depends on javadb, (derby)):
I tried using Chouser's "-Dclojure.assert-if-lazy-seq=please" facility. While I was able to trigger an exception from it using sample code, it wasn't triggered during the hang.
I'd like to figure this out.
- Has anyone gotten past this already? - Does anyone see the problem by inspecting the lib code? - This seems like an opportunity for me to use a Java debugger with Clojure for the first time. Has anyone written about using JSwat or another debugger with Clojure?
I would appreciate hearing any tips for getting to the cause of this.
On Feb 15, 2009, at 4:30 PM, Stephen C. Gilardi wrote:
> - This seems like an opportunity for me to use a Java debugger with > Clojure for the first time. Has anyone written about using JSwat or > another debugger with Clojure?
On Feb 15, 4:30 pm, "Stephen C. Gilardi" <squee...@mac.com> wrote:
> On Feb 15, 2009, at 12:18 PM, Rich Hickey wrote:
> > I am looking for feedback from people willing to read and understand
> > the linked-to documentation and the fully lazy model, and especially
> > from those trying the lazy branch code and porting some of your own.
> I'm trying svn rev 1282 with the following test (which depends on
> javadb, (derby)):
Here's an example of what I think will be the worst kind of breakage resulting from changing the meaning of rest from seq-on-the-next-item-if-any-else-nil to possibly-empty-collection-of-the-remaining-items:
This is a bit like the builtin interpose, except it takes multiple args instead of a collection, and it returns a vector with the interposed value surrounding all the others:
(my-interpose 'x 'a 'b 'c) -> [x a x b x c x]
At least that's what it does in svn 1282 trunk. In 1282 lazy branch, it's an infinite loop. Can you spot the problem?
When discussing this yesterday in IRC, I was pretty firmly against Rich's preferred names, for exactly this reason. And worse than trying to fix my own code would be the potential confusion over which versions of examples, libs, etc. work with which versions of Clojure.
...but my position has softened, as I tried to construct an example for this post that actually broke in a bad way. My first several attempts produced code that worked in both versions.
For example, my-interpose above takes multiple args so that I could safely assume that 'coll' is a seq. My first (unposted) version took a collection as a second argument, but in that case a simple "(if coll" is probably already an error, in case a user passed in an empty vector, or some other collection. The solution would be to test the seq of the coll:
(defn my-interpose [x coll] (loop [v [x] coll coll] (if (seq coll) ; Don't assume coll is a seq-or-nil (recur (-> v (conj (first coll)) (conj x)) (rest coll)) v)))
That also happens to solve the lazy-branch infinite-loop problem -- what's more correct in trunk is more correct in lazy, in this case. So I kept refining my-interpose, trying to get a version that was correct in trunk but caused a non-exception error in lazy. After several iterations I finally got the one at the top of this post.
...but even that one can be caught easily by turning on clojure.assert-if-lazy-seq, in which case you get an exception pointing directly to the line that needs to be changed:
java.lang.Exception: LazySeq used in 'if'
So the same changes that will already have to be made to nil puns for the other seq functions would now have to be made for uses of the new 'rest' function.
Sorry if this has been a bit long-winded, but I wanted to explain why I've changed my mind a bit -- changing the meaning of 'rest' may not be as bad as I had been thinking.
> I've been working on this for a few months, in lieu of more > interesting things, because I knew it would be a breaking change and > we're trying to get the biggest of those behind us. I appreciate any > effort you spend in trying to provide informed input.
For those who want to play with this without keeping two versions of their source code files, I have added a new macro lazy-and-standard- branch to clojure.contrib.macros. Here is an example of how to use it:
(lazy-and-standard-branch
(defn value-seq [f seed] (lazy-seq (let [[value next] (f seed)] (cons value (value-seq f next)))))
(defn value-seq [f seed] (let [[value next] (f seed)] (lazy-cons value (value-seq f next))))
On Sun, Feb 15, 2009 at 1:44 PM, Chouser <chou...@gmail.com> wrote: > (defn my-interpose [x coll] > (loop [v [x] coll coll] > (if (seq coll) ; Don't assume coll is a seq-or-nil > (recur (-> v (conj (first coll)) (conj x)) (rest coll)) > v)))
You know, there is an empty? predicate. Why not write it as: (defn my-interpose [x coll] (loop [v [x] coll coll] (if (empty? coll) v ; Don't assume coll is a seq-or-nil (recur (-> v (conj (first coll)) (conj x)) (rest coll)))))
I know that your first version is viewed as more idiomatic in Clojure, but I've never understood why Rich and others prefer that style. It assumes that converting something to a seq is guaranteed to be a computationally cheap operation, and I see no reason to assume that will always be the case. I can certainly imagine seq-able collections that take some time seq-ify, so converting to a seq to test for empty, and then just throwing it away causing it to be recomputed in rest doesn't seem as future-proof as just using empty?.
Based on it working for you, the current theory I'm working to verify is that this was caused by a clojure-contrib.jar compiled with trunk interacting with a clojure.jar from lazy 1282.
Should we branch contrib and do the fixups on a lazy branch? Chouser, have you already fixed it enough to compile with clojure contrib's build.xml?
> For those who want to play with this without keeping two versions of > their source code files, I have added a new macro lazy-and-standard- > branch to clojure.contrib.macros. Here is an example of how to use it:
BTW, my library modules in clojure.contrib (accumulators, monads, probabilities) now work with the lazy branch as well as with the standard one. The changes were minor and quick to do. The nil-punning compiler flag was quite helpful.
On Feb 15, 2009, at 5:03 PM, Stephen C. Gilardi wrote:
> Based on it working for you, the current theory I'm working to > verify is that this was caused by a clojure-contrib.jar compiled > with trunk interacting with a clojure.jar from lazy 1282.
I've confirmed this. Thanks for the help. The test I wrote about is now working for me with lazy 1282.
> Here's an example of what I think will be the worst kind of breakage > resulting from changing the meaning of rest from > seq-on-the-next-item-if-any-else-nil to > possibly-empty-collection-of-the-remaining-items:
> This is a bit like the builtin interpose, except it takes multiple > args instead of a collection, and it returns a vector with the > interposed value surrounding all the others:
> (my-interpose 'x 'a 'b 'c) > -> [x a x b x c x]
> At least that's what it does in svn 1282 trunk. In 1282 lazy branch, > it's an infinite loop. Can you spot the problem?
> When discussing this yesterday in IRC, I was pretty firmly against > Rich's preferred names, for exactly this reason. And worse than > trying to fix my own code would be the potential confusion over which > versions of examples, libs, etc. work with which versions of Clojure.
While not knowing if sample code has been ported will still be an issue, anyone following the porting recipe:
> ...but my position has softened, as I tried to construct an example > for this post that actually broke in a bad way. My first several > attempts produced code that worked in both versions.
> For example, my-interpose above takes multiple args so that I could > safely assume that 'coll' is a seq. My first (unposted) version took > a collection as a second argument, but in that case a simple > "(if coll" is probably already an error, in case a user passed in an > empty vector, or some other collection. The solution would be to test > the seq of the coll:
> That also happens to solve the lazy-branch infinite-loop problem -- > what's more correct in trunk is more correct in lazy, in this case. > So I kept refining my-interpose, trying to get a version that was > correct in trunk but caused a non-exception error in lazy. After > several iterations I finally got the one at the top of this post.
> ...but even that one can be caught easily by turning on > clojure.assert-if-lazy-seq, in which case you get an exception > pointing directly to the line that needs to be changed:
> java.lang.Exception: LazySeq used in 'if'
> So the same changes that will already have to be made to nil puns for > the other seq functions would now have to be made for uses of the new > 'rest' function.
I would just clarify that to say that the best route is *not* to structurally change code that uses rest, just have it call next instead (unless you are writing a lazy-seq body). Using next is going to let you preserve your code structure and yields the simplest idioms - since next (still) nil puns!
core.clj e.g. is full of code that presumes it is walking a seq chain, and so contains lots of next calls:
There's nothing wrong with that idiom. I do not recommend that people leave their rest calls and 'fix' the nil puns - instead, change your rest calls to next, then deal with your own lazy-cons calls (possibly restoring some rest calls in lazy-seq bodies), then try the clojure.assert-if-lazy-seq flag to find any conditional use of lazy sequences.
> On Sun, Feb 15, 2009 at 1:44 PM, Chouser <chou...@gmail.com> wrote: >> (defn my-interpose [x coll] >> (loop [v [x] coll coll] >> (if (seq coll) ; Don't assume coll is a seq-or-nil >> (recur (-> v (conj (first coll)) (conj x)) (rest coll)) >> v)))
> You know, there is an empty? predicate. Why not write it as: > (defn my-interpose [x coll] > (loop [v [x] coll coll] > (if (empty? coll) v ; Don't assume coll is a seq-or-nil > (recur (-> v (conj (first coll)) (conj x)) (rest coll)))))
> I know that your first version is viewed as more idiomatic in Clojure, > but I've never understood why Rich and others prefer that style. It > assumes that converting something to a seq is guaranteed to be a > computationally cheap operation, and I see no reason to assume that > will always be the case. I can certainly imagine seq-able > collections that take some time seq-ify, so converting to a seq to > test for empty, and then just throwing it away causing it to be > recomputed in rest doesn't seem as future-proof as just using empty?.
When walking a chain of seqs, empty? made no sense as there is no such thing as an empty seq. Now that rest returns a collection, this makes more sense (although still not my preference), but to each his own. Let's please not get bogged down in a style discussion now. You should be quite happy for this:
(rest [1]) -> () ;an empty sequence, note - not a canonic/sentinel value!
On Sun, Feb 15, 2009 at 5:03 PM, Stephen C. Gilardi <squee...@mac.com> wrote:
> Should we branch contrib and do the fixups on a lazy branch? Chouser, have > you already fixed it enough to compile with clojure contrib's build.xml?
I don't ever compile clojure-contrib, I just put its src dir in my classpath. I've fixed a couple functions here and there using a macro something like Konrads, but I think that's going to make the code cluttered pretty quickly. A branch of clojure-contrig is probably quite sensible at this point.
>> For those who want to play with this without keeping two versions of >> their source code files, I have added a new macro lazy-and-standard- >> branch to clojure.contrib.macros. Here is an example of how to use >> it:
> BTW, my library modules in clojure.contrib (accumulators, monads, > probabilities) now work with the lazy branch as well as with the > standard one. The changes were minor and quick to do. The nil-punning > compiler flag was quite helpful.
