How to use "realized?" on a lazy sequence?

[George Kangas]

Here's a REPL session, wherein I try to use "realized?" on a
lazy-seq.

Clojure 1.3.0

Define the lazy-seq:

user=> (def naturals (iterate inc 0))
#'user/naturals

Force realization of the first 1 + 123456 elements:

user=> (time (nth naturals 123456))
"Elapsed time: 481.349 msecs"
123456

Due to previous realization, the same expression now eval's
quickly:

user=> (time (nth naturals 123456))
"Elapsed time: 15.571 msecs"
123456

Now I try to use "realized?" on 123456th element:

user=> (realized? (nth naturals 123456))
ClassCastException java.lang.Long cannot be cast to
clojure.lang.IPending clojure.core/realized? (core.clj:6505)

Ouch! I guess "realized?" isn't a macro. Next try:

user=> (realized? (drop 123456 naturals))
false

Hmmm... could I be off by one? Let's leave lots of room for error:

user=> (realized? (drop 12345 naturals))
false
user=> (realized? (drop 0 naturals))
false

Huh? How do I get "realized?" to tell me what I want to know?

What I want to know, is wether the element at the nth index
has been computed and cached. Maybe something like "realized-
length", just for lazy-seq's, which would report how far along
the sequence has realization occurred.

Thanks,
George Kangas

[Tassilo Horn]

George Kangas <gwka...@gmail.com> writes:

> Now I try to use "realized?" on 123456th element:
>
> user=> (realized? (nth naturals 123456))
> ClassCastException java.lang.Long cannot be cast to
> clojure.lang.IPending clojure.core/realized? (core.clj:6505)

Hm, that's strange indeed. I also get

user> (realized? (iterate int 0))
;clojure.lang.Cons cannot be cast to clojure.lang.IPending
; Evaluation aborted.

where I'd expect it to return false. However, it works just fine with
`range' in place of iterate:

user> (realized? (range 0 10))
false
user> (realized? (doall (range 0 10)))
true

Bye,
Tassilo

> Ouch! I guess "realized?" isn't a macro. Next try:

No, it's a function. But why should that be important. Ok, before a
funcall all args are evaluated, but the result of evaluating a lazy seq
is still the same lazy seq. Evaluation doesn't realize anything, only
calling first/rest on the seq does.

> Huh? How do I get "realized?" to tell me what I want to know?
>
> What I want to know, is wether the element at the nth index
> has been computed and cached.

Ditto.

Bye,
Tassilo

[Alan Malloy]

Can't you write that function yourself?

(defn realized-length [xs]
(loop [n 0 xs xs]
(if (realized? xs)
(recur (inc n) (rest xs))
n)))

drop returns a new lazy sequence, with no realized elements, so
naturally you can't ask if the sequence "under" it is realized. If you
want to work at such a low level you can't build a new lazy sequence;
the above avoids that problem by working with the sequence primitives.

user=> (def naturals (rest (iterate inc 0)))
#'user/naturals
user=> (realized-length naturals)
0
user=> (take 10 naturals)
(1 2 3 4 5 6 7 8 9 10)
user=> (realized-length naturals)
10

Note that this does not work for the "base" case of an iterated
sequence, because that is not a lazy-seq but a cons. Seems a bit weird
to me, but then realized? itself is a bit weird...

[George Kangas]

On Oct 7, 5:02 pm, Alan Malloy <a...@malloys.org> wrote:
> Can't you write that function yourself?
>
> (defn realized-length [xs]
>   (loop [n 0 xs xs]
>     (if (realized? xs)
>       (recur (inc n) (rest xs))
>       n)))

Thanks, Alan!

>
> drop returns a new lazy sequence, with no realized elements,

I didn't "realize" that! I expected that "drop" would call "rest"
recursively, like your "realized-length" does. But apparently, it's
lazier than that:

user=> (def c (drop 12345678 naturals))
#'user/c
user=> (time (first c))
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
(more java.lang.barfing....)

> user=> (def naturals (rest (iterate inc 0)))
> #'user/naturals

You introduce another subtlety here: "realized?" won't work on the
result of "iterate" (as Tassilo Horn found), but it will work on
"(rest (iterate ...))". And this is because:

> Note that this does not work for the "base" case of an iterated
> sequence, because that is not a lazy-seq but a cons. Seems a bit weird
> to me, but then realized? itself is a bit weird...

"realized?" doesn't seem so weird, when it deals with promises,
delays and futures. Those objects have explicit API's for their
creation and realization, and "realized?" has a natural and
predictable role in that API.

Lazy sequences are more magic, and less explicit about creation and
realization (or so it seems to me). Implementors of functions such as
"iterate", "drop", "rest", "range", etc., have some freedom to decide
times of creation/realization. Then, when you try to use "realized?"
on lazy sequences, those decisions (which you may not have expected)
come to the surface.

Maybe "realized?" doesn't really belong in the lazy sequence API, but
it is fun to play around with it. Thanks for showing me how!

regards,

George