On laziness and with-open

[Mike]

I wanted to grab bytes out of a stream, and didn't see an analogue to
reader from duck-streams, so I made my own:

(defn byte-seq
"Returns the bytes from stream as a lazy sequence of ints.
stream must implement java.io.InputStream."
[#^java.io.InputStream stream]
(lazy-seq
(let [b (. stream (read))]
(if (>= b 0)
(cons b (byte-seq stream))))))

Then I did a simple lazy operation on a stream of bytes, say, to drop
the first 5:

(with-open [st (FileInputStream. "mike.clj")] (drop 5 (byte-seq st)))

I was a little surprised at first getting java.io.IOException: Bad
file descriptor, but of course it hit me: the laziness persists well
beyond the .close() in the with-open.

I modified my byte-seq to close the stream when EOF is reached, but
this is an awful ugly solution having the inner thing know when to
close the outer concern's thing. (What if the outer thing wants to
rewind? etc.)

Is there a pattern out there in Clojure for handling laziness at the
same time as handling resource lifecycle (with-open, etc.)?

Thanks again in advance,
Mike

[B Smith-Mannschott]

I'd just like to pile-on to say that I ran into the same question last
night in a slightly different form.

I have a file containing a sequence of clojure forms, which I want to
read in sequentially. I don't have to have them all in memory at the
same time for my task. A lazy sequence which yielded parsed forms
while consuming bytes from the underlying file would be a perfect
solution, but I ran into the same issue.

I close the file when EOF is reached. I can (but don't yet, because it
was a quick hack late at night) catch IOExceptions and close the
underlying file under those circumstances, but this is ugly.

What happens if the caller grabs such a lazy sequence and then just
drops it on the floor? Does the file ever get closed before the VM
exits? Do InputStreams register a finalizer?

Needed: a pattern for handing a lazy sequence backed by a closeable resource.

// ben

[Janico Greifenberg]

Hi Mike,

I think you have two ways to deal with this: (1) Include all your
processing inside the with-open body, or (2) realize the seq by
calling doall.
With (1) you can take advantage of the laziness by using only the
parts of the seq you really need, but you have to have all your
processing in there which may seem a bit messy when there is a lot of
processing code.
With (2) you can return the complete seq for processing in other
places, so you can encapsulate the reading in a function, but you have
all the data in memory.

You could also combine these approaches, by doing some preprocessing
(like dropping 5 elements) and then returning the realized result seq.
(with-open [st (FileInputStream. "msghub.clj")] (doall (drop 5 (byte-seq st))))

I hope this helps
Janico

[eyeris]

I ran the code you pasted here. It didn't throw an IOException for me.
I am running 1.0.

[B Smith-Mannschott]

On Thu, Jul 9, 2009 at 16:43, eyeris<drewp...@gmail.com> wrote:
>
> I ran the code you pasted here. It didn't throw an IOException for me.
> I am running 1.0.

I suspect that You're Doing it Wrong.

You'll see the exception only if you actually try to evaluate the lazy
sequence returned by byte-seq.

(import [java.io InputStream FileInputStream])

(defn byte-seq

[#^java.io.InputStream stream]
(lazy-seq
(let [b (. stream (read))]
(if (>= b 0)
(cons b (byte-seq stream))))))

(defn most []
(with-open [st (FileInputStream. "/home/smithma/.bashrc")]
(drop 5 (byte-seq st))))

(def x (most)) ;; this is OK, it just binds x to the lazy seq
returned by (most)

(first x) ;; not ok, as it tries to get the first element of x, which dies
;; since the file we're reading from is already closed.

[Stuart Sierra]

On Jul 9, 6:10 am, Mike <cki...@gmail.com> wrote:
> Is there a pattern out there in Clojure for handling laziness at the
> same time as handling resource lifecycle (with-open, etc.)?

Not yet, but it is planned, in something called "scopes." http://clojure.org/todo

-SS

[Sean Devlin]

That pages says the scopes system is already designed. To you have
any preliminary design docs posted somewhere?