self-closing streams

[Graham Fawcett]

Hi folks,

I'm thinking about the following pattern:

(with-open f (open-some-stream) (doall (foo f)))

...where (doall) forces the evaluation of the entire stream before
(with-open) closes it, and the outer expression returns a strict
sequence of the stream elements (or whatever else (foo f) might
return). If foo's result is lazy, then forgetting the (doall) will
lead to errors when the result is evaluated outside of the with-open
scope.

Would it be possible to wrap a stream in some way so that the stream
would be automatically closed when its last element was consumed? This
would put the burden for closing the stream on the consumer -- it
would have to consume the whole stream eventually, or presumably a
finalize() handler would close the stream as a last resort. So one
could write:

(bar (with-open-lazy f (open-some-stream) (foo f)))

...and f would be closed when f reached its end-of-stream, regardless
of when or in what scope the last element was reached.

Safety considerations aside, is such a wrapper function/macro
possible?

Thanks,
Graham

[Kevin Downey]

it looks like with-open from boot.clj is what you are looking for.

(defmacro with-open
"Evaluates body in a try expression with name bound to the value of
init, and a finally clause that calls (. name (close))."
[name init & body]
`(let [~name ~init]
(try
~@body
(finally
(. ~name (close))))))

--
The Mafia way is that we pursue larger goals under the guise of
personal relationships.
Fisheye

[Graham Fawcett]

On Jun 25, 3:38 pm, "Kevin Downey" <redc...@gmail.com> wrote:
> On Wed, Jun 25, 2008 at 11:30 AM, Graham Fawcett
> <graham.fawc...@gmail.com> wrote:
> >
> > I'm thinking about the following pattern:
> >
> > (with-open f (open-some-stream) (doall (foo f)))
> >
> > ...where (doall) forces the evaluation of the entire stream before
> > (with-open) closes it, and the outer expression returns a strict
> > sequence of the stream elements (or whatever else (foo f) might
> > return). If foo's result is lazy, then forgetting the (doall) will
> > lead to errors when the result is evaluated outside of the with-open
> > scope.
> >
> > Would it be possible to wrap a stream in some way so that the stream
> > would be automatically closed when its last element was consumed? This
> > would put the burden for closing the stream on the consumer -- it
> > would have to consume the whole stream eventually, or presumably a
> > finalize() handler would close the stream as a last resort.
>

> it looks like with-open from boot.clj is what you are looking for.

Thanks, but it's an alternative to (with-open) that I'm looking
for. Consider this:

(last (with-open f
(java.io.BufferedReader.
(java.io.StringReader. "hello\nworld"))
(line-seq f)))

It will raise an exception, "java.io.IOException: Stream closed",
because (with-open) closes the stream before (last) gets a crack at
it. What I'm looking for is a way to tell a stream, "look, you've just
consumed the last element, so close yourself". With-open's semantics
are more like "look, we're leaving the scope of with-open, so close
the stream".

Best,
Graham

[Stuart Sierra]

On Jun 25, 4:01 pm, Graham Fawcett <graham.fawc...@gmail.com> wrote:
> Thanks, but it's an alternative to (with-open) that I'm looking
> for. Consider this:
>
> (last (with-open f
> (java.io.BufferedReader.
> (java.io.StringReader. "hello\nworld"))
> (line-seq f)))
>
> It will raise an exception, "java.io.IOException: Stream closed",
> because (with-open) closes the stream before (last) gets a crack at
> it. What I'm looking for is a way to tell a stream, "look, you've just
> consumed the last element, so close yourself". With-open's semantics
> are more like "look, we're leaving the scope of with-open, so close
> the stream".

Hi Graham,
I've run into this too, haven't found a perfect solution. Here's one
thing I've done:

(defn lazy-stream [r] ; r must be a BufferedReader
(if-let line (. r (readLine))
(lazy-cons line (lazy-stream r))
(. r (close))))

But if you don't consume the entire sequence, the stream never gets
closed.

We need something like a destructor, which neither Java nor Clojure
supports, as far as I know. Maybe some kind of special metadata on
the sequence ...?

-Stuart

[mac]

Stuart wrote:
> We need something like a destructor, which neither Java nor Clojure
> supports, as far as I know. Maybe some kind of special metadata on
> the sequence ...?

Would it not be possible to combine them, as long as closing multiple
times will not cause any errors?
Like so:

(defn lazy-stream [r] ; r must be a BufferedReader
(if-let line (. r (readLine))
(lazy-cons line (lazy-stream r))
(. r (close))))

(def my-stream (new java.io.BufferedReader (new java.io.StringReader
"Hello\nWorld!")))

(with-open s my-stream
(last (lazy-stream s)))

I had some similar problems myself when using other resources that
have to be explicitly "deleted" so I decided to create a sort of
simple cache system that supported running a supplied "destructor" for
a given data object when it was removed from the cache.
That way I was able to control the lifetime of all the objects in the
cache by way of a with macro on the whole cache at the "top" of my
scope. So that any resources not freed before the stack unwound would
at least be guaranteed to be freed at some time. Sort of like
providing finalizers but with more fine grained (albeit manual)
control.
What I was looking for was something similar to the RAII idiom in C++
but since that's not really possible without deterministic destruction
I made that cache thing instead. I'm sure my implementation is pretty
bad from a lispers point of view and I haven't tried it out much but
I'm pretty happy with it anyway :)
I could post it somewhere if anyone wants a look at it.

/mac

On Jun 25, 11:22 pm, Stuart Sierra <the.stuart.sie...@gmail.com>
wrote:

[Graham Fawcett]

On Jun 25, 5:22 pm, Stuart Sierra <the.stuart.sie...@gmail.com> wrote:
> On Jun 25, 4:01 pm, Graham Fawcett <graham.fawc...@gmail.com> wrote:
>
> > Thanks, but it's an alternative to (with-open) that I'm looking
> > for. Consider this:
>
> > (last (with-open f
> >            (java.io.BufferedReader.
> >             (java.io.StringReader. "hello\nworld"))
> >          (line-seq f)))
>
> > It will raise an exception, "java.io.IOException: Stream closed",
> > because (with-open) closes the stream before (last) gets a crack at
> > it. What I'm looking for is a way to tell a stream, "look, you've just
> > consumed the last element, so close yourself". With-open's semantics
> > are more like "look, we're leaving the scope of with-open, so close
> > the stream".
>
> Hi Graham,
> I've run into this too, haven't found a perfect solution.  Here's one
> thing I've done:
>
> (defn lazy-stream [r]  ; r must be a BufferedReader
>   (if-let line (. r (readLine))
>     (lazy-cons line (lazy-stream r))
>     (. r (close))))
>
> But if you don't consume the entire sequence, the stream never gets
> closed.

Thanks Stuart, I like that, it's a good general solution.

On the drive home yesterday, I remembered that PLT Scheme recently
added a feature called "custodians" for fine-grained resource
management:

http://download.plt-scheme.org/doc/371/html/reference/Evaluation_Model.html#(part%20mz:custodian-model)

I don't think custodians can be 'ported' to Clojure directly (creation
of stream objects can occur in Java code, and would not be
automatically aware of the custodian system). But I do like the idea
that one could define a wider scope in which resources could be
managed. Leaving the scope closes the managed resources.

I'm going to read up on the 'custodian' system and see if I can
propose a Clojure-friendly workalike. A quick sketch might be like
this:

*custodian* -- a Var holding the current set of managed resources
(manage f) -- add f to the current custodian
(cleanup) -- close all objects in the current custodian

and some syntax:

(in-custodian ...)
-- establish a new custodian (binding *custodian* to a new,
empty set), and a lexical scope outside of which the managed
resources are closed.

(with n v body)
-- sugar for (let [n v] body), this is optional, but might make
converting (with-open ...) code easier.

Then one might write:

(let [open-stream #(manage (java.io.BufferedReader.
(java.io.StringReader. %)))]
(in-custodian
...
(doseq line (take 3 (open-stream "hello\nworld"))
(print line))
...)
;; at this point, the stream(s) would be closed.
...)

(I put open-stream outside of the in-custodian lexical scope, just to
suggest the dynamic scope of the custodian.)

It deserves more thought, and some good use-cases. But the general
idea of extending the useful scope of a stream wider than (with-open),
while maintaining a "management scope" for it, is a good idea IMO.

Best,
Graham

[Graham Fawcett]

On Jun 26, 2:21 am, mac <markus.gustavs...@gmail.com> wrote:
> Stuart wrote:
> > We need something like a destructor, which neither Java nor Clojure
> > supports, as far as I know.  Maybe some kind of special metadata on
> > the sequence ...?
>
> Would it not be possible to combine them, as long as closing multiple
> times will not cause any errors?
> Like so:
> (defn lazy-stream [r]  ; r must be a BufferedReader
>   (if-let line (. r (readLine))
>     (lazy-cons line (lazy-stream r))
>     (. r (close))))
>
> (def my-stream (new java.io.BufferedReader (new java.io.StringReader
> "Hello\nWorld!")))
>
> (with-open s my-stream
>         (last (lazy-stream s)))

Here, your 'last' is within the 'with-open', so the stream is open
when you force its last element. Essentially your 'lazy-stream'
behaves like 'line-seq', and you'll still get an error if you put the
'last' outside of the 'with-open' scope:

(last (with-open s my-stream
(lazy-stream s)))

I think your caching technique is sort of like the custodian concept I
just mentioned in a different message. I hope the Clojure community is
interested in fleshing out the idea.

Best,
Graham

[Rich Hickey]

> http://download.plt-scheme.org/doc/371/html/reference/Evaluation_Mode...)

It would be easy and useful to do something like this, but it seems to
me just another flavor of with-open. Your original problem, returning
a seq on a resource needing cleanup _outside_ a known resource-
management block, remains.

There are 2 possibilities along your first line of thought, that could
work alone or together. One is auto-cleanup on end-of-seq (eos), the
other is a seq that had a close/done method.

I imagine one could write a simple seq-on-seq wrapper:

(on-eos [aseq eos-expr] ...) -> a seq

which would invoke eos-expr on end-of-seq or any exception thrown by
the wrapped seq, and use it like:

(on-eos (take 100 (line-seq f)) (.close f))

For (optional) manual management, the wrapper seq could have a close
function of its own.

This is a general approach not tied to files/streams.

Rich

[Graham Fawcett]

On Jun 26, 11:13 am, Rich Hickey <richhic...@gmail.com> wrote:
> On Jun 26, 9:58 am, Graham Fawcett <graham.fawc...@gmail.com> wrote:
> > I don't think custodians can be 'ported' to Clojure directly (creation
> > of stream objects can occur in Java code, and would not be
> > automatically aware of the custodian system). But I do like the idea
> > that one could define a wider scope in which resources could be
> > managed. Leaving the scope closes the managed resources.

[snip]

> > It deserves more thought, and some good use-cases. But the general
> > idea of extending the useful scope of a stream wider than (with-open),
> > while maintaining a "management scope" for it, is a good idea IMO.
>
> It would be easy and useful to do something like this, but it seems to
> me just another flavor of with-open. Your original problem, returning
> a seq on a resource needing cleanup _outside_ a known resource-
> management block, remains.
>
> There are 2 possibilities along your first line of thought, that could
> work alone or together. One is auto-cleanup on end-of-seq (eos), the
> other is a seq that had a close/done method.

I particularly like the close/done method.

> I imagine one could write a simple seq-on-seq wrapper:
>
> (on-eos [aseq eos-expr] ...) -> a seq
>
> which would invoke eos-expr on end-of-seq or any exception thrown by
> the wrapped seq, and use it like:
>
> (on-eos (take 100 (line-seq f)) (.close f))

Yes, though that's awfully close to (with-open) and could be expressed
with it. I think this is more what I would hope for:

(take 100 (on-eos (line-seq f) (.close f)))

...which could register (.close f) as the eos and done/close callbacks
for the seq. In this example, if the seq had more than 100 items,
manual closing would be required, but a reference to the seq would be
enough to make that happen. The compelling difference from (with-open)
is that the closing behaviour would be part of the seq's private
environment, so the seq could be passed around and closed in whatever
scope made best sense.

> For (optional) manual management, the wrapper seq could have a close
> function of its own.

Yes, and if it's implemented as a method, then one could write:

(with-open s some-sequence (foo s))

...so (with-open) could manage sequences as well! Handy.

> This is a general approach not tied to files/streams.

Yes, I really like that. Streams and seqs have similar semantics, and
the parallel is worth exploiting. (For what it's worth, Gambit Scheme
uses 'custom ports' to implement lazy sequences (e.g. Gambit's
directory-ports), so the semantics there are identical.)

After some reading, I might still play with the custodian idea, but I
don't think that requires any core changes, and could be implemented
as a library.

Best,
Graham

> > Rich

[Rich Hickey]

No, the idea is that you are then passing the return value of on-eos
to someone else.

> (take 100 (on-eos (line-seq f) (.close f)))
>
> ...which could register (.close f) as the eos and done/close callbacks
> for the seq. In this example, if the seq had more than 100 items,
> manual closing would be required, but a reference to the seq would be
> enough to make that happen. The compelling difference from (with-open)
> is that the closing behaviour would be part of the seq's private
> environment, so the seq could be passed around and closed in whatever
> scope made best sense.

We're saying the same thing. IMO, if you know when you are calling on-
eos that you want to return only 100, wrap the take with the on-eos
call.

>
> > For (optional) manual management, the wrapper seq could have a close
> > function of its own.
>
> Yes, and if it's implemented as a method, then one could write:
>
> (with-open s some-sequence (foo s))
>
> ...so (with-open) could manage sequences as well! Handy.

Yes, definitely.

Rich

[Graham Fawcett]

Oh, I see. Yes, we're on the same page.

> > (take 100 (on-eos (line-seq f) (.close f)))
>
> > ...which could register (.close f) as the eos and done/close callbacks
> > for the seq. In this example, if the seq had more than 100 items,
> > manual closing would be required, but a reference to the seq would be
> > enough to make that happen. The compelling difference from (with-open)
> > is that the closing behaviour would be part of the seq's private
> > environment, so the seq could be passed around and closed in whatever
> > scope made best sense.
>
> We're saying the same thing. IMO, if you know when you are calling on-
> eos that you want to return only 100, wrap the take with the on-eos
> call.
>
> > > For (optional) manual management, the wrapper seq could have a close
> > > function of its own.
>
> > Yes, and if it's implemented as a method, then one could write:
>
> > (with-open s some-sequence (foo s))
>
> > ...so (with-open) could manage sequences as well! Handy.
>
> Yes, definitely.

This gets my +1. Thanks for your response.

Graham

>
> Rich