[Haskell-cafe] Are handles garbage-collected?

[Peter Simons]

What happens when a System.IO.Handle falls out of scope
without being explicitly hClosed? Is that a resource leak?
Or will the RTS close the handle for me?

Peter

_______________________________________________
Haskell-Cafe mailing list
Haskel...@haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe

[Tomasz Zielonka]

On Sun, Oct 24, 2004 at 01:39:06AM +0200, Peter Simons wrote:
> What happens when a System.IO.Handle falls out of scope
> without being explicitly hClosed? Is that a resource leak?
> Or will the RTS close the handle for me?

AFAIK, Handles have finalisers which close them, but I don't know if GHC
triggers garbage collection when file descriptors run out. If not, you
will have problems if you manage to run out of fds between GCs.

How about using bracket to introduce explicit close on end of scope?

Best regards,
Tom

--
.signature: Too many levels of symbolic links

[Sven Panne]

Tomasz Zielonka wrote:
> AFAIK, Handles have finalisers which close them, but I don't know if GHC
> triggers garbage collection when file descriptors run out. If not, you
> will have problems if you manage to run out of fds between GCs.

Yes, in GHC and Hugs handles are flushed and closed automatically during GC
if they are garbage and not already closed. But no GC is triggered when one
runs out of native file descriptors. I'm not exactly sure about NHC, but I
guess it's the same there.

> How about using bracket to introduce explicit close on end of scope?

I would recommend this, too, e.g. via something like:

withFile :: FilePath -> IOMode -> (Handle -> IO a) -> IO a
withFile fileName mode = bracket (openFile fileName mode) hClose

This ensures that the handle is closed immediately even in the presence of
exceptions.

Cheers,
S.

[Peter Simons]

Tomasz Zielonka writes:

> AFAIK, Handles have finalisers which close them, but I
> don't know if GHC triggers garbage collection when file
> descriptors run out. If not, you will have problems if
> you manage to run out of fds between GCs.

Thank you for answering.

Now there is only one problem: Assuming I could _not_ use
'bracket', 'withFile', 'finally' or any of the other usual
scope-guarding techniques, what would I do? (The handle has
to be passed up to the outside of the scope in which it was
opened.)

If I stored the handle in an 'MVar' and attached an
MVar-finalizer to that, would that work better? Would the
MVar's finalizer be run any sooner than the one attached to
the handle anyway?

Or can I explicitly trigger garbage collection somehow? Say,
in case I receive an exception telling me that file
descriptors are running out? Which exception would I even
get in this case?

Peter

[Remi Turk]

On Sun, Oct 24, 2004 at 02:16:50PM +0200, Peter Simons wrote:
> Tomasz Zielonka writes:
>
> > AFAIK, Handles have finalisers which close them, but I
> > don't know if GHC triggers garbage collection when file
> > descriptors run out. If not, you will have problems if
> > you manage to run out of fds between GCs.
>
> Thank you for answering.
>
> Now there is only one problem: Assuming I could _not_ use
> 'bracket', 'withFile', 'finally' or any of the other usual
> scope-guarding techniques, what would I do? (The handle has
> to be passed up to the outside of the scope in which it was
> opened.)

Refactoring comes to the mind... ;)

> If I stored the handle in an 'MVar' and attached an
> MVar-finalizer to that, would that work better? Would the
> MVar's finalizer be run any sooner than the one attached to
> the handle anyway?

Both are just finalizers, so it won't make any difference AFAICS.

> Or can I explicitly trigger garbage collection somehow? Say,
> in case I receive an exception telling me that file
> descriptors are running out? Which exception would I even
> get in this case?
>
> Peter

You could try using something like this instead of the normal
openFile:
(or make it even less portable by importing GHC.IOBase and only
retrying on ResourceExhausted)

module Main where

import IO
import System.Mem

myOpenFile path mode
= catch (openFile path mode) $ \_ -> do
putStrLn "==> Open failed. Retrying <=="
performGC
openFile path mode

open = openFile "/tmp/foo" ReadMode
myOpen = myOpenFile "/tmp/foo" ReadMode

Example:

-- Let's eat all available filedescriptors.

*Main> sequence (repeat open)
Loading package haskell98 ... linking ... done.
*** Exception: /tmp/foo: openFile: resource exhausted (Too many open files)

-- They are indeed all gone.

*Main> h <- open
*** Exception: /tmp/foo: openFile: resource exhausted (Too many open files)

-- What about performing garbage collection?

*Main> h <- myOpen
==> Open failed. Retrying <==

*Main> h
{handle: /tmp/foo}

*Main>

Groeten,
Remi

--
Nobody can be exactly like me. Even I have trouble doing it.

[Peter Simons]

Remi Turk writes:

>> Assuming I could _not_ use 'bracket', 'withFile',
>> 'finally' or any of the other usual scope-guarding

>> techniques [...]

> Refactoring comes to the mind... ;)

I wish that were possible! I use bracket-style resource
allocation wherever I can, but in this case the Handle is
the _result_ of a rather complex computation, so it has to
leave the scope in which is was opened. A finalizer is all I
can use.

> performGC

That is the function I was looking for.

Well, it's actually not at all the function I was looking
for, but given the circumstances, it seems to be the only
solution. Alternatives are highly sought after, though!

Peter

[Marcin 'Qrczak' Kowalczyk]

Peter Simons <sim...@cryp.to> writes:

> I wish that were possible! I use bracket-style resource
> allocation wherever I can, but in this case the Handle is
> the _result_ of a rather complex computation, so it has to
> leave the scope in which is was opened. A finalizer is all I
> can use.

It does not necessarily make bracket unavailable. Just put it
somewhere outside, when it's known when the file will need to be
closed.

The whole action which opens the file inside it should be the
"acquire" part of the bracket. Asynchronous exceptions will be blocked
during executing it, so it should not be too long unless it unblocks
them for longer parts.

Of course there are also some cases when the file handle is deeply
buried in some data structures, and we don't know when they will be
freed.

--
__("< Marcin Kowalczyk
\__/ qrc...@knm.org.pl
^^ http://qrnik.knm.org.pl/~qrczak/

[Peter Simons]

Marcin 'Qrczak' Kowalczyk writes:

> It does not necessarily make bracket unavailable. Just
> put it somewhere outside, when it's known when the file
> will need to be closed.

The problem is that there is a gap of coverage, so to speak.
It works like this in my code:

input -> iodriver --> connection handler --+
| /|\ |
output <--+ +-----------------------------+
(Event, State)

The I/O driver is a StateT monad which calls the (various)
connection handlers every time new input is available from
the network. The handlers are StateT monads, too, and their
respective state is stored in the I/O driver's state when
they return.

Now I have the case where a connection handler needs open a
handle -- and keep it open for the life-time of the
connection. But what happens if I receive an exception after
the handle has been opened but before the connection handler
has returned? The handle is _supposed_ to leave the
connection handler, so it cannot attach any 'finally'
clauses or 'bracket' calls. And doing it in I/O driver is
too late. Neither will catching the exception in the I/O
driver help, because the intermediate state, which contained
the open handle, is lost.

The only solution I would know is to put the entire state of
the connection handler into an MVar, so that any changes to
it are immediate for the I/O driver. But that approach feels
like overkill to me. I can't judge what it would to the
performance of the whole application either. Doing every
little state access through an MVar can't be fast, can it?

If there is another solution, then I'd be delighted to hear
about it.

Peter

[Keean Schupke]

> The _result_ of a rather complex computation, so it has to

return a function that returns the handle.

for example:

handleOpener = complex function
bracket handleOpener close (\h -> do

Keean

[Peter Simons]

Keean Schupke writes:

>> The _result_ of a rather complex computation

> return a function that returns the handle.

The idea is good. :-) You'll find my other posting explains
that a bit more: My main driver doesn't know about the
handle; that's just one more entry in the state of the
connection handler. So the connection handler cannot return
a complex function, because _it_ is the one who wants to use
the handle, not the function that called it. What I return
is a state which contains the handle, not the handle per se.

[Keean Schupke]

You return the handle in the state, are you saying the complex
funtion does some IO on the handle first? I don't see why this
excludes doing it the other way round, passing the work function
to your complex function as an argument.

Keean

[Conal Elliott]

I'm puzzled why explicit bracketing is seen as an acceptable solution.
It seems to me that bracketing has the same drawbacks as explicit memory
management, namely that it sometimes retains the resource (e.g., memory
or file descriptor) longer than necessary (resource leak) and sometimes
not long enough (potentially disastrous programmer error). Whether the
resource is system RAM, file descriptors, video memory, fonts, brushes,
bitmaps, graphics contexts, 3D polygon meshes, or whatever, I'd like GC
to track the resource use and free unused resources correctly and
efficiently.

Cheers,

- Conal

[Marcin 'Qrczak' Kowalczyk]

"Conal Elliott" <co...@conal.net> writes:

> I'm puzzled why explicit bracketing is seen as an acceptable solution.
> It seems to me that bracketing has the same drawbacks as explicit memory
> management, namely that it sometimes retains the resource (e.g., memory
> or file descriptor) longer than necessary (resource leak) and sometimes
> not long enough (potentially disastrous programmer error).

If used correctly, it will retain it about the right amount of time,
while with garbage collection it's *impossible* to ensure that it will
not be retained too long. Most GC schemes don't guarantee promptness
of collection.

It doesn't matter for pure memory, because GC will be performed as
soon as enough memory has been allocated, but it matters for other
resources, except for programs which don't open many files at all.

--
__("< Marcin Kowalczyk
\__/ qrc...@knm.org.pl
^^ http://qrnik.knm.org.pl/~qrczak/

[Glynn Clements]

Conal Elliott wrote:

> > > What happens when a System.IO.Handle falls out of scope
> > > without being explicitly hClosed? Is that a resource leak?
> > > Or will the RTS close the handle for me?
> >
> > AFAIK, Handles have finalisers which close them, but I don't know if GHC
> > triggers garbage collection when file descriptors run out. If not, you
> > will have problems if you manage to run out of fds between GCs.
> >
> > How about using bracket to introduce explicit close on end of scope?
>

> I'm puzzled why explicit bracketing is seen as an acceptable solution.
> It seems to me that bracketing has the same drawbacks as explicit memory
> management, namely that it sometimes retains the resource (e.g., memory
> or file descriptor) longer than necessary (resource leak) and sometimes
> not long enough (potentially disastrous programmer error). Whether the
> resource is system RAM, file descriptors, video memory, fonts, brushes,
> bitmaps, graphics contexts, 3D polygon meshes, or whatever, I'd like GC
> to track the resource use and free unused resources correctly and
> efficiently.

File descriptors aren't simply a "resource" in the sense that memory
is. Closing a descriptor may have significance beyond the process
which closes it. If it refers to the write end of a pipe or socket,
closing it may cause the reader to receive EOF; if it refers to a
file, any locks will be released; and so on.

--
Glynn Clements <glynn.c...@virgin.net>

[Sven Panne]

Conal Elliott wrote:
> I'm puzzled why explicit bracketing is seen as an acceptable solution.
> It seems to me that bracketing has the same drawbacks as explicit memory
> management, namely that it sometimes retains the resource (e.g., memory
> or file descriptor) longer than necessary (resource leak) and sometimes
> not long enough (potentially disastrous programmer error). Whether the
> resource is system RAM, file descriptors, video memory, fonts, brushes,
> bitmaps, graphics contexts, 3D polygon meshes, or whatever, I'd like GC
> to track the resource use and free unused resources correctly and
> efficiently.

The lifetime problems are worse without explicit bracketing IMHO: One has
to wait for the next GC to free them (resource leaks) and even without
bracketing there's nothing to stop you from using a Handle after closing.

Furthermore, the main difference between simple RAM and more "external
things" like Handles is that the effect of keeping them occupied is
observable from outside our happy Haskell world, e.g. a server socket
might stay open, taking away a port number, an external device with
exclusive access is unusable until the next GC, my valuable texture memory
on the graphics card contains unused textures until the GC runs, etc. etc.
Finalizers are not a solution for these problems, a fact that e.g. a lot
of novice Java programmers have to learn when they do their first "real
world" programs...

IMHO it would be best to use explicit bracketing where possible, and hope
for the RTS/GC to try its best when one runs out of a given resource.
Admittedly the current Haskell implementations could be improved a little
bit in the last respect.

Cheers,
S.

[Remi Turk]

On Sun, Oct 24, 2004 at 12:19:59PM -0700, Conal Elliott wrote:
> I'm puzzled why explicit bracketing is seen as an acceptable solution.
> It seems to me that bracketing has the same drawbacks as explicit memory
> management, namely that it sometimes retains the resource (e.g., memory
> or file descriptor) longer than necessary (resource leak) and sometimes
> not long enough (potentially disastrous programmer error). Whether the
> resource is system RAM, file descriptors, video memory, fonts, brushes,
> bitmaps, graphics contexts, 3D polygon meshes, or whatever, I'd like GC
> to track the resource use and free unused resources correctly and
> efficiently.
>
> Cheers,
>
> - Conal

IMO, it does indeed have those same drawbacks. (Although the
traditional "explicit memory management model" is alloc/free,
which is much worse than bracket/withFile)
However, Garbage Collection is usually based only on memory.
Using GC for file-handle-closing therefore means that one will
close garbage file-handles when memory is getting low, instead of
when file-handles are almost used up...
The theoretical solution (and probably _only_ theoretical) is
implementing a lot of garbage collectors: one for memory, one for
open files, one for sockets, one for 3D polygon meshes etc etc...

Greetings,
Remi

--
Nobody can be exactly like me. Even I have trouble doing it.

[Conal Elliott]

Thanks, Remi (and other responders). I did indeed mean that GC of
non-RAM resources would be triggered by _those_ resources getting low
rather than RAM. That is, there would be many per-type collectors, each
with its own trigger. And maybe really only a small number different
collection algorithms, but clearly separated spaces and triggers. This
idea sounds fairly simple to me, so maybe not "only theoretical".

More comments?

- Conal

-----Original Message-----
From: Remi Turk [mailto:rt...@science.uva.nl]
Sent: Sunday, October 24, 2004 1:08 PM
To: Conal Elliott
Cc: haskel...@haskell.org
Subject: Re: [Haskell-cafe] Are handles garbage-collected?

[Stefan Monnier]

> File descriptors aren't simply a "resource" in the sense that memory
> is. Closing a descriptor may have significance beyond the process
> which closes it. If it refers to the write end of a pipe or socket,
> closing it may cause the reader to receive EOF; if it refers to a
> file, any locks will be released; and so on.

Yes, I think this is the crux of the matter: closing a file-descriptor is
not just a question of releasing resources, but it has actual side-effects
visible by the outside world (e.g. closing a pipe, or actual write-back if
the file is on NFS, ...) so it is important for programs to close FDs
explicitly when it needs to happen, rather than leave it to the GC to do it
whenever it feels like (which can be much much later if you get unlucky with
your generations).

Stefan

[Ketil Malde]

Remi Turk <rt...@science.uva.nl> writes:

> IMO, [bracket] does indeed have those same drawbacks. (Although the

> traditional "explicit memory management model" is alloc/free,
> which is much worse than bracket/withFile)

Isn't bracket more like stack allocated memory? And most problems
with explicit memory management related to heap (as you indicate)?

> The theoretical solution (and probably _only_ theoretical) is
> implementing a lot of garbage collectors: one for memory, one for
> open files, one for sockets, one for 3D polygon meshes etc etc...

...or have a number of available file handles that is limited by
memory? :-)

-kzm
--
If I haven't seen further, it is by standing in the footprints of giants

[Simon Marlow]

On 24 October 2004 20:51, Sven Panne wrote:

> IMHO it would be best to use explicit bracketing where possible, and
> hope for the RTS/GC to try its best when one runs out of a given
> resource. Admittedly the current Haskell implementations could be
> improved a little bit in the last respect.

Indeed, GHC could/should try to free up file descriptors when they run
out. It's a bit tricky though.

At the moment performGC doesn't actually run any finalizers. It
schedules a thread to run the finalizers, and you hope the thread runs
soon. So if you're running performGC for the purposes of finalization,
then almost certainly (performGC >> yield) is better. I've been
wondering whether having a more synchronous kind of finalizer would be a
good thing.

Nevertheless, I agree with the general sentiment on this thread that
file descriptors shouldn't be treated as a resource in the same way as
memory.

Cheers,
Simon

[Marcin 'Qrczak' Kowalczyk]

"Simon Marlow" <simo...@microsoft.com> writes:

> At the moment performGC doesn't actually run any finalizers. It
> schedules a thread to run the finalizers, and you hope the thread runs
> soon. So if you're running performGC for the purposes of finalization,
> then almost certainly (performGC >> yield) is better. I've been
> wondering whether having a more synchronous kind of finalizer would be a
> good thing.

In my language Kogut I have two kinds of finalizers:

- Some objects are implemented in C with C finalizers attached, e.g.
raw files. These finalizers should not access other objects and they
can't call back Kogut; they are used to free external resources or
explicitly managed memory (e.g. bignums, or the payload of arrays
which is malloced). These finalizers are run during GC.

- You can also attach GHC-style weak references with finalizers to
arbitrary objects. These finalizers are Kogut functions. They may
resurrect the object being finalized, and as far as I know they
can't be used to crash the runtime. They are executed in a separate
thread spawned by the garbage collector. They may use synchronization
primitives to wait until global variables are in a consistent state.

If a finalizer throws an exception, the exception is ignored and
the finalizing thread proceeds with other objects.

There is a small problem with their implementation: if a finalizer
blocks forever, other finalizers from the same GC round will never
be executed.

This implies that if you forget about a buffered file used for
writing, then the file destriptor will be closed, but buffers will
not be flushed.

Opening a file tries to GC once if it gets an error about too many
open files. It works because files use the first kind of finalizers.

Finalizers of the first kind are run at program end, because it was
easy to do. It doesn't matter anyway for things they are currently
used for, as all of them would be freed by the OS anyway. Finalizers
of the second kind are not run at program end, and I don't know
whether there is a sane way to do that.

You may register actions to be run at program end. I think it's
possible to use them to implement temporary files which are guaranteed
to be deleted at some time: either when they are no longer referenced,
or at program end. Implementing this would require a global set of
weak references to these files, and a single action will delete them
all. It's not possible to unregister these actions, and I'm not
convinced that it's needed in general.

* * *

BTW, I managed to get opening of named fifos working in the presence
of green threads. Opening them in non-blocking mode doesn't work;
GHC documentation describes a workaround for opening them for reading.
It's yet worse for writing (the system just returns ENXIO).

So I open all files in blocking mode, and switch the descriptor to
non-blocking mode after it has been obtained. The timer signal is
ITIMER_REAL (SIGALRM) rather than ITIMER_VIRTUAL (SIGVTALRM), which
causes the timer to tick during waiting for opening a file too. This
causes open() to fail with EINTR, in which case I process signals or
reschedule as appropriate and go back to trying open().

This also allows waiting for chold processes concurrently with waiting
for I/O or timeout. The thread waiting for a process or trying to open
a named fifo wastes its time slice though. I don't think it's possible
to implement this without this problem using Unix API, without native
threads.

While it would be possible to avoid waking it up N times per second if
it's the only thread which can do some work, I'm not sure it would be
worth the effort. The CPU usage is minimal.

Another downside of ITIMER_REAL is that third-party libraries might
not be prepared for EINTR from various system calls. For example my
Kogut<->Python bridge has to disable the timer when entering Python,
and enable it when entering Kogut back - otherwise Python's blocking
I/O will just fail. It still fails when a signal arrives, even if it's
then ignored (because ignoring a signal is done by having a signal
handler which effectively does nothing, rather than raw SIG_IGN).

* * *

These libraries might not be prepared for EAGAIN resulting from
non-blocking mode, in case the same descriptor is used by several
runtimes or across an exec, in particular stdin/stdout/stderr.
This requires explicit programmer intervention - I have no idea how
to do this fully automatically, so I require to do this almost fully
manually to avoid confusion when it's being done by the runtime.

A program tries to restore the original mode when it finishes, instead
of blindly setting them to the blocking mode, so a fork & exec of a
Kogut program from another doesn't disturb the blocking state. OTOH
the programmer must set them to blocking mode manually before exec'ing
a program which is not prepared for non-blocking mode.

Instead of resetting them to non-blocking mode on SIGCONT, as GHC does,
my default handler for SIGTSTP looks like this:

restore blocking mode on std handles
set SIGTSTP signal handler to SIG_DFL
raise(SIGTSTP) // This causes the process to stop until SIGCONT
set SIGTSTP signal handler back to my runtime's handler
set non-blocking mode on std handles

This means that it works on ^Z, and whatever parent process notices
the stop, it will not be confused by non-blocking mode of descriptors.
This is about the only case, apart from program start & exit and
opening a new file, when the blocking mode is changed by the runtime.

It does not work well though when somebody sends a SIGSTOP signal
instead of hitting ^Z. Then the shell starts using the terminal,
possibly sets blocking mode, and the program is continued without
restoring the mode. But since it will not work anyway in other cases
where the descriptor shared with other processes changes its mode,
I'm leaving it this way.

--
__("< Marcin Kowalczyk
\__/ qrc...@knm.org.pl
^^ http://qrnik.knm.org.pl/~qrczak/

[Remi Turk]

On Mon, Oct 25, 2004 at 08:46:41AM +0200, Ketil Malde wrote:
> Remi Turk <rt...@science.uva.nl> writes:
>
> > IMO, [bracket] does indeed have those same drawbacks. (Although the
> > traditional "explicit memory management model" is alloc/free,
> > which is much worse than bracket/withFile)
>
> Isn't bracket more like stack allocated memory? And most problems
> with explicit memory management related to heap (as you indicate)?

I think you're right. (WRT usage (only): AFAICS even
Foreign.Marshal.Alloc.alloca doesn't actually use the stack.)

> > The theoretical solution (and probably _only_ theoretical) is
> > implementing a lot of garbage collectors: one for memory, one for
> > open files, one for sockets, one for 3D polygon meshes etc etc...
>
> ...or have a number of available file handles that is limited by
> memory? :-)

Would be nice if OSes actually worked that way.
Then again, I don't think _everything_ can be made dependent only
on available memory.

Groetjes,
Remi

--
Nobody can be exactly like me. Even I have trouble doing it.

[Remi Turk]

On Mon, Oct 25, 2004 at 02:14:28PM +0100, Simon Marlow wrote:
> On 24 October 2004 20:51, Sven Panne wrote:
>
> > IMHO it would be best to use explicit bracketing where possible, and
> > hope for the RTS/GC to try its best when one runs out of a given
> > resource. Admittedly the current Haskell implementations could be
> > improved a little bit in the last respect.
>
> Indeed, GHC could/should try to free up file descriptors when they run
> out. It's a bit tricky though.

Hm, I'm not sure about the "should". Garbage collection is meant
for memory, and anything making that less clear makes people
more likely to depend on incorrect assumptions.
And redefining GC to be a collection of _all_ garbage, instead of
just memory doesn't sound so fantastic either.

> At the moment performGC doesn't actually run any finalizers. It
> schedules a thread to run the finalizers, and you hope the thread runs
> soon. So if you're running performGC for the purposes of finalization,
> then almost certainly (performGC >> yield) is better. I've been
> wondering whether having a more synchronous kind of finalizer would be a
> good thing.

Synchronous finalizers seem to be difficult to implement in e.g.
the JVM, so may make a JVM-backend more difficult. (I'm thinking
about how CPython vs Jython go about closing file-handles...
CPython uses (primarily) reference-counting, so files get closed
as soon as they aren't referenced anymore, which lots of people
now depend on. Jython uses Java-GC, so some CPython programs
may suddenly fail...)

> Nevertheless, I agree with the general sentiment on this thread that
> file descriptors shouldn't be treated as a resource in the same way as
> memory.
>
> Cheers,
> Simon

Groeten,
Remi

P.S. Why do so many people (including me) seem to come to Haskell
from Python? It can't be just the indentation, can it? ;)

--
Nobody can be exactly like me. Even I have trouble doing it.

[Tomasz Zielonka]

On Mon, Oct 25, 2004 at 08:55:46PM +0200, Remi Turk wrote:
> P.S. Why do so many people (including me) seem to come to Haskell
> from Python? It can't be just the indentation, can it? ;)

How many? I don't.

Best regards,
Tom

--
.signature: Too many levels of symbolic links

[Marcin 'Qrczak' Kowalczyk]

Remi Turk <rt...@science.uva.nl> writes:

> Hm, I'm not sure about the "should". Garbage collection is meant
> for memory, and anything making that less clear makes people
> more likely to depend on incorrect assumptions.
> And redefining GC to be a collection of _all_ garbage, instead of
> just memory doesn't sound so fantastic either.

I would not be surprised if relying on GC to close open files would
be generally considered kosher in a few years - in cases when it has
little visible effects outside, i.e. excluding network connections,
but including reading configuration files.

All which is needed is that the OS doesn't run out of system-wide
resources when only a given process opens many files; and that the
language implementation can actually force a garbage collection and
run finalizers of file objects immediately in the rare case when the
limit of per-process descriptors is reached.

Well, this has some impact on mixing languages, because when a module
implemented in one language runs out of file descriptors, it should
cause a program-wide GC of all runtimes. Maybe this style would apply
only to environments like .NET.

--
__("< Marcin Kowalczyk
\__/ qrc...@knm.org.pl
^^ http://qrnik.knm.org.pl/~qrczak/

[Remi Turk]

On Mon, Oct 25, 2004 at 09:28:23PM +0200, Tomasz Zielonka wrote:
> On Mon, Oct 25, 2004 at 08:55:46PM +0200, Remi Turk wrote:
> > P.S. Why do so many people (including me) seem to come to Haskell
> > from Python? It can't be just the indentation, can it? ;)
>
> How many? I don't.
>
> Best regards,
> Tom

At least one. (Me) And, judging from the amount of references to
Python in these mailing-lists, I really doubt I'm the only one.

I actually met Haskell mostly by reading about it in the python
mailinglist/newsgroup. (in e.g. Alex Martelli's posts)

Someone else (who shall remain anonymous. SCNR) wrote:
> Speculation I'm afraid to post: People are drawn to Python because they
> hear it is a clean language, but slowly find that it's really pretty
> messy internally. Haskell is beautiful on the outside and the inside.
> :-)

The last sentence I can definitely agree with, but I'm not so
sure Python really is that messy. (Messier-than-Haskell sure, but
messy? ;o)

Groetjes,
Remi

P.S. Hm, it _is_ haskell-cafe, but maybe it's about time for a
[Off-topic] note? ;)

--
Nobody can be exactly like me. Even I have trouble doing it.

[Marcin 'Qrczak' Kowalczyk]

Tomasz Zielonka <t.zie...@students.mimuw.edu.pl> writes:

>> P.S. Why do so many people (including me) seem to come to Haskell
>> from Python? It can't be just the indentation, can it? ;)
>
> How many? I don't.

And I don't either.

But indeed I've seen more references to Haskell on Python lists than
on other lists:

http://groups.google.com/groups?&q=group%3Acomp.lang.python+haskell
- 1590 posts
http://groups.google.com/groups?&q=group%3Acomp.lang.python
- 219 000 threads

http://groups.google.com/groups?&q=group%3Acomp.lang.java+haskell
- 73 posts
http://groups.google.com/groups?&q=group%3Acomp.lang.java
- 85 100 threads

http://groups.google.com/groups?&q=group%3Acomp.lang.perl+haskell
- 35 posts
http://groups.google.com/groups?&q=group%3Acomp.lang.perl
- 52 700 threads

http://groups.google.com/groups?&q=group%3Acomp.lang.c+haskell
- 197 posts
http://groups.google.com/groups?&q=group%3Acomp.lang.c
- 435 000 threads

http://groups.google.com/groups?&q=group%3Acomp.lang.scheme+haskell
- 1330 posts
http://groups.google.com/groups?&q=group%3Acomp.lang.scheme
- 44 800 threads

I don't know how to get the total number of posts from a group
archived by Google. Assuming that average threads on these groups
have the same size, Haskell is
- 8 times more popular on c.l.python than c.l.java,
- 11 times more popular on c.l.python than c.l.perl,
- 16 times more popular on c.l.python than c.l.c, but finally
- 4 times *less* popular on c.l.python than c.l.scheme,
i.e. the order of Haskell-awareness is Scheme > Python > Java > Perl > C.

--
__("< Marcin Kowalczyk
\__/ qrc...@knm.org.pl
^^ http://qrnik.knm.org.pl/~qrczak/

[ol...@pobox.com]

Simon Marlow wrote:
> I've been wondering whether having a more synchronous kind of
> finalizer would be a good thing.

Hans Boehm in his POPL2003 paper "Destructors, Finalizers, and
Synchronization" persuasively argued that finalizers _must_ be
asynchronous. That assertion is the title of Section 3.5 of the paper.

Peter Simons wrote:

> input -> iodriver --> connection handler --+
> | /|\ |
> output <--+ +-----------------------------+
> (Event, State)

> ...

> The I/O driver is a StateT monad which calls the (various)
> connection handlers every time new input is available from
> the network. The handlers are StateT monads, too, and their
> respective state is stored in the I/O driver's state when
> they return.

> ...

> The only solution I would know is to put the entire state of
> the connection handler into an MVar, so that any changes to
> it are immediate for the I/O driver. But that approach feels
> like overkill to me.

It seems you don't need to store the whole state in MVar: it's enough
to store a `clean-up' action. Suppose there exists an MVar known to
both a connection handler and the iodriver. The type of MVar's value
is Maybe (IO ()). When the connection handler opens a Handle, it
places a Just action that will close the handle. If the connection
handler makes it to the end without getting an exception, it, right
before returning to the iodriver, replaces that cleanup action with
Nothing. The iodriver, upon getting control back, checks the MVar and
executes whatever action is outstanding. The action can refer to any
context it needs for the clean-up.

Come to think of it, perhaps MVar is overkill. From your description,
it seems that each connection handle has its own private piece of
state inside the iodriver state. Perhaps it is not so much to ask to
add an IORef (Maybe (IO ())) to that state. The iodriver will check
that IORef upon getting control back from the control handler; Or the
driver can scan all these IORefs upon receiving an event, an
exception, etc.

Perhaps a more general approach is a special Handle leasing
component. Can we assume that a connection handler has a specific ID?
Can we assume that all alive connection handlers can be enumerated (it
seems so from the above description: iodriver is aware of all alive
connection handlers). If these assumptions hold, we can introduce one
level of indirection. Whenever a connection handler needs a Handle, it
applies to the Lord of Handles (LoH), passing its own id and the file
name. The LoH gives an id of a handle (or the handle itself). The LoH
also stores the allocated Handle and the component ID in its
tables. From time to time (or on certain events) the LoH checks all
the open handles to see if the corresponding connection handlers are
still alive.

[David Menendez]

Marcin 'Qrczak' Kowalczyk writes:

> I would not be surprised if relying on GC to close open files would
> be generally considered kosher in a few years - in cases when it has
> little visible effects outside, i.e. excluding network connections,
> but including reading configuration files.

One of the goals of ReiserFS, if I recall correctly, is to create a
filesystem API that doesn't involve file handles. I wonder how far one
could get without explicitly dealing with file handles outside of the IO
library implementation.
--
David Menendez <zedn...@psualum.com> <http://www.eyrie.org/~zednenem/>

[Sun Yi Ming]

From: David Menendez <zedn...@psualum.com>

Subject: Re: [Haskell-cafe] Are handles garbage-collected?

Date: Mon, 25 Oct 2004 22:51:34 -0400

[Ketil Malde]

Marcin 'Qrczak' Kowalczyk <qrc...@knm.org.pl> writes:

> - 8 times more popular on c.l.python than c.l.java,
> - 11 times more popular on c.l.python than c.l.perl,
> - 16 times more popular on c.l.python than c.l.c, but finally
> - 4 times *less* popular on c.l.python than c.l.scheme,
> i.e. the order of Haskell-awareness is Scheme > Python > Java > Perl > C.

I'm not sure whether this is accounted for in the tally, but I would
guess more threads are crossposted between c.l.scheme and
c.l.functional than between c.l.f and the other groups, and thus more
likely to be read by Haskellites.

-kzm
--
If I haven't seen further, it is by standing in the footprints of giants

[Graham Klyne]

At 20:55 25/10/04 +0200, Remi Turk wrote:
>P.S. Why do so many people (including me) seem to come to Haskell
> from Python? It can't be just the indentation, can it? ;)

I did. (Or: from Java via Python.)

I don't think it's the indentation. At least, not *just* that.

FWIW, I speculate:

1. Python users have already chosen expressivity over efficiency

2. For all that it's a "rapid" development language, and being dynamically
typed, Python doesn't completely abandon discipline in program construction.

#g

------------
Graham Klyne
For email:
http://www.ninebynine.org/#Contact

[Henning Thielemann]

On Tue, 26 Oct 2004, Graham Klyne wrote:

> At 20:55 25/10/04 +0200, Remi Turk wrote:
> >P.S. Why do so many people (including me) seem to come to Haskell
> > from Python? It can't be just the indentation, can it? ;)
>
> I did. (Or: from Java via Python.)
>
> I don't think it's the indentation. At least, not *just* that.

Maybe it is the fact that Python adapted some of the techniques that make
Haskell convenient, like map, filter, list construction, lambda and so on.
Perhaps people like to know the origin of these extensions?

[Simon Marlow]

On 26 October 2004 03:51, ol...@pobox.com wrote:

> Simon Marlow wrote:
>> I've been wondering whether having a more synchronous kind of
>> finalizer would be a good thing.
>
> Hans Boehm in his POPL2003 paper "Destructors, Finalizers, and
> Synchronization" persuasively argued that finalizers _must_ be
> asynchronous. That assertion is the title of Section 3.5 of the paper.

I didn't mean fully synchronous, just "more synchronous". For example,
the finalization routine could be run directly after garbage collection.
Hugs & nhc98 already do this, because they don't support Haskell
finalizers.

Just a thought, anyway (and I've read that paper, it's great).

Cheers,
Simon

[Malcolm Wallace]

"Simon Marlow" <simo...@microsoft.com> writes:

> I didn't mean fully synchronous, just "more synchronous". For example,
> the finalization routine could be run directly after garbage collection.
> Hugs & nhc98 already do this, because they don't support Haskell
> finalizers.

Actually, nhc98 supports both C and Haskell finalisers, with the same
signatures as in ghc:

newForeignPtr :: FinalizerPtr a -> Ptr a -> IO (ForeignPtr a)
newConcForeignPtr :: IO () -> Ptr a -> IO (ForeignPtr a)

C finalisers are run during garbage collection, Haskell finalisers
are run immediately afterwards. In fact, you cannot run a Haskell
finaliser during GC, because you don't have a valid heap to run it in.

Regards,
Malcolm

[Sam Mason]

Marcin 'Qrczak' Kowalczyk wrote:
>I would not be surprised if relying on GC to close open files would
>be generally considered kosher in a few years - in cases when it has
>little visible effects outside, i.e. excluding network connections,
>but including reading configuration files.

The best idea I've seen is another one from Microsoft Research.
It's an extension to C that allows the the programmer to use the
type system to specify the lifetime of things. Seems like a pretty
good idea:

http://research.microsoft.com/vault/

Not sure how you'd do something like that in Haskell though!

Cheers,
Sam

[Simon Marlow]

On 26 October 2004 10:42, Malcolm Wallace wrote:

> "Simon Marlow" <simo...@microsoft.com> writes:
>
>> I didn't mean fully synchronous, just "more synchronous". For
>> example, the finalization routine could be run directly after
>> garbage collection. Hugs & nhc98 already do this, because they don't
>> support Haskell finalizers.
>
> Actually, nhc98 supports both C and Haskell finalisers, with the same
> signatures as in ghc:
>
> newForeignPtr :: FinalizerPtr a -> Ptr a -> IO (ForeignPtr a)
> newConcForeignPtr :: IO () -> Ptr a -> IO (ForeignPtr a)
>
> C finalisers are run during garbage collection, Haskell finalisers
> are run immediately afterwards. In fact, you cannot run a Haskell
> finaliser during GC, because you don't have a valid heap to run it in.

But don't you run into problems in the implementation of
newConcForeignPtr, when the Haskell finalizer needs the value of a thunk
that was already under evaluation when the GC happened? ie. the need
for concurrency that stopped this from becoming part of the official
FFI.

Cheers,
Simon

[Ben Rudiak-Gould]

David Menendez wrote:

>One of the goals of ReiserFS, if I recall correctly, is to create a
>filesystem API that doesn't involve file handles. I wonder how far one
>could get without explicitly dealing with file handles outside of the IO
>library implementation.

I tried to do this last year with my File and Directory abstractions
[1], but the consensus was that it would be impossible to implement on
top of Posix or Win32. It might be possible on top of the native NT API,
though.

For this to really work properly the whole OS needs to be designed
around it. Such OSes exist -- they're called "capability-based" -- but
like so many other good ideas this one hasn't generated much interest
in industry. I think we're stuck with explicit file handle management
for the next couple of decades.

-- Ben

[1] http://www.mail-archive.com/has...@haskell.org/msg13071.html

[Peter Simons]

oleg writes:

> It seems you don't need to store the whole state in MVar:
> it's enough to store a `clean-up' action.

Yes, that is good advice, I'll do that. Thank you, Oleg.

Peter

[Malcolm Wallace]

"Simon Marlow" <simo...@microsoft.com> writes:

> > C finalisers are run during garbage collection, Haskell finalisers
> > are run immediately afterwards. In fact, you cannot run a Haskell
> > finaliser during GC, because you don't have a valid heap to run it in.
>
> But don't you run into problems in the implementation of
> newConcForeignPtr, when the Haskell finalizer needs the value of a thunk
> that was already under evaluation when the GC happened? ie. the need
> for concurrency that stopped this from becoming part of the official FFI.

Yeah, sure, there are necessary restrictions on the nature of Haskell
finalisers in our implementation, and if we wanted to fix them,
e.g. with locks, it could get quite messy internally.

My point wasn't really about Haskell finalisers, more about your
statement that Hugs and nhc98 fun their finalisers immediately post-GC
as a direct consequence of them *not* being written in Haskell,
which I think was a slightly misleading statement.

Regards,
Malcolm

[John Goerzen]

On 2004-10-25, Tomasz Zielonka <t.zie...@students.mimuw.edu.pl> wrote:
> On Mon, Oct 25, 2004 at 08:55:46PM +0200, Remi Turk wrote:
>> P.S. Why do so many people (including me) seem to come to Haskell
>> from Python? It can't be just the indentation, can it? ;)
>
> How many? I don't.

I'm one.

I look at Haskell at what Python should be in a lot of ways. Haskell's
type system is static and helpful, yet inobtrusive like I like my
Python. Python has added a lot of FP-like features lately, and they're
great. Things like generators, etc.

But why go only half-way? Haskell doesn't need a special generator
because a list is lazy. I like that.

Haskell's greatest deficiency compared to Python, IMHO, is the breadth
of good libraries available for it. I'm working to fix that.

One down (ftplib), many more (ConfigParser, etc) to go :-)

--
John Goerzen
Author, Foundations of Python Network Programming
http://www.amazon.com/exec/obidos/tg/detail/-/1590593715

[Marcin 'Qrczak' Kowalczyk]

Sam Mason <ma...@f2s.com> writes:

> The best idea I've seen is another one from Microsoft Research.
> It's an extension to C that allows the the programmer to use the
> type system to specify the lifetime of things.

I'm worried about putting too many thing in types. For example many
people believe exception specifications in Java are a mistake.

Such problems often don't show up until someone tries to write big
programs. It's not sufficient that it appears to work for toy
programs.

Putting many things in types is generally problematic for all kinds of
polymorphism.

--
__("< Marcin Kowalczyk
\__/ qrc...@knm.org.pl
^^ http://qrnik.knm.org.pl/~qrczak/

[Shae Matijs Erisson]

Ben Rudiak-Gould <Benjamin.R...@cl.cam.ac.uk> writes:

> For this to really work properly the whole OS needs to be designed
> around it. Such OSes exist -- they're called "capability-based" -- but
> like so many other good ideas this one hasn't generated much interest
> in industry. I think we're stuck with explicit file handle management
> for the next couple of decades.

hOp/House doesn't have a filesystem yet.
See http://www.cse.ogi.edu/~hallgren/House/ (source recently released)
Maybe we won't always require explicit file handles.
--
Shae Matijs Erisson - Programmer - http://www.ScannedInAvian.org/
"I will, as we say in rock 'n' roll, run until the wheels come off,
because I love what I do." -- David Crosby

[Shae Matijs Erisson]

Remi Turk <rt...@science.uva.nl> writes:

> At least one. (Me) And, judging from the amount of references to
> Python in these mailing-lists, I really doubt I'm the only one.

At least two. I also came to Haskell from Python.

> I actually met Haskell mostly by reading about it in the python
> mailinglist/newsgroup. (in e.g. Alex Martelli's posts)

I met Haskell because I started writing all of my Python code with a single
return point, was overusing reduce and map, building function pipelines, and
finally someone asked me if I'd used Haskell before.

>> Haskell is beautiful on the outside and the inside. :-)

Yes, Haskell is beautiful inside and outside. That's something that has kept me
interested in it for years. 'Conceptually pure' is what I call it.

--
Shae Matijs Erisson - Programmer - http://www.ScannedInAvian.org/
"I will, as we say in rock 'n' roll, run until the wheels come off,
because I love what I do." -- David Crosby

_______________________________________________

[William Lee Irwin III]

Ben Rudiak-Gould <Benjamin.R...@cl.cam.ac.uk> writes:
>> For this to really work properly the whole OS needs to be designed
>> around it. Such OSes exist -- they're called "capability-based" -- but
>> like so many other good ideas this one hasn't generated much interest
>> in industry. I think we're stuck with explicit file handle management
>> for the next couple of decades.

On Tue, Oct 26, 2004 at 09:22:58PM +0200, Shae Matijs Erisson wrote:
> hOp/House doesn't have a filesystem yet.
> See http://www.cse.ogi.edu/~hallgren/House/ (source recently released)
> Maybe we won't always require explicit file handles.

It doesn't seem to have demand paging either. Hmm. Maybe I should look
at this sometime.

-- wli

[Bryn Keller]

Shae Matijs Erisson wrote:

>Remi Turk <rt...@science.uva.nl> writes:
>
>
>
>>At least one. (Me) And, judging from the amount of references to
>>Python in these mailing-lists, I really doubt I'm the only one.
>>
>>
>
>At least two. I also came to Haskell from Python.
>
>

Me three.

>>I actually met Haskell mostly by reading about it in the python
>>mailinglist/newsgroup. (in e.g. Alex Martelli's posts)
>>
>>
>
>I met Haskell because I started writing all of my Python code with a single
>return point, was overusing reduce and map, building function pipelines, and
>finally someone asked me if I'd used Haskell before.
>
>

I met Haskell by first meeting Vyper, which was a more
functionally-slanted variant of Python, which was written in Ocaml. This
led me to Haskell, which I liked better.

>>> Haskell is beautiful on the outside and the inside. :-)
>>>
>>>
>
>Yes, Haskell is beautiful inside and outside. That's something that has kept me
>interested in it for years. 'Conceptually pure' is what I call it.
>
>

Yep.

Bryn

[William Lee Irwin III]

Remi Turk <rt...@science.uva.nl> writes:
>> At least one. (Me) And, judging from the amount of references to
>> Python in these mailing-lists, I really doubt I'm the only one.
>

On Tue, Oct 26, 2004 at 09:21:10PM +0200, Shae Matijs Erisson wrote:
> At least two. I also came to Haskell from Python.

I made a conscious effort at some point in college to learn a variety
of programming languages. Haskell arose as a particularly "nice" ML
variant in this exploration. I still find it useful for expressing
mathematical things for various computations to help with things.

-- wli

[Tomasz Zielonka]

On Mon, Oct 25, 2004 at 09:28:23PM +0200, Tomasz Zielonka wrote:

> On Mon, Oct 25, 2004 at 08:55:46PM +0200, Remi Turk wrote:
> > P.S. Why do so many people (including me) seem to come to Haskell
> > from Python? It can't be just the indentation, can it? ;)
>
> How many? I don't.

My road to Haskell went from C, C++ and Perl through Ocaml, Clean and
Erlang. I was mainly motivated by dissatisfaction with languages I used
at the moment. I am very happy with Haskell and I think I'll be using it
for some time.

I still use C++ at work, mostly because of performance requirements and
participation in multi-developer projects (multi stands for 2-3 here).
To be honest, I am very happy with what we managed to achieve in C++,
but I still get angry when I code some more complicated but
less-performance-critical parts and I think how easy it would be in
Haskell :)

[William Lee Irwin III]

On Tue, Oct 26, 2004 at 10:22:23PM +0200, Tomasz Zielonka wrote:
> My road to Haskell went from C, C++ and Perl through Ocaml, Clean and
> Erlang. I was mainly motivated by dissatisfaction with languages I used
> at the moment. I am very happy with Haskell and I think I'll be using it
> for some time.
> I still use C++ at work, mostly because of performance requirements and
> participation in multi-developer projects (multi stands for 2-3 here).
> To be honest, I am very happy with what we managed to achieve in C++,
> but I still get angry when I code some more complicated but
> less-performance-critical parts and I think how easy it would be in
> Haskell :)

All C and assembly programming for kernels and firmware at work here.

-- wli

[Conal Elliott]

Thanks for the explanation and example. I think the goal is to close
the pipe (for instance) asap after writing is finished. GC imposes a
delay, but so does bracketing, especially where modularity is desired.
Consider that the bracketing IO code calls separately defined IO code,
which makes the final write to a pipe but then do more work before
returning. In that case, a GC-like solution may close the pipe, release
the lock or whatever, sooner than the bracket solution, especially if
the descriptor GC were particularly eager. Just as with memory
management, stack-based allocation and freeing thwarts modularity or
efficiency or both, and automatic GC is a possible solution, if it
performs well enough.

Cheers,

- Conal

-----Original Message-----
From: Glynn Clements [mailto:gl...@gclements.plus.com]
Sent: Sunday, October 24, 2004 12:50 PM
To: Conal Elliott
Cc: haskel...@haskell.org
Subject: RE: [Haskell-cafe] Are handles garbage-collected?

Conal Elliott wrote:

> > > What happens when a System.IO.Handle falls out of scope
> > > without being explicitly hClosed? Is that a resource leak?
> > > Or will the RTS close the handle for me?
> >
> > AFAIK, Handles have finalisers which close them, but I don't know if
GHC
> > triggers garbage collection when file descriptors run out. If not,
you
> > will have problems if you manage to run out of fds between GCs.
> >
> > How about using bracket to introduce explicit close on end of scope?
>
> I'm puzzled why explicit bracketing is seen as an acceptable solution.
> It seems to me that bracketing has the same drawbacks as explicit
memory
> management, namely that it sometimes retains the resource (e.g.,
memory
> or file descriptor) longer than necessary (resource leak) and
sometimes
> not long enough (potentially disastrous programmer error). Whether
the
> resource is system RAM, file descriptors, video memory, fonts,
brushes,
> bitmaps, graphics contexts, 3D polygon meshes, or whatever, I'd like
GC
> to track the resource use and free unused resources correctly and
> efficiently.

File descriptors aren't simply a "resource" in the sense that memory
is. Closing a descriptor may have significance beyond the process
which closes it. If it refers to the write end of a pipe or socket,
closing it may cause the reader to receive EOF; if it refers to a
file, any locks will be released; and so on.

--
Glynn Clements <glynn.c...@virgin.net>