Resource management and >>

[Patrick Wheeler]

I have been following the recent discussions on reddit about resource management and in an effort to simplify/understand some of the code written by Tekmo I came up with the following:

runSafeConsumerT p = runSafeT $ runEffect $ hoist lift catFromLifted >-> p
runSafeProducerT p = runSafeT $ runEffect $ p >-> hoist lift catToLifted
runSafePipeT     p = runSafeT $ runEffect $
                       hoist lift catFromLifted >-> p >-> hoist lift catToLifted

catFromLifted = go
  where
    go = do
        x <- lift await
        yield x
        go

catToLifted = go
  where
    go = do
        x <- await
        lift $ yield x
        go

runSafe*T creates a temporary new monad transform layer to runSafeT over so that resource management can be isolated from the rest of the pipe.

This seems to work under some conditions and provide some type safety(forgetting runSafe*T results in type error).

main =
        runEffect $ runSafeProducerT (readFile'' "input.dat") >-> loop 0
  where
loop i = do
          let fp = "out-pipes/" ++ show i
          runSafeConsumerT $ P.take 50 >-> writeFile'' fp
          loop $ i + 1

however if I compose a P.take on the outside of runSafeConsumerT it tries to open all of the files at once just like if I was using P.writeFile with out runSafeConsumerT.

Example:

loop i = do
          let fp = "out-pipes/" ++ show i
          P.take 1 >-> runSafeConsumerT  (P.take 50 >-> writeFile''' fp)
          loop $ i + 1

This loop tries to open all of the files first before closing any. I will try to expand this out step by step later. I would like to understand why composing P.take on the outside of runSafeConsumerT is causing this different allocation pattern.

Patrick

--
Patrick Wheeler
Patrick.Jo...@gmail.com
Patrick....@rice.edu
Patrick...@colorado.edu

[Patrick Wheeler]

forgot to link to the reddit discussion:

http://www.reddit.com/r/haskell/comments/1nw7ji/pipes_resource_problems/cco303j?context=3

After a little testing it looks like when the pipe terminates on the external P.take the resource is not cleaned up, but when it terminates on inside of runSafeConsumerT the resource is cleaned up.  This can be easily checked by making the amount the outside `P.take` takes larger or small then the internal one.

It looks like I am escaping the bracket and runSafeT without calling the finalization.

The full code is:

http://lpaste.net/94067

This needs the latest pipes-safe code from the repo.

[Patrick Wheeler]

Forgot to include some examples:

out side P.take 1, inside P.take 2, external limit 6

runhaskell test.hs 

{out-pipes/0 open}

{input.dat open}

{out-pipes/1 open}

{out-pipes/2 open}

{out-pipes/3 open}

{out-pipes/4 open}

{out-pipes/5 open}

{out-pipes/6 open}

out side P.take 2, inside P.take 2, limit 6

runhaskell test.hs 

{out-pipes/0 open}

{input.dat open}

{out-pipes/0 closed}

{out-pipes/1 open}

{out-pipes/1 closed}

{out-pipes/2 open}

{out-pipes/2 closed}

{out-pipes/3 open}

It looks like the last file here is not closing either.

[Gabriel Gonzalez]

Ok, I found the problem.  So I'll first begin by explaining what I intended to happen in this scenario: I thought the finalizers would be delayed until the outer `SafeT` block closed.

The reason they don't is because `runSafeT` uses `bracket` from `Control.Monad.Catch` (from the `exceptions` package), not the `pipes-safe` `bracket`.  The `bracket` from `Control.Monad.Catch` is not termination safe, unlike the `bracket` from `pipes-safe`.

The reason why the `bracket` from `exceptions` is not termination safe is because it is defined in terms of `onException` from `exception`s, which is also not termination safe.

Unfortunately, it's not sufficient to just add `onException` to the `MonadCatch` class and have `pipes-safe` overload it.  The reason why is that the `pipes-safe` `onException` function also depends on `register` and `release`, which are not part of the `MonadCatch` class, and I highly doubt Edward would approve of adding those to the `MonadCatch`

However, there are two solutions.  They are:

1) Short-term solution:

Duplicate the `MonadCatch` class so that includes the `register` and `release` functions, too.  Then all the `pipes-safe` functions would have a `MonadCatch` constraint instead of a `MonadSafe` constraint (but for backwards compatibility you can make `MonadSafe` just a sub-class of `MonadCatch`), and then nested `runSafeT`'s would do the correct thing by using the `pipes-safe` `bracket` function.  However, even so this would still be a non-backwards compatible change if users import any functionality from `Control.Monad.Catch`.

2) Long-term solution:

Michael and I have been discussing a long-term solution for a `MonadCatch` class with resourcet-like behavior (i.e. `register` and `release`) that both libraries could share so in the long term that may be the better solution.

Which solution do you guys prefer?  Or, are there any other solutions that somebody would like to suggest?

--
You received this message because you are subscribed to the Google Groups "Haskell Pipes" group.
To unsubscribe from this group and stop receiving emails from it, send an email to haskell-pipe...@googlegroups.com.
To post to this group, send email to haskel...@googlegroups.com.

[Dan Burton]

Michael and I have been discussing a long-term solution for a `MonadCatch` class with resourcet-like behavior (i.e. `register` and `release`) that both libraries could share so in the long term that may be the better solution.

I want this. :) I'd say just use resourcet.

-- Dan Burton

[Gabriel Gonzalez]

Alright, then I will try to continue to hash this out with Michael.

[Dag Odenhall]

On Thu, Oct 10, 2013 at 10:45 PM, Gabriel Gonzalez <gabri...@gmail.com> wrote:

Michael and I have been discussing a long-term solution for a `MonadCatch` class with resourcet-like behavior (i.e. `register` and `release`) that both libraries could share so in the long term that may be the better solution.

Ideally I'd like two APIs here:

• The main API which is basically SafeT or ResourceT plus a state index

• An optional full-on indexed monad API with implicit region subtyping and all that juice

Is there any chance of making that work in a way that's decoupled from both pipes and conduit but also works with both?

[Patrick Wheeler]

Thanks for the feed back. I don't think I have a good perspective on weather options 1 or 2 is the better solution. I will voice that I have liked how the functionality of the pipes ecosystem has been separated out in general and would rather not sacrifice that if possible. A long term shared solution without that sacrifice would be positive in my opinion.

[Gabriel Gonzalez]

I'm not sure, yet.  This is all still very hypothetical.  However, if anybody is interested in joining in on those discussions just let me know and I will CC you all.

--

[Dan Burton]

I second the desire for indexed monad stuff. Indexed monads are currently ugly to use in Haskell, especially when mixing them with "regular" monads. We should work on changing this, because indexed monads can add a level of type safety and flexibility that could be really useful for solving various real problems.

-- Dan Burton

On Thu, Oct 10, 2013 at 2:07 PM, Dag Odenhall <dag.od...@gmail.com> wrote:

--

[Dag Odenhall]

On Thu, Oct 10, 2013 at 11:15 PM, Gabriel Gonzalez <gabri...@gmail.com> wrote:

However, if anybody is interested in joining in on those discussions just let me know and I will CC you all.

How about moving any private discussions to the streaming list?

[Gabriel Gonzalez]

So the #1 problem that users of the old `Frame`s API came across when using indexed monads was that you can't use both indexed monads and ordinary monads within the same module.  The workaround was to lift ordinary monad operations to become indexed monad operations, but it was both inefficient and ugly.

However, I agree that indexed monads are theoretically beautiful and open up the possibility for many cool new features.

[Dag Odenhall]

Perhaps this could be solved by a new GHC extension, or (at least in the mean time) a quasi-quoter. I don‘t really like quasi-quoters as they have a number of problems (such as you can’t nest them, and to parse Haskell you need a third-party parser, and then you have problems with fixity resolution, …) but they might still be better than nothing in certain situations.

And apparently someone already made a package for this.

[Patrick Wheeler]

@Gabriel

> I'm not sure, yet.  This is all still very hypothetical.  However, if anybody is interested in joining in on those discussions just let me know and I will CC you all.

I would be interested in following along, though I doubt I will have much to add.

[Gabriel Gonzalez]

That's a good idea.  I will do that.

[Bardur Arantsson]

Is that list on GMANE by any chance? (If so, what's it named?)

[Gabriel Gonzalez]

It's a Google group:

https://groups.google.com/forum/m/?fromgroups#!forum/streaming-haskell

... but I think there is some way to add it to GMane.  I know Gergely did this for `haskell-pipes`.