Pipes and state and lenses and zoom (newbie question)

[Adam Conner-Sax]

Hi All,

I am new to Haskell and new to functional programming so I apologize in advance for the various idiocies in the questions below.

I have a monad transformer stack of just RS:

type App = ReaderT Env (StateT StateA Identity)  

with the identity made explicit at the bottom so it's clearer for me.

I wanted to do logging within the monadic functions so, based on Gabriel's cool example on his blog, I added a (pipes) Producer as in

type AppIO = ReaderT Env (StateT StateA IO)

type PApp = Producer LogEntry AppIO

type CApp = Consumer LogEntry AppIO

where

data LogLevel = Debug | Info   deriving (Show,Eq)

data LogEntry = LogEntry { _leLevel::LogLevel, _leMsg::String }

and I have a Consumer which allows me to choose which LogLevels I want to see:

printLog::[LogLevel]->CApp a

printLog levels = do

  le <- await

  let level = le ^. leLevel 

      msg = le ^. leMsg

      shouldLog = elem level levels

  if shouldLog 

    then liftIO $ putStrLn (show level ++ ": " ++ msg)

    else printLog levels  

  printLog levels        

(which works like a charm but feels clunky!)

The first problem was lifting (?) App functions into PApp.  I finally figured that out by importing Monad.Morph and defining

liftApp = lift . (hoist (hoist generalize))

(which I almost understand, depending on the exact moment you ask me), would do the trick.  

So far so good.  And it all works fine.  Excellent. Actually sort of dazzling.  Very cool.

Now the problem:  my larger application is a simulation of something evolving over time.  So the StateA of App is really a subset of a larger state, let's call it StateAB, which some, but not all monadic functions need access to.

so I generalized:

type App2 s = ReaderT Env (StateT s Identity)

type App2IO s = ReaderT Env (StateT s IO)  

type PApp2 s = Producer LogEntry App2IO s

type CApp2 s = Consumer LogEntry App2IO s

now I want to be able to run functions only requiring StateA inside a function requiring StateAB. Enter lenses and zoom. Let's assume that StateAB has a lens called A to get the StateA part

so, if I have f::App2 StateA ()

and 

g::App2 StateAB ()

g = do

   ...

   zoom A f

   ...

that works.  So far so good.  But once I add the producer, this no longer works.

so

f'::PApp2 StateA ()

g'::PApp2 StateAB ()

g'  = do

   ...

   zoom A f'

   ..

gives me a compile error which begins:

Couldn't match type ‘Control.Lens.Internal.Zoom.Zoomed

                           (Proxy X () () App2.LogEntry (App.AppIO StateAB))’

                  with ‘Control.Lens.Internal.Zoom.Zoomed

                          (Proxy X () () App2.LogEntry (App.AppIO StateA))’

    NB: ‘Control.Lens.Internal.Zoom.Zoomed’ is a type function, and may not be injective

and I find this confusing but largely because I don't quite understand how it all works.  I sort of see how zoom works and could maybe even write it in the case of any particular stack.  But it's not working here and I don't understand why well enough to figure out how to fix it.

Help?

And a more general question if I haven't worn out your patience already.

In my simulation I have the computation to move a single run forward one unit in time.  This modifies StateA.  Then I want to move it many steps forward and use StateB (also within StateAB) to collect statistics about this.  Then, eventually, I will want to run it many times with different initial conditions (or whatever) and collect those statistics so eventually there will be StateABC.  

That's okay if I can get all the zoom (and magnify) working.  Though the ways that is tricky sometimes make me just want to promote all computations to run in App2 StateABC, which seems like bad form since it's good if the signature makes clear what can and cannot be modified.  

But what if a computation needs to see StateB but not modify it while being able to modify StateA?

So in general, I'd like to be able to mix and match which things are passed as part of the Reader and which as State and have that vary from monadic function call to monadic function call.  So many would need to be lifted (transformed?) into the larger monadic context when called.  I can almost see how to do it for my plain SR stack, by writing my own lifts using get and ask and then runReaderT and runStateT.  But it will fall apart once the Producer is in there. Also there must be a better way. 

And also, am I going about this all wrong?

Any thoughts would be most appreciated!

As a C++ and perl programmer but once, long ago, a theoretical physicist, I am loving Haskell and all the smart people who comment on all these threads.  I'd be nowhere near as far along as I am without the combined expertise I find daily.  So thanks everybody!

Adam

[Gabriel Gonzalez]

Responses inline below:

Yep, that's exactly the recommended way to do this.  This is guaranteed to do "the right thing" (read: satisfy the monad morphism laws) because:

* `generalize` is a monad morphism
* therefore, `hoist generalize` is a monad morphism
* therefore, `hoist (hoist generalize)` is a monad morphism
* `lift` is a monad morphism
* therefore `lift . hoist (hoist generalize)` is a monad morphism

This is a confusing error because of how `lens`a type class to try to auto-`hoist` `zoom` for you.  The type error would probably be much better if you used the `zoom` from `lens-family-core`, which has the more concrete type:

    zoom :: Monad m => Lens' a b -> StateT b m r -> StateT a m r

Then it would have just complained that `StateT` is not the same as `Producer`, which would have then suggested the correct solution: `hoist` the zoom:

    _A :: Lens' StateAB StateA

    hoist (hoist (zoom _A)) :: PApp2 StateA r -> PApp2 StateAB r

That's also guaranteed to satisfy the monad morphism laws because:

* `zoom _A` satisfies the monad morphism laws (this is one of the consequences of the lens laws)
* therefore, `hoist (zoom _A)` satisfies the monad morphism laws
* therefore, `hoist (hoist (zoom _A))` satisfies the monad morphism laws

All that the `Zoomed` type class from lens is doing is auto-`hoist`ing things for you, but I prefer to be more explicit and `hoist` it myself to avoid confusing type errors.  You could even implement the `Zoomed` type class for `Producer`, but that would require `pipes` taking on a `lens` dependency, which I try to avoid.

And a more general question if I haven't worn out your patience already.

In my simulation I have the computation to move a single run forward one unit in time.  This modifies StateA.  Then I want to move it many steps forward and use StateB (also within StateAB) to collect statistics about this.  Then, eventually, I will want to run it many times with different initial conditions (or whatever) and collect those statistics so eventually there will be StateABC.  

That's okay if I can get all the zoom (and magnify) working.  Though the ways that is tricky sometimes make me just want to promote all computations to run in App2 StateABC, which seems like bad form since it's good if the signature makes clear what can and cannot be modified.  

But what if a computation needs to see StateB but not modify it while being able to modify StateA?

So in general, I'd like to be able to mix and match which things are passed as part of the Reader and which as State and have that vary from monadic function call to monadic function call.  So many would need to be lifted (transformed?) into the larger monadic context when called.  I can almost see how to do it for my plain SR stack, by writing my own lifts using get and ask and then runReaderT and runStateT.  But it will fall apart once the Producer is in there. Also there must be a better way.

There's a very elegant solution to this problem, which is to define this monad morphism:

    readOnly :: Monad m => ReaderT s m r -> StateT s m r
    readOnly readerT = StateT $ \s -> do
        r <- runReaderT readerT s
        return (r, s)

That satisfies the monad morphism laws (it's a fun exercise to prove them for `readOnly`), so then if you want to make a certain part of your pipeline `readOnly`, you would just define it as:

    example :: Producer LogEntry (ReaderT Env (ReaderT s IO)) r

That type prevents it from modifying the state, and if you wanted to incorporate it within a larger stateful computation, you would just do:

    hoist (hoist readOnly) example :: Producer LogEntry (ReaderT env (StateT s IO)) r

I added an issue to `mmorph` to remind myself to add `readOnly` to the next release of `mmorph` because I find myself reaching for this `readOnly` function all the time for exactly the reason you just described: to define a subroutine that can only read but not write the state:

https://github.com/Gabriel439/Haskell-MMorph-Library/issues/new

More generally, you're doing the right thing by limiting each computation to the minimum features it needs, such as:

* Limiting things to `Identity` when they don't need `IO`
* Limiting things to `ReaderT` when they don't need `StateT`
* Limiting things to just `App` when they don't need logging

You don't always have to do things this way (sometimes it's more convenient to dump everything into one monolithic monad transformer stack if you are in a hurry), but it's generally good practice to limit things to the bare minimum feature set they need, for the same reason that we try not to sprinkle the `IO` monad everywhere within our program.

And also, am I going about this all wrong?

Any thoughts would be most appreciated!

As a C++ and perl programmer but once, long ago, a theoretical physicist, I am loving Haskell and all the smart people who comment on all these threads.  I'd be nowhere near as far along as I am without the combined expertise I find daily.  So thanks everybody!

Adam

--
You received this message because you are subscribed to the Google Groups "Haskell Pipes" group.
To unsubscribe from this group and stop receiving emails from it, send an email to haskell-pipe...@googlegroups.com.
To post to this group, send email to haskel...@googlegroups.com.

[Gabriel Gonzalez]

Oops, wrong link to the Github issue:

https://github.com/Gabriel439/Haskell-MMorph-Library/issues/16

[Adam Conner-Sax]

Thanks!  I should have realized it needed more hoisting!  I've been thinking of hoist as approximately something that puts one stack level between the outer monadic context and it's target.  So that totally makes sense.

I'll have to think more about readOnly but that looks like the perfect solution.

You're very kind to reply so thoroughly and so quickly.  

I'm barely scratching the surface with pipes but I love how simple and flexible it has made the logging and now that I have the zoom right I can apply it only where needed.

Thanks Again!

Adam

[Gabriel Gonzalez]

You're welcome!

As a side note, if this general pattern interests you then you might want to read this post I wrote up a while ago, which talks about this design principle at length:

http://www.haskellforall.com/2012/09/the-functor-design-pattern.html

It even digs into the specific topic of monad morphisms.

[Adam Conner-Sax]

Thanks again.  I read that and the Category post with great interest.  I think I need to do more actual work to get it more clearly but I do like the simplicity that category and functor structure give you to work with.

A question from your previous suggestion of readOnly.  I can see how that would work if I made my stack a stack of several StateT transformers and then I could selectively make parts readOnly.  But that get's super messy if I have 3 or 4 components to the state.  Is there a way to do that with one composite state?  So suppose I have 

data MegaState = MegaState {_A::StateA, _B::StateB, _C::StateC, _D::StateD } 

makeLenses ''MegaState

MegaApp::StateT MegaState Identity

MegaAppIO::StateT MegaState IO

etc.

now if I have a function

f::ReaderT StateA (StateT StateC Identity)

how do I call it from MegaApp?  If I had a stack of four StateT as MegaApp I can see how some hoists and readOnly's and zooms would probably do it.  But how do I fuse the pieces back together to make one StateT?

Thanks.

[Gabriel Gonzalez]

I'd probably decompose this into three separate monad transformations:

* First, transform the `ReaderT` layer to a `StateT` layer using `readOnly`
* Then, combine the two `StateT` layers into one layer using the following transformation:

    -- There's only one way to implement `mult`, and it will be a monad morphism
    mult :: Monad m => StateT a (StateT b m) r -> StateT (a, b) m r

* Then, use `zoom` to embed `StateT (StateA, StateB)` within `StateT MegaState`, by defining a traversal from `MegaState` to (StateA, StateB)

For the last step, I think you would have to write that lens by hand.  I don't know of a standard way to combine two existing lenses into a larger lens.

As a side note, `mult` transformation I mentioned is inspired by category theory and it has a partner function named unit, which would have this type:

    -- a specialization of `lift`
    unit :: Monad m => m r -> StateT () m r

This pattern might seem more familiar if you know the `Applicative` type class in Haskell, which is equivalent to the following type class:

    class Functor f => Monoidal f where
        unit ::         () -> f     ()
        mult :: (f a, f b) -> f (a, b)

The `mult` and `unit` for `StateT` are just higher-order generalizations of that same pattern and those examples are in turn special cases of a more genera pattern which is called a lax monoidal functor:

http://en.wikipedia.org/wiki/Monoidal_functor

To a zeroth-order approximation, you can think of category theory as a way of manipulating and reasoning about types "positionally".  The more you learn category theory the more you start thinking about program transformations in terms of spatial metaphors like:

* "I'll just squish these two `StateT` layers together using `mult`"
* "I can lift this operation over the `ReaderT` using `hoist`"
* "I will insert a new `Producer`layer here using `lift`"

This turns out to be a nice solution to an age-old programming problem: naming things.  If you can describe program transformations positionally, you don't have to come up with interesting or unique names for the transformations.  It just becomes an exercise in applying various permutations of the same toolkit of standard and reusable rearrangement operations.

[Adam Conner-Sax]

Thanks again!  It's starting to make more sense. 

Here's what I've got for mult:

mult::Monad m => StateT a m (StateT b m r) -> StateT (a,b) m r

mult sAsB = StateT $ \s -> do

  (sB,newA) <- runStateT sAsB (fst s)

  (mr,newB) <- runStateT sB (snd s)

  return (mr,(newA,newB))

The monad morphisms make choosing the exact boundaries among all these things less of an issue but I still (coming from a C++ background) have trouble thinking clearly about design in Haskell.  I end up coding and then seeing where it doesn't fit well and re-designing. However, one of the great things about Haskell is that the re-factoring is always easier than I think it will be, I think because the components interact in such well defined ways.

Anyway, thank you for the help and all the great posts on your blog!

Adam