Pipes vs Foldl: where to end one and start the other

[Chris Pollard]

Hello,

I am rewriting some old code, and I have a design question regarding the "right" way to interface pipes (or generic streams) and folds that may themselves be quite complex. Currently my setup goes something like this (this is pseudocode--haven't checked that it compiles/runs):

- I have one pipe that produces Events, where a simplified event might look like

data Event a b = Event { myA :: a, myBs :: [b] }

- I want to loop over events and store statistics of type r; I do this by implementing Fold(M)s:

foldA :: Fold a r

foldB :: Fold b r

- I combine the statistics using the Applicative or Monoid instances of Fold(M):

foldEvent :: Fold (Event a b) [r]

foldEvent = sequenceA

  [ premap myA foldA

  , premap myBs $ handles folded foldB

  ]

- I then connect these stats-gathering folds to my Event Producer, which works quite nicely.

My problem with this setup is one of complexity. In fact my Events are not so simple, and I want to categorize them in arbitrarily complicated ways and then feed them into the r statistics for each category. Currently all the complexity lives in the Fold side of my code, but it seems like Pipes are much better suited to handling non-trivial flows of information. I wonder if it's possible to have a complicated set of Pipes that connect (at the last second) to a bunch of Folds, which I then combine applicatively. There's a lot of Pipes infrastructure that could make the flow easier to understand, and I think the Pipes Monad instance may help clean up a lot of hard-to-read Fold code. In my silly example, Pipes.Prelude.map and Pipes.Prelude.mapFoldable are much easier to reason about (in my mind) than Foldl.premap and "handles folded".

Unfortunately, I can't see how to implement something as simple as my foldEvent using Pipes and still get the correct streaming properties. Is there a solution that I am missing, or should I focus on building up more composable primitives on the Fold side of my program?

Sorry if this is too vague: I don't want to dump a huge codebase on you, but I can provide more examples. Questions and comments welcome!

Cheers,

Chris

[Daniel Díaz]

>  There's a lot of Pipes infrastructure that could make the flow easier to understand, and I think the Pipes Monad instance may help clean up a lot of hard-to-read Fold code. 

Indeed, writing functions that "pull" from a source tends to be easier and more natural that writing "push-style" consumers.

This might not be exactly what you are looking for, but the Streaming.Eversion.Pipes module from my streaming-eversion package has functions that transform operations working on pipes into folds and fold transformers (transducers).

[Chris Pollard]

Thanks Daniel. I expected a mail in my inbox, so I just saw this. I'll have a look at your library.

Perhaps a more general question: is there anywhere a guide to what is in Pipes.Core? There are some nice (small) insights on in the haddocks, but it seems like quite a beautiful setup, so I'm wondering if there is a more complete introduction to this module a-la what is in Pipes.Tutorial.

Chris