Is it possible to applicatively lift `Pipes a b m r` into `Pipes (ZipList a) (ZipList b) m r`?

[Louis Pan]

Say I have a `Pipes a b m r` called 'p'.

How do I convert it to a `Pipes (ZipList a) (ZipList b) m r`?

That is, I want to apply the pipe 'p' and apply zip-wise to an input ziplist, without losing the state of each pipe in the ziplist.

The reason I want this, is that I want to run multiple attoparsec parsers in parallel in one pass of a file.

My approach for this is to use Control.Foldl  to combine the attoparsec parsers into a `Control.Foldl.Fold ByteString [PartialParseResult]`, then convert it (using Control.Foldl.purely Pipes.Prelude.scan) to a `Pipe ByteString [PartialParseResult]`

My other requirement is that I want to look at intermediate parser output (eg. each line of a file as it's parsed), in order to summarise each line, in a stateful way as I'm keeping track of the current line number.

My approach for this is to use Control.Foldl.Fold for the summary logic, and then convert it to a pipe. I am able to create a `Pipe PartialParseResult AnalysisResult m r` for this.

However, how do I connect a `Pipe ByteString [PartialParseResult] m r` to a `Pipe PartialParseResult AnalysisResult m r`?

Regards,

Louis

[Louis Pan]

One approach I can think of is the use Pipes.Concurrent to manually split the ZipList and then combine then results. Is there a nicer way?

[Gabriel Gonzalez]

I would do what you're already doing with `Control.Foldl.Fold`.  That's the approach I usually recommend for this.

--
You received this message because you are subscribed to the Google Groups "Haskell Pipes" group.
To unsubscribe from this group and stop receiving emails from it, send an email to haskell-pipe...@googlegroups.com.
To post to this group, send email to haskel...@googlegroups.com.

[Louis Pan]

Sorry, could you please elaborate? I'm a bit slow.

How do I use Control.Foldl.Fold to convert `Pipes a b m r` into `Pipes (ZipList a) (ZipList b) m r` ?

[Louis Pan]

I managed to convert between `P.Producer (ZipList a) m r` and `ZipList (P.Producer a m ()` using the StateT trick used in Pipes.Parse

Does the following code make sense? Am I breaking any laws?

import Control.Applicative

import Control.Monad.Trans.Class

import Control.Monad.Trans.State.Strict

import qualified Pipes as P

import qualified Pipes.Lift as PL

-- ZipList Traversable is only in 4.9.0.0

sequenceAZipList :: Applicative f => ZipList (f a) -> f (ZipList a)

sequenceAZipList xs = ZipList <$> sequenceA (getZipList xs)

-- | Similar to Pipes.Parse.Parser, except it stores a ZipList of Producers.

type ZipParser a m r = forall x . StateT (ZipList (P.Producer a m x)) m r

-- | Draw one element from each underlying Producer, returning 'Nothing' if any of the producers are empty

drawZ :: Monad m => ZipParser a m (Maybe (ZipList a))

drawZ = do

  ps <- get

  rs <- lift (sequenceAZipList (P.next <$> ps))

  case sequenceAZipList rs of

    Left _ -> pure Nothing

    Right rs' -> do

      put $ snd <$> rs'

      pure . Just $ fst <$> rs'

-- | Push back a Ziplist element onto the underlying ZipList of Producers

unDrawZ :: Monad m => ZipList a -> ZipParser a m ()

unDrawZ as = modify (\ps -> appendA <$> ps <*> as)

 where

  appendA p a = do

    r <- p

    P.yield a

    pure r

toZipList :: Monad m => P.Producer (ZipList a) m r -> m (ZipList (P.Producer a m ()))

toZipList p = execStateT (toZipParser p) (pure (pure ()))

 where

  toZipParser :: Monad m => P.Producer (ZipList a) m r -> ZipParser a m r

  toZipParser p' = do

    r <- lift $ P.next p'

[Louis Pan]

My code doesn't stream properly - it forces consumption of the original producer before outputting anything. :(

[Louis Pan]

I think I understand what you mean by using Control.Fold.Foldl.

It's not possible to "applicatively" lift a Pipe from "Pipe a b m r" to "Applicative f => Pipe (f a) (f a) m r", but it is possible to lift a Control.Foldl.Fold from "Fold a b" to "Applicative f => Fold a b -> Fold (f a) (f b)" (see below).

This means it's better in my case to code the consumer as a Fold (and then convert it to a Pipes Parse), as opposed to a Pipe Consumer.

-- requires a function to force the applicative to prevent space leaks.

liftFold :: Applicative f => (forall x. f x -> f x) -> L.Fold a b -> L.Fold (f a) (f b)

liftFold f (L.Fold step begin done) = L.Fold step' begin' done'

 where

  step' xs as = f (step <$> xs <*> as)

  begin' = pure begin

  done' xs = done <$> xs

forceZipList :: ZipList a -> ZipList a

forceZipList (ZipList xs) = ZipList (forceFoldable xs)

-- https://ro-che.info/articles/2015-05-28-force-list

forceFoldable :: Foldable t => t a -> t a

forceFoldable xs = case foldr seq () xs of

      () -> xs

liftZipList ::  L.Fold a b -> L.Fold (ZipList a) (ZipList b)

liftZipList = liftFold forceZipList

I've confirmed this prevents space leaks by profiling with "+RTS -h -p" and using hp2ps.

However, I'm a novice with the significance of using seq and $!.

Am I doing anything dangerously? Is there something I should be careful of?

[Gabriel Gonzalez]

Yeah, what you wrote looks correct.  The general idea is that you compose `Fold`s and then convert to a `Pipe` at the last minute.

There's another related type which also has nice properties for this sort of thing, which is basically a mealy machine.  I usually encode it like this:

{-# LANGUAGE ExistentialQuantification #-}

data Scan a b = forall m . Monoid m => Scan { begin :: s, step :: s -> a -> (s, b) }

instance Category Scan where ...

instance Arrow Scan where ...

instance Functor (Scan a) where ...

instance Applicative (Scan a) where ...

focus

    :: (forall x. LensLike (State x) s t a b)

    -> Scan a b

    -> Scan s t

focus k (Scan s f) = Scan s (Control.Lens.mapAccumLOf k f)

Then you can use `focus traverse` to lift a `Scan` to work on anything that is `Traversable` (like `ZipList`):

focus traverse :: Traversable t => Scan a b -> Scan (t a) (t b)

[Louis Pan]

Hi Gabriel, thank you for taking the time to answer my questions.

Is your Scan encoding available anywhere? The closest thing I can find is ekmett's "strict Mealy Machine" at https://hackage.haskell.org/package/folds-0.7.1/docs/Data-Fold-L1'.html but it looks quite different.

[Gabriel Gonzalez]

The closest thing is the mealy machine type from this library, which is equivalent, although possibly less efficient (and missing the `focus` utility):

https://hackage.haskell.org/package/machines-0.6.1/docs/Data-Machine-Mealy.html

This is something I've been meaning to write up in a small library at some point, but I keep forgetting.

[Mitchell Rosen]

This library would be awesome to have!