Streaming parser with pipes-parse

[Tran Ma]

Hi all,

I'm using pipes-parse to write a streaming parser that process incremental JSON data coming from a pipe, and stream out the parsed results. The input to this parser may look like this:

[{"a"
:0,"b":
1}
,{

...so on

where each line is a ByteString. My parser looks like this:

import qualified Pipes.Aeson.Unchecked as PU
import Control.Lens (zoom)

data Foo = Foo { a :: Int, b :: Int } deriving Show

parseFoo :: MonadIO m => Producer PS.ByteString m () -> Producer Foo m ()
parseFoo input = do
  fmap snd $ runStateT (zoom PU.decoded parser) (hoist lift input)
  return ()
  where parser :: Monad m => Parser Foo (Producer Foo m) ()
        parser = foldAllM (const $ yield) (return ()) return

This parser uses pipes-aeson to decode the streamed JSON, then folds over the result, yielding each Foo to the result producer underneath. The problem is, while there is definitely input coming in to ``parseFoo``, there is nothing coming in for ``parser`` (i.e. if I try to ``draw`` in ``parser``, the result is ``Nothing``).

I must be seriously misunderstanding something here, I'd appreciate any help =(

The complete working example can be found here: test data, json server, parser. In this example, I'm getting the JSON data from an http server on 3000 and parses it, then stream out the result again to another http server on 8080.

Thanks heaps!

[Gabriel Gonzalez]

Your code looks correct to me.  This makes me suspect that either (A) PU.decoded is not decoding correctly or (B) Your JSON is in a subtly different format than PU.decoded expects.

As a side note, your code can be simplified to:

    parseFoo = view PU.decoded

I would test that out, too.  If that still doesn't work (i.e. it creates an empty `Producer`) then it strongly implicates either your JSON or PU.decoded.

--
You received this message because you are subscribed to the Google Groups "Haskell Pipes" group.
To unsubscribe from this group and stop receiving emails from it, send an email to haskell-pipe...@googlegroups.com.
To post to this group, send email to haskel...@googlegroups.com.

[Renzo Carbonara]

Tran,


The problem is that the JSON you are trying to parse is not a
representation of `Foo`, but of `[Foo]`. That is, your `parseFoo`
would parse an input made of consecutive JSON representations for
`Foo`:

{"a":0,"b":1}{"a":1,"b":5}

But it wouldn't parse a single JSON list as:

[{"a":0,"b":1},{"a":1,"b":5}]

Re-reading the documentation for `pipes-aeson`, I can see that this is
not quite clear. I'll try and improve that documentation.


Regards,

Renzo Carbonara.

[Daniel Díaz]

It would be nice to have a function to stream the elements of a JSON list; I suspect it is a common use case. 

I wrote one in this gist: https://gist.github.com/danidiaz/305c0382782d1cefe2fa

The implementation hinges on a Parser whose base monad is itself a Producer. Maybe there's a less convoluted way to do it?