RFC: pipes-opaleye

[Bas van Dijk]

Hello,

I just started playing with pipes and I really like it so far. For my first project I would like to create a Producer for opaleye query results. I came up with the following:

  https://github.com/basvandijk/pipes-opaleye/blob/master/src/Pipes/Opaleye/RunQuery.hs

Is that the correct way of doing it? What can be improved?

Thanks,

Bas

[Gabriel Gonzalez]

Looks pretty straightforward to me

The only other suggestion I'd provide is to also provide a `Managed` version of the same `Producer`, with this type:

    query

        :: (Default QueryRunner columns haskells, MonadIO m, MonadMask m)
        => Connection -> Query columns -> Managed (Producer haskells m ())

Also, since it doesn't use the return value, you could even use the simpler `ListT` type instead:

    query

        :: (Default QueryRunner columns haskells, MonadIO m, MonadMask m)
        => Connection -> Query columns -> Managed (ListT m haskells)

... and then you can always convert that back to a `Producer` using `enumerate` if necessary

--
You received this message because you are subscribed to the Google Groups "Haskell Pipes" group.
To unsubscribe from this group and stop receiving emails from it, send an email to haskell-pipe...@googlegroups.com.
To post to this group, send email to haskel...@googlegroups.com.

[Bas van Dijk]

I like the ListT suggestion and just implemented it.

I don't understand how I can get rid of SafeT by using Managed. You're talking about Managed from your managed package right?

To unsubscribe from this group and stop receiving emails from it, send an email to haskell-pipes+unsubscribe@googlegroups.com.

To post to this group, send email to haskel...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Haskell Pipes" group.

To unsubscribe from this group and stop receiving emails from it, send an email to haskell-pipes+unsubscribe@googlegroups.com.

[Oliver Charles]

You might also want to look at streaming-postgresql-simple for similar prior art.

Ollie

On Sun, 30 Jul 2017, 3:47 pm Bas van Dijk, <v.dij...@gmail.com> wrote:

I like the ListT suggestion and just implemented it.

I don't understand how I can get rid of SafeT by using Managed. You're talking about Managed from your managed package right?

On 30 July 2017 at 15:58, Gabriel Gonzalez <gabri...@gmail.com> wrote:

Looks pretty straightforward to me

The only other suggestion I'd provide is to also provide a `Managed` version of the same `Producer`, with this type:

    query

        :: (Default QueryRunner columns haskells, MonadIO m, MonadMask m)
        => Connection -> Query columns -> Managed (Producer haskells m ())

Also, since it doesn't use the return value, you could even use the simpler `ListT` type instead:

    query

        :: (Default QueryRunner columns haskells, MonadIO m, MonadMask m)
        => Connection -> Query columns -> Managed (ListT m haskells)

... and then you can always convert that back to a `Producer` using `enumerate` if necessary

On Jul 29, 2017, at 2:54 PM, Bas van Dijk <v.dij...@gmail.com> wrote:

Hello,

I just started playing with pipes and I really like it so far. For my first project I would like to create a Producer for opaleye query results. I came up with the following:

  https://github.com/basvandijk/pipes-opaleye/blob/master/src/Pipes/Opaleye/RunQuery.hs

Is that the correct way of doing it? What can be improved?

Thanks,

Bas

--
You received this message because you are subscribed to the Google Groups "Haskell Pipes" group.

To unsubscribe from this group and stop receiving emails from it, send an email to haskell-pipe...@googlegroups.com.

To post to this group, send email to haskel...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Haskell Pipes" group.

To unsubscribe from this group and stop receiving emails from it, send an email to haskell-pipe...@googlegroups.com.

To post to this group, send email to haskel...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Haskell Pipes" group.

To unsubscribe from this group and stop receiving emails from it, send an email to haskell-pipe...@googlegroups.com.

[Tom Ellis]

I'm curious about all the MVar business. Is this a typical thing to do in
the pipes world? I thought there would be a more pure approach.

[Oliver Charles]

It's necessary because the type of `fold` given in `postgresql-simple` is fairly useless. It can't be massaged into the necessary shape of a free monad, so the best we can do is recover the steps by repeatedly pushing into an MVar and blocking. In my streaming-postgresql-simple library I write the fold's my self (using libpq) to avoid needing to introduce concurrency.

[Tom Ellis]

Interesting. I've been wondering what we can do about the type of `fold`.
Are there any strange consequences to generalising to `MonadIO`? For
example, what happens if we try to use this with `ListT IO`? I presume it
behaves in a not very useful way.

[Gabriel Gonzalez]

Note that you can turn the type of `fold` into a `Turtle.Shell`

The type of `postgresql-simple`'s `fold` forces you to consume the stream exactly once, whereas `Producer`s, `ListT`s, and similar types let you traverse the stream or any element multiple times

`Turtle.Shell` also has the advantage that it has built-in support for `Managed`, however, it has the major disadvantage that you have to consume the stream in its entirety, unlike `Producer`/`ListT` which let you partially consume a stream

On Sun, Jul 30, 2017 at 12:50 PM, Tom Ellis <tom-lists-hask...@jaguarpaw.co.uk> wrote:

Interesting.  I've been wondering what we can do about the type of `fold`.
Are there any strange consequences to generalising to `MonadIO`?  For
example, what happens if we try to use this with `ListT IO`?  I presume it
behaves in a not very useful way.

On Sun, Jul 30, 2017 at 06:42:16PM +0000, Oliver Charles wrote:
> It's necessary because the type of `fold` given in `postgresql-simple` is
> fairly useless. It can't be massaged into the necessary shape of a free
> monad, so the best we can do is recover the steps by repeatedly pushing
> into an MVar and blocking. In my streaming-postgresql-simple library I
> write the fold's my self (using libpq) to avoid needing to introduce
> concurrency.
>
> On Sun, Jul 30, 2017 at 5:42 PM Tom Ellis <

> tom-lists-haskell-pipes-2016@jaguarpaw.co.uk> wrote:
>
> > On Sat, Jul 29, 2017 at 11:54:33PM +0200, Bas van Dijk wrote:
> > > I just started playing with pipes and I really like it so far. For my
> > first
> > > project I would like to create a Producer for opaleye query results. I
> > came
> > > up with the following:
> > >
> > >
> > https://github.com/basvandijk/pipes-opaleye/blob/master/src/Pipes/Opaleye/RunQuery.hs
> > >
> > > Is that the correct way of doing it? What can be improved?
> >
> > I'm curious about all the MVar business.  Is this a typical thing to do in
> > the pipes world?  I thought there would be a more pure approach.

--
You received this message because you are subscribed to the Google Groups "Haskell Pipes" group.

To unsubscribe from this group and stop receiving emails from it, send an email to haskell-pipes+unsubscribe@googlegroups.com.

[Oliver Charles]

Depending on how you do that streaming, that might not be a disadvantage - "single row mode" streaming requires all rows are processed before another query is issued. On my library I use MonadResource to hook into stream destruction and make sure any remaining rows are fetched. Not great, but workable

To unsubscribe from this group and stop receiving emails from it, send an email to haskell-pipe...@googlegroups.com.

To post to this group, send email to haskel...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Haskell Pipes" group.

To unsubscribe from this group and stop receiving emails from it, send an email to haskell-pipe...@googlegroups.com.

[Bas van Dijk]

Note that I created a PR for opaleye[1] giving it a cursor interface. With it I can write a Producer which doesn't need to fork a thread and push all rows through a MVar.

Bas

[1] https://github.com/tomjaguarpaw/haskell-opaleye/pull/314

On 31 July 2017 at 10:36, Oliver Charles <ol...@ocharles.org.uk> wrote:

Depending on how you do that streaming, that might not be a disadvantage - "single row mode" streaming requires all rows are processed before another query is issued. On my library I use MonadResource to hook into stream destruction and make sure any remaining rows are fetched. Not great, but workable

On Mon, 31 Jul 2017, 2:05 am Gabriel Gonzalez, <gabri...@gmail.com> wrote:

Note that you can turn the type of `fold` into a `Turtle.Shell`

The type of `postgresql-simple`'s `fold` forces you to consume the stream exactly once, whereas `Producer`s, `ListT`s, and similar types let you traverse the stream or any element multiple times

`Turtle.Shell` also has the advantage that it has built-in support for `Managed`, however, it has the major disadvantage that you have to consume the stream in its entirety, unlike `Producer`/`ListT` which let you partially consume a stream

On Sun, Jul 30, 2017 at 12:50 PM, Tom Ellis <tom-lists-haskell-pipes-2016@jaguarpaw.co.uk> wrote:

Interesting.  I've been wondering what we can do about the type of `fold`.
Are there any strange consequences to generalising to `MonadIO`?  For
example, what happens if we try to use this with `ListT IO`?  I presume it
behaves in a not very useful way.

On Sun, Jul 30, 2017 at 06:42:16PM +0000, Oliver Charles wrote:
> It's necessary because the type of `fold` given in `postgresql-simple` is
> fairly useless. It can't be massaged into the necessary shape of a free
> monad, so the best we can do is recover the steps by repeatedly pushing
> into an MVar and blocking. In my streaming-postgresql-simple library I
> write the fold's my self (using libpq) to avoid needing to introduce
> concurrency.
>
> On Sun, Jul 30, 2017 at 5:42 PM Tom Ellis <

> tom-lists-haskell-pipes-2016@jaguarpaw.co.uk> wrote:
>
> > On Sat, Jul 29, 2017 at 11:54:33PM +0200, Bas van Dijk wrote:
> > > I just started playing with pipes and I really like it so far. For my
> > first
> > > project I would like to create a Producer for opaleye query results. I
> > came
> > > up with the following:
> > >
> > >
> > https://github.com/basvandijk/pipes-opaleye/blob/master/src/Pipes/Opaleye/RunQuery.hs
> > >
> > > Is that the correct way of doing it? What can be improved?
> >
> > I'm curious about all the MVar business.  Is this a typical thing to do in
> > the pipes world?  I thought there would be a more pure approach.

--
You received this message because you are subscribed to the Google Groups "Haskell Pipes" group.

To unsubscribe from this group and stop receiving emails from it, send an email to haskell-pipes+unsubscribe@googlegroups.com.

To post to this group, send email to haskel...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Haskell Pipes" group.

To unsubscribe from this group and stop receiving emails from it, send an email to haskell-pipes+unsubscribe@googlegroups.com.

To post to this group, send email to haskel...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Haskell Pipes" group.

To unsubscribe from this group and stop receiving emails from it, send an email to haskell-pipes+unsubscribe@googlegroups.com.

[Bas van Dijk]

Hi Tom,

I now wrote a version which doesn't use a separate thread to push all the rows through an MVar. It's using the new cursor interface that I added to opaleye:

https://github.com/basvandijk/pipes-opaleye/blob/master/src/Pipes/Opaleye/RunQuery.hs

Bas

[Tom Ellis]

Looks really nice!

On Wed, Aug 02, 2017 at 12:46:20AM +0200, Bas van Dijk wrote:
> I now wrote a version which doesn't use a separate thread to push all the
> rows through an MVar. It's using the new cursor interface that I added to
> opaleye:
>
> https://github.com/basvandijk/pipes-opaleye/blob/master/src/Pipes/Opaleye/RunQuery.hs
>

[Bas van Dijk]

Ollie, your streaming-postgresql-simple is really interesting. I didn't know about "single row mode" streaming[1]. I can imagine it's faster than streaming using a cursor since you don't have to send back a "FETCH FORWARD $chunkSize" command for every chunk that you want to fetch. Instead you only read from a socket and push each result on the stream.

[1] https://www.postgresql.org/docs/current/static/libpq-single-row-mode.html

To unsubscribe from this group and stop receiving emails from it, send an email to haskell-pipes+unsubscribe@googlegroups.com.

To post to this group, send email to haskel...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Haskell Pipes" group.

To unsubscribe from this group and stop receiving emails from it, send an email to haskell-pipes+unsubscribe@googlegroups.com.

To post to this group, send email to haskel...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Haskell Pipes" group.

To unsubscribe from this group and stop receiving emails from it, send an email to haskell-pipes+unsubscribe@googlegroups.com.

To post to this group, send email to haskel...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Haskell Pipes" group.

To unsubscribe from this group and stop receiving emails from it, send an email to haskell-pipes+unsubscribe@googlegroups.com.

[Bas van Dijk]

I realise we should also be able to combine cursor streaming and single-result-mode streaming. This would have the advantage of also streaming the results in a chunk from a cursor and, in case of termination, you only need to drain[1] the results for the current chunk.

[1] https://github.com/ocharles/streaming-postgresql-simple/blob/53184571a26cc4e18c1f7c5881b09fe770f62545/Database/PostgreSQL/Simple/Streaming.hs#L164

[Oliver Charles]

It would have that advantage, but nullifies the other advantage of using a cursor - being able to issue another query at the same time on the same connection handle.

To unsubscribe from this group and stop receiving emails from it, send an email to haskell-pipe...@googlegroups.com.

To post to this group, send email to haskel...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Haskell Pipes" group.

To unsubscribe from this group and stop receiving emails from it, send an email to haskell-pipe...@googlegroups.com.

To post to this group, send email to haskel...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Haskell Pipes" group.

To unsubscribe from this group and stop receiving emails from it, send an email to haskell-pipe...@googlegroups.com.

To post to this group, send email to haskel...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Haskell Pipes" group.

To unsubscribe from this group and stop receiving emails from it, send an email to haskell-pipe...@googlegroups.com.

[Bas van Dijk]

Good point.

To unsubscribe from this group and stop receiving emails from it, send an email to haskell-pipes+unsubscribe@googlegroups.com.

To post to this group, send email to haskel...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Haskell Pipes" group.

To unsubscribe from this group and stop receiving emails from it, send an email to haskell-pipes+unsubscribe@googlegroups.com.

To post to this group, send email to haskel...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Haskell Pipes" group.

To unsubscribe from this group and stop receiving emails from it, send an email to haskell-pipes+unsubscribe@googlegroups.com.

To post to this group, send email to haskel...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Haskell Pipes" group.

To unsubscribe from this group and stop receiving emails from it, send an email to haskell-pipes+unsubscribe@googlegroups.com.