Can anybody review this `pipes` functional pearl?

[Gabriel Gonzalez]

One of the reasons I've been busy lately is that I've been busy writing
up a small `pipes` functional pearl for the Haskell symposium. I would
be really grateful if any of you guys have time to review it.

[Andrew Cowie]

First impression: it's not really a pearl; reads more like expanded
documentation of a library API — which is sorely needed (I've been
fighting to keep up with the melange of blog posts, reddit comments,
and .Tutorial packages to understand enough of pipes to work with it,
and losing) — but my sense of the Functional Pearls over time has been
that they show a solution to an interesting [practical] problem; ie,
Suduko solver, the Financial Contract DSLs, etc. I think even the
original pretty print paper might have been one.

So if you were to take what you have and beef it up with a practical (vs
contrived) example of all the theory being applied to a problem
neophytes can understand (say, oh, I don't know, applying pipes to
process a stock ticker feed or something).

This represents several more pages of writing, of course; not what you
really wanted to hear.

++

Now that said, I look forward to reading it so I can learn something!
Does this use all the new 4.0 types?

Can you [in preprint form] output it as a single column? Double column
PDFs are a pain to read on mobile devices.

AfC
Sydney

[Dan Burton]

It uses the 1.0 types. :)

Looks good to me. I would recommend using the Conduit nomenclature for the 4-connection-operators section, since conduit used to be implemented just like that.

$$, $=, =$, and =$=

-- Dan Burton

[Gabriel Gonzalez]

On Thu, Jun 13, 2013 at 8:12 AM, Andrew Cowie <and...@operationaldynamics.com> wrote:

On Thu, 2013-06-13 at 07:25 -0700, Gabriel Gonzalez wrote:
> One of the reasons I've been busy lately is that I've been busy writing
> up a small `pipes` functional pearl for the Haskell symposium.  I would
> be really grateful if any of you guys have time to review it.

First impression: it's not really a pearl; reads more like expanded
documentation of a library API — which is sorely needed (I've been
fighting to keep up with the melange of blog posts, reddit comments,
and .Tutorial packages to understand enough of pipes to work with it,
and losing) — but my sense of the Functional Pearls over time has been
that they show a solution to an interesting [practical] problem; ie,
Suduko solver, the Financial Contract DSLs, etc. I think even the
original pretty print paper might have been one.

So in this case the practical problem was supposed to be the initial "How do I build Python-style generators in Haskell" (i.e. the first section).  However, I didn't want to title the paper after that because that's only the first 1/3 of the paper.  However, maybe I can change the abstract so that it's obvious that is the initial starting point.

 

So if you were to take what you have and beef it up with a practical (vs
contrived) example of all the theory being applied to a problem
neophytes can understand (say, oh, I don't know, applying pipes to
process a stock ticker feed or something).

Maybe I should dress up the initial generator example a bit so it is less trivial, then, such as adding a prompt for the user before each input.

However, keep in mind that functional pearls are supposed to be self-contained, so I have to limit myself to code examples that can fit within the paper and that are not overly reliant on libraries.  Is a stock market ticker easy to implement from scratch?

 

This represents several more pages of writing, of course; not what you
really wanted to hear.

Of course it is what I want to hear!  I wouldn't submit it for review if I didn't intend to make changes.  :)

 

++

Now that said, I look forward to reading it so I can learn something!
Does this use all the new 4.0 types?

No, this uses the original 1.0 API, like Dan said.  This is not documentation for the library at all, but rather a simple tutorial for how to build effectful streams in Haskell.  I used the original Pipes API for teaching purposes since it is simpler and very easy to implement.

If you want to learn the library `Control.Proxy.Tutorial` is still the official recommended place to learn and it is a self-contained document that has pretty much every last detail you need to know to learn `pipes` (and it will get smaller when `pipes-4.0.0` comes out because it will no longer need to explain the proxy transformer stuff and the bidirectionality section will be shorter).

 

Can you [in preprint form] output it as a single column? Double column
PDFs are a pain to read on mobile devices.

Of course!  I've attached the single column version.

[Gabriel Gonzalez]

Maybe this abstract will capture the application better:

Lazy linked lists sit at the intersection of Haskell's two biggest strengths:

laziness and purity.  However, they are strictly less powerful than Python's

generators because they cannot embed effects.  We require effectful streams if

we wish to meaningfully communicate with the outside world.

Effectful streams come in three flavors: input streams, output streams, and

stream transformers.  Most streaming libraries treat input streams

or output streams as first-class citizens, building rich domain-specific languages for them, but then

allow stream transformers to languish as second-class citizens.  Here we shall also

begin from input streams by implementing Python-style generators in Haskell, but

then gravitate towards an approach that places transformations front and center,

generalizing other approaches in the process.

This transformations-first approach is not only general, but also theoretically

inspired.  We draw a parallel between streaming transformations and the

morphisms of category theory and show how the category laws correspond to

streaming best practices.

[Gabriel Gonzalez]

Ok, I think I've settled on this version of the abstract:

Haskell's lazy lists are less powerful than Python's generators because pure

lists do not embed effects. We require effectful streams if we wish to

compete with Python and meaningfully communicate with the outside world.

Effectful streams come in three flavors: input streams, output streams, and

stream transformers. Most streaming libraries treat input streams

or output streams as first-class citizens, building rich domain-specific languages for 

them, but then allow stream transformers to languish as second-class citizens.

Here we also begin from input streams by reimplementing Python-style generators

in Haskell, but then gravitate towards using stream transformers as the central

abstraction. This transformations-first approach not only generalizes other

approaches, but also draws inspiration from theory. We draw a parallel between

transformations and the morphisms of category theory and show how the category

laws correspond to streaming best practices.

[Gabriel Gonzalez]

On Thu, Jun 13, 2013 at 8:20 AM, Dan Burton <danburt...@gmail.com> wrote:

It uses the 1.0 types. :)

Looks good to me. I would recommend using the Conduit nomenclature for the 4-connection-operators section, since conduit used to be implemented just like that.

$$, $=, =$, and =$=

The reason I avoided the conduit operators was to deliberately to avoid the paper being misinterpreted as a dig against conduit.  I was trying to keep the tone of the paper as neutral as possible and not single out other libraries.

[Renzo Carbonara]

It's a fun, clear and lightweight article, just like Functional Pearls
are meant to be.

I am satisfied with the concrete use case that is presented as an
example (i.e, reading from STDIN, transforming the stream and writing
back to STDOUT).

I am quite familiar with Python generators and I can follow the Python
references quite fine. However, I doubt the same will be true for many
other readers. Perhaps you could replace the Python comparisons with
some stand alone descriptions of the “generator” idea and just mention
that Python implements part of this idea using their generator
support.


Regards,

Renzo Carbonara.

[Gabriel Gonzalez]

That sounds like a good idea.  I will try to summarize what a generator is in a language neutral way so that other readers don't get lost.  I will also see if I can tie in references to generators in other languages, too, like Javascript and C#, although people familiar with Java will probably need the language-neutral description.

--
You received this message because you are subscribed to the Google Groups "Haskell Pipes" group.
To unsubscribe from this group and stop receiving emails from it, send an email to haskell-pipe...@googlegroups.com.
To post to this group, send email to haskel...@googlegroups.com.

[Dan Burton]

The reason I avoided the conduit operators was to deliberately to avoid the paper being misinterpreted as a dig against conduit.  I was trying to keep the tone of the paper as neutral as possible and not single out other libraries.

That's understandable, but changing the names just makes it a lightly veiled dig against conduit. Rather than flippantly dismissing each of the previous suggestions with "Next!", how about using a gentler and more compelling transition, such as "Can we do better?" Your paper should acknowledge conduit's contribution in this space, rather than trying to pretend it doesn't exist.

-- Dan Burton

[Gabriel Gonzalez]

Alright, I'll try to make the segues more positive and then mention `conduit` explicitly.

--

[Gabriel Gonzalez]

So I just ended up removing the "Next!"s and changed all the pre-Pipes operators to `conduit` versions.  I also changed the end of the conduit section to reflect its contribution:

This approach (pioneered by conduit [4]) uses three streaming

types and four connection operators. This API is a little large,

though, so we need to think about how we can compress the number

of abstractions

I've attached the latest draft which incorporates most changes suggested so far.

[Pierre R]

Hi,

I have read the first part with attention and skimmed the last parts. Here are some remarks (from a Haskell beginners). 

1° In the Generator section, the two first definitions didn't help much. "data List" and "data Step" sounded like extra noise to me.

2° To stick with the "Python-style", I would use "next" instead of "stepG" in the record syntax of data Generator. Then I would rename "next" into "stepG" in the Sinks section. I am not even sure "stepG/stepS" are the best names (I much prefer request/respond or something alike).

3° data Generator cannot use "newtype" ?

4° I find it easier to read StepG first and then Generator (page one). Small point but it makes a difference for a newbie like me.

5° The inline of "getFirst" is confusing at this point. I would remove that section (and make a note).

6° Again because this is not real code, the suffix "G" is noisy (knowing the purpose is to emulate "Python-style generator"). Maybe it could be introduced later from the Sink section ?

7° The inlining process of "oneTwo" is not much fun (but the monadic instance is)

8° Beginning of page 3, "echoAndLoop" is rather a strange example at that point. Maybe it should be a simple echo and then in the Sink section "echoAndLoop" would be introduced.

9° The Sinks transition is a bit abrupt. The "python" conductor line has been dropped without notice. We are leaving the strict realm of "input stream" after all. 

I am a bit disappointed that the article (given its title) does not say anything about the Proxy, Request or Respond category.

Overall I have found the current text more difficult to read than the Pipes tutorials. I think it should be possible to add some more fun in there.

Thanks for the read,

Cheers,

[Gabriel Gonzalez]

On Fri, Jun 14, 2013 at 8:01 AM, Pierre R <p.rade...@gmail.com> wrote:

Hi,

I have read the first part with attention and skimmed the last parts. Here are some remarks (from a Haskell beginners). 

1° In the Generator section, the two first definitions didn't help much. "data List" and "data Step" sounded like extra noise to me.

They are mainly there to point out that the difference between a generator and a list is not that big.  That's actually not obvious to a lot of people who still think that you need continuations to implement generators in Haskell thanks to Oleg.

 

2° To stick with the "Python-style", I would use "next" instead of "stepG" in the record syntax of data Generator. Then I would rename "next" into "stepG" in the Sinks section. I am not even sure "stepG/stepS" are the best names (I much prefer request/respond or something alike).

I like the `next` idea.  I think I will also rename `stepS` and the `step` for `Pipe`, too, as soon as I figure equally intuitive names.

 

3° data Generator cannot use "newtype" ?

It can use newtype and I will fix this.  Same thing for the other types.

 

4° I find it easier to read StepG first and then Generator (page one). Small point but it makes a difference for a newbie like me.

I agree.  I'll swap them around.

 

5° The inline of "getFirst" is confusing at this point. I would remove that section (and make a note).

I would like to keep that.  For a lot of beginners, the way Haskell streaming libraries work appears to be magic and I wanted to dispel that.  I welcome any suggestions for how to make it clearer.

 

6° Again because this is not real code, the suffix "G" is noisy (knowing the purpose is to emulate "Python-style generator"). Maybe it could be introduced later from the Sink section ?

I can get rid of it for the `Yield` and `Await` constructors, but I would still need to differentiate their `Return` constructors for the `($$)` operator.

Another reason they have the `G` suffix is that I wanted to differentiate them from `Pipe` because in approach 3 they still conceptually coexist and I wanted to avoid confusion.  I like to encourage people to compile and test these code examples themselves, and without suffixes it is difficult to do so because papers don't have a module system.

 

7° The inlining process of "oneTwo" is not much fun (but the monadic instance is)

Do you mean that the last half (inlining the `yield 2` and `return ()`) was not as interesting?  The main reason I have that last part was that it's not obvious what the "normal form" of a generator might look like when fully evaluated, so I wanted to include that in one of the examples, and felt that would be the best place to put it.

 

8° Beginning of page 3, "echoAndLoop" is rather a strange example at that point. Maybe it should be a simple echo and then in the Sink section "echoAndLoop" would be introduced.

There already is a simple echo example: stdin.  I decided not to do redo the stdin example by that point in the pearl because:

* It's not shorter than the hand-written version (and therefore doesn't properly motivate why we want an imperative DSL)

* I show how to redo it in the `Pipe` section later on

 

9° The Sinks transition is a bit abrupt. The "python" conductor line has been dropped without notice. We are leaving the strict realm of "input stream" after all. 

Yeah, unfortunately there are no good analogs (that I know of) to first class sink DSLs in mainstream languages so I have to drop that line eventually no matter what.  However, I will try to make the transition more gradual.

 

I am a bit disappointed that the article (given its title) does not say anything about the Proxy, Request or Respond category.

Overall I have found the current text more difficult to read than the Pipes tutorials. I think it should be possible to add some more fun in there.

This is not supposed to be a `pipes` tutorial.  The paper is supposed to teach a library-independent lesson of not biasing libraries towards input or output streams.  I only use `pipes` terms only because that's my own library, but the paper could have just as well used `conduit` types and terms for the entire presentation and it would still convey my intended lesson.

This is the reason I stick to the simpler `pipes-1.0` implementation and don't use any modern features like the `Proxy` type, or other pipe categories.  They are not at all related to the message of the tutorial and the `Pipe` type is nothing more than a vehicle for the actual lesson.

There will be a technical write-up of `pipes` later on (semantics, usage, and theory), but that is not what this paper is supposed to be.  It's a functional pearl, which is supposed to be about solving a simple problem and learning something about functional programming in the process.

 

Thanks for the read,

Thank you for the feedback!  This has been very helpful. 

[Gabriel Gonzalez]

Ok, so this is the final draft I've come up with based on everybody's comments.  There were some suggestions I ended up not implementing, such as removing sufixes, but I think I covered most points.

Based off of Pierre's feedback I walked the reader more slowly through the first example instead of doing a big equation dump.  I've also structured the paper to make it less likely that the reader mistakes it to be a `pipes` tutorial.

I just wanted to thank all of you guys for your help and I've added all of you to the acknowledgements section.  Thanks for taking the time to read it and offer very helpful feedback.  I'm much happier with the paper after the changes you all suggested.

[Pierre R]

Love the changes, particularly the first paragraph of the Abstract which nicely introduces the subject.

A small point. It might be worth to add a comment on the last line of "stdin" because the recursive call is not that obvious to imperative eyes (but I guess for most readers of functional pearls it is piece of cake ;-)

Thanks for your comment about list and continuation. It made me realize that some of my initial confusion on my first read came from the fact that continuation was not mentioned. Coming from the imperative style it is still easier for me to understand generators in term of continuation. I can then go along from continuation to co-routine or to monad and list.

[Gabriel Gonzalez]

Alright, I added an extra comment, "-- Recurse!", to `stdin` and uploaded the final version.  The die is cast!  Thank you! :)