ANN: hyperdrive on github

[Jeremy Shaw]

I am mildly ashamed to admit that hyperdrive is now on github:

https://github.com/stepcut/hyperdrive

I'd almost prefer it if you didn't look at it right now because pretty much everything about it is in a state of disrepair. 

The pong server will actually serve a single PONG request (pretty sure pipelining is broken).

But, this code has been dragged through so many iterations of the pipes library that there is not much beauty left in it :)

Here is my short term plan:

 1. get the infrastructure in place

I want to start off right by having proper tests and benchmarks in place. So, I need to figure out how to properly get quickcheck, hunit, criterion, and hpc all integrated.

I want to be able to automatically run some test suite and know that the code is not only doing things correctly, but that the performance has remained consistent as well.

I always want to make sure that the documentation is easy to regenerate and publish.

If it is not easy (and automatic) to run tests, measure performance, and update the published documentation, then it won't get done and that is bad!

 2. work on the core API and structure

Once we have a pleasant environment to work in -- I want to make sure that the internal structure of the server is right (which is certainly is not right now).

Among other things, this means that it needs to be:

 1. easy to add HTTPS support

 2. easy to add web socket support 

 3. easy to provide your own accept loop

 4. easy to supply an alternative parser

 5. easy to understand and follow

 6. other stuff

 3. implement a very trustworthy parser 

The next step will be to implement a parser that we have a very high degree of confidence is correct -- even if it is not the fastest. I have some interesting ideas on how to do this which we can discuss when the time is right.

 4. implement a fast parser

Once we have a known to be correct parser, we can experiment with alternative parsers and have a  correct reference implementation to quickcheck against.

 5. create entries for benchmark games

Add those entries to the test suite and use them to help identify, analyze, and solve performance issues.

The key emphasis of this projects is being thoughtful at every step and doing things right. There is no rush to get something out there as soon as possible. And, since nobody is using it, there is not yet any pressure to retain backwards compatibility.

If it doesn't feel beautiful -- then it's not done yet :)

- jeremy

[Oliver Charles]

On Fri, Jul 5, 2013 at 10:33 PM, Jeremy Shaw <jer...@n-heptane.com> wrote:
>
> I am mildly ashamed to admit that hyperdrive is now on github:
>
> https://github.com/stepcut/hyperdrive
>
> I'd almost prefer it if you didn't look at it right now because pretty much everything about it is in a state of disrepair.

Dude, tell me you're kidding! This rocks! Now you get to deal with me
and Renzo bombarding you with pull requests ;)

> Here is my short term plan:
>

> [..]

Sounds sane to me. Not sure how much time I have immediately, but I'll
be following with keen eyes and might look at some benchmarking
suites.

> If it doesn't feel beautiful -- then it's not done yet :)

You're in good company :)

- Ollie

[Jack Henahan]

I haven't worked on any serious pipes projects before, but it is my stream ecosystem of choice. I'll definitely be forking, and I'll try to contribute something worthwhile. I really like all the work everyone gets done on this list. Great discussions, too.

- Jack

> --
> You received this message because you are subscribed to the Google Groups "Haskell Pipes" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to haskell-pipe...@googlegroups.com.
> To post to this group, send email to haskel...@googlegroups.com.
>
>

[Renzo Carbonara]

On Fri, Jul 5, 2013 at 6:58 PM, Oliver Charles <ol...@ocharles.org.uk> wrote:
> On Fri, Jul 5, 2013 at 10:33 PM, Jeremy Shaw <jer...@n-heptane.com> wrote:
>>
>> I am mildly ashamed to admit that hyperdrive is now on github:
>>
>> https://github.com/stepcut/hyperdrive

Great news! I look forward to seeing hyperdrive grow and succeed!

>> I'd almost prefer it if you didn't look at it right now because pretty much everything about it is in a state of disrepair.
>
>
> Dude, tell me you're kidding! This rocks! Now you get to deal with me
> and Renzo bombarding you with pull requests ;)

Yes, this is excellent! We are all still discovering the best ways to
deal with many Pipes scenarios, so discussing and building this in the
open at these early stages will benefit hyperdrive and everyone else
:)

Also, I think it was a good decision to move the hyperdrive
development to GitHub so that you can reduce the barrier to entry for
new contributors; even if you like Darcs more.

>> If it doesn't feel beautiful -- then it's not done yet :)
>
> You're in good company :)

Yeah, no one in here will complain about that :)


Regards,

Renzo Carbonara.

[Gabriel Gonzalez]

Sorry I took so long to respond. I was busy traveling to meet family
yesterday and just got settled in.

I really like the switch to Github. You won't regret it, trust me! The
biggest advantage of having a Github repository is having a transparent
development process, which is very valuable to users of your library.

I just wanted to briefly comment on performance since you mentioned
benchmarks. If you want great performance when its benchmarks time then
I can definitely guarantee that. In my hands I can get really excellent
performance, using the style I outlined in this post:

http://www.haskellforall.com/2013/04/pipes-and-io-streams.html

The reason that particular style produces really high-performance code
is because it does not use recursion, so the compiler optimizes it
incredibly well. In fact, this is generally true of all
high-performance Haskell code: you program the inner loop
non-recursively, because GHC is generally not good at optimizing
recursive loops. For most `pipes` users this overhead does not matter
that much so I don't discuss it a lot, but if you really want to win the
benchmarks game then this is the way to go.

Anyway, I will walk you through this when the time comes, but for now
you should just program in the classic recursive `pipes` style and not
worry about this just yet. Don't prematurely optimize for this until we
actually know that it's a bottle-neck after it's done.

Also, once you get the necessary infrastructure in place, it might be
worthwhile factoring that out into a skeleton that other people starting
new Haskell projects can use. I think Edward Kmett may have done
something similar (you might even be able to reuse his), but I couldn't
find it just now.

[Richard Wallace]

Awesome! I look forward to seeing this develop into the besr haskell http server library.

One thing I noticed is that you've recreated types for things like http method, headers, status, etc. Any reason not to use the types in http-types for these?  Would you accept a pull request?

Rich

[Jeremy Shaw]

On Sat, Jul 6, 2013 at 3:24 PM, Richard Wallace <rwal...@thewallacepack.net> wrote:

Awesome! I look forward to seeing this develop into the besr haskell http server library.

 sweet!

One thing I noticed is that you've recreated types for things like http method, headers, status, etc. Any reason not to use the types in http-types for these?  Would you accept a pull request?

It's too early to decide on that. Obviously if we can use it that would be good for interoperating with other libraries that use http-types.

- jeremy

 

[Richard Wallace]

On Jul 6, 2013 1:43 PM, "Jeremy Shaw" <jer...@n-heptane.com> wrote:
>
> On Sat, Jul 6, 2013 at 3:24 PM, Richard Wallace <rwal...@thewallacepack.net>

>> One thing I noticed is that you've recreated types for things like http method, headers, status, etc. Any reason not to use the types in http-types for these?  Would you accept a pull request?
>
> It's too early to decide on that. Obviously if we can use it that would be good for interoperating with other libraries that use http-types.
>

It's fair enough that it might not be practical in the long run, but might it be better to start with the intention of using it, helping to evolve it where necessary, and only dropping use of it if it becomes absolutely necessary? I think the interoperability and collaboration with other, similar efforts would be of tremendous benefit for everyone.

Rich

[Gabriel Gonzalez]

> It's fair enough that it might not be practical in the long run, but
> might it be better to start with the intention of using it, helping to
> evolve it where necessary, and only dropping use of it if it becomes
> absolutely necessary? I think the interoperability and collaboration
> with other, similar efforts would be of tremendous benefit for everyone.
>
> Rich
>

I'd like to hear from Andrew Cowie because he didn't use http-types for
http-streams and maybe he can comment on why.

[Jeremy Shaw]

Ok, #1 is done for now I think. But, will still need future improvements.I have hooked into cabal's test and benchmark stuff. So you can now do:

 $ cabal configure --enable-tests --enable-benchmarks

 $ cabal build

 $ cabal test

 $ cabal bench

Soon we will need to actually put in some useful tests and benchmarks. I would also like to figure out how to record, publish, and compare benchmarks so we can tell if we have suddenly lost performance..

I updated to the version of parse-attoparsec that uses ErrorT instead of EitherT.

I have also monkeyed around with the code enough to be able to fake the pong benchmark. The request parser being used is a total hack job and very wrong. But right enough to fake that benchmark ;) I have something special in mind for the request parser, so there is no point in doing any real work on the code that is there -- it is just a dummy parser to hold the space until the real deal comes along.

Next I will likely focus on the stuff in Serve.hs for a bit. Right now it is not easy to do anything of things that should be easy like supply a different parser, supply a different accept loop, add support for HTTPS or and support for web sockets, etc. That doesn't mean the stuff anywhere is correct yet :) Just that I'd like to start with Serve because I think it will dictate some low-level changes.

Part of that likely includes the integration of the pipes-safe stuff. Right now when something goes wrong, nothing good at all happens.

After that, I want to examine the Request and Response types and think about how things should actually work. 

- jeremy

[Andrew Cowie]

On Sat, 2013-07-06 at 14:03 -0700, Gabriel Gonzalez wrote:

> I'd like to hear from Andrew Cowie because he didn't use http-types for
> http-streams and maybe he can comment on why.

Mostly because it's just a bunch of typedefs; there's no ADT for the
HTTP methods, for example. Pretty ugly.

At the time I was working to reuse Snap's types instead; that didn't
quite work out, but I *AM* about to factor the types I used out of
http-streams to be a common dependency for http-streams and pipes-http,
which I'll certainly mention here once I've done. By design they are
already client/server agnostic.

AfC
Sydney

[Richard Wallace]

There are more than just typedefs. There is StdMethod, HttpVersion, and Status. HeaderName and Header are just typedefs, but there are useful functions for working with the types and many of the standard values are defined. I'm curious what a new library would do differently.

Rich

[Jeremy Shaw]

On Sat, Jul 6, 2013 at 3:49 PM, Richard Wallace <rwal...@thewallacepack.net> wrote:

It's fair enough that it might not be practical in the long run, but might it be better to start with the intention of using it, helping to evolve it where necessary, and only dropping use of it if it becomes absolutely necessary? I think the interoperability and collaboration with other, similar efforts would be of tremendous benefit for everyone.

We just need to wait 2-3 weeks. All the types in the code right now are just place holders. I've used the http-types library in some of my packages like web-routes. Deciding if it is right for hyperdrive will happen -- but not quite yet. I can only think about so many things at once :)

- jeremy

[Kẏra ​]

is Hyperdrive still planned to be the server for Happstack/Clckwrks?

[Jeremy Shaw]

Yes! I've been lost of in the lands of client-side framework
development and other distractions. But I think about working on
hyperdrive all the time. Now is actually a good time (for me) to start
up work on it again -- so I'll start work this week. I am going to
blog the development process so that there is a lot of opportunity to
get feedback early on.

- jeremy