Current Stack

[Ryan Riley]

I added links to the current parts of the stack as I see them on the welcome page. These include FHPS -> HttpMachine (I need to talk with Benjamin van der Veen as I finish this) -> Frack (optional) -> Frank (optional) -> WebSharper. Frack is an implementation of OWIN, which seems to still facilitate the possibility of running on multiple server platforms without much loss of readability or excess noise. We'll need to do some perf testing. Frank is still working off of the WCF Web APIs, but I'm about to refactor that to the latest version of Frack.

Adam, we still need to determine how best to use Frank to route requests to WebSharper side-by-side with other handlers.

Dave, have you done anything with HTTP parsing? My HttpMachine lib is following the node and mongrel style parsers that parse parts of the request and execute a callback (currently planned to be an IObservable). I would like to work with you more closely on that as I get more of the parser combinators completed.

I'm really excited to be working with you all on this. Hopefully you'll find my middle layers useful, though I could see starting off with FHPS -> HttpMachine -> WebSharper as an initial goal.

Cheers!
Ryan

[7sharp9]

So what are the plans for a full web stack? Will it be a plugin for
websharper or just an experiment or showcase?

[Ryan Riley]

On Wed, Feb 16, 2011 at 5:40 PM, 7sharp9 <kukulca...@gmail.com> wrote:

So what are the plans for a full web stack?  Will it be a plugin for
websharper or just an experiment or showcase?

Great question. Here are my thoughts:

1. I want to show off F# as the best language on .NET
2. You can build very high performance servers in a minimal amount of code and maintain readability
3. Show off the ease of building DSLs to wire up components, from configuration to UI (frank + WebSharper)

I could go on, as well as add more detail above, but those are my top goals. I also want to use this stuff for rebuilding the Community for F# site.

What do you see as motivations?

Ryan

[7sharp9]

Im probably going to be integrating pipeline processing into fhps at
some point, I may well experiment with a parser implementation in that
too...

[7sharp9]

That was supposed to be a reply to one of the points in the first
post, I post my motivations in the morning, need to get to bed now its
2:30am zzzz

[7sharp9]

I havent done that much HTTP parsing but Ill get up to speed on that
when I get a chance.

As for my motivations:

I suppose to be noticed in the F# sphere.
Increase my prospects of working in F# for real money, its really very
niche at the moment.
Ill probably blog about it too at some point.
I enjoy F# so working on it will be fun.

Im quite enjoying writing the articles although my latest ones on
pipelines hasn't really had much interest which is disappointing :(

D.

[7sharp9]

What kind of performance do you get from http machine is it efficient?

[Ryan Riley]

On Thu, Feb 17, 2011 at 1:55 PM, 7sharp9 <kukulca...@gmail.com> wrote:

What kind of performance do you get from http machine is it efficient?

I don't really have any way to compare it yet. I am trying to finish the http parser, then I'll get with Benjamin to compare his Ragel-based parser with mine. I don't think it will take me too long now that I finally understand the correct way to combine the parser combinators.

[7sharp9]

What are you using?

On Feb 17, 10:46 pm, Ryan Riley <ryan.ri...@panesofglass.org> wrote:

[Ryan Riley]

On Thu, Feb 17, 2011 at 2:57 PM, 7sharp9 <kukulca...@gmail.com> wrote:

What are you using?

Do you mean for a parsing library or performance testing? I haven't looked at perf testing tools. I'm using the cashel library as a base for parsing. For reference, Benjamin has an HttpMachine, as well, based on node.js and Manos de Mono, as well as Mongrel2, which also uses Ragel.

Ryan

[Dave Thomas]

I have used loads of perf testing tools don't worry about those, I meant Cashel, for some reason I was searching for Cashew :~  lol

Have you got a quick brain dump or reference project that will give me a high level over view that I can be thinking about?

I guess you want me to finish FHPS too, Is Cachel working using a stream FHPS produces a payload at a time based on the contents of the pinned memory buffers in the TCP stack.

Theres FParsec too although the documentation looks to have a steep learning curve...

D.

[Ryan Riley]

On Thu, Feb 17, 2011 at 7:01 PM, Dave Thomas <kukulca...@gmail.com> wrote:

I have used loads of perf testing tools don't worry about those, I meant Cashel, for some reason I was searching for Cashew :~  lol

Had some of those as a snack today. ;)
 

Have you got a quick brain dump or reference project that will give me a high level over view that I can be thinking about?

I've got specs in my Cashel repo that might help.
 

I guess you want me to finish FHPS too, Is Cachel working using a stream FHPS produces a payload at a time based on the contents of the pinned memory buffers in the TCP stack.

I set up my HttpMachine implementation (well, the UriParser, anyway) to use ArraySegment<byte>, which looked to be a good corollary to the SocketAsyncEventArgs. My thought was that I could take the SAEA buffer, offset and count, and create a related ArraySegment. The parser progresses through the ArraySegment as it would use functional lists. Again, tests are available to demonstrate that in Cashel. Along with updating to the newer naming conventions and .NET 4.0, the ArraySegment combinators were part of my contribution.
 

Theres FParsec too although the documentation looks to have a steep learning curve...

FParsec is very powerful, uses Streams of chars with a custom iterator, and is just amazing... until you want to do something asynchronously. It's highly optimized for synchronous parsing. That's the primary reason I went with Cashel. When I started with Cashel, I actually converted the parser monad to wrap an async at first, but then I realized I was releasing the thread all over the place when I didn't want to do that.

Ryan

[Dave Thomas]

I guess you want me to finish FHPS too, Is Cachel working using a stream FHPS produces a payload at a time based on the contents of the pinned memory buffers in the TCP stack.

I set up my HttpMachine implementation (well, the UriParser, anyway) to use ArraySegment<byte>, which looked to be a good corollary to the SocketAsyncEventArgs. My thought was that I could take the SAEA buffer, offset and count, and create a related ArraySegment. The parser progresses through the ArraySegment as it would use functional lists. Again, tests are available to demonstrate that in Cashel. Along with updating to the newer naming conventions and .NET 4.0, the ArraySegment combinators were part of my contribution.

The problem may be that under load you get one buffer full of a message and there will be another almost immediately afterwards.  If you are parsing half a message does it cope well with this?  \

One of the reasons I pipelined the output was that there were stages which split messages up producing a payload which was deemed to be one complete message, I was dealing with a different domain though, SDR (Short Data Router) messages which were 140bytes in size, and raw xml in another instance.

 

Theres FParsec too although the documentation looks to have a steep learning curve...

FParsec is very powerful, uses Streams of chars with a custom iterator, and is just amazing... until you want to do something asynchronously. It's highly optimized for synchronous parsing. That's the primary reason I went with Cashel. When I started with Cashel, I actually converted the parser monad to wrap an async at first, but then I realized I was releasing the thread all over the place when I didn't want to do that.

Im trying to get a picture in my mind of how it would work asynchronously, are different threads working on different pattern matches, which are then combined?

Dave.

[7sharp9]

Oh yeah the FHPS interface will return you an
Action<ArraySegment<Byte>> sort of thing anyway, there a function
hooks for connect, disconnect, sent, receive.

The implementation on my blog had all that removed for clarity.

[Ryan Riley]

The idea I'm following is a state machine parser that returns once a certain portion of the message is matched. So you would need a state machine parser instance per connection. The connection would pass each segment it receives to the parser state machine, which would run on each segment, then take any leftover bytes and add them into its next pass. The parser needs to also respond or return each piece as it is returned, so the app using the connection will be receiving status line, headers, etc. as they are parsed and will be able to manage what it wants and how it wants to proceed, i.e. pause, parse other, etc. I'm still trying to understand the whole approach, but the state machine makes sense to me. I'm planning to use a MailboxProcessor for it, though I might use a continuation-based machine. I'm working on that right now, actually. :)

Ryan

[7sharp9]

It depends on how heavyweight the state machine parser is and how long
the uncompleted data lingers in there to how scalable and performant
the solution is.

If you have 300 connections on a single host what would that mean
thread wise for the parsing, is it a single thread per parse, or are
there multiple?

So your saying that there are delegates or functions attached to the
parser:

Status
Header
Body

What are the typical downstream actions associated with these that you
have encountered so far?

My particular approach to this would still be pipelines, although I
probably need to look through a complete end to end application...

[Adam Granicz]

Hi guys, catching up on emails... Ok, so my motivations are:

* Connect up WebSharper sitelets with a fully F#-based web stack
(Frack + Frank)
* Eliminate the need for IIS to run WebSharper applications
* Increase the adoption of WebSharper
* All-in-all drive people to develop web applications with F#

Our piece in this puzzle is establishing a connector for expressing
first-class web sites (sitelets), web forms and UI in general
(formlets, flowlets), and web functionality/behavior (pagelets) on the
Frack/Frank stack - and thus enable developing markupless, dynamic,
client-based web and mobile applications with nothing but F# code.
This will change the way people write web applications, not only
because it's cool and easy, but also because of the strong impact
functional programming carries. Being able to write client-side code
in F# is a strong point (and the wealth of WebSharper connectors to
third-party JavaScript libraries is another), but writing the backend
logic side-by-side as well without having to worry about how to
interface with the client-side is a clear and unbeatable benefit.

One test case I would like to create is a visualization-heavy,
enterprise-grade application that uses asynchronous and higher-order
functional programming to process data obtained from various sources,
serve it to clients in an interactive and visually highly appealing
way, enable access over mobile phones, and allow for thousands of
concurrent users. This should be under 1000 lines of nothing but F#
code.

--
Adam Granicz, IntelliFactory
www.intellifactory.com

[7sharp9]

The 1000 lines in question is user code? not the API code to allow
it?

The FHPS project is currently around 200 lines as it stands, I cant
help feeling that there is a more elegant way of constructing the FHPS
code.

What do we need for the connecter between sitelets and Frack/Frank, I
guess we splice together what we have and they look at streamlining
while keeping to the Interfaces in OWIN, or is that open to
modification?

I need to get up to speed on Websharper too, I have it installed on my
VM but I think the project files were all missing or something.

In terms of the test case Im all for data visualisation experiments, I
got a penchant fro Neuromancer type data imagery :)

D.

[Adam Granicz]

The 1000 lines is the user code (all F#, except for the designer
template - that's in Xml and auto-converted to F# code) - all the code
a user has to write to use WebSharper and the related extensions to
implement this application. We would likely use Infovis or Protovis
to visualize data (or anything we can wrap reasonably easy), these are
available as WebSharper extensions already. Usually, similar small
demo-type apps are around a couple hundred lines, but here I would be
aiming to implement an enterprise-grade application that could even
sell (perhaps visualizing stock prices, or whatever), hence the larger
code base - but still, darned impressive compared to similar ASP.NET
or MVC apps that are at least an order of magnitude larger in code
size. The beauty will be mostly in how the data is retrieved,
filtered/mapped, tranformed, and then visualized.

Re the sitelets to Frack/Frank connector - on the high-level, we need
a function that lifts a sitelet to an application that Frack can
serve.

Sitelets have strong URL management capabilities (safe and computed
URLs, infering from action types, etc.) and can also match (in a
type-safe way) arbitrary requests (and respond to anything from plain
GET requests to full RESTful services) - so they offer the "full
enchilada" when it comes to defining dynamic sites.

--
Adam Granicz, IntelliFactory
www.intellifactory.com

[Ryan Riley]

On Sat, Feb 19, 2011 at 3:41 PM, 7sharp9 <kukulca...@gmail.com> wrote:

What do we need for the connecter between sitelets and Frack/Frank, I
guess we splice together what we have and they look at streamlining
while keeping to the Interfaces in OWIN, or is that open to
modification?

I'm ripping out OWIN, so we don't need to work with that. Current signature is IDictionary<string, obj> -> Async<string * IDictionary<string, string> * seq<obj>>. I'm thinking of moving the final seq<obj> to either AsyncSeq<obj> or AsyncSeq<ArraySegment<byte>>. The latter directly correlates with both the request body and the AsyncSeqEventArgs.

Ryan

[Ryan Riley]

On Sat, Feb 19, 2011 at 4:11 PM, Adam Granicz <adam.g...@gmail.com> wrote:

The 1000 lines is the user code (all F#, except for the designer
template - that's in Xml and auto-converted to F# code) - all the code
a user has to write to use WebSharper and the related extensions to
implement this application.  We would likely use Infovis or Protovis
to visualize data (or anything we can wrap reasonably easy), these are
available as WebSharper extensions already.  Usually, similar small
demo-type apps are around a couple hundred lines, but here I would be
aiming to implement an enterprise-grade application that could even
sell (perhaps visualizing stock prices, or whatever), hence the larger
code base - but still, darned impressive compared to similar ASP.NET
or MVC apps that are at least an order of magnitude larger in code
size.  The beauty will be mostly in how the data is retrieved,
filtered/mapped, tranformed, and then visualized.

Re the sitelets to Frack/Frank connector - on the high-level, we need
a function that lifts a sitelet to an application that Frack can
serve.

Can we get a sitelet to fit into IDictionary<string, obj> (* request *) -> Async<string (* status code *) * IDictionary<string, string> (* headers *) * seq<obj> (* body *)> or the same with AsyncSeq<ArraySegment<byte>> for the body?
 

Sitelets have strong URL management capabilities (safe and computed
URLs, infering from action types, etc.) and can also match (in a
type-safe way) arbitrary requests (and respond to anything from plain
GET requests to full RESTful services) - so they offer the "full
enchilada" when it comes to defining dynamic sites.

Two options for Frank: allow the sitelet to register its routes or allow the user to load a sitelet into a uri-space, e.g. domain is www.websharper.com and uri-space for the sitelet is under /mysitelet.

Ryan

[Dave Thomas]

If we want to use a config based setup then I guess registering routes maybe the way to go.

Ryan if you get chance could you do me a zip of what you have so far?

Im on the train to London tonight so I might crack on with FHPS for an hour or two.  At the minute Im using an Assembla subversion repo for that.

D.

[Ryan Riley]

Have so far with regard to what? Frack? Frank? Both? If you've pulled them from github, that's the latest working bits. I'm working on demos for a talk next weekend today.

Sent from my iPhone

[Dave Thomas]

I tried to build frank but was missing references even after the git commands you gave and trying to build everything.

D.

[Ryan Riley]

On Tue, Feb 22, 2011 at 3:33 PM, Dave Thomas <kukulca...@gmail.com> wrote:

I tried to build frank but was missing references even after the git commands you gave and trying to build everything.

D.

I'll have to look into it. It's been awhile as I've been focusing primarily on frack. I think that's almost ready. I need to work out some kinks; then I'll get to frank. I still need to finish my parser pieces, though, as they are becoming more necessary in other spots. Lots of work to do. I didn't realize how much more I actually had until this became something of a core component. :) Any chance of getting sitelets running directly on top of your FHPS?

Ryan

[Dave Thomas]

Im in London at the moment with only my phone for internet connection, just sorted out a flat to rent for 6 months so me and the wife can move down here while I work. 

As soon as that's sorted then I can get stuck in.  until then it will just be drips and drabs.  I don't currently have a working websharper either. 

While I was travelling down on the train I just couldn't figure out how to design the pipeline that I'm working on, it feels like I have slipped back into imperative design mode, I can see it plain as day in that mode of thought not sure it fits functional design... :(

D.

[Adam Granicz]

Can you guys send me what errors/issues you are getting with
WebSharper? We are aware of a Win7 issue around a failing obfuscated
DLL, but so far haven't been able to find a reasonable obfuscator that
can be substituted. If your machine is not a Win7 machine I
definitely would like to know about any other issues so we can
investigate.

--
Adam Granicz, IntelliFactory
www.intellifactory.com