Windows support in master

[Jackson Harper]

Hey guys got some awesome news,

I pushed some patches from Carlo Kok yesterday to get Manos working on
Windows. He's bundled up libev with Manos and included instructions
for how to build on Windows in the README.

I haven't had a chance to try it yet myself, but it looks like a
pretty simple process based on his instructions.

Please give it a shot and email the list if you have any problems.

Cheers,
Jackson

[Jackson Harper]

Just as a little update to this i changed the DISABLE_SENDFILE define
to DISABLE_POSIX and wrapped the seteuid call in manos.exe with that
too. So I think manos.exe should build properly without Mono.Posix
now.

Jackson

[Tadeusz Wójcik]

Hi, I've tried to run Manos on Windows 7 but had some problems.
Firstly I couldn't build project because of Error error C1083: Cannot
open include file: 'config.h': No such file or directory d:\git\manos
\src\libev-4.00\ev.c . I've copied config.h to libev-4.0.0 folder and
build succesfully project. When tried to run server I get :

Unhandled Exception: System.TypeInitializationException: The type
initializer fo
'Manos.Tool.Driver' threw an exception. --->
System.TypeLoadException: Could n
t load type 'Libev.Loop' from assembly 'manos, Version=0.0.0.0,
Culture=neutral
PublicKeyToken=null'.
at Manos.Tool.Environment..ctor()
at Manos.Tool.Driver..cctor() in D:\git\manos\src\manos.exe
\Driver.cs:line 43

--- End of inner exception stack trace ---
at Manos.Tool.Driver.Main(String[] args)


Maybe I'm doing something wrong...

[Jackson Harper]

2010/11/3 Tadeusz Wójcik <tadeus...@gmail.com>:

>
> Unhandled Exception: System.TypeInitializationException: The type
> initializer fo
>  'Manos.Tool.Driver' threw an exception. --->
> System.TypeLoadException: Could n
> t load type 'Libev.Loop' from assembly 'manos, Version=0.0.0.0,

Is libev.dll in the same directory as manos.exe and Manos.dll ?

[Andrew Tierney]

Hi,

Any thoughts on implementing something like the following with Manos ?

==========================================================

BigPipe: Pipelining web pages for high performance

Site speed is one of the most critical company goals for Facebook. In 2009, we successfully made Facebook site twice as fast, which was blogged in this post. Several key innovations from our engineering team made this possible. In this blog post, I will describe one of the secret weapons we used called BigPipe that underlies this great technology achievement.

BigPipe is a fundamental redesign of the dynamic web page serving system. The general idea is to decompose web pages into small chunks called pagelets, and pipeline them through several execution stages inside web servers and browsers. This is similar to the pipelining performed by most modern microprocessors: multiple instructions are pipelined through different execution units of the processor to achieve the best performance. Although BigPipe is a fundamental redesign of the existing web serving process, it does not require changing existing web browsers or servers; it is implemented entirely in PHP and JavaScript.

Motivation

To understand BigPipe, it's helpful to take a look at the problems with the existing dynamic web page serving system, which dates back to the early days of the World Wide Web and has not changed much since then. Modern websites have become dramatically more dynamic and interactive than 10 years ago, and the traditional page serving model has not kept up with the speed requirements of today's Internet. In the traditional model, the life cycle of a user request is the following:

1. Browser sends an HTTP request to web server.
   
2. Web server parses the request, pulls data from storage tier then formulates an HTML document and sends it to the client in an HTTP response.
   
3. HTTP response is transferred over the Internet to browser.
   
4. Browser parses the response from web server, constructs a DOM tree representation of the HTML document, and downloads CSS and JavaScript resources referenced by the document.
   
5. After downloading CSS resources, browser parses them and applies them to the DOM tree.
   
6. After downloading JavaScript resources, browser parses and executes them.
   

The traditional model is very inefficient for modern web sites, because a lot of the operations in the system are sequential and can’t be overlapped with each other. Some optimization techniques such as delaying JavaScript downloading, parallelizing resource downloading etc. have been widely adopted in the web community to overcome some of the limitations. However, very few of these optimizations touch the bottleneck caused by the web server and browser executing sequentially. When the web server is busy generating a page, the browser is idle and wasting its cycles doing nothing. When web server finishes generating the page and sends it to the browser, the browser becomes the performance bottleneck and the web server cannot help any more. By overlapping the web server’s generation time with the browser’s rendering time, we can not only reduce the end-to-end latency but also make the initial response of the web page visible to the user much earlier, thereby significantly reducing user perceived latency.

The overlapping of web server generation time and browser’s render time is especially useful for content-rich web sites like Facebook. A typical Facebook page contains data from many different data sources: friend list, new feeds, ads, and so on. In a traditional page rendering model a user would have to wait until all of these queries have returned data before building the final document and sending it to the user's computer. One slow query holds up the train for everything else.

How BigPipe works

To exploit the parallelism between web server and browser, BigPipe first breaks web pages into multiple chunks called pagelets. Just as a pipelining microprocessor divides an instruction’s life cycle into multiple stages (such as “instruction fetch”, “instruction decode”, “execution”, “register write back” etc.), BigPipe breaks the page generation process into several stages:

1. Request parsing: web server parses and sanity checks the HTTP request.
   
2. Data fetching: web server fetches data from storage tier.
   
3. Markup generation: web server generates HTML markup for the response.
   
4. Network transport: the response is transferred from web server to browser.
   
5. CSS downloading: browser downloads CSS required by the page.
   
6. DOM tree construction and CSS styling: browser constructs DOM tree of the document, and then applies CSS rules on it.
   
7. JavaScript downloading: browser downloads JavaScript resources referenced by the page.
   
8. JavaScript execution: browser executes JavaScript code of the page.
   

The first three stages are executed by the web server, and the last four stages are executed by the browser. Each pagelet must go through all these stages sequentially, but BigPipe enables several pagelets to be executed simultaneously in different stages.

Pagelets in Facebook home page. Each rectangle corresponds to one pagelet.

The picture above uses Facebook’s home page as an example to demonstrate how web pages are decomposed into pagelets. The home page consists of several pagelets: “composer pagelet”, “navigation pagelet”, “news feed pagelet”, “request box pagelet”, “ads pagelet”, “friend suggestion box” and “connection box”, etc. Each of them is independent of each. When the "navigation pagelet" is displayed to the user, the "news feed pagelet" can still be being generated at the server.


In BigPipe, the life cycle of a user request is the following: The browser sends an HTTP request to web server. After receiving the HTTP request and performing some sanity check on it, web server immediately sends back an unclosed HTML document that includes an HTML <head> tag and the first part of the <body> tag. The <head> tag includes BigPipe’s JavaScript library to interpret pagelet responses to be received later. In the <body> tag, there is a template that specifies the logical structure of page and the placeholders for pagelets. For example:


<div>
<div id=”left_column”>
<div id=”pagelet_navigation”></div>
</div>
<div id=”middle_column”>
<div id=”pagelet_composer”></div>
<div id=”pagelet_stream”></div>
</div>
<div id=”right_column”>
<div id=”pagelet_pymk”></div>
<div id=”pagelet_ads”></div>
<div id=”pagelet_connect”></div>
</div>
</div>


After flushing the first response to the client, web server continues to generate pagelets one by one. As soon as a pagelet is generated, its response is flushed to the client immediately in a JSON-encoded object that includes all the CSS, JavaScript resources needed for the pagelet, and its HTML content, as well as some meta data. For example:


<script type="text/javascript">
big_pipe.onPageletArrive({id: “pagelet_composer”, content=<HTML>, css=[..], js=[..], …})
</script>


At the client side, upon receiving a pagelet response via “onPageletArrive” call, BigPipe’s JavaScript library first downloads its CSS resources; after the CSS resources are downloaded, BigPipe displays the pagelet by setting its corresponding placeholder div’s innerHTML to the pagelet’s HTML markup. Multiple pagelets’ CSS can be downloaded at the same time, and they can be displayed out-of-order depending on whose CSS download finishes earlier. In BigPipe, JavaScript resource is given lower priority than CSS and page content. Therefore, BigPipe won’t start downloading JavaScript for any pagelet until all pagelets in the page have been displayed. After that all pagelets’ JavaScript are downloaded asynchronously. Pagelet’s JavaScript initialization code would then be executed out-of-order depending on whose JavaScript download finishes earlier.

The end result of this highly parallel system is that several pagelets are executed simultaneously in different stages. For example, the browser can be downloading CSS resources for three pagelets while rendering the content for another pagelet, and meanwhile the server is still generating the response for yet another pagelet. From the user’s perspective, the page is rendered progressively. The initial page content becomes visible much earlier, which dramatically improves user perceived latency of the page. To see the difference for yourself, you can try the following links: Traditional model and BigPipe. The first link renders the page in the traditional single flush model. The second link renders the page in BigPipe’s pipeline model. The difference between the two pages’ load times will be much more significant if your browser version is old, your network speed is slow, and your browser cache is not warmed.

Performance results

The graph below shows the performance data comparing the 75th percentile user perceived latency for seeing the most important content in a page (e.g. news feed is considered the most important content on Facebook home page) on traditional model and BigPipe. The data is collected by loading Facebook home page 50 times using browsers with cold browser cache. The graph shows that BigPipe reduces user perceived latency by half in most browsers.

Time to interact latency on Facebook's home page

It is worth noting that BigPipe was inspired by pipelining microprocessors. However, there are some differences between the pipelining performed by them. For example, although most stages in BigPipe can only operate on one pagelet at a time, some stages such as CSS downloading and JavaScript downloading can operate on multiple pagelets simultaneously, which is similar to superscalar microprocessors. Another important difference is that in BigPipe, we have implemented a ‘barrier’ concept borrowed from parallel programming, where all pagelets have to finish a particular stage, e.g. pagelet displaying stage, before any one of them can proceed further to download JavaScript and execute them.

At Facebook, we encourage thinking outside the box. We are constantly innovating on new technologies to make our site faster.

[Tadeusz Wójcik]

Hi,
Yes, threa all are in the sam dir (in build directory), maybe I
compiled something wrong? I also used DISABLE_POSIX flag maybe thats
the problem

2010/11/3 Jackson Harper <jack...@gmail.com>:

[Tadeusz Wójcik]

Hi,
I think I can't run manos because of 64bit version of my Windows 7? Is
it even possible to run it on x64 OS ?

W dniu 4 listopada 2010 08:45 użytkownik Tadeusz Wójcik
<tadeus...@gmail.com> napisał:

[Jackson Harper]

Yes, I do all my development and testing on 64bit Linux.

It could be the problem on windows though, maybe the libev wrapper
stuff isn't getting compiled properly.

2010/11/4 Tadeusz Wójcik <tadeus...@gmail.com>:

[Christopher Deutsch]

I get the same error (64bit Windows 7).

I'm guessing it's an issue with compiling libev.

Out of the box the compile fails with the error "cannot open include file: 'config.h'"

If I put a config.h file in the spot Visual Studio is looking, it compiles successfully but I'm guessing the correct flags aren't being set or something.

Christopher

2010/11/3 Tadeusz Wójcik <tadeus...@gmail.com>

[Tadeusz Wójcik]

I wonder if it would be possible to provide compiled libev dll somehow....

2010/11/6 Christopher Deutsch <crde...@gmail.com>:

[Jackson Harper]

Eventually there will be packages.

Please remember Manos isn't even at 0.1 right now. For the most part,
people that are downloading and compiling it are either playing around
with it or contributing back to the project. Its not ready for
general consumption.

Cheers,
Jackson

2010/11/6 Tadeusz Wójcik <tadeus...@gmail.com>:

[Ged Wed]

This thread seems to have turned into a C discussion.

But about BigPipe.

Yes i agree its very smart way. Its also a good design fro larger web sites because it allow CMS and Portal style of architectecture.

For example, say you have a website and you want to add a blog into it.

You could make a portle representing the blog, and then just call the blog inside a hidden IFrame.

Yes i knwo IFrames are Evil, but Google uses it alot.

Anyway i am digressing.

I fully support big pipe.

Its a very old pattern that i developed 10 years ago before ajax was invented using Iframes as the transport between client and server.

Oh i am sounding like an old fart now.

There is a decent implementation over here now:

https://github.com/dynamoid/bigpipe

I can see 2 low hanging fruit improvements also:

1. Client side resource dependency tracking. This is pretty easy using an array on the client to track what's loaded, and dont load it if its already in.

- It also means that the way you design your css is scoped to only affect the html inside your pagelet. Its a nice pattern anyway for encapsulation

2. Server side make a formal pattern, where by you can describe a pagelets resource requirements and even routing.

- this is an obvious little thing, but its nice.

- You could then save the data that drives this pattern in a DB, and before you know it you will have a decent basic CMS.

But this is too adventurous for now, and i think just having an example implementation as part of Manos examples would be nice.

I don't think it should be at all re-factored into Manos; thats just making the same old monolithic mistake that MS does, but i do think that manos should be capable of supporting this example.

Regards

Gerard

[Ged Wed]

I just pulled Master and got an error trying to compile the C code.

Error 2 error C1083: Cannot open source file: '..\libev-4.00\ev.c': No such file or directory C:\Documents and Settings\Administrator\Desktop\jacksonh-manos-ee505a8\src\libev\c1 libev

Windows Server 2003

No Mono installed

Visual Studio 2010.

Any ideas people ??

[Jackson Harper]

Did you follow this part of the windows instructions?

Download the libev package from: http://software.schmorp.de/pkg/libev.html

Extract it and place it next to the libev in the source dir, so
both libev-4.00 and libev are in the same directory.

Make sure you get the libev-4.00 package. Eventually I'll probably
bundle this source with manos, just to make the build a little easier
for windows people.

[Ged Wed]

Yes got it now working.

Am pushing all these developer issues via GitHub issues so others can stand on my giant shoulders :)

Ged

[Ged Wed]

Ok so next problem is the Posix references.

I dont have Mono and i am on Windows.

So, should i just install it and be done with it for now OR wait for a miracle workaround ?

Regards

Ged

[Jackson Harper]

Just define DISABLE_POSIX

[Ged Wed]

Oh yeah i forgot. Ok will give it a go.

G

[Ged Wed]

works like a charm..

G

[Henning]

I'm having problems building Manos on my 32bit windows 7 system. Using
VS 2010.
Just pulled the newest manos source and libev-4.00. Firstly, the Manos
project didn't properly reference Mono.CSharp.dll and Manos.Tests
didn't reference nunit.framework.ddl properly causing the build to
fail. After adding a reference, I now get the error Cannot open
include file: 'syscall.h': No such file or directory c:\source\manos
\src\libev-4.00\ev.c

What am I doing wrong?

Henning

[Jackson Harper]

master probably wont build on windows right now, the build is in the
midsts of being completely redone, partly to make it easier for
windows users.

Here is a tarball of a recent windows build I did, should work fine:

http://dl.dropbox.com/u/234197/jacksonh-manos-17d8cfe.zip

Good luck,
Jackson

[Henning]

Awesome. I'll check it out tomorrow.

Henning