[erlang-questions] Erlang Web Servers challenge
Zabrane Mickael
Loïc Hoguin
> Hi guys,
>
> I'd like your help for this small Web servers challenge.
>
> The idea is ver simple: measure "how fast" you can serve a "100 bytes"
> file (attached as "small.html")
> in "less than 01 second" by using one of the following web servers:
>
> - Yaws (http://yaws.hyber.org/download/yaws-1.90.tar.gz)
> - Mochiweb (https://github.com/mochi/mochiweb)
> - Misultin (https://github.com/ostinelli/misultin)
> - Cowboy (https://github.com/extend/cowboy)
> - Bare metal web server using "gen_tcp"
I don't really understand where the challenge is.
Also yaws should be faster considering it supports sendfile whereas the
others don't without much additional work beforehand. So regardless of
the server being used you'd probably just be benchmarking sendfile and
not the servers.
--
Loïc Hoguin
Dev:Extend
_______________________________________________
erlang-questions mailing list
erlang-q...@erlang.org
http://erlang.org/mailman/listinfo/erlang-questions
Zabrane Mickael
The challenge as stated is very really simple.
How fast you can serve a 100 bytes files with your web server "Cowboy" for example in less than a second.
> Also yaws should be faster considering it supports sendfile whereas the
> others don't without much additional work beforehand. So regardless of
> the server being used you'd probably just be benchmarking sendfile and
> not the servers.
Don't suppose anything (sendfile) before benchmarking it Loïc.
Loïc Hoguin
>>> I don't really understand where the challenge is.
>
> The challenge as stated is very really simple.
> How fast you can serve a 100 bytes files with your web server "Cowboy" for example in less than a second.
Only yaws is suited for this "challenge", the others aren't meant to
serve files and would be better coupled with something like cherokee or
nginx deployed on multiple servers each using separate domains.
Even yaws with sendfile would probably be beaten by the C servers for
that purpose. Though I guess measuring that one would be interesting.
>> Also yaws should be faster considering it supports sendfile whereas the
>> others don't without much additional work beforehand. So regardless of
>> the server being used you'd probably just be benchmarking sendfile and
>> not the servers.
>
> Don't suppose anything (sendfile) before benchmarking it Loïc.
I don't suppose anything. Kernel cache is faster than userland cache.
--
Loïc Hoguin
Dev:Extend
Zabrane Mickael
Also yaws should be faster considering it supports sendfile whereas theothers don't without much additional work beforehand. So regardless ofthe server being used you'd probably just be benchmarking sendfile andnot the servers.Don't suppose anything (sendfile) before benchmarking it Loïc.
I don't suppose anything. Kernel cache is faster than userland cache.
Joe Armstrong
> Hi guys,
> I'd like your help for this small Web servers challenge.
> The idea is ver simple: measure "how fast" you can serve a "100 bytes" file
> (attached as "small.html")
Why measure this? Serving static content is one of the least
interesting things I can think of
to study. Any network worth the name won't bother to access the "root"
server but will cache the
content upstream.
How fast you can serve up dynamic content is far more interesting,
since it can't be cached.
Something that evokes a computation on the server is far more
interesting, and something that stresses
the concurrency.
How about processing send a GET /fac/N command (ie return factorial N)
The server should
1) send factorial N
2) wait 30 seconds
3) send factorial N (again)
4) close the socket
The 30 seconds gap will cause lots of nastiness in the server. Such a
benchmark tests many useful things
(for example the inter-session protection mechanism, as soon as we
perform computation on behalf of
the client we have to protect server sessions against each other,
secondly handling concurrency. If we wait
30 sections and do 10K requests/sec, we going to build up 300K
processes, which tests a lot of internal stuff)
/Joe
> in "less than 01 second" by using one of the following web servers:
> - Yaws (http://yaws.hyber.org/download/yaws-1.90.tar.gz)
> - Mochiweb (https://github.com/mochi/mochiweb)
> - Misultin (https://github.com/ostinelli/misultin)
> - Cowboy (https://github.com/extend/cowboy)
> - Bare metal web server using "gen_tcp"
> The rules are very simple :
> 1. The first time you served the file, you MUST read it from disk.
> All subsequent requests to that file can use any caching mechanism!
> 2. Any (hidden) trick is allowed (eg. prim_inet, SMP enabled ...)
> 3. Erlang versions R14B to R14B03
> 4. The HTTP client to use for this benchmark needs to be "Apache AB"
> (http://httpd.apache.org/docs/2.0/programs/ab.html) with:
> $ ab -n 200000 -c 100 -t1 -k "http://127.0.0.1:8080/small.html"
> I hope that the authors if these web servers will also contribute to this
> bench as they
> know their software better than anybody else.
> Challenge's open ;-)
> Regards,
> Zabrane
>
>
Zabrane Mickael
> interesting things I can think of
> to study. Any network worth the name won't bother to access the "root"
> server but will cache the content upstream.
I disagree. That's not what the challenge is about.
Why don't just try it and avoid (useless) questions!
> How fast you can serve up dynamic content is far more interesting,
> since it can't be cached.
This will be part of my second challenge of course.
For now, let's stick with the first challenge rules please.
Regards
/Zab
Amy Lear
>
> > Why measure this? Serving static content is one of the least
> > interesting things I can think of
> > to study. Any network worth the name won't bother to access the "root"
> > server but will cache the content upstream.
>
> I disagree. That's not what the challenge is about.
> Why don't just try it and avoid (useless) questions!
These aren't useless questions. They're genuine concerns about the
validity and applicability of your 'challenge'.
As Joe notes, anything that doesn't have a short TTL can be presumed
to be cached for at least some of the clients at least some of the
time. In some cases, it could be all of the time for some of the
clients; most significant ISPs make some attempt at this to limit
outside bandwidth. Caching within the network is often cheaper, after
all.
For some time now, the useful places to be applying effort are
non-trivial results, where the actual content being received is either
in flux, or a direct result of the client's actions.
You're wanting to measure something that's inaccurate, unnecessary,
and uninformative.
Could you give us a problem space defininition covering what exactly
you're trying to achieve?
Jesper Louis Andersen
> The idea is ver simple: measure "how fast" you can serve a "100 bytes" file
> (attached as "small.html")
What are you trying to show with this benchmark? That is, what is the
goal of it? Serving a static file is something nginx and a couple of
Haskell HTTP servers are very good at (see the Warp-project). The main
problem is that what you are trying to show might not be reflected
very well from your test.
> The rules are very simple :
> 4. The HTTP client to use for this benchmark needs to be "Apache AB"
> (http://httpd.apache.org/docs/2.0/programs/ab.html) with:
> $ ab -n 200000 -c 100 -t1 -k "http://127.0.0.1:8080/small.html"
This violates Rule 1 in Mark Nottingham's http benchmark advice:
http://www.mnot.net/blog/2011/05/18/http_benchmark_rules
Quote: "The most common mistake I see people making is benchmarking a
server on the same box where the load is generated. This doesn’t just
put your results out a little bit, it makes them completely
unreliable, because the load generator’s “steal” of resources varies
depending on how the server handles the load, which depends on
resource availability."
Is there any specific reason why you want to break that advice? Also,
Steve Vinoski did a talk at the recent Erlang Factory on Yaws and the
last part are benchmark and performance advice:
http://www.erlang-factory.com/upload/presentations/409/vinoski-ef-london-2011.pdf
I am with Joe here. Dynamically generated content is much more
interesting for real problems. Static files can be outsourced to a CDN
or Varnish easily. The reason you still serve them from the same
webserver is convenience.
--
J.
Zabrane Mickael
What are you trying to show with this benchmark? That is, what is the
goal of it?
Serving a static file is something nginx and a couple of
Haskell HTTP servers are very good at (see the Warp-project).
This violates Rule 1 in Mark Nottingham's http benchmark advice:
http://www.mnot.net/blog/2011/05/18/http_benchmark_rules
Ivan Uemlianin
>> What are you trying to show with this benchmark? That is, what is the
>> goal of it?
>
> At the end of the day, I'll try to compare Erlang Web servers with others
> competitors (not written in Erlang).
>
>> Serving a static file is something nginx and a couple of
>> Haskell HTTP servers are very good at (see the Warp-project).
>
> Please, just follow and stick to the rules.
> Nginx or Varnish (https://www.varnish-cache.org/) or any HTTP
> accelerator are out of this challenge scope.
Why is nginx outside of the challenge scope? If you want "to compare
Erlang Web servers with other competitors (not written in Erlang)",
surely nginx is one of the main competitors?
Best wishes
Ivan
--
============================================================
Ivan A. Uemlianin
Speech Technology Research and Development
iv...@llaisdy.com
www.llaisdy.com
llaisdy.wordpress.com
www.linkedin.com/in/ivanuemlianin
"Froh, froh! Wie seine Sonnen, seine Sonnen fliegen"
(Schiller, Beethoven)
============================================================
Zabrane Mickael
Why is nginx outside of the challenge scope?
If you want "to compare Erlang Web servers with other competitors (not written in Erlang)", surely nginx is one of the main competitors?
Joe Armstrong
Because if you propose a new game then people will question and try to
change the rules.
That's how baseball got invented, and I know cricket is far better,
but that is a bit off-topic here.
In the particular case just about the last thing I want to know about
is how fast I can serve static content - there is an entire industry devoted
to exactly this question.
Erlang was designed for building fault-tolerant systems, it was *not* designed
for building fast systems. I would expect therefore that it would do badly
at things that non-fault tolerant systems are good at (like serving
static content).
(( I'm sure we could build a fast content delivery network in Erlang -
but this is not easy - the low hanging
fruit was taken year ago - and it's got nothing to do with Erlang, its
all about algorithms, caching, locality
peering agreements etc.))
Like it or not, the results of such a measurement will be
misinterpreted and abused.
The fastest ever Erlang web server should be way slower than the
fastest non-fault
tolerant web server.
A result of the form "the fastest Erlang web server is way slower than
the fastest non fault-tolerant
server" will be widely quoted and blogged, and the results will be
taken out-of-context.
For me I like to "make things work" before I "make them fast" - so the
question of exactly what happens
under stress, just before things fail is extremely interesting.
I am very interested to know exactly how and when things fail when you
stress them.
Sending 100 bytes from a file seems to measure nothing interesting.
Since it's the same file
I expect it to be read *once* then cached in memory. It's so small
that I expect no IP fragmentation at all.
You won't really be measuring the performance of the web server, but
rather the perfomance of
a tiny sub-set of gen_tcp and file. If the authors of the various web
serveres havn't made some catestripohuic
mistales in their implamentaion I expect their code to be just a thin
layer of the standard librarie rountines and
I expect the results to be broadly similar.
I always like to ask the question:
What would you do with the results of such a measurement?
Because unless you can give a good answer to this, there does not seem
much point in doing the experiment.
/joe
Ivan Uemlianin
from the same flaw: a partisan of server X tests a highly tuned and
optimised install of their favourite server against default installs of
competitor servers. These kinds of benchmarks are really not worth
anything.
I think it's a good idea to set a challenge, and invite server
developers to show how their own project can meet the challenge.
The first step is to devise a challenge, and this is not a trivial step.
This step is actually listing the essential functional requirements of
a web server. The ability to serve a static file promptly from disk is
one requirement, but these days I shouldn't think it's the only or even
the main one.
I think it might be interesting to agree on a small set of requirements
and then devise a challenge which could test those requirements.
Best wishes
Ivan
Joe Armstrong
> Most of the web server benchmarks I've come across on the web suffer from
> the same flaw: a partisan of server X tests a highly tuned and optimised
> install of their favourite server against default installs of competitor
> servers. These kinds of benchmarks are really not worth anything.
>
> I think it's a good idea to set a challenge, and invite server developers to
> show how their own project can meet the challenge.
>
> The first step is to devise a challenge, and this is not a trivial step.
> This step is actually listing the essential functional requirements of a
> web server. The ability to serve a static file promptly from disk is one
> requirement, but these days I shouldn't think it's the only or even the main
> one.
I know that sounds fine, but it's the non-functional requirements that
always seem
to make or break a project.
How good is the documentation?
How mature is the product?
How easy is it to maintain?
Can I understand how the thing works?
Is it covered by the blue-river-snowy public license?
...
The answers to these questions are just not quantifiable.
Not being covered by the blue-river-snowy license and having no
documentation would be a very good reason not to use a web server
even if it could serve pages faster than a shark can bite off your leg.
/Joe
David Goehrig
I disagree. That's not what the challenge is about.
Why don't just try it and avoid (useless) questions!
Jesper Louis Andersen
> What are you trying to show with this benchmark? That is, what is the
> goal of it?
>
> At the end of the day, I'll try to compare Erlang Web servers with others
> competitors (not written in Erlang).
That was not the answer I had in mind when phrasing the question. I'll
try to be more precise: What are you going to measure? If you are
going to count requests, then the problem is that a 100 bytes transfer
is way less than the size of the average HTTP header. In other words,
you will be measuring who cheats and does not provide full headers as
per the HTTP spec. If you are going to measure throughput, on a single
machine, then 'ab' and whatever web server you have will fight for the
resources and thus show nothing at all.
What I am after is why you think your benchmark will give you any
indication of the goal you had in mind. What is performance to you? I
am interested in your answer because it is important to me if I want
to use the results for anything. It may be flawed or not, that is not
the concern for me. I am interested in the reason for why you laid out
the rules of the game as your did.
I disagree that you can see web servers as competitors to each other.
There is always a greater purpose behind writing a web server. From
the point where it is a personal exercise (can I do it?), to providing
a web server in your favorite language for ease of integration, to
writing a server because you want to earn money. It is not as much a
competition as it is a convenience of use. Or said otherwise, many of
the competitors are not even aware they are competing :)
> I'm aware of that rule (Steve Vinoski
> post: http://steve.vinoski.net/blog/2011/05/09/erlang-web-server-benchmarking/).
> Thanks anyway.
What are you going to do about it? Those who will have multiple CPUs
in their machine and pins 'ab' to one and gives Erlang free reign over
the others will definitely be at an advantage here. In fact, those who
disable SMP from the Erlang runtime might also be faster. If you ran
many tests and then used a bootstrapping technique, then you might be
able to tell if the localhost test generator tool interferes too much
with the results for instance. That is one way to circumvent that
problem.
> Why it's so hard to just play the game guys?
We are playing the game on another level. Our game is "The scientific
method and experimental testing". To be precise, I think your test has
way to many confounding factors to underpin your hypothesis that your
experiment can be used to show anything about web servers in
comparison to each other. When you design a simple experiment, it is
mandatory in our game to point out exactly what the confounding
factors are and what trouble there will be when drawing conclusions
from the test itself. I have pointed out some factors and Joe has
pointed out some different factors. That is all.
David Mercer
> Erlang was designed for building fault-tolerant systems, it was *not*
> designed
> for building fast systems
+1. Salient point worth repeating. This often gets forgotten.
> The fastest ever Erlang web server should be way slower than the
> fastest non-fault
> tolerant web server.
Cheers,
DBM
Garrett Smith
different time and place:
http://www.youtube.com/watch?v=xEJ1n13soWU
Very slightly NSFW language.
Garrett
P.S. I hope this isn't viewed as piling on. I just couldn't resist.
This is absolutely not personally directed. It's just a little fun
poking at a certain ideology that we've all been guilty of adopting :)
Zabrane Mickael
Le 7 juil. 2011 à 00:35, Garrett Smith a écrit :
> Interesting thread. Reminds me of a conversation I overhead, in
> different time and place:
>
> http://www.youtube.com/watch?v=xEJ1n13soWU
;-)
Joe Armstrong
> Interesting thread. Reminds me of a conversation I overhead, in
> different time and place:
>
> http://www.youtube.com/watch?v=xEJ1n13soWU
>
> Very slightly NSFW language.
Wonderfull - any ideas how this movie was made?
/Joe
Ivan Uemlianin
And the synthetic voices aren't bad either. xtranormal looks pretty cool.
Ivan
Sent from iPhone.
On 7 Jul 2011, at 07:34, Joe Armstrong <erl...@gmail.com> wrote:
> On Thu, Jul 7, 2011 at 12:35 AM, Garrett Smith <g...@rre.tt> wrote:
>> Interesting thread. Reminds me of a conversation I overhead, in
>> different time and place:
>>
>> http://www.youtube.com/watch?v=xEJ1n13soWU
>>
>> Very slightly NSFW language.
>
> Wonderfull - any ideas how this movie was made?
>
> /Joe
_______________________________________________
Jesse Gumm
--
Jesse Gumm
Sigma Star Systems
414.940.4866
gu...@sigma-star.com
http://www.sigma-star.com