Exercising the Plaintext Benchmark

[ricardo.dan...@gmail.com]

Hello,

I'm trying to understand how the test is performed for the simple plaintext case, where the http server must respond with "Hello World".

My questions are:

0. Once I have my http server coded, what is the procedure do you use to stress test it?

1. The best one is currently Vert.x with number 656,119 (100%) What does 656,119 mean? (from https://www.techempower.com/benchmarks/#section=data-r8&hw=i7&test=plaintext)

2. Are you making http requests over loopback (127.0.0.1) or over the network?

3. Is the client keeping the socket connection open (keep-alive) and making all requests through the same socket or opening a new socket on each request?

So basically I have coded the plaintext case with my framework and I want to execute it on my environment. I'll then procede to execute the same test with the source code of some alternative frameworks. That way I'll be able to have a comparison of the frameworks running on my controlled environment.

Can someone help?

-Ricardo

[Brian Hauer]

Hi Ricardo,

Welcome to the project!  Others may be able to answer a bit more specifically on some of the points, but I'll try to respond to your questions:

0. For rapid-fire testing of your implementation, my personal advice is to your load generator directly.  Running the full benchmark suite to test each tweak you make to your implementation may be a bit too cumbersome for development.  Generally speaking, if a change improves performance in spot testing with wrk (or similar), it will likely improve the final results in a real benchmark run as well.  Very few changes would make for illusory gains in spot testing that vanish in a real run.

1. In that particular case, that means that Vert.x generated 656,119 plaintext HTTP responses per second on our (old) i7 physical hardware test environment.  Note that the i7 hardware environment was retired after Round 8.  It may help to see the raw output from Wrk for that particular test in Round 8:

https://github.com/TechEmpower/TFB-Round-8/blob/master/i7/round-8-final/20131214101855/plaintext/vertx/raw

In that raw data, you can see that Vert.x performed best at 256 concurrency, and produced 9841798 responses in 15 seconds.  9841798 / 15 ~= 656,119.

2. No.  In our benchmark runs, load is generated from a separate computer or virtual machine and sent over the network.  In the Peak environment, the network is 10 gigabit Ethernet and in the EC2 environment, it's 1 gigabit Ethernet.

3. In the plaintext test, not only is the socket kept alive, but we allow for HTTP pipelining.  Pipelining allows for a huge number of requests to be processed without a lot of lock-step coordination between the client and server.  You can read the requirements for the plaintext test and others here:

https://www.techempower.com/benchmarks/#section=code

Let me know if you have any more questions!