FrameworkBenchmarks Round 11 results are available

[Ludovic Gasc]

Hi everybody,

For now, I can't contribute like I want because I'm handling a lot of personal and professional changes.

However, I continue to keep an eye.

FrameworkBenchmarks Round 11 results are available:

https://www.techempower.com/benchmarks/#section=data-r11&hw=peak&test=fortune&l=1kw

If you don't know what is FrameworkBenchmarks: https://www.techempower.com/benchmarks/#section=intro

No big surprises, AsyncIO+aiohttp continues to have good results, except for plain text, need to dig where is the bottleneck.

For the next round, I've a small todo list:

1. Add tests with MySQL, because MySQL is more optimized by techempower than PostgreSQL setup and all other Python frameworks with better results use MySQL.

2. Upgrade aiohttp, because the latest version has several performance improvements: https://github.com/KeepSafe/aiohttp/releases/tag/v0.20.0

3. Use Python 3.5 with async/await. Apparently, I could hope some performance improvements.

4. If I have the time, take a try with MicroPython.

If somebody wants to help me or suggests other improvements, be my guest.

Happy holidays.

[Ludovic Gasc]

Hi,

This is some values I've found during the update attempt of the AsyncIO test suite for FrameworkBenchmarks: https://www.techempower.com/benchmarks/#section=intro

Long story short: After Python 3.5 update, dependencies update and async/await syntax usage, on my test setup, I loose around 15% of efficiency.

You can see the difference between old test setup and now: https://github.com/Eyepea/FrameworkBenchmarks/commit/a40c82ed720a53e04ebdafd7db90302ca67b5226

If somebody has any suggestion to find my mistake or how to find the new bottleneck, be my guest.

However, even if the absolute values aren't exact, the relative values between each test should give us an idea of the evolution.

For each test, I've launched 5 times, and I've taken the best one.

The values on this setup before to change something in the test setup (python 3.4.2, aiohttp 0.16.3, aiopg 0.7.0):

$ wrk -t8 -c256 -d1m http://127.0.0.1:8080/queries?queries=20

Running 1m test @ http://127.0.0.1:8080/queries?queries=20

  8 threads and 256 connections

  Thread Stats   Avg      Stdev     Max   +/- Stdev

    Latency   216.97ms  105.14ms   1.14s    75.31%

    Req/Sec   148.88     17.97   206.00     69.38%

  71647 requests in 1.00m, 54.98MB read

Requests/sec:   1194.19

Transfer/sec:      0.92MB

Now, I upgrade only Python 3.4 to Python 3.5 (python 3.5.1, aiohttp 0.16.3, aiopg 0.7.0):

$ wrk -t8 -c256 -d1m http://127.0.0.1:8080/queries?queries=20

Running 1m test @ http://127.0.0.1:8080/queries?queries=20

  8 threads and 256 connections

  Thread Stats   Avg      Stdev     Max   +/- Stdev

    Latency   237.25ms  118.33ms   1.17s    74.74%

    Req/Sec   134.77     13.24   171.00     66.98%

  65051 requests in 1.00m, 49.92MB read

Requests/sec:   1084.09

Transfer/sec:    851.93KB

And now, I update aiohttp and aiopg  (python 3.5.1, aiohttp 0.21.1, aiopg 0.9.2)

$ wrk -t8 -c256 -d1m http://127.0.0.1:8080/queries?queries=20

Running 1m test @ http://127.0.0.1:8080/queries?queries=20

  8 threads and 256 connections

  Thread Stats   Avg      Stdev     Max   +/- Stdev

    Latency   254.25ms  181.80ms   1.56s    75.51%

    Req/Sec   129.43     24.09   204.00     68.37%

  62122 requests in 1.00m, 46.25MB read

Requests/sec:   1035.44

Transfer/sec:    789.32KB

And now, I use async/await syntax:

$ wrk -t8 -c256 -d1m http://127.0.0.1:8080/queries?queries=20

Running 1m test @ http://127.0.0.1:8080/queries?queries=20

  8 threads and 256 connections

  Thread Stats   Avg      Stdev     Max   +/- Stdev

    Latency   259.27ms  121.86ms 842.74ms   70.02%

    Req/Sec   126.29     17.70   207.00     75.82%

  60740 requests in 1.00m, 45.22MB read

Requests/sec:   1014.01

Transfer/sec:    772.99KB

To be sure it isn't a problem with my hardware (CPU too hot after a while...) at the end, I've relaunched the first test without any updates:

$ wrk -t8 -c256 -d1m http://127.0.0.1:8080/queries?queries=20

Running 1m test @ http://127.0.0.1:8080/queries?queries=20

  8 threads and 256 connections

  Thread Stats   Avg      Stdev     Max   +/- Stdev

    Latency   220.24ms   98.73ms 798.70ms   72.78%

    Req/Sec   147.85     19.10   215.00     70.30%

  70967 requests in 1.00m, 54.46MB read

Requests/sec:   1183.54

Transfer/sec:      0.91MB

Thanks for your remarks.

[Yury Selivanov]

Hi Ludovic,

I’m usually highly sceptical about any network benchmarks over localhost. I’d suggest you to rerun your benchmarks using two different machines connected over a gigabit network. Please make sure that your server process loads the CPU 100%, and ideally you should use several processes for clients.

Yury

[Ludovic Gasc]

Hi Yury,

Thanks for the remark, I'll test with two different machines during the next week.

Have a nice day.

--

Ludovic Gasc (GMLudo)

http://www.gmludo.eu/