Cakephp vs Laravel: weird results

98 views
Skip to first unread message

Matthias Vanooteghem

unread,
Apr 30, 2019, 8:46:16 AM4/30/19
to framework-benchmarks
Hi all,

I'm trying to recreate the benchmark tests on my local server. I git cloned the repo, didn't change anything and ran the tests for CakePHP vs Laravel (Full Stack PHP frameworks), but the results I'm getting aren't even close to the results in Round 17.

My results for json-serialization at 16 concurrency

"cakephp": [
        {
          "latencyAvg": "8.23ms", 
          "latencyMax": "68.38ms", 
          "latencyStdev": "5.01ms", 
          "totalRequests": 30431, 
          "startTime": 1553070286, 
          "endTime": 1553070301
        },
...

"laravel": [
        {
          "latencyAvg": "11.67ms", 
          "latencyMax": "63.74ms", 
          "latencyStdev": "6.12ms", 
          "totalRequests": 20962, 
          "startTime": 1553071165, 
          "endTime": 1553071180
        }, 
////

So according to my results, the json serialization test for CakePHP has a totalRequests value that is 50% more than Laravel, while in the official Round 17 results CakePHP is 25% slower than Laravel. (Link: https://www.techempower.com/benchmarks/#section=data-r17&hw=ph&test=json&f=zik0zj-zik073-zik0zj-zik0vz-zik0zj-zik0zj-zik0zj-zik0zj-e7 )

My server:
CPU: Intel  i5-750
Cores: 4
CPU: 2.67GHz
RAM: 16GB DDR3 1333 MHz
Hard disk speed: 7200 RPM

Docker version: 1.29
Debian 9.8 Stretch


What could cause this? Am I doing something very wrong? I didn't change anything in the repo, just cloned it.

Thanks in advance for any help in this.

Matthias

zloster

unread,
May 1, 2019, 3:33:47 AM5/1/19
to framework-...@googlegroups.com

Hi Matthias,

see my notes below.
--
You received this message because you are subscribed to the Google Groups "framework-benchmarks" group.
To unsubscribe from this group and stop receiving emails from it, send an email to framework-benchm...@googlegroups.com.
Visit this group at https://groups.google.com/group/framework-benchmarks.
For more options, visit https://groups.google.com/d/optout.

There are changes in both framework configurations since the last official results. Check here:
https://github.com/TechEmpower/FrameworkBenchmarks/commits/master/frameworks/PHP/cakephp
https://github.com/TechEmpower/FrameworkBenchmarks/commits/master/frameworks/PHP/laravel
Also you have to check if the external dependencies are pinned. If not - this is an additional moving/changing part.

How to find out which code is benchmarked for given result set?
The logs from the official runs are stored on separate server. Here is the "results.json" file for Round 17: http://tfb-logs.techempower.com/round-17/final/citrine-results/20180903024112/results.json
There you will find the following:
"environmentDescription": "Citrine", 
  "git": {
    "commitId": "b20ac7b1bad9e0e5e629d3d4cf1d5970387bb608", 
    "repositoryUrl": "https://github.com/TechEmpower/FrameworkBenchmarks.git", 
    "branchName": "master"
  },
The existence of this information is discussed in some of the issues on GitHub. You could use the following two issues as a starting point:
https://github.com/TechEmpower/FrameworkBenchmarks/issues/3326
https://github.com/TechEmpower/FrameworkBenchmarks/issues/3587

About the hardware:
The official tests setup is using 3 (three) separate computers and they are connected via Ethernet network. Additional details:
https://www.techempower.com/benchmarks/#section=environment
https://www.techempower.com/benchmarks/#section=motivation (particularly Question 21)
https://github.com/TechEmpower/FrameworkBenchmarks/wiki/Project-Information-Environment (this is very recent addition)

I'm hoping this will help you.

Best regards,
zloster

Joan Miquel

unread,
May 10, 2019, 7:39:33 AM5/10/19
to framework-benchmarks
Hi Matthias,

The Techempower bencmark is alive.
It's changing all the time: new frameworks, new configs, new platforms, new versions, ....

But they need to stop at any time to release the results to a public round.

In round 17 cakephp was using PHP5.6 and laravel PHP7.2.
and click the tab "Framework overhead". You will see that one is using php(v7) and the other php-php5.

The round 17 was released to the public at 2018-10-30 using the github at 2018-09-01 https://github.com/TechEmpower/FrameworkBenchmarks/tree/b20ac7b1bad9e0e5e629d3d4cf1d5970387bb608

So the difference you have is normal. You are comparing new code with old one.
For example the next round 18 will be faster for all. New changes: php7.3, nginx, configs, versions, ...
What would be strange is that a new one was slower than the old one.


To check the last results live, go to https://tfb-status.techempower.com 

Thank you.

And you can check the results:
Interestingly cakephp 3.7 is failing in the fortunes test.

I'm hoping this will help you.

Best regards,
Joanhey

PD: I wanted to link to the branch or tag in github for round 17, but not one exist. I will create a issue.

 


Reply all
Reply to author
Forward
0 new messages