The number of sql connections used by a framework has a huge impact on its ranking.

[Matthieu Garrigues]

Hi all,

Did you notice that the number of sql connections used by a framework has a huge impact on its performances ?

Some frameworks are using the optimal number, some are not. This brings a huge bias in the ranking of the framework:

I was able to bring lithium from rank #40 to rank #2 on the update test by just lowering the number of sql connections, and all

sql tests are impacted.

To avoid this bias and to actually rank frameworks according to their performances, I propose that we constraint all frameworks

to use the same number of connections. What do you think ?

[Shay Rojansky]

Isn't the number of SQL connections a legitimate environmental parameter of each framework? Database drivers may be implemented in different ways, which require more or less connections - why discriminate against any of them by forcing a sub-optimal value on them? Isn't the end performance the thing that counts?

--
You received this message because you are subscribed to the Google Groups "framework-benchmarks" group.
To unsubscribe from this group and stop receiving emails from it, send an email to framework-benchm...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/framework-benchmarks/05289634-fb7d-4688-8760-2f4c07b8c6f9%40googlegroups.com.

[Matthieu Garrigues]

Indeed, so if the optimal value varies on each framework, could we
give at least recommandations ?
I'm posting this because there is probably lots of framework that are
actually using a suboptimal number of connections, causing their
ranking not to reflect their real performance.

Another idea: the benchmark could test the framework with different
number of sql connections, and report
the highest performance (like we do today for the number of http
connections on some tests).

Matthieu Garrigues

[Shay Rojansky]

> Indeed, so if the optimal value varies on each framework, could we give at least recommandations ?

Well, since the optimal number is quite framework-specific, what kind of recommendations could we give except "be sure to test and optimize this"? This is somewhat similar to a give benchmark scenario not using its framework correctly code-wise, e.g. forgetting to prepare statements where that would result in a big perf gain. Isn't this the responsibility of each framework benchmark writer?

> the benchmark could test the framework with different number of sql  connections, and report the highest performance (like we do today for the number of http connections on some tests).

That's certainly possible (and would definitely make for an interesting chart). The problem is that unlike with client connections, every framework/language/driver has this configuration in a completely different form - benchmarks would need to somehow expose this value in a standardized way, which seems like a lot of trouble (and brittleness) for little gain.

Also, keep in mind that this isn't always a simple, specific number. Different database drivers may support different concepts around how many database connections they hold, and how they use it.

IMHO, the bottom line is that unlike client connections, database connections really does seem to be a pretty internal matter that's between the database driver and the database...