Round 13 is now available
618 views
Skip to first unread message
Brian Hauer
Nov 16, 2016, 10:34:06 AM11/16/16
to framework-benchmarks
Hi everyone,
We are happy to announce that Round 13 results are now available! This round's results have been gathered from two new hardware environments: a physical environment provided by ServerCentral and a cloud environment provided by Microsoft Azure.
http://www.techempower.com/benchmarks/#section=data-r13
Notes and observations at our blog:
http://www.techempower.com/blog/2016/11/16/framework-benchmarks-round-13/
New with this round is "continuous benchmarking," which means our physical hardware environment will be continuously running and gathering results. We hope this will help us deliver future rounds more quickly.
Thank you to everyone who has contributed!
We are happy to announce that Round 13 results are now available! This round's results have been gathered from two new hardware environments: a physical environment provided by ServerCentral and a cloud environment provided by Microsoft Azure.
http://www.techempower.com/benchmarks/#section=data-r13
Notes and observations at our blog:
http://www.techempower.com/blog/2016/11/16/framework-benchmarks-round-13/
New with this round is "continuous benchmarking," which means our physical hardware environment will be continuously running and gathering results. We hope this will help us deliver future rounds more quickly.
Thank you to everyone who has contributed!
Matthieu Garrigues
Nov 17, 2016, 3:48:06 AM11/17/16
to framework-benchmarks
Great Job!
Continuous benchmarking is a really good news. It would be amazing to push daily or weekly previews, Is it you plan?
zloster
Nov 17, 2016, 12:26:11 PM11/17/16
to framework-...@googlegroups.com
Great news. Thank you for your efforts.
An one question: I can't find the dstat output for the round 13 results. What happened to them?
Hint: Round 12 has them in
http://tfb-logs.techempower.com/round-12/peak/linux/2016-02-22-final/20160222131230/fortune/dropwizard-postgres/
Paulo Lopes
Nov 17, 2016, 2:22:14 PM11/17/16
to framework-benchmarks
Hi,
During the preview phase I saw vertx in the top performance frameworks (for text and json) and now with the final results it is marked as "did not complete". What changed in between the last preview and the final result?
Zhong Yu
Nov 18, 2016, 1:56:04 PM11/18/16
to Brian Hauer, framework-benchmarks
Thanks Brian.
My framework (bayou.io) has the misfortune of being marked as 'stripped', otherwise it would be in top 10 in plaintext tests. Let me say a few words about it.
The most peculiar thing about the test load is that requests are densely pipelined. The best throughput can only be achieved by read and write as many requests and responses in one batch, because tcp read/write is the leading cost here.
However I think this test scenario is unrealistic in most real world applications; and the approach to pin on one connection while ignoring other outstanding requests in other connections is unfair, and quite dangerous. Therefore my framework by default does not do that; rather, after a request is processed, it favors the requests of other connections.
My framework does have an option to favor pipelined requests of the same connection, in case that's good for some use cases. I turned on this option for this test [1], because it's the same strategy taken by many comparable frameworks. I guess that is the primary reason why my framework is marked as 'stripped'. I strongly disagree that this counts as any sort of "carefully crafted" maneuver to cheat the test.
However I think this test scenario is unrealistic in most real world applications; and the approach to pin on one connection while ignoring other outstanding requests in other connections is unfair, and quite dangerous. Therefore my framework by default does not do that; rather, after a request is processed, it favors the requests of other connections.
My framework does have an option to favor pipelined requests of the same connection, in case that's good for some use cases. I turned on this option for this test [1], because it's the same strategy taken by many comparable frameworks. I guess that is the primary reason why my framework is marked as 'stripped'. I strongly disagree that this counts as any sort of "carefully crafted" maneuver to cheat the test.
Also, it would be nice if original authors are notified when their tests are branded as 'stripped', instead of being surprised at the last minute. It'd show more respect towards your contributors.
Best regards,
Zhong Yu[1] https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/frameworks/Java/bayou/src/main/java/bayou/BayouServer.java#L18
--
You received this message because you are subscribed to the Google Groups "framework-benchmarks" group.
To unsubscribe from this group and stop receiving emails from it, send an email to framework-benchmarks+unsub...@googlegroups.com.
Visit this group at https://groups.google.com/group/framework-benchmarks.
For more options, visit https://groups.google.com/d/optout.
Daniel Nicoletti
Nov 18, 2016, 2:32:46 PM11/18/16
to Zhong Yu, framework-benchmarks, Brian Hauer
I also think there should be a non keep alive test, with TCP recycle and reuse.
For Cutelyst plain text or JSON tests have nearly the same performance json being a little slower of course but the pipelined nature of the test make the result be the opposite...
Or even send 10 pipelined resquests then close the connection.
To unsubscribe from this group and stop receiving emails from it, send an email to framework-benchmarks+unsubscrib...@googlegroups.com.
Visit this group at https://groups.google.com/group/framework-benchmarks.
For more options, visit https://groups.google.com/d/optout.
jul...@julienviet.com
Nov 19, 2016, 5:03:18 AM11/19/16
to framework-benchmarks
Hi,
all Vert.x benchmarks are marked as "Did not complete" without any reason. They used to run fine in previous rounds.
Can you tell us what happened and how we can change this ?
thanks
Julien
Fredrik Widlund
Nov 19, 2016, 6:57:09 PM11/19/16
to Brian Hauer, framework-benchmarks
Hi,
I would propose that adding a variable to both the plaintext and json categories would to some degree prevent unrealistic optimisations, for example making sure that the json object is instantiated and serialized on the fly. GET /json/1 -> {"message":"Hello, 1!"} or something similar.
Kind regards,
Fredrik Widlund
To unsubscribe from this group and stop receiving emails from it, send an email to framework-benchmarks+unsubscrib...@googlegroups.com.
Visit this group at https://groups.google.com/group/framework-benchmarks.
For more options, visit https://groups.google.com/d/optout.
Joe Duarte
Nov 20, 2016, 7:08:35 AM11/20/16
to framework-benchmarks
Hi Brian – Now that you've got Azure instances and ASP.NET Core, what will it take to include Windows Server? It looks like the ASP.NET benchmarks for Round 13 are all running on Linux.
FreeBSD and the Joyent platforms would be interesting too, especially with the release of FreeBSD 11. (I'm confused by the different platforms Joyent offers, and which is an actual OS vs. a container runtimes or hypervisor, but Brendan Gregg's performance blog makes Joyent's platforms very interesting.)
Last thing: A user authentication workload benchmark would be cool.
Cheers,
Joe Duarte
Aliaksandr Valialkin
Nov 21, 2016, 4:38:06 PM11/21/16
to framework-benchmarks
FYI, it looks like fasthttp-postgres results are missing on the fortunes benchmark results.
jul...@julienviet.com
Nov 22, 2016, 7:06:24 AM11/22/16
to framework-benchmarks
Hi,
I looked at the logs and they don't show anything significant.
Plus I also ran the latest Vert.x benchmark on my machine using the steps advocated by the README and it worked fine.
Is there a better way to know what happens and spend time with you to investigate the reality of the situation ?
thanks
Julien
zloster
Nov 22, 2016, 9:16:17 AM11/22/16
to jul...@julienviet.com, framework-benchmarks
Hi,
On 22.11.2016 14:06, jul...@julienviet.com wrote:
> I looked at the logs and they don't show anything significant.
maybe you are not checking the right logs.
For example here:
http://tfb-logs.techempower.com/round-13/final/azure/logs/vertx-web-postgres/out.txt.gz
There are a lot like the following:
--------------------------------------------------------------------------------
Benchmarking vertx-web-postgres
--------------------------------------------------------------------------------
Server vertx-web-postgres: Nov 14, 2016 6:50:20 AM
io.vertx.core.net.impl.ConnectionBase
Server vertx-web-postgres: SEVERE: java.io.IOException: Connection reset
by peer
Later there are also DB transaction deadlocks (if I understand the
exceptions right)
Server vertx-web-postgres: Nov 14, 2016 6:55:42 AM
io.vertx.ext.web.impl.RoutingContextImplBase
Server vertx-web-postgres: SEVERE: Unexpected exception in route
Server vertx-web-postgres:
com.github.mauricio.async.db.postgresql.exceptions.GenericDatabaseException:
ErrorMessage(fields=Map(Detail -> Process 63666 waits for ShareLock on
transaction 17086277; blocked by process 63583.
Server vertx-web-postgres: Process 63583 waits for ShareLock on
transaction 17086300; blocked by process 63590.
Server vertx-web-postgres: Process 63590 waits for ShareLock on
transaction 17086287; blocked by process 63666., Line -> 956, Hint ->
See server log for query details., File -> deadlock.c, SQLSTATE ->
40P01, Routine -> DeadLockReport, Message -> deadlock detected, Severity
-> ERROR))
Server vertx-web-postgres: at
com.github.mauricio.async.db.postgresql.PostgreSQLConnection.onError(PostgreSQLConnection.scala:172)
Server vertx-web-postgres: at
com.github.mauricio.async.db.postgresql.codec.PostgreSQLConnectionHandler.channelRead0(PostgreSQLConnectionHandler.scala:206)
Server vertx-web-postgres: at
io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
Server vertx-web-postgres: at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
Server vertx-web-postgres: at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:334)
Server vertx-web-postgres: at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:326)
Server vertx-web-postgres: at
io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293)
Server vertx-web-postgres: at
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267)
Server vertx-web-postgres: at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
Server vertx-web-postgres: at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:334)
Server vertx-web-postgres: at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:326)
Server vertx-web-postgres: at
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1320)
Server vertx-web-postgres: at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
Server vertx-web-postgres: at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:334)
Server vertx-web-postgres: at
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:905)
Server vertx-web-postgres: at
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:123)
Server vertx-web-postgres: at
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:563)
Server vertx-web-postgres: at
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:504)
Server vertx-web-postgres: at
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:418)
Server vertx-web-postgres: at
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:390)
Server vertx-web-postgres: at
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:742)
Server vertx-web-postgres: at java.lang.Thread.run(Thread.java:745)
Take a look to other out.txt.gz files. Maybe this will help you with the
investigation.
zloster
On 22.11.2016 14:06, jul...@julienviet.com wrote:
> I looked at the logs and they don't show anything significant.
For example here:
http://tfb-logs.techempower.com/round-13/final/azure/logs/vertx-web-postgres/out.txt.gz
There are a lot like the following:
--------------------------------------------------------------------------------
Benchmarking vertx-web-postgres
--------------------------------------------------------------------------------
Server vertx-web-postgres: Nov 14, 2016 6:50:20 AM
io.vertx.core.net.impl.ConnectionBase
Server vertx-web-postgres: SEVERE: java.io.IOException: Connection reset
by peer
Later there are also DB transaction deadlocks (if I understand the
exceptions right)
Server vertx-web-postgres: Nov 14, 2016 6:55:42 AM
io.vertx.ext.web.impl.RoutingContextImplBase
Server vertx-web-postgres: SEVERE: Unexpected exception in route
Server vertx-web-postgres:
com.github.mauricio.async.db.postgresql.exceptions.GenericDatabaseException:
ErrorMessage(fields=Map(Detail -> Process 63666 waits for ShareLock on
transaction 17086277; blocked by process 63583.
Server vertx-web-postgres: Process 63583 waits for ShareLock on
transaction 17086300; blocked by process 63590.
Server vertx-web-postgres: Process 63590 waits for ShareLock on
transaction 17086287; blocked by process 63666., Line -> 956, Hint ->
See server log for query details., File -> deadlock.c, SQLSTATE ->
40P01, Routine -> DeadLockReport, Message -> deadlock detected, Severity
-> ERROR))
Server vertx-web-postgres: at
com.github.mauricio.async.db.postgresql.PostgreSQLConnection.onError(PostgreSQLConnection.scala:172)
Server vertx-web-postgres: at
com.github.mauricio.async.db.postgresql.codec.PostgreSQLConnectionHandler.channelRead0(PostgreSQLConnectionHandler.scala:206)
Server vertx-web-postgres: at
io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
Server vertx-web-postgres: at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
Server vertx-web-postgres: at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:334)
Server vertx-web-postgres: at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:326)
Server vertx-web-postgres: at
io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293)
Server vertx-web-postgres: at
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267)
Server vertx-web-postgres: at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
Server vertx-web-postgres: at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:334)
Server vertx-web-postgres: at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:326)
Server vertx-web-postgres: at
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1320)
Server vertx-web-postgres: at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
Server vertx-web-postgres: at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:334)
Server vertx-web-postgres: at
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:905)
Server vertx-web-postgres: at
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:123)
Server vertx-web-postgres: at
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:563)
Server vertx-web-postgres: at
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:504)
Server vertx-web-postgres: at
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:418)
Server vertx-web-postgres: at
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:390)
Server vertx-web-postgres: at
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:742)
Server vertx-web-postgres: at java.lang.Thread.run(Thread.java:745)
Take a look to other out.txt.gz files. Maybe this will help you with the
investigation.
zloster
Julien Viet
Nov 22, 2016, 9:18:18 AM11/22/16
to zloster, framework-benchmarks
Hi,
how about the simple string and json tests ?
how about the simple string and json tests ?
zloster
Nov 22, 2016, 9:30:54 AM11/22/16
to Julien Viet, framework-benchmarks
On 22.11.2016 16:18, Julien Viet wrote:
> Hi,
>
> how about the simple string and json tests ?
>
http://tfb-logs.techempower.com/round-13/final/azure/logs/vertx/out.txt.gz
> Hi,
>
> how about the simple string and json tests ?
>
A lot of again:
--------------------------------------------------------------------------------
Benchmarking vertx
--------------------------------------------------------------------------------
Server vertx: Nov 14, 2016 6:18:55 AM io.vertx.core.net.impl.ConnectionBase
Server vertx: SEVERE: java.io.IOException: Connection reset by peer
zloster
zloster
Nov 22, 2016, 9:42:08 AM11/22/16
to Aliaksandr Valialkin, framework-benchmarks
Hi,
On 21.11.2016 23:38, Aliaksandr
Valialkin wrote:
FYI, it looks like fasthttp-postgres results are missing on the fortunes benchmark results.
It seems that only the base variant of fasthttp has Fortune test
implemented. Check the configuration
file.
However I'm not sure why only fasthttp-mysql-prefork has a results for the Fortune test in the Round 13.
Cheers,
zloster
However I'm not sure why only fasthttp-mysql-prefork has a results for the Fortune test in the Round 13.
Cheers,
zloster
Julien Viet
Nov 22, 2016, 9:43:57 AM11/22/16
to zloster, framework-benchmarks
That does not mean anything
Connection reset by peer is expected in all situations (successful or not) with “wrk”, they happen when wrk close the connections after the test.
When I run the benchmark locally (and it passes) I also get the connections reset by peer logging.
the problem is more that we don’t know what happened from the Client perspective (wrk)
Connection reset by peer is expected in all situations (successful or not) with “wrk”, they happen when wrk close the connections after the test.
When I run the benchmark locally (and it passes) I also get the connections reset by peer logging.
the problem is more that we don’t know what happened from the Client perspective (wrk)
jul...@julienviet.com
Nov 23, 2016, 11:35:08 PM11/23/16
to framework-benchmarks
Hi,
as I said in this thread previously, we are still waiting for a clear explanation of what happened to the Vert.x related benchmarks.
We believe that in the situation where a benchmark produces no results and thus discards the candidates, Techempower is accountable to provide a clear explanation and furthermore should warn the contender and provide an opportunity to fix the problem in order to compete fairly.
Julien
On Wednesday, November 16, 2016 at 4:34:06 PM UTC+1, Brian Hauer wrote:
Aliaksandr Valialkin
Nov 28, 2016, 6:42:00 AM11/28/16
to framework-benchmarks, val...@gmail.com, mo...@edno.moe
Hi,
On 21.11.2016 23:38, Aliaksandr Valialkin wrote:
It seems that only the base variant of fasthttp has Fortune test implemented. Check the configuration file.FYI, it looks like fasthttp-postgres results are missing on the fortunes benchmark results.
Fortune test is implemented in all the fasthttp variants. However, it has been deleted by an accident during the cleanup at https://github.com/TechEmpower/FrameworkBenchmarks/commit/d581f1d7a99556f5135360ed4929ccb0e6e7fba8 .
Re-added fortune test for fasthttp-postgresql in the PR https://github.com/TechEmpower/FrameworkBenchmarks/pull/2380 .
zloster
Nov 28, 2016, 9:18:15 AM11/28/16
to Aliaksandr Valialkin, framework-benchmarks
On 28.11.2016 13:41, Aliaksandr Valialkin wrote:
> Fortune test is implemented in all the fasthttp variants. However, it has
> been deleted by an accident during the cleanup
> at https://github.com/TechEmpower/FrameworkBenchmarks/commit/d581f1d7a99556f5135360ed4929ccb0e6e7fba8
> .
>
> Re-added fortune test for fasthttp-postgresql in the
> PR https://github.com/TechEmpower/FrameworkBenchmarks/pull/2380 .
No more mystery. Thank you.
> Fortune test is implemented in all the fasthttp variants. However, it has
> been deleted by an accident during the cleanup
> at https://github.com/TechEmpower/FrameworkBenchmarks/commit/d581f1d7a99556f5135360ed4929ccb0e6e7fba8
> .
>
> Re-added fortune test for fasthttp-postgresql in the
> PR https://github.com/TechEmpower/FrameworkBenchmarks/pull/2380 .
Brian Hauer
Nov 28, 2016, 1:17:44 PM11/28/16
to framework-benchmarks
Hi Matthieu,
> Continuous benchmarking is a really good news. It would be amazing to push daily or weekly previews, Is it you plan?
With respect to conrtinuous benchmarking, the short-term plan is to take it simple. Once we have Round 14 PRs processed, we'll resume the continuous benchmarker and share preview results as soon as we have some that we believe are accurate.
The longer-term goal is to automatically capture results and post them to a web app for viewing. What we have in mind is the ability to see performance trends over time and get earlier recognition of significant changes that may indicate problems.
The longer-term goal is to automatically capture results and post them to a web app for viewing. What we have in mind is the ability to see performance trends over time and get earlier recognition of significant changes that may indicate problems.
Brian Hauer
Nov 28, 2016, 1:25:56 PM11/28/16
to framework-benchmarks
Hi Paulo,
We removed Vert.x from the Round 13 results based on our arrangement from previous rounds, wherein the Vert.x maintainers had asked us to hide its results, and our understanding from your issue:
https://github.com/TechEmpower/FrameworkBenchmarks/issues/2117
The results web site shows anything that is missing as "did not complete" by default.
I apologize for misunderstanding your wishes here; I am guessing that changes that were made since then supersede #2117. I'll see if I can restore the Vert.x results in the Round 13 results file.
This is perhaps an inevitable challenge of manually hiding results as we are doing here. In the future, I would prefer that benchmarks that do not want to be included in the results remove their implementations outright or that we add a flag indicating the implementation is disable/deprecated.
We removed Vert.x from the Round 13 results based on our arrangement from previous rounds, wherein the Vert.x maintainers had asked us to hide its results, and our understanding from your issue:
https://github.com/TechEmpower/FrameworkBenchmarks/issues/2117
The results web site shows anything that is missing as "did not complete" by default.
I apologize for misunderstanding your wishes here; I am guessing that changes that were made since then supersede #2117. I'll see if I can restore the Vert.x results in the Round 13 results file.
This is perhaps an inevitable challenge of manually hiding results as we are doing here. In the future, I would prefer that benchmarks that do not want to be included in the results remove their implementations outright or that we add a flag indicating the implementation is disable/deprecated.
Brian Hauer
Nov 28, 2016, 1:55:12 PM11/28/16
to framework-benchmarks, teona...@gmail.com
Hi Zhong,
When we reviewed implementations in Round 13, we marked several implementations as stripped. The spectrum from realistic to stripped is one full of interpretation. Based on your description below, I am satisfied with switching your test back to realistic and we'll modify the metadata to reflect that.
In the future, we will try to create Github issues whenever we mark a test as stripped.
When we reviewed implementations in Round 13, we marked several implementations as stripped. The spectrum from realistic to stripped is one full of interpretation. Based on your description below, I am satisfied with switching your test back to realistic and we'll modify the metadata to reflect that.
In the future, we will try to create Github issues whenever we mark a test as stripped.
Brian Hauer
Nov 28, 2016, 2:05:32 PM11/28/16
to framework-benchmarks, zhong...@gmail.com, teona...@gmail.com
Hi Daniel,
Although creating new test types has not received the attention it deserves, I've nevertheless added your suggestion to the list here:
https://github.com/TechEmpower/FrameworkBenchmarks/issues/133
Although creating new test types has not received the attention it deserves, I've nevertheless added your suggestion to the list here:
https://github.com/TechEmpower/FrameworkBenchmarks/issues/133
Brian Hauer
Nov 28, 2016, 2:06:35 PM11/28/16
to framework-benchmarks
Hi Julien,
Please see my reply to Paulo Lopes elsewhere in this thread. I will attempt to restore the Vert.x results.
Please see my reply to Paulo Lopes elsewhere in this thread. I will attempt to restore the Vert.x results.
Brian Hauer
Nov 28, 2016, 2:10:56 PM11/28/16
to framework-benchmarks
Hi Joe,
There is quite a bit of work involved in making the test suite run on Windows since it's heavily dependent on bash, ssh, and several other Linux assumptions. However, it was done in the past and could be done again with sufficient effort.
FreeBSD would also be interesting. Presumably that would be an easier effort than Windows, but still not trivial.
And I like the idea of an authenticated user test. In the following comment ( https://github.com/TechEmpower/FrameworkBenchmarks/issues/133#issuecomment-16295825 ) you'll see that we have that captured as a recommendation. There would be a lot to do there, but it would be a very interesting test.
There is quite a bit of work involved in making the test suite run on Windows since it's heavily dependent on bash, ssh, and several other Linux assumptions. However, it was done in the past and could be done again with sufficient effort.
FreeBSD would also be interesting. Presumably that would be an easier effort than Windows, but still not trivial.
And I like the idea of an authenticated user test. In the following comment ( https://github.com/TechEmpower/FrameworkBenchmarks/issues/133#issuecomment-16295825 ) you'll see that we have that captured as a recommendation. There would be a lot to do there, but it would be a very interesting test.
Brian Hauer
Nov 28, 2016, 2:50:32 PM11/28/16
to framework-benchmarks
Hi Julien,
I have restored the Vert.x Round 13 results on the physical hardware; if you've viewed the results recently, you may need to forcefully reload to see Vert.x. I am presently not sure if I can restore the Vert.x results for the cloud environment. But we aim to have Round 14 on a faster timeframe so there should be new cloud results soon.
Again, we removed Vert.x based on our (apparently incorrect) understanding that the Vert.x maintainers wanted it hidden. This process of manually hiding results has been error-prone—previously I forgot to hide Vert.x when releasing a new round. So I would ask that in the future anyone who wants results hidden should simply remove their implementation. Or we could add a "skip" flag to the metadata files.
I have restored the Vert.x Round 13 results on the physical hardware; if you've viewed the results recently, you may need to forcefully reload to see Vert.x. I am presently not sure if I can restore the Vert.x results for the cloud environment. But we aim to have Round 14 on a faster timeframe so there should be new cloud results soon.
Again, we removed Vert.x based on our (apparently incorrect) understanding that the Vert.x maintainers wanted it hidden. This process of manually hiding results has been error-prone—previously I forgot to hide Vert.x when releasing a new round. So I would ask that in the future anyone who wants results hidden should simply remove their implementation. Or we could add a "skip" flag to the metadata files.
Paulo Lopes
Nov 29, 2016, 7:50:14 AM11/29/16
to framework-benchmarks
Hi Brian,
I am one of the core developers and I did request to unlist **vertx-web** (https://github.com/TechEmpower/FrameworkBenchmarks/issues/2117).
Note though that vertx-web and vertx are 2 different benchmark implementations :)
Vertx-web is not vertx but a sub-project which main focus in on simplicity of web development and the round #12 implementation was done with some tech preview code and therefore I asked it to be unlisted.
Thanks!
Paulo
I am one of the core developers and I did request to unlist **vertx-web** (https://github.com/TechEmpower/FrameworkBenchmarks/issues/2117).
Note though that vertx-web and vertx are 2 different benchmark implementations :)
Vertx-web is not vertx but a sub-project which main focus in on simplicity of web development and the round #12 implementation was done with some tech preview code and therefore I asked it to be unlisted.
Thanks!
Paulo
Brian Hauer
Nov 29, 2016, 2:37:25 PM11/29/16
to framework-benchmarks
Aha! Thanks for the clarification.
For the time being, are you comfortable having both listed in Round 13 (on physical hardware)? I'd rather not change things again if I can avoid it.
For the time being, are you comfortable having both listed in Round 13 (on physical hardware)? I'd rather not change things again if I can avoid it.
Roger Pack
Dec 1, 2016, 7:07:38 PM12/1/16
to framework-benchmarks
On Wednesday, November 16, 2016 at 8:34:06 AM UTC-7, Brian Hauer wrote:
Hi everyone,
We are happy to announce that Round 13 results are now available! This round's results have been gathered from two new hardware environments: a physical environment provided by ServerCentral and a cloud environment provided by Microsoft Azure.
http://www.techempower.com/benchmarks/#section=data-r13
It says some crystal benchmarks "did not complete" is there a way to see the full output there? Something doesn't seem quite on there...
jnet...@techempower.com
Dec 2, 2016, 11:02:27 AM12/2/16
to framework-benchmarks
Hi Roger,
You can find the crystal the logs here. Hope that helps!
Reply all
Reply to author
Forward
0 new messages