C10K Websocket Test

[Jaybill McCarthy]

I know this list isn't too big on benchmarks, but I just saw this on HN and thought people might find it of interest. Apparently, Go did really well in a C10K Websocket Test. From the article:

"This benchmark starts a new client every 1ms, each client sends a timestamp message to the server every second and the server echos that message back to the client."

Erlang won, Go was a close second. (Neat!) Java, Haskell, Node & Python failed.

Here's the whole thing:

https://github.com/ericmoritz/wsdemo/blob/results-v1/results.md 

[andrey mirtchovski]

https://groups.google.com/d/topic/golang-nuts/bchGjWI39nY/discussion

[Jaybill McCarthy]

Dang! I searched for it on the list, too!

Oh, well. Sorry about the repeat, then. 

--Jaybill

On Saturday, June 16, 2012 7:38:32 PM UTC-7, aam wrote:

https://groups.google.com/d/topic/golang-nuts/bchGjWI39nY/discussion

[Dave Cheney]

This benchmark was taken on shared virtualized hardware running over a shared network segment. I do not believe the author has sufficiently eliminated the background noise present in the environment and thus the results do not represent a valid base for comparison. 

Dave

[Mikael Gustavsson]

Jaybill:

Some values are promising from Go's perspective (total messages processed and concurrent connections).

But the Go server actually fared worse than the Haskell and Java ones, look at the latency values.

On Sunday, June 17, 2012 10:41:29 AM UTC+8, Dave Cheney wrote:

This benchmark was taken on shared virtualized hardware running over a shared network segment. I do not believe the author has sufficiently eliminated the background noise present in the environment and thus the results do not represent a valid base for comparison. 

The Go server had 18 seconds of mean latency, it's unlikely that's because of background noise.

Clearly, all of the server implementations were under very heavy load, and the Erlang server handled it best.

The actual values (like 18s for Go) is almost meaningless when a system is overloaded, but you can make a statement about whether the system was able to handle the load or not, and this implementation of the server in Go didn't handle it nearly as good as the Erlang server.

[kortschak]

How do you measure the latency of a dropped messages? and what does this do to the claim that haskell and java fared better?

[André Moraes]

>
> The Go server had 18 seconds of mean latency, it's unlikely that's because
> of background noise.
> Clearly, all of the server implementations were under very heavy load, and
> the Erlang server handled it best.
> The actual values (like 18s for Go) is almost meaningless when a system is
> overloaded, but you can make a statement about whether the system was able
> to handle the load or not, and this implementation of the server in Go
> didn't handle it nearly as good as the Erlang server.
>

Quoting the author of the benchmark
(http://www.reddit.com/r/programming/comments/v5ap8/web_server_benchmark_haskellsnap_erlang/c51wv9i):
"I wrote the thing and I can tell you that yes, these results are
pretty much crap. I incorrectly used Folsom, the project I used to
collect the stats. Folsom is meant for realtime monitoring and not
what I used it for. It only keeps 60 second windows of the data it
produces. That means all times prior to the fine 60 second window was
thrown out.

I tweeted the results at 3am to get feedback from folks I know.
Unfortunately it spread across the net really fast.
"

Also, using the standard go websocket client/server in localhost, the
time is measured in micro-seconds with only a few milliseconds.

This benchmark is wrong.


--
André Moraes
http://amoraes.info

[tomwilde]

He posted this now:

"I'm sorry I was mistaken that Folsom only keeps a 60 second window. It has support for sliding windows but its default is to use the entire dataset for histograms and percentiles. I take back my statement that the result were pretty much crap. They're only slightly crappy because I used a m1.medium box.
I'm spending my free time this week to make the benchmark more automated so that it is easier for someone who volunteered to run it in their datacenter on real hardware. That should produce a much fairer result."