Discussion of the Go Programming Language ()

Jaybill McCarthy | 17 Jun 2012 04:31

C10K Websocket Test

I know this list isn't too big on benchmarks, but I just saw this on HN and thought people might find it of interest. Apparently, Go did really well in a C10K Websocket Test. From the article:

"This benchmark starts a new client every 1ms, each client sends a timestamp message to the server every second and the server echos that message back to the client."

Erlang won, Go was a close second. (Neat!) Java, Haskell, Node & Python failed.

Here's the whole thing:

https://github.com/ericmoritz/wsdemo/blob/results-v1/results.md

headers

andrey mirtchovski | 17 Jun 2012 04:38

Re: C10K Websocket Test

https://groups.google.com/d/topic/golang-nuts/bchGjWI39nY/discussion

headers

Jaybill McCarthy | 17 Jun 2012 04:40

Re: C10K Websocket Test

Dang! I searched for it on the list, too!

Oh, well. Sorry about the repeat, then.

--Jaybill

On Saturday, June 16, 2012 7:38:32 PM UTC-7, aam wrote:

https://groups.google.com/d/topic/golang-nuts/bchGjWI39nY/discussion

headers

Dave Cheney | 17 Jun 2012 04:41

Re: C10K Websocket Test

This benchmark was taken on shared virtualized hardware running over a shared network segment. I do not believe the author has sufficiently eliminated the background noise present in the environment and thus the results do not represent a valid base for comparison.

Dave

On 17/06/2012, at 12:31, Jaybill McCarthy <heybill <at> jaybill.com> wrote:

I know this list isn't too big on benchmarks, but I just saw this on HN and thought people might find it of interest. Apparently, Go did really well in a C10K Websocket Test. From the article:

"This benchmark starts a new client every 1ms, each client sends a timestamp message to the server every second and the server echos that message back to the client."

Erlang won, Go was a close second. (Neat!) Java, Haskell, Node & Python failed.

Here's the whole thing:

https://github.com/ericmoritz/wsdemo/blob/results-v1/results.md

headers

Mikael Gustavsson | 18 Jun 2012 03:07

Re: C10K Websocket Test

Jaybill:

Some values are promising from Go's perspective (total messages processed and concurrent connections).

But the Go server actually fared worse than the Haskell and Java ones, look at the latency values.

On Sunday, June 17, 2012 10:41:29 AM UTC+8, Dave Cheney wrote:

This benchmark was taken on shared virtualized hardware running over a shared network segment. I do not believe the author has sufficiently eliminated the background noise present in the environment and thus the results do not represent a valid base for comparison.

The Go server had 18 seconds of mean latency, it's unlikely that's because of background noise.

Clearly, all of the server implementations were under very heavy load, and the Erlang server handled it best.

The actual values (like 18s for Go) is almost meaningless when a system is overloaded, but you can make a statement about whether the system was able to handle the load or not, and this implementation of the server in Go didn't handle it nearly as good as the Erlang server.

headers

kortschak | 18 Jun 2012 05:20

Re: C10K Websocket Test

How do you measure the latency of a dropped messages? and what does this do to the claim that haskell and java fared better?

On Monday, 18 June 2012 10:37:41 UTC+9:30, Mikael Gustavsson wrote:

Jaybill:

Some values are promising from Go's perspective (total messages processed and concurrent connections).

But the Go server actually fared worse than the Haskell and Java ones, look at the latency values.

On Sunday, June 17, 2012 10:41:29 AM UTC+8, Dave Cheney wrote:
This benchmark was taken on shared virtualized hardware running over a shared network segment. I do not believe the author has sufficiently eliminated the background noise present in the environment and thus the results do not represent a valid base for comparison.

The Go server had 18 seconds of mean latency, it's unlikely that's because of background noise.

Clearly, all of the server implementations were under very heavy load, and the Erlang server handled it best.

The actual values (like 18s for Go) is almost meaningless when a system is overloaded, but you can make a statement about whether the system was able to handle the load or not, and this implementation of the server in Go didn't handle it nearly as good as the Erlang server.

headers

André Moraes | 19 Jun 2012 01:27

Re: C10K Websocket Test

>
> The Go server had 18 seconds of mean latency, it's unlikely that's because
> of background noise.
> Clearly, all of the server implementations were under very heavy load, and
> the Erlang server handled it best.
> The actual values (like 18s for Go) is almost meaningless when a system is
> overloaded, but you can make a statement about whether the system was able
> to handle the load or not, and this implementation of the server in Go
> didn't handle it nearly as good as the Erlang server.
>
Quoting the author of the benchmark
(http://www.reddit.com/r/programming/comments/v5ap8/web_server_benchmark_haskellsnap_erlang/c51wv9i):
"I wrote the thing and I can tell you that yes, these results are
pretty much crap. I incorrectly used Folsom, the project I used to
collect the stats. Folsom is meant for realtime monitoring and not
what I used it for. It only keeps 60 second windows of the data it
produces. That means all times prior to the fine 60 second window was
thrown out.

I tweeted the results at 3am to get feedback from folks I know.
Unfortunately it spread across the net really fast.
"

Also, using the standard go websocket client/server in localhost, the
time is measured in micro-seconds with only a few milliseconds.

This benchmark is wrong.

--

-- 
André Moraes
http://amoraes.info

headers

tomwilde | 19 Jun 2012 05:09

Re: C10K Websocket Test

He posted this now:

"I'm sorry I was mistaken that Folsom only keeps a 60 second window. It has support for sliding windows but
its default is to use the entire dataset for histograms and percentiles. I take back my statement that the
result were pretty much crap. They're only slightly crappy because I used a m1.medium box.
I'm spending my free time this week to make the benchmark more automated so that it is easier for someone who
volunteered to run it in their datacenter on real hardware. That should produce a much fairer result."