100K concurrent TCP connections?

[Monnand]

Hi all,

I just had an interesting conversation with a friend. I think the
problem we are talking about is quite open ended. So I would like to
share it and see what others say about it.

Let's first take a look at the problem: My friend is working on a
service which needs to handle lots of concurrent TCP connections. As you
can imagine, I suggest him to use Go.

He then told me that his company would like to handle ~100K connections
per server. His colleague made a quick benchmark in Go the other day and
the Go program handles 5K connections by utilizing 200% of the CPU.

5K connections is apparently not the maximum that a Go program can get.
(As far as I know, vitess claims it can handle 10K connections.)
Moreover, such benchmark relies on the underlying hardware, OS
configuration, etc. (I'm unable to get details about the benchmark or
the configuration information.)

Although it's not a new problem to this field, his problem made me
google some existing solutions/suggestions about handling large amount
of concurrent TCP connections. For something like node.js, we need some
tweaks[1]. Eventually, "The kernel isn’t the solution. The kernel is the
problem."

Specific to Go, I didn't find anything related to handling large amount
of concurrent TCP connections. But I believe there must be someone who
has the experience of working on such scale. It would be great if you
can share here. Any suggestion/discussion is welcome. As I mentioned,
this is an open ended problem. I won't mind if you throw me some crazy idea.

[1]
http://blog.caustik.com/2012/04/08/scaling-node-js-to-100k-concurrent-connections/
[2]
http://highscalability.com/blog/2013/5/13/the-secret-to-10-million-concurrent-connections-the-kernel-i.html

[Dave Cheney]

We've heard on this mailing list form people who (I believe) run large
chat servers that host upwards of half a million concurrent
connections. The main problem the OP had was the amount of per
connection memory his application used. From memory it was 28k per
connection.

> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[Dmitry Vyukov]

Go is not as efficient as C yet. But 100K should be possible on a
beefy machine (both RAM and CPU).

I've tested vitess with 10K connections and it runs fine. However I
was more interested in total throughput (I was able to achieve ~120K
fat (including external request to memcached) requests per second).

[Rajiv Kurian]

Is this an implementation detail of the net package or more of a state of the go compiler?

[Dave Cheney]

Neither. 100k connections consumes lots of memory in both the kernel
and the application. If you are using one goroutine per connection,
you need to account for at least 4k per connection, plus overhead for
actually reading and processing data. On the kernel side you need to
account for memory for SKBs and other data structures.

On Mon, Jul 22, 2013 at 8:15 PM, Rajiv Kurian <geet...@gmail.com> wrote:
> Is this an implementation detail of the net package or more of a state of the go compiler?
>

[Rajiv Kurian]

100k is a very possible number using epoll straight up or even using java NIO. These numbers have been achieved on very bland EC2 machines - http://urbanairship.com/blog/2010/08/24/c500k-in-action-at-urban-airship/

Given that this is not an OS limitation, I'll assume that the overhead of a go routine per connection is a limitation.

[Dave Cheney]

This is a problem of memory. Goroutines consume memory, so do threads,
so do buffers, and so on.

On Mon, Jul 22, 2013 at 8:28 PM, Rajiv Kurian <geet...@gmail.com> wrote:
> 100k is a very possible number using epoll straight up or even using java NIO. These numbers have been achieved on very bland EC2 machines - http://urbanairship.com/blog/2010/08/24/c500k-in-action-at-urban-airship/
>
> Given that this is not an OS limitation, I'll assume that the overhead of a go routine per connection is a limitation.
>

[Dmitry Vyukov]

On Mon, Jul 22, 2013 at 2:28 PM, Rajiv Kurian <geet...@gmail.com> wrote:
> 100k is a very possible number using epoll straight up or even using java NIO. These numbers have been achieved on very bland EC2 machines - http://urbanairship.com/blog/2010/08/24/c500k-in-action-at-urban-airship/
>
> Given that this is not an OS limitation, I'll assume that the overhead of a go routine per connection is a limitation.

A goroutine adds ~6.5K. We need to reduce it in long term...

[Rajiv Kurian]

Yeah, so the go routine memory overhead then.

[Robert Melton]

On Mon, Jul 22, 2013 at 12:28 AM, Monnand <mon...@gmail.com> wrote:

He then told me that his company would like to handle ~100K connections per server. His colleague made a quick benchmark in Go the other day and the Go program handles 5K connections by utilizing 200% of the CPU.

Any chance he could post that test?  So far, my experience with semi-useless toys tests has been easy (effortless actually) scaling into the six digits concurrent assuming the memory is available.  Maybe he hit a hidden gotcha and wasn't aware of it?  

-- 

Robert Melton

[nga...@gmail.com]

1000k connections is ok if you got enough memory. 

bug gc is the problem, you program may pause ~3 seconds

在 2013年7月21日星期日UTC-8下午8时28分34秒，Monnand写道：

[atomly]

honestly if he needs to handle 100k concurrent connections easily, then i would say he would have a very easy time with anything based around the standard event multiplexing models (e.g. epoll on linux) such as java.nio/netty...

:: atomly ::

[ ato...@atomly.com : www.atomly.com  : http://blog.atomly.com/ ...
[ atomiq records : new york city : +1.347.692.8661 ...
[ e-mail atomly-new...@atomly.com for atomly info and updates ...

On Mon, Jul 22, 2013 at 12:28 AM, Monnand <mon...@gmail.com> wrote:

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.

To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.

[Monnand]

Thank you all your replies! All of them are very useful.

On 07/22/2013 12:28 AM, Monnand wrote:
<snip>

> He then told me that his company would like to handle ~100K connections
> per server. His colleague made a quick benchmark in Go the other day and
> the Go program handles 5K connections by utilizing 200% of the CPU.
>

I'm pretty sure that this guy made a wrong benchmark. According to my
friend's description, he just wanted to make a simple program and see
how many connections it can handle.

I wrote a quite simple echo server and tested on my own laptop. It can
easily handle 20K connections on my laptop running both server and
clients (this means 40K fds in total. Considering I was using firefox
watching youtube, thunderbird receiving email, etc, it should work much
faster on a dedicated server. No need to say more advanced hardware on
those servers -- I'm pretty sure they are not using EC2 or any publicly
available cloud instances) I will contact my friend to know more details
about the benchmark. I think it is totally safe to simply ignore the
results of his colleague's benchmark.

The source code of my test programs (both server and client) is
available here:

https://github.com/monnand/c10k-go

With only 46 lines of code, the server scales linearly with the number
of connections. Goroutines have very low footprint.

Regards,
-Monnand

[Sugu Sougoumarane]

https://groups.google.com/forum/#!searchin/golang-nuts/Garbage$20collecting/golang-nuts/S9goEGuoMRM/FZyd2M6uiVMJ

This person was using 600k connections in a production environment, and he was using a very old version of go. Things are even better now.

[Monnand]

This is very useful! Thank you very much!

-Monnand