Tuning documentation for a high connection count configuration?

[John Kalucki]

I'd like to test APE to scale to the maximum number of connections per
host. I can't find any tuning documentation or tutorials on large-
scale deploys.

What OS changes are needed to support 100,000 or more connections?
Are there any APE changes required?
Any success stories with running with 100k connections in production?
What are the hard limits on connections? Why not 500k connections per
host?

-John Kalucki
jkal...@gmail.com

[Anthony Catel]

Hi John,

This limit really depends on a large number of factors :

- How many updates/sec are you going to send to your users?
- Does users can directly interact on updates? (Does a user can send something to APE)
- What is the average of IE users (Long polling => TCP flood / HTTP overhead => Bandwidth / CPU / Memory usage)
- Are you going to use server-side Javascript?

Basically, here is what I've tuned on my kernel :

- /proc/sys/net/core/somaxconn
- /proc/sys/net/netfilter/nf_conntrack_max
- /proc/sys/net/ipv4/tcp_*

We are doing all of our benchmark with Tsung. Btw, as far as I know, none of our user has such a platform (I mean with 100k concurents users) to give us feedback ;)

If you plan to use it in a very large platform, I'll be very excited to help you. Feel free to drop me an email ;)

Anthony Catel

Le 19/12/09 16:38, John Kalucki a écrit :

[John Kalucki]

Well, I have 8 or 16 core boxes with gigabit outbound for FEs, so
there's a lot of CPU and bandwidth to fill. I was wondering if there
were any hard limits found, or, what, if anything would keep a server
from serving more than 100,000 connections...

All of the clients will be custom HTTP clients, and use long lived
connections (> 60 minutes), users send nothing after connecting. Each
user will receive a few updates a minute, at most.

-John

On Dec 20, 10:10 am, Anthony Catel <a.ca...@weelya.com> wrote:
> Hi John,
>
> This limit really depends on a large number of factors :
>
> - How many updates/sec are you going to send to your users?
> - Does users can directly interact on updates? (Does a user can send
> something to APE)
> - What is the average of IE users (Long polling => TCP flood / HTTP
> overhead => Bandwidth / CPU / Memory usage)
> - Are you going to use server-side Javascript?
>
> Basically, here is what I've tuned on my kernel :
>
> - /proc/sys/net/core/somaxconn
> - /proc/sys/net/netfilter/nf_conntrack_max
> - /proc/sys/net/ipv4/tcp_*
>
> We are doing all of our benchmark with Tsung. Btw, as far as I know,
> none of our user has such a platform (I mean with 100k concurents users)
> to give us feedback ;)
>
> If you plan to use it in a very large platform, I'll be very excited to
> help you. Feel free to drop me an email ;)
>
> Anthony Catel
>

> Le 19/12/09 16:38, John Kalucki a �crit :

>
>
>
> > I'd like to test APE to scale to the maximum number of connections per
> > host. I can't find any tuning documentation or tutorials on large-
> > scale deploys.
>
> > What OS changes are needed to support 100,000 or more connections?
> > Are there any APE changes required?
> > Any success stories with running with 100k connections in production?
> > What are the hard limits on connections? Why not 500k connections per
> > host?
>
> > -John Kalucki

> > jkalu...@gmail.com

[carldoemarchis]

I'm also very interested in using APE server for a huge number of
connected clients. Those clients will be in read only mode getting
updates from a central logic. Will need to understand how to architect
such a solution to be scalable - maybe on AWS with multiple nodes.

[Anthony Catel]

Indeed, there is no hard limit.
Btw if you build a custom HTTP user agent, I guess you are not going to
use XHRStreaming or Long polling.

I'm working on a new transport method for long lived connection. It will
be available on the next release (1.0.1) this week.

Anthony

Le 21/12/09 05:02, John Kalucki a �crit :

>> Le 19/12/09 16:38, John Kalucki a �crit :