Max outbound TCP Connections on Windows Server 2008
Dave
windows server 2008 box. The application performs large numbers of
outbound connections to TCP servers located all over the world.
I seem to be maxing out at about 1600 connections, and about 1250
(sustained for over 15 min) connections. I’m trying to connection to
over 7K addresses / ports, some are bad but most are good and expected
my sustained connection number be well over 2000. The box has 16 GB
of memory with 2 quad cores, and isn’t near CPU or memory bound.
I’ve ensured the MaxUserPort is 65534, and TcpTimedWaitDelay is 30
(secs). MaxFreeTcbs and MaxHashTableSize are deprecated in vista /
server 2k8 I believe. (http://smallvoid.com/article/winnt-tcpip-max-
limit.html was helpful, although dated,
http://www.microsoft.com/downloads/details.aspx?FamilyId=12AC9780-17B5-480C-AEF7-5C0BDE9060B0&displaylang=en
was helpful but have not spotted missing registry values yet.)
The application is decently coded, ensuring to not immediately retry
to reconnect upon failures, and waits a configurable amount of time
(now about 15 minutes).
If you need more information I’d be glad to provide it, and any help
would be appreciated, thanks,
dave
David Schwartz
> I seem to be maxing out at about 1600 connections, and about 1250
> (sustained for over 15 min) connections. I’m trying to connection to
> over 7K addresses / ports, some are bad but most are good and expected
> my sustained connection number be well over 2000. The box has 16 GB
> of memory with 2 quad cores, and isn’t near CPU or memory bound.
What happens when you max out? Why are you limited to 1,600
connections? What happens when your code asks for connection 1,601?
Does some function take a very long time? If so, what function?
> The application is decently coded, ensuring to not immediately retry
> to reconnect upon failures, and waits a configurable amount of time
> (now about 15 minutes).
What does your I/O model look like? How do you wait for all the
outstanding connection requests to complete? How do you detect if
they've succeeded or failed? Are you using completion routines?
Completion ports? Or what?
DS
Dave
> On Mar 26, 12:47 pm, Dave <djohanns...@gmail.com> wrote:
>
> > I seem to be maxing out at about 1600 connections, and about 1250
> > (sustained for over 15 min) connections. I’m trying to connection to
> > over 7K addresses / ports, some are bad but most are good and expected
> > my sustained connection number be well over 2000. The box has 16 GB
> > of memory with 2 quad cores, and isn’t near CPU or memory bound.
>
> What happens when you max out? Why are you limited to 1,600
> connections? What happens when your code asks for connection 1,601?
> Does some function take a very long time? If so, what function?
I'm not sure that i'm maxing out, but my connections are much lower
than expected at 1600, no visible errors at 1601 or anything like
that, but i'm seeing an error in my AsyncSendCallback() method: Cannot
access a disposed object. Its super hard to tell if one function is
taking a long time over another as the app is multithreaded / using
IOCP. i'm not instrumenting these values, i'd hoped to avoid it and
find the possible issue without that as i know that would take a while
to code. i'm not seeing any timeout errors in my error logs, and i'm
logging all errors.
>
> > The application is decently coded, ensuring to not immediately retry
> > to reconnect upon failures, and waits a configurable amount of time
> > (now about 15 minutes).
>
> What does your I/O model look like? How do you wait for all the
> outstanding connection requests to complete? How do you detect if
> they've succeeded or failed? Are you using completion routines?
> Completion ports? Or what?
>
> DS
i'm using async sockets with BeginConnect(), EndConnect(),
BeginSend(), EndSend(), ... and detecting / logging all socket errors
(connect / callback, recv / callback, send / callback, etc). so yes
using completion methods.
let me know what other info you need to be able to help out.
thanks, dave
David Schwartz
> > What does your I/O model look like? How do you wait for all the
> > outstanding connection requests to complete? How do you detect if
> > they've succeeded or failed? Are you using completion routines?
> > Completion ports? Or what?
> i'm using async sockets with BeginConnect(), EndConnect(),
> BeginSend(), EndSend(), ... and detecting / logging all socket errors
> (connect / callback, recv / callback, send / callback, etc). so yes
> using completion methods.
Honestly, I'm not that familiar with these .NET functions, but it
sounds like they're inefficient for the same reason completion
routines are. I'm assuming a callback always runs by the same thread
that initiated the operation. So you may see a ridiculous (and
senseless) number of context switches to get the "right thread" to run
to complete each operation.
That said, 1,600 is usually not the wall you hit with callback
routines, it's usually more like 4,000 or so. IOCP is usually good to
16,000 or so.
DS
Dave
yeah i did IOCP with C++ app in the past and we got it up to 30K+
connections after we fully optimized it, so i agree IOCP is of course
better. this app is decently coded though, and the completion
routines do seem to hum along pretty well with the completion
routimes, save my little possible resource issue (but i don't think
completion methods are to blame).
thanks so much for your time, hard to diagnose remotely!
dave