Re: [ace-users] ACE MAXIMUM FILE DESCRIPTORS LIMIT PROBLEM

Steve Huston

unread,

Dec 13, 2009, 7:42:14 PM12/13/09

to Arkady, ace-...@list.isis.vanderbilt.edu

Hi Arkady,

The source for ACE_Dev_Poll_Reactor is in ACE_wrappers/ace/Dev_Poll_Reactor.cpp - your first two questions can be researched there.

Starting 5000 threads is not a good way to get responsiveness, unless you have about 5000 processors. Try it with one thread to start and see how that goes.

-Steve

--
Steve Huston, Riverace Corporation
Total Lifecycle Support for Your Networked Applications
http://www.riverace.com

-----Original Message-----
From: Arkady [mailto:ark...@pineapp.com]
Sent: Sunday, December 13, 2009 6:32 AM
To: Steve Huston
Cc: ace-...@list.isis.vanderbilt.edu
Subject: Re: [ace-users] ACE MAXIMUM FILE DESCRIPTORS LIMIT PROBLEM

Hi Steve,
Before I finally release my Http proxy with ACE_Dev_Poll_Reactor I will appreciate if you answer to some questions:
1. Where can i see ACE_Dev_Poll_Reactor source code .
2. What thread model ACE_Dev_Poll_Reactor uses (Leader/Follower as ACE_TP_Reactor or something else)
3. I am going create 5000 threads with the following code ACE_Reactor::instance ()->run_reactor_event_loop (); to allow maximum responsiveness . What is you opinion about it ?

Thanks in advance ,Arkady

Steve Huston wrote:

It's stable and reliable, but not as extensible as you are looking for. For 10K connections, I recommend you use ACE_Dev_Poll_Reactor on Linux. You can use ACE_TP_Reactor (if you alter the FD_SETSIZE limit beyond 1024) but performance may become an issue.

-Steve

--
Steve Huston, Riverace Corporation
Total Lifecycle Support for Your Networked Applications
http://www.riverace.com

-----Original Message-----
From: Arkady [mailto:ark...@pineapp.com]
Sent: Thursday, December 10, 2009 12:17 PM
To: Steve Huston
Cc: ace-...@list.isis.vanderbilt.edu
Subject: Re: [ace-users] ACE MAXIMUM FILE DESCRIPTORS LIMIT PROBLEM

Hi,thanks a lot for the answer, but does it means that I do not have stable and reliable ACE solution now ?

Arkady
Steve Huston wrote:

Please see:

http://www.riverace.com/newsletters/March2005.htm

--
Steve Huston, Riverace Corporation
Total Lifecycle Support for Your Networked Applications
http://www.riverace.com

-----Original Message-----
From: ace-user...@list.isis.vanderbilt.edu [mailto:ace-user...@list.isis.vanderbilt.edu] On Behalf Of Arkady
Sent: Thursday, December 10, 2009 11:14 AM
To: ace-...@list.isis.vanderbilt.edu
Subject: [ace-users] ACE MAXIMUM FILE DESCRIPTORS LIMIT PROBLEM

Hi,
I apologize if this is the second mail concerning this problem,but I am not sure that the first one was really send.

I write some HTTP proxy using ACE 5.7 Acceptor-Connector framework and ACE_TP_Reactor .
My computer parameters are:
Linux surfsecure.chimi.co.il 2.6.20.21-Surf-SeCure #11 SMP Thu Jun 4 16:07:27 IDT 2009 i686 i686 i386 GNU/Linux

I want to be able open about 10000 sockets , but default ACE installation allow about 1024 .

ulimit command output is :
unlimited

So what I supposed to do/change in ACE for achieve my goal ?

Thanks in advance ,Arkady

Arkady

unread,

Dec 14, 2009, 3:06:50 AM12/14/09

to Steve Huston, ace-...@list.isis.vanderbilt.edu

Hi,Steve

Do you have idea why I get 99.9 CPU once I start my reactor loop ?

This scare me .

Steve Huston

unread,

Dec 14, 2009, 7:20:22 AM12/14/09

to Arkady, ace-...@list.isis.vanderbilt.edu

I have seen a report about high CPU usage in ACE_Dev_Poll_Reactor. I have not yet seen a fix.

Arkady

unread,

Dec 14, 2009, 7:32:59 AM12/14/09

to Steve Huston, ace-...@list.isis.vanderbilt.edu

Hi Steve,
I apologize for keep bothering but if I create more then 2 threads (I have 2 CPUs) then ACE_Dev_Poll_Reactor enter to 99.9 %CPU .
Is it expected result ?

The problem is that my IO threads not only receive and send data,but use different other SLOW IO synchronous components ,and this why I need a lot of threads.
If I am forced to use only 2 IO threads , it will force to design changes,and I do not like it

Does I really have no choice ?

Douglas C. Schmidt

unread,

Dec 14, 2009, 8:40:11 AM12/14/09

to ark...@pineapp.com, ace-...@cse.wustl.edu

Hi Arkady,

If you're having issues with the ACE_Dev_Poll_Reactor I recommend
you either spend some time trying to debug it or contact one of the
companies listed at

http://www.dre.vanderbilt.edu/support.html

for commercial support.

Thanks,

Doug

>I apologize for keep bothering but if I create more then 2 threads (I
>have 2 CPUs) then ACE_Dev_Poll_Reactor enter to 99.9 %CPU .
>Is it expected result ?
>
>The problem is that my IO threads not only receive and send data,but use
>different other SLOW IO synchronous components ,and this why I need a
>lot of threads.
>If I am forced to use only 2 IO threads , it will force to design
>changes,and I do not like it
>
>Does I really have no choice ?
>
>Arkady
>
>
>Steve Huston wrote:
>> Hi Arkady,
>>
>> The source for ACE_Dev_Poll_Reactor is in
>> ACE_wrappers/ace/Dev_Poll_Reactor.cpp - your first two questions can
>> be researched there.
>>
>> Starting 5000 threads is not a good way to get responsiveness, unless
>> you have about 5000 processors. Try it with one thread to start and
>> see how that goes.
>>
>> -Steve
>>
>> --
>> Steve Huston, Riverace Corporation
>> Total Lifecycle Support for Your Networked Applications

>> http://www.riverace.com <http://www.riverace.com/>
>>
>> -----Original Message-----
>> *From:* Arkady [mailto:ark...@pineapp.com]
>> *Sent:* Sunday, December 13, 2009 6:32 AM
>> *To:* Steve Huston
>> *Cc:* ace-...@list.isis.vanderbilt.edu
>> *Subject:* Re: [ace-users] ACE MAXIMUM FILE DESCRIPTORS LIMIT PROBLEM

>>
>> Hi Steve,
>> Before I finally release my Http proxy with
>> ACE_Dev_Poll_Reactor I will appreciate if you answer to some
>> questions:
>> 1. Where can i see ACE_Dev_Poll_Reactor source code .
>> 2. What thread model ACE_Dev_Poll_Reactor uses (Leader/Follower as
>> ACE_TP_Reactor or something else)
>> 3. I am going create 5000 threads with the following code

>> *ACE_Reactor::instance ()->run_reactor_event_loop ()*; to allow

>> maximum responsiveness . What is you opinion about it ?
>>

>> *Thanks in advance ,Arkady*

>>
>>
>>
>>
>>
>> Steve Huston wrote:
>>> It's stable and reliable, but not as extensible as you are
>>> looking for. For 10K connections, I recommend you use
>>> ACE_Dev_Poll_Reactor on Linux. You can use ACE_TP_Reactor (if you
>>> alter the FD_SETSIZE limit beyond 1024) but performance may
>>> become an issue.
>>>
>>> -Steve
>>>
>>> --
>>> Steve Huston, Riverace Corporation
>>> Total Lifecycle Support for Your Networked Applications

>>> http://www.riverace.com <http://www.riverace.com/>
>>>
>>> -----Original Message-----
>>> *From:* Arkady [mailto:ark...@pineapp.com]
>>> *Sent:* Thursday, December 10, 2009 12:17 PM
>>> *To:* Steve Huston
>>> *Cc:* ace-...@list.isis.vanderbilt.edu
>>> *Subject:* Re: [ace-users] ACE MAXIMUM FILE DESCRIPTORS LIMIT
>>> PROBLEM
>>>
>>> Hi,thanks a lot for the answer, but does it means that I _*do
>>> not have* *stable* *and reliable*_ ACE solution now ?

>>>
>>> Arkady
>>> Steve Huston wrote:
>>>> Please see:
>>>> http://www.riverace.com/newsletters/March2005.htm
>>>>
>>>> --
>>>> Steve Huston, Riverace Corporation
>>>> Total Lifecycle Support for Your Networked Applications

>>>> http://www.riverace.com <http://www.riverace.com/>
>>>>
>>>> -----Original Message-----
>>>> *From:* ace-user...@list.isis.vanderbilt.edu
>>>> [mailto:ace-user...@list.isis.vanderbilt.edu] *On
>>>> Behalf Of *Arkady
>>>> *Sent:* Thursday, December 10, 2009 11:14 AM
>>>> *To:* ace-...@list.isis.vanderbilt.edu
>>>> *Subject:* [ace-users] ACE MAXIMUM FILE DESCRIPTORS

>>>> LIMIT PROBLEM
>>>>
>>>> Hi,
>>>> I apologize if this is the second mail concerning this
>>>> problem,but I am not sure that the first one was really
>>>> send.
>>>>
>>>> I write some HTTP proxy using ACE 5.7 Acceptor-Connector
>>>> framework and ACE_TP_Reactor .
>>>> My computer parameters are:
>>>> Linux surfsecure.chimi.co.il 2.6.20.21-Surf-SeCure #11
>>>> SMP Thu Jun 4 16:07:27 IDT 2009 i686 i686 i386 GNU/Linux
>>>>

>>>> *I want to be able open about 10000 sockets , but
>>>> default ACE installation allow about 1024 .*
>>>>
>>>> *ulimit* command output is :
>>>> *unlimited

>>>>
>>>> So what I supposed to do/change in ACE for achieve my goal ?
>>>>
>>>> Thanks in advance ,Arkady
>>>>

>>>> *
>>>>
>>>
>>
>
>
>--------------090405080101030708020701
>Content-Type: text/html; charset="ISO-8859-1"
>Content-Transfer-Encoding: 7bit
>
><!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
><html>
><head>
> <meta content="text/html;charset=ISO-8859-1" http-equiv="Content-Type">
></head>
><body bgcolor="#ffffff" text="#000000">
><meta http-equiv="CONTENT-TYPE" content="text/html; charset=utf-8">
><title></title>
><meta name="GENERATOR" content="OpenOffice.org 2.4 (Linux)">
><style type="text/css">
> </style>
><meta http-equiv="CONTENT-TYPE" content="text/html; charset=utf-8">
><title></title>
><meta name="GENERATOR" content="OpenOffice.org 2.4 (Linux)">
><style type="text/css">
> 
> </style>
>Hi Steve,

>I apologize for keep
>bothering but if I create more then 2 threads (I have 2 CPUs) then

>ACE_Dev_Poll_Reactor
>enter to 99.9 %CPU . face="Arial"> 
> size="2">Is
>it expected result ? face="Arial"> 
> 
> size="2">The

>problem is that my IO threads not only receive and send data,but use
>different other SLOW IO synchronous components ,and this why I need a

>lot of threads. 
>If I am forced  to  use only 2  IO 
>threads  , it will  force to design changes,and I do not
>like it  
> size="2">Does I really have no choice ?
> size="2">Arkady
> size="2"> 
>Steve Huston wrote:
><blockquote cite="mid:4AA2ECDFDF904F6AA543875A037C0AA8@hope" type="cite">
> <meta http-equiv="Content-Type" content="text/html; ">
> <title>Message</title>
> <meta name="GENERATOR" content="MSHTML 8.00.6001.18854">
> <div> face="Arial" size="2">Hi Arkady,</div>
> <div> </div>
> <div> face="Arial" size="2">The source for ACE_Dev_Poll_Reactor is in

>ACE_wrappers/ace/Dev_Poll_Reactor.cpp - your first two questions can be

>researched there.</div>
> <div> </div>
> <div> face="Arial" size="2">Starting 5000 threads is not a good way to get

>responsiveness, unless you have about 5000 processors. Try it with one

>thread to start and see how that goes.</div>
> <div> </div>
> <div> face="Arial" size="2">-Steve</div>
>
> -- 
>Steve Huston, Riverace Corporation 
>Total Lifecycle Support for Your Networked Applications 
> <a moz-do-not-send="true" href="http://www.riverace.com/">http://www.riverace.com</a> 
> 
> <blockquote style="margin-right: 0px;">
> <div dir="ltr" class="OutlookMessageHeader" align="left"
> lang="en-us">-----Original Message----- 
> From: Arkady [<a class="moz-txt-link-freetext" href="mailto:ark...@pineapp.com">mailto:ark...@pineapp.com</a>] 
> Sent: Sunday, December 13, 2009 6:32 AM 
> To: Steve Huston 
> Cc: <a class="moz-txt-link-abbreviated" href="mailto:ace-...@list.isis.vanderbilt.edu">ace-...@list.isis.vanderbilt.edu</a> 
> Subject: Re: [ace-users] ACE MAXIMUM FILE DESCRIPTORS LIMIT
>PROBLEM 
> 
> </div>
>Hi Steve, 
>Before I finally  release my  Http proxy  with class="181172317-10122009">ACE_Dev_Poll_Reactor 
>I will appreciate if you answer to some questions: 
>1. Where can i see class="181172317-10122009">ACE_Dev_Poll_Reactor
> face="Arial" size="2">source code  . 
>2. What thread model class="181172317-10122009"> color="#0000ff" face="Arial" size="2">ACE_Dev_Poll_Reactor uses
>(Leader/Follower as ACE_TP_Reactor or something else) 
>3. I am going create 5000 threads with the following code  ACE_Reactor::instance
>()->run_reactor_event_loop (); to allow maximum responsiveness .
>What is you opinion about it ? 
> 
> Thanks in advance ,Arkady 
> 
> 
> 
> 
> 
>Steve Huston wrote:
> <blockquote cite="mid:C1FEE3E696874907B87F5E8AA1DA22E7@hope"
> type="cite">
> <meta name="GENERATOR" content="MSHTML 8.00.6001.18854">
> <div> face="Arial" size="2">It's stable and reliable, but not as extensible

>as you are looking for. For 10K connections, I recommend you use
>ACE_Dev_Poll_Reactor on Linux. You can use ACE_TP_Reactor (if you alter

>the FD_SETSIZE limit beyond 1024) but performance may become an issue.</div>
> <div> </div>
> <div> face="Arial" size="2">-Steve</div>
>
> -- 
>Steve Huston, Riverace Corporation 
>Total Lifecycle Support for Your Networked Applications 
> <a href="http://www.riverace.com/" moz-do-not-send="true">http://www.riverace.com</a> 
> 
> <blockquote
> style="border-left: 2px solid rgb(0, 0, 255); padding-left: 5px; margin-left: 5px; margin-right: 0px;">
> <div dir="ltr" class="OutlookMessageHeader" align="left"
> lang="en-us">-----Original Message----- 
> From: Arkady [<a moz-do-not-send="true"
> class="moz-txt-link-freetext" href="mailto:ark...@pineapp.com">mailto:ark...@pineapp.com</a>]
> 
> Sent: Thursday, December 10, 2009 12:17 PM 
> To: Steve Huston 
> Cc: <a moz-do-not-send="true"
> class="moz-txt-link-abbreviated"
> href="mailto:ace-...@list.isis.vanderbilt.edu">ace-...@list.isis.vanderbilt.edu</a> 
> Subject: Re: [ace-users] ACE MAXIMUM FILE DESCRIPTORS
>LIMIT PROBLEM 
> 
> </div>
>Hi,thanks a lot for the answer, but does it means that I do not
>have stable and reliable ACE solution now ? 
> 
>Arkady 
>Steve Huston wrote:
> <blockquote cite="mid:C33A60DDB4584E6B98E4C91D688C56BA@hope"
> type="cite">
> <meta name="GENERATOR" content="MSHTML 8.00.6001.18854">
> <div> class="967234116-10122009">Please see:</div>
> <div><a
> href="http://www.riverace.com/newsletters/March2005.htm"
> moz-do-not-send="true">http://www.riverace.com/newsletters/March2005.htm</a></div>
>
> -- 
>Steve Huston, Riverace Corporation 
>Total Lifecycle Support for Your Networked Applications 
> <a href="http://www.riverace.com/" moz-do-not-send="true">http://www.riverace.com</a> 
> 
> <blockquote
> style="border-left: 2px solid rgb(0, 0, 255); padding-left: 5px; margin-left: 5px; margin-right: 0px;">
> <div dir="ltr" class="OutlookMessageHeader" align="left"
> lang="en-us">-----Original Message----- 
> From: <a class="moz-txt-link-abbreviated"
> href="mailto:ace-user...@list.isis.vanderbilt.edu"
> moz-do-not-send="true">ace-user...@list.isis.vanderbilt.edu</a> [<a
> class="moz-txt-link-freetext"
> href="mailto:ace-user...@list.isis.vanderbilt.edu"
> moz-do-not-send="true">mailto:ace-user...@list.isis.vanderbilt.edu</a>]
> On Behalf Of Arkady 
> Sent: Thursday, December 10, 2009 11:14 AM 
> To: <a class="moz-txt-link-abbreviated"
> href="mailto:ace-...@list.isis.vanderbilt.edu" moz-do-not-send="true">ace-...@list.isis.vanderbilt.edu</a> 
> Subject: [ace-users] ACE MAXIMUM FILE DESCRIPTORS
>LIMIT PROBLEM 
> 
> </div>
>Hi,

>I apologize if this is the second mail concerning this problem,but I am

>not sure that the first one was really send. 
>

>I write some HTTP proxy using ACE 5.7 Acceptor-Connector framework and

>ACE_TP_Reactor . 
>My computer parameters are:

>Linux surfsecure.chimi.co.il 2.6.20.21-Surf-SeCure #11 SMP Thu Jun 4

>16:07:27 IDT 2009 i686 i686 i386 GNU/Linux 
> 
> I want to be able open about 10000 sockets , but default
>ACE installation allow about 1024 . 
> 
> ulimit command output is : 
> unlimited 
> 
>So what I supposed to do/change in ACE for achieve my goal ? 
> 
>Thanks in advance ,Arkady 
>  
> 
> 
> </blockquote>
> </blockquote>
> 
> </blockquote>
> </blockquote>
> 
> </blockquote>
></blockquote>
> 
></body>
></html>
>
>--------------090405080101030708020701--

--
Dr. Douglas C. Schmidt Professor and Associate Chair
Electrical Engineering and Computer Science TEL: (615) 343-8197
Vanderbilt University WEB: www.dre.vanderbilt.edu/~schmidt
Nashville, TN 37203 NET: d.sc...@vanderbilt.edu

Vadym Krevs

unread,

Dec 14, 2009, 9:29:20 AM12/14/09

to ace-...@cse.wustl.edu

Sounds like a similar issue with ACE_Dev_Poll_Reactor from ACE 5.7 which Sergio Ruiz reported back in June:

http://stevehuston.wordpress.com/2009/06/15/revised-ace_dev_poll_reactor-fixes-multithread-issues-and-more-on-linux/#comment-44

Regards,
Vadym

2009/12/14 Douglas C. Schmidt <sch...@dre.vanderbilt.edu>

Hi Arkady,

If you're having issues with the ACE_Dev_Poll_Reactor I recommend
you either spend some time trying to debug it or contact one of the
companies listed at

http://www.dre.vanderbilt.edu/support.html

for commercial support.

Thanks,

Doug

>I apologize for keep bothering but if I create more then 2 threads (I
>have 2 CPUs) then ACE_Dev_Poll_Reactor enter to 99.9 %CPU .
>Is it expected result ?
>
>The problem is that my IO threads not only receive and send data,but use
>different other SLOW IO synchronous components ,and this why I need a
>lot of threads.
>If I am forced to use only 2 IO threads , it will force to design
>changes,and I do not like it
>
>Does I really have no choice ?
>
>Arkady
>
>
>Steve Huston wrote:

>> Hi Arkady,
>>
>> The source for ACE_Dev_Poll_Reactor is in
>> ACE_wrappers/ace/Dev_Poll_Reactor.cpp - your first two questions can
>> be researched there.
>>
>> Starting 5000 threads is not a good way to get responsiveness, unless
>> you have about 5000 processors. Try it with one thread to start and
>> see how that goes.
>>
>> -Steve
>>
>> --
>> Steve Huston, Riverace Corporation
>> Total Lifecycle Support for Your Networked Applications

>> http://www.riverace.com <http://www.riverace.com/>

>>
>> -----Original Message-----
>> *From:* Arkady [mailto:ark...@pineapp.com]
>> *Sent:* Sunday, December 13, 2009 6:32 AM
>> *To:* Steve Huston
>> *Cc:* ace-...@list.isis.vanderbilt.edu

>> *Subject:* Re: [ace-users] ACE MAXIMUM FILE DESCRIPTORS LIMIT PROBLEM

>>
>> Hi Steve,
>> Before I finally release my Http proxy with
>> ACE_Dev_Poll_Reactor I will appreciate if you answer to some
>> questions:
>> 1. Where can i see ACE_Dev_Poll_Reactor source code .
>> 2. What thread model ACE_Dev_Poll_Reactor uses (Leader/Follower as
>> ACE_TP_Reactor or something else)
>> 3. I am going create 5000 threads with the following code

>> *ACE_Reactor::instance ()->run_reactor_event_loop ()*; to allow

>> maximum responsiveness . What is you opinion about it ?
>>

>> *Thanks in advance ,Arkady*

>>
>>
>>
>>
>>
>> Steve Huston wrote:
>>> It's stable and reliable, but not as extensible as you are
>>> looking for. For 10K connections, I recommend you use
>>> ACE_Dev_Poll_Reactor on Linux. You can use ACE_TP_Reactor (if you
>>> alter the FD_SETSIZE limit beyond 1024) but performance may
>>> become an issue.
>>>
>>> -Steve
>>>
>>> --
>>> Steve Huston, Riverace Corporation
>>> Total Lifecycle Support for Your Networked Applications

>>> http://www.riverace.com <http://www.riverace.com/>

>>>
>>> -----Original Message-----
>>> *From:* Arkady [mailto:ark...@pineapp.com]

>>> *Sent:* Thursday, December 10, 2009 12:17 PM

>>> *To:* Steve Huston
>>> *Cc:* ace-...@list.isis.vanderbilt.edu

>>> *Subject:* Re: [ace-users] ACE MAXIMUM FILE DESCRIPTORS LIMIT
>>> PROBLEM
>>>

>>> Hi,thanks a lot for the answer, but does it means that I _*do

>>> not have* *stable* *and reliable*_ ACE solution now ?

>>>
>>> Arkady
>>> Steve Huston wrote:
>>>> Please see:
>>>> http://www.riverace.com/newsletters/March2005.htm
>>>>
>>>> --
>>>> Steve Huston, Riverace Corporation
>>>> Total Lifecycle Support for Your Networked Applications

>>>> http://www.riverace.com <http://www.riverace.com/>

>>>>
>>>> -----Original Message-----
>>>> *From:* ace-user...@list.isis.vanderbilt.edu
>>>> [mailto:ace-user...@list.isis.vanderbilt.edu] *On
>>>> Behalf Of *Arkady
>>>> *Sent:* Thursday, December 10, 2009 11:14 AM
>>>> *To:* ace-...@list.isis.vanderbilt.edu

>>>> *Subject:* [ace-users] ACE MAXIMUM FILE DESCRIPTORS

>>>> LIMIT PROBLEM
>>>>
>>>> Hi,
>>>> I apologize if this is the second mail concerning this
>>>> problem,but I am not sure that the first one was really
>>>> send.
>>>>
>>>> I write some HTTP proxy using ACE 5.7 Acceptor-Connector
>>>> framework and ACE_TP_Reactor .
>>>> My computer parameters are:
>>>> Linux surfsecure.chimi.co.il 2.6.20.21-Surf-SeCure #11
>>>> SMP Thu Jun 4 16:07:27 IDT 2009 i686 i686 i386 GNU/Linux
>>>>

>>>> *I want to be able open about 10000 sockets , but
>>>> default ACE installation allow about 1024 .*
>>>>
>>>> *ulimit* command output is :
>>>> *unlimited
>>>>

>>>> So what I supposed to do/change in ACE for achieve my goal ?
>>>>
>>>> Thanks in advance ,Arkady
>>>>

>I apologize for keep
>bothering but if I create more then 2 threads (I have 2 CPUs) then

>ACE_Dev_Poll_Reactor
>enter to 99.9 %CPU . face="Arial"> 
> size="2">Is
>it expected result ? face="Arial"> 
> 
><font

> size="2">The

>problem is that my IO threads not only receive and send data,but use
>different other SLOW IO synchronous components ,and this why I need a

>lot of threads. 
>If I am forced  to  use only 2  IO 
>threads  , it will  force to design changes,and I do not
>like it  
> size="2">Does I really have no choice ?
> size="2">Arkady
> size="2"> 
>Steve Huston wrote:
><blockquote cite="mid:4AA2ECDFDF904F6AA543875A037C0AA8@hope" type="cite">
> <meta http-equiv="Content-Type" content="text/html; ">
> <title>Message</title>
> <meta name="GENERATOR" content="MSHTML 8.00.6001.18854">
> <div> face="Arial" size="2">Hi Arkady,</div>
> <div> </div>
> <div><font color="#0000ff"

> face="Arial" size="2">The source for ACE_Dev_Poll_Reactor is in

>ACE_wrappers/ace/Dev_Poll_Reactor.cpp - your first two questions can be

>researched there.</div>
> <div> </div>
> <div><font color="#0000ff"

> face="Arial" size="2">Starting 5000 threads is not a good way to get

>responsiveness, unless you have about 5000 processors. Try it with one

> face="Arial" size="2">It's stable and reliable, but not as extensible

>as you are looking for. For 10K connections, I recommend you use
>ACE_Dev_Poll_Reactor on Linux. You can use ACE_TP_Reactor (if you alter

>I apologize if this is the second mail concerning this problem,but I am

>not sure that the first one was really send. 
>

>I write some HTTP proxy using ACE 5.7 Acceptor-Connector framework and

>ACE_TP_Reactor . 
>My computer parameters are:

>Linux surfsecure.chimi.co.il 2.6.20.21-Surf-SeCure #11 SMP Thu Jun 4

_______________________________________________
ace-users mailing list

ace-...@list.isis.vanderbilt.edu

http://list.isis.vanderbilt.edu/mailman/listinfo/ace-users

Arkady

unread,

Dec 14, 2009, 9:36:14 AM12/14/09

to Douglas C. Schmidt, ace-...@cse.wustl.edu

Hi Steve

It seems that I some how succeeded to irritate you - it was not my intention ...

My real goal is to check if I am able to work with ACE_Dev_Poll_Reactor according with your recommendation.
So I want to know if the behavior that I described - is ACE_Dev_Poll_Reactor problem or may be I do not use it correctly.
BTW this phenomena (99.9 % CPU with 500 threads ) doesn't happen with ACE_TP_Reactor .

Respectfully ,Arkady

Douglas C. Schmidt

unread,

Dec 14, 2009, 9:47:31 AM12/14/09

to Arkady, ace-...@cse.wustl.edu

Hi Arkady,

> Hi Steve

I'm not Steve, but the point here is that if things aren't working the
way you expected then you either need to dig into it yourself or get
help from someone who has time to look into this. The most predictable
way to do this is via the commercial support providers.

Thanks,

Doug

Olivier Langlois

unread,

Dec 14, 2009, 10:15:57 AM12/14/09

to Arkady, ace-...@list.isis.vanderbilt.edu

Hi Arkady,

Having a 100% CPU usage while inside a reactor is usually a symptom that a event handler does not handle completely some events thus the reactor thread will always loop.

Hope this helps

Olivier Langlois | STREAMTHEWORLD | t. 514 448 4037 ext. 675

StreamTheWorld launches its new BlackBerry application. Learn more

From: ace-user...@list.isis.vanderbilt.edu [mailto:ace-user...@list.isis.vanderbilt.edu] On Behalf Of Arkady

Sent: Monday, December 14, 2009 7:33 AM
To: Steve Huston
Cc: ace-...@list.isis.vanderbilt.edu

Subject: Re: [ace-users] ACE MAXIMUM FILE DESCRIPTORS LIMIT PROBLEM

Hi Steve,

I apologize for keep bothering but if I create more then 2 threads (I have 2 CPUs) then ACE_Dev_Poll_Reactor enter to 99.9 %CPU .
Is it expected result ?

The problem is that my IO threads not only receive and send data,but use different other SLOW IO synchronous components ,and this why I need a lot of threads.
If I am forced to use only 2 IO threads , it will force to design changes,and I do not like it

Does I really have no choice ?

Arkady

Steve Huston wrote:

Hi Arkady,

The source for ACE_Dev_Poll_Reactor is in ACE_wrappers/ace/Dev_Poll_Reactor.cpp - your first two questions can be researched there.

Starting 5000 threads is not a good way to get responsiveness, unless you have about 5000 processors. Try it with one thread to start and see how that goes.

-Steve

--
Steve Huston, Riverace Corporation
Total Lifecycle Support for Your Networked Applications
http://www.riverace.com

-----Original Message-----

From: Arkady [mailto:ark...@pineapp.com]
Sent: Sunday, December 13, 2009 6:32 AM
To: Steve Huston
Cc: ace-...@list.isis.vanderbilt.edu

Subject: Re: [ace-users] ACE MAXIMUM FILE DESCRIPTORS LIMIT PROBLEM

Hi Steve,
Before I finally release my Http proxy with ACE_Dev_Poll_Reactor I will appreciate if you answer to some questions:
1. Where can i see ACE_Dev_Poll_Reactor source code .
2. What thread model ACE_Dev_Poll_Reactor uses (Leader/Follower as ACE_TP_Reactor or something else)

3. I am going create 5000 threads with the following code ACE_Reactor::instance ()->run_reactor_event_loop (); to allow maximum responsiveness . What is you opinion about it ?

Thanks in advance ,Arkady

Steve Huston wrote:

It's stable and reliable, but not as extensible as you are looking for. For 10K connections, I recommend you use ACE_Dev_Poll_Reactor on Linux. You can use ACE_TP_Reactor (if you alter the FD_SETSIZE limit beyond 1024) but performance may become an issue.

-Steve

--
Steve Huston, Riverace Corporation
Total Lifecycle Support for Your Networked Applications
http://www.riverace.com

-----Original Message-----
From: Arkady [mailto:ark...@pineapp.com]
Sent: Thursday, December 10, 2009 12:17 PM
To: Steve Huston
Cc: ace-...@list.isis.vanderbilt.edu
Subject: Re: [ace-users] ACE MAXIMUM FILE DESCRIPTORS LIMIT PROBLEM

Hi,thanks a lot for the answer, but does it means that I do not have stable and reliable ACE solution now ?

Arkady
Steve Huston wrote:

Please see:

http://www.riverace.com/newsletters/March2005.htm

--
Steve Huston, Riverace Corporation
Total Lifecycle Support for Your Networked Applications
http://www.riverace.com

-----Original Message-----

From: ace-user...@list.isis.vanderbilt.edu [mailto:ace-user...@list.isis.vanderbilt.edu] On Behalf Of Arkady

Sent: Thursday, December 10, 2009 11:14 AM
To: ace-...@list.isis.vanderbilt.edu
Subject: [ace-users] ACE MAXIMUM FILE DESCRIPTORS LIMIT PROBLEM

Hi,
I apologize if this is the second mail concerning this problem,but I am not sure that the first one was really send.

I write some HTTP proxy using ACE 5.7 Acceptor-Connector framework and ACE_TP_Reactor .
My computer parameters are:
Linux surfsecure.chimi.co.il 2.6.20.21-Surf-SeCure #11 SMP Thu Jun 4 16:07:27 IDT 2009 i686 i686 i386 GNU/Linux

I want to be able open about 10000 sockets , but default ACE installation allow about 1024 .

ulimit command output is :
unlimited

Arkady

unread,

Dec 14, 2009, 10:40:40 AM12/14/09

to Olivier Langlois, ace-...@list.isis.vanderbilt.edu

Hi Olivier
Thanks for the answer.

It happen only when the number of threads more then number of CPUs - 2 in my case .
It seems that this code from Dev_Poll_Reactor.cpp executed endlessly
do
    {
      result = this->work_pending_i (max_wait_time);
      if (result == -1 && (this->restart_ == 0 || errno != EINTR))
        ACE_ERROR ((LM_ERROR, ACE_TEXT("%t: %p\n"), ACE_TEXT("work_pending_i")));
    }
while (result == -1 && this->restart_ != 0 && errno == EINTR);

Function name is:

int
ACE_Dev_Poll_Reactor::handle_events_i (ACE_Time_Value *max_wait_time,
                                       Token_Guard &guard)

But why ???

Arkady

Olivier Langlois

unread,

Dec 14, 2009, 10:55:33 AM12/14/09

to Arkady, ace-...@list.isis.vanderbilt.edu

Ok then your epoll call is continually interrupted by actions performed by your other threads. You should mask most signals in your reactor threads when you start them because otherwise they will pretty much be interrupted by every signals. They are certainly good examples on how to do it in ACE books.

If you are not sure what I am talking about, I would recommend you to check this book:

http://blog.olivierlanglois.net/index.php/2007/10/25/programming_with_posix_threads

It explains very well the concept of signals and pthreads.

Hope this helps

Olivier Langlois | STREAMTHEWORLD | t. 514 448 4037 ext. 675

StreamTheWorld launches its new BlackBerry application. Learn more

Arkady

unread,

Dec 14, 2009, 11:32:26 AM12/14/09

to Olivier Langlois, ace-...@list.isis.vanderbilt.edu

I have to correct myself: On another Linux computer (Linux arkadys-desktop 2.6.24-19-generic #1 SMP Fri Jul 11 23:41:49 UTC 2008 i686 GNU/Linux) this happens with single thread too. And nobody do anything yet. It is only main startup .

Arkady.

Steve Huston

unread,

Dec 14, 2009, 11:34:32 AM12/14/09

to Arkady, ace-...@list.isis.vanderbilt.edu

Hi Arkady,

> I apologize for keep bothering but if I create more then 2 threads
> (I have 2 CPUs) then ACE_Dev_Poll_Reactor enter to 99.9 %CPU .
> Is it expected result ?

No, it's a problem. It should be fixed. If you can develop a fix, that
would be great.

> The problem is that my IO threads not only receive and send data,but
> use different other SLOW IO synchronous components ,and this why I
need
> a lot of threads.

You could also look into using half-sync/half-async as a better way to
keep your I/O threads freed up for sockets.

> If I am forced to use only 2 IO threads , it will force to
design
> changes,and I do not like it
> Does I really have no choice ?

You could work on a fix for the ACE_Dev_Poll_Reactor issue, switch to
something like half-sync/half-async, or, ideally, both.

Vadym Krevs

unread,

Dec 15, 2009, 12:34:14 PM12/15/09

to Steve Huston, Arkady, ace-...@list.isis.vanderbilt.edu

Sounds like a similar issue with ACE_Dev_Poll_Reactor from ACE 5.7 which Sergio Ruiz reported back in June:

http://stevehuston.wordpress.com/2009/06/15/revised-ace_dev_poll_reactor-fixes-multithread-issues-and-more-on-linux/#comment-44

Regards,
Vadym

2009/12/14 Steve Huston <shu...@riverace.com>

Olivier Langlois

unread,

Dec 15, 2009, 2:07:41 PM12/15/09

to Vadym Krevs, Steve Huston, Arkady, ace-...@list.isis.vanderbilt.edu

Interesting,

then if this is an issue new with 5.7, I might suspect the refactoring that made epoll_wait() return a single event per call + 2 calls to epoll_ctl() to disable/enable event reporting while handling the event to synchronize access to events in a TP setup to be a probable cause of the reported behavior…

Olivier Langlois | STREAMTHEWORLD | t. 514 448 4037 ext. 675

StreamTheWorld launches its new BlackBerry application. Learn more

From: ace-user...@list.isis.vanderbilt.edu [mailto:ace-user...@list.isis.vanderbilt.edu] On Behalf Of Vadym Krevs
Sent: Tuesday, December 15, 2009 12:34 PM
To: Steve Huston; Arkady; ace-...@list.isis.vanderbilt.edu
Subject: Re: [ace-users] ACE MAXIMUM FILE DESCRIPTORS LIMIT PROBLEM

Sounds like a similar issue with ACE_Dev_Poll_Reactor from ACE 5.7 which Sergio Ruiz reported back in June:

http://stevehuston.wordpress.com/2009/06/15/revised-ace_dev_poll_reactor-fixes-multithread-issues-and-more-on-linux/#comment-44

Regards,
Vadym

Arkady

unread,

Dec 31, 2009, 4:01:22 AM12/31/09

to Steve Huston, ace-...@list.isis.vanderbilt.edu

Hi,Steve

I didn't managed to fix bug,but now I know exactly what the problem, and where it is.
The problem in notification mechanism .
In situation when:
One thread is waiting for events in epoll_wait system call
Notification thread sent pipe notification to wakeup epoll_wait
Even if there is no other activity at all THE ENDLESS LOOP begin: before notification handler_input is called a thread that is waiting on threads queue is waked up (before upcall).
Before go to sleep on thread entry queue notification via pipe is sent ((this->sleep_hook ();) and so on and so on .
I tried to move sending pipe notification code only when socket is added to epoll handler but it doesn't work somehow.
I feel that I understand code well enough to understand the problem but not good enough to fix it . :-)
Can you help me ?

Thanks in advance,Arkady

Steve Huston wrote:

]
Sent: Sunday, December 13, 2009 6:32 AM
To: Steve Huston
Cc: ace-...@list.isis.vanderbilt.edu
Subject: Re: [ace-users] ACE MAXIMUM FILE DESCRIPTORS LIMIT PROBLEM

Hi Steve,
Before I finally  release my  Http proxy  with ACE_Dev_Poll_Reactor  I
will appreciate if you answer to some questions:
1. Where can i see ACE_Dev_Poll_Reactor source code  .
2. What thread model ACE_Dev_Poll_Reactor uses (Leader/Follower as
ACE_TP_Reactor or something else)
3. I am going create 5000 threads with the following code
ACE_Reactor::instance ()->run_reactor_event_loop (); to allow maximum
responsiveness . What is you opinion about it ?

Thanks in advance ,Arkady

Steve Huston wrote:
It's stable and reliable, but not as extensible as you are looking for.
For 10K connections, I recommend you use ACE_Dev_Poll_Reactor on Linux.
You can use ACE_TP_Reactor (if you alter the FD_SETSIZE limit beyond
1024) but performance may become an issue.

-Steve
--
Steve Huston, Riverace Corporation
Total Lifecycle Support for Your Networked Applications

http://www.riverace.com

-----Original Message-----
From: Arkady [mailto:ark...@pineapp.com

]
Sent: Thursday, December 10, 2009 12:17 PM
To: Steve Huston
Cc: ace-...@list.isis.vanderbilt.edu
Subject: Re: [ace-users] ACE MAXIMUM FILE DESCRIPTORS LIMIT PROBLEM

Hi,thanks a lot for the answer, but does it means that I do not have
stable and reliable ACE solution now ?

Arkady
Steve Huston wrote:
Please see:

http://www.riverace.com/newsletters/March2005.htm
--
Steve Huston, Riverace Corporation
Total Lifecycle Support for Your Networked Applications
http://www.riverace.com