Jetty 503 Too Many Concurrent Requests

[Chuck Boecking]

Hi Everyone,

I am receiving the below Jetty error message during peak login times (beginning of day). When this happens, the dedicated webui server is approaching 100 concurrent users. Under steady-state, it performs like a champ. The login process seems to cross some threshold.

The machine has the below Webui Server Details:

• 8 core
• 24GB RAM
• 16GB XMS/XMX. Most of the heap is typically free according to the About dialog => Info tab => Free Heap value.
• Thread count typically hovers between 170 and 250. I do not know this value when the below error is shown.
  

When the above error occurs, all existing sessions remain intact and functional, and new sessions are met immediately with the 503 error.

Based on my research, the above error is either related to connections or threads. I believe it is related to threads (specifIcally the default threadpool size defined in /opt/idempiere-server/jettyhome/etc/jetty.xml). I updated the below values accordingly as a means to test/experiment:

•        <Set name="minThreads" type="int"><Property name="jetty.threadPool.minThreads" deprecated="threads.min" default="100"/></Set> # from 10
•        <Set name="maxThreads" type="int"><Property name="jetty.threadPool.maxThreads" deprecated="threads.max" default="600"/></Set> # from 200

I do not have a lot of logic behind why I chose these new values. I made the following assumptions:

• It seems reasonable to increase the max by 3x as a starting point given how much memory is available to java.
• I believe jetty is configured to use the same /opt/idempiere-server/utils/myEnvironment.sh => XMX/XMS settings; therefore, the system should have plenty of memory to support the increased threadpool size.
  

If I am wrong, and the issue is connection related (not thread related), I believe the desired value to change is the following setting in /opt/idempiere-server/jettyhome/etc/jetty-http.xml:

•          <Set name="acceptQueueSize"><Property name="jetty.http.acceptQueueSize" deprecated="http.acceptQueueSize" default="0"/></Set>
• where default=0 gets a system level default. I could not find what the default is for 64bit linux in the jetty 9.4 codebase.
• I am leaving this value unchanged for now.
  

There was a conversation many years ago around the below setting; however, I do not believe this is relevant any more.

• cat /proc/sys/kernel/threads-max

If you have any additional insight, please share. I hope this helps, and I will keep you posted with my findings.

Regards,


Chuck Boecking
512.850.6068 (office and cell)
ch...@chuboe.com
ChuckBoecking.com
chuck.boecking (skype)

@ChuckBoecking

http://www.linkedin.com/pub/chuck-boecking/10/970/17b