SEVERE: All threads are currently busy.....

[Sharon Deminsky]

Hi,
We get this error every once in a while. We receive an email and the
server has to be bounced in order for the website to be used again.
We increased the maxThreads and it still has happened but, not often.
I have read that that is not necessarily the thing to do. Suggestions
to see where the requests are taking too long or not returning to the
pool are recommended.
We have installed javamelody in hopes this might shed some light but,
we are having a hard time making sense of the charts. Do you have any
suggestions for troubleshooting this when it doesn't happen very
often. Is there any good documentation I could read to learn what the
charts are telling us?

Thank you in advance,
Sharon

[Vernat Emeric]

Hi,

When you sometimes have "All threads are currently busy", I do not
suggest to look particularly at charts, except perhaps the chart of the
number of active threads.
The charts would give various trends about requests or about other
variables over times (memory, cpu, number of sessions...), but the
charts will only rarely give the cause of some long requests.

Case 1
When you are in the situation of not enough threads (or preferrably,
when you think that it is going to have not enough threads),
I suggest to look at the monitoring page of javamelody and check the
"Current requests" part, which is after the "Statistics" part.
You can click on "Details" on the right to display all of them, if you
have more than one current request.
And if possible for the server, you can refresh the page 3 or 5 times to
see if the current requests are not always the same and if some are more
often there than others.

If you are lucky to have the display of the long current requests, copy
and paste the whole html table in OOo, Word or Notepad right away.
You will get:
- the current stack-trace of this thread to see what it is currently doing
- the http request with parameters
- depending on the configuration, the logged in user
- depending on the configuration, the services and sql requests
currently running
- and for how long, these http request, services and sql requests are
running

In the threads part at the bottom of the monitoring page, you can also
check the stack-traces of all the threads to see what the threads are
doing in general (for example, computing some dirty hibernate objects,
waiting to log on disk or waiting for the database).
Among all the threads, ignore the idle threads of the app server and of
the JVM.
You can also take threads dump as text with the link below the threads
table (take 5 threads dump to have a better idea).

Or, if there is currently no problem on the server, go to case 2.

Case 2
After the fact, you can browse the statistics of the http requests (and
services and sql requests depending on the configuration).
First, choose the period (day, week, month...) of the statistics to
display at the top of the monitoring page.
Click the "Details" on the right of the statistics to see the list of
requests.
Check the first requests. You can also sort the requests by mean
execution time and eventually by max execution time.
Depending on the configuration, you can then click on some of the http
requests, to see what services or sql requests they have executed
including how many times.

Knowing exactly which are statistically the long http requests and which
services or sql requests are the cause for these http requests, you can
then analyze and start to optimize.
Note that if the very long requests have never ended and if you have
killed the server before their ends, then those long requests were
possibly never recorded in the statistics.

bye,
Emeric

Le 13/01/2012 17:01, Sharon Deminsky a �crit :

[Vernat Emeric]

Hi,

The mail below contains some advices to find and analyze long requests
and their root causes.
So I submit it again with a better title.

[Sharon Deminsky]

Emeric,

 

Thank you so much for getting back to me.  While reading over your insructions, I realized that we are not getting anything for Statistics sql and Current requests.  When I set up javamelody, it was the bare minimum instructions of the 2 .jar files and the section of code added to the web.xml.  What do I need to do to get these results?  I have attached my javamelody monitoring page.

 

Sharon

[David Karlsen]

Sharon - did you read my email (sent from another address than this
one) - I suggested two different possibilities for you there...

2012/1/25 Sharon Deminsky <sdem...@gmail.com>:

--
--
David J. M. Karlsen - http://www.linkedin.com/in/davidkarlsen

[Vernat Emeric]

Hi,

The user guide to monitor sql requests is here:
http://code.google.com/p/javamelody/wiki/UserGuide#7._JDBC

If that does not help enough, you can say how your webapp connects to
the database (datasource from jndi, datasource from spring, hibernate ...)

Have you made some progress otherwise?

bye,
Emeric

Le 25/01/2012 19:05, David Karlsen a �crit :

>>>> Le 13/01/2012 17:01, Sharon Deminsky a �crit :

[Sharon Deminsky]

Emeric,

 

I have not figured out what to do yet.  We do not have a datasource.  We use jdbc to connect to an oracle database.  We are using Tomcat 5.5.

 

Sharon

On Sun, Jan 29, 2012 at 7:16 AM, Vernat Emeric <eve...@free.fr> wrote:

Hi,

The user guide to monitor sql requests is here:
http://code.google.com/p/javamelody/wiki/UserGuide#7._JDBC

If that does not help enough, you can say how your webapp connects to the database (datasource from jndi, datasource from spring, hibernate ...)

Have you made some progress otherwise?

bye,
Emeric

[Sharon Deminsky]

Emeric,

 

I have read and re-read the jdbc instructions.  I cannot figure out what to do to get the sql stats and the current requests to show up.  We use tomcat 5.5 and we do not have a named datasource.  We have not had a problem for 3 months.  Then, on Sunday, the threads spiked at 2:00 am and brought the server to it's knees.  It was able to recover on its own but, we need to find out what is going on. 

 

Sharon

[Vernat Emeric]

Hi,

The graphic you have given is typical, because it shows what is often the case: when it goes bad, it goes bad very very fast. The server was in a bad state, from just a few to 500 active threads, probably in less than 15 minutes.

Given that the problem is in the past, we can't look at the list of the current requests to see which they are.
And first, I suggest to look at what is already available in the monitoring page: the statistics of http requests.
Choose the period at the top to to display the statistics values including the problem: the week period or better a custom period from 4/15/12 to 4/16/12 using the button at the top of the page on the right. After choosing the period, open the list of the http requests, using the "+ Details" on the right below the global statistics of http requests.

Some execution times of requests in those statistics may just be caused by the fact that your server was in a bad state.
But are there some particularly long http requests and times in the top ten? (you can also look with a sort by max time after clicking on the column header)
Is there not enough in those top ten requests to identify an interesting request which you can then analyze in the code of your application?

And during the problem, how where the graphics of used memory, %cpu, %Garbage Collector time, system load (in "Other Charts")? Were they ok? (you can give a screenshot of the graphics)

And what is the database used? (Some databases have tools to know what were "slow queries" and we could try them.)

And second, we can look at why you do not have statistics for the sql requests in the monitoring page.
For that, how the application connects to the database?
1. direct connection with jdbc driver (something like DriverManager.getConnection)
2. direct connection with commons-dbcp or c3p0
3. connection defined in hibernate.cfg.xml
4. datasource defined in a Spring context xml file
5. datasource from jndi
6. other

And also what is your javamelody version ?
(it is displayed at the bottom of the monitoring page)

bye,
Emeric

[Sharon Deminsky]

Emeric,

 

Thank you again for responding.  You don't know how much we appreciate your help.

This is a snippet of the TisConfig.java - we have an oracle database and this is how we connect.  We have JavaMelody 1.32.1

In the attached .pdf, we are seeing severe 404.  Is that due to the state of the server or is that the cause?  Where would you recommend we look next?

We received many email notices with the following stack trace...

java.lang.NullPointerException
        at edu.uwsa.OIS.Model.Database.ToplinkMapper.getWizardList(ToplinkMapper.java:196)
        at edu.uwsa.tis.Model.DataObjects.Database.TisMapper.getWizActiveFromInstList(TisMapper.java:2453)
        at edu.uwsa.tis.Wizards.Actions.ActionReverseInstitutionList.executeAction(ActionReverseInstitutionList.java:32)
        at edu.uwsa.tis.SystemWide.BaseAction.execute(BaseAction.java:46)
        at org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:484)
        at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:274)
        at edu.uwsa.OIS.SystemWide.OISRequestProcessor.process(OISRequestProcessor.java:116)
        at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1482)
        at org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:507)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:627)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
        at org.displaytag.filter.ResponseOverrideFilter.doFilter(ResponseOverrideFilter.java:125)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
        at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:185)
        at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:159)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174)
        at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:875)
        at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665)
        at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528)
        at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81)
        at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)
        at java.lang.Thread.run(Thread.java:619)


Any you see here and can further direct us will be appreciated.

 

Sharon

 

 

 

 

 

 

/*
 * Created on Aug 27, 2007
 *
 * TODO To change the template for this generated file go to
 * Window - Preferences - Java - Code Style - Code Templates
 */
package edu.uwsa.tis.SystemWide;

import java.sql.ResultSet;
import java.sql.Statement;

import javax.servlet.ServletContext;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpSession;

import oracle.jdbc.driver.OracleConnection;
import oracle.toplink.sessions.DatabaseLogin;
import oracle.toplink.sessions.Session;

import org.apache.log4j.Logger;
import org.apache.struts.action.ActionError;
import org.apache.struts.action.ActionErrors;
import org.apache.struts.action.ActionMessage;
import org.apache.struts.action.ActionMessages;
import org.apache.struts.action.ActionServlet;

import edu.uwsa.OIS.Model.Database.JDBCConnector;
import edu.uwsa.OIS.Model.Database.ToplinkMapper;
import edu.uwsa.OIS.SystemWide.OISAppPlugin;
import edu.uwsa.OIS.SystemWide.OISEmail;

/**
 * @author mary seitz
 *
 * TODO To change the template for this generated type comment go to
 * Window - Preferences - Java - Code Style - Code Templates
 */
public class TisConfig {

 private static final String devURL = "tis.uwsa.edu:1521:devTis";
 private static final String TisURL = "tis.uwsa.edu:1521:tis";
 private static final String accURL = "tis.uwsa.edu:1521:accTis";
 private static final String devJDBCURL = "jdbc:oracle:thin:@tis.uwsa.edu:1521:devTis";
 private static final String TisJDBCURL = "jdbc:oracle:thin:@tis.uwsa.edu:1521:tis";
 private static final String accJDBCURL = "jdbc:oracle:thin:@tis.uwsa.edu:1521:accTis"; 
 private static String toplinkURL = null;
 private static String jdbcURL = null;
 private static String dbUserId = null;
 private static String appServer = null;
 private static String dbSessionsXMLName = null;
 private static DatabaseLogin dbLogin = new DatabaseLogin();
 private static ServletContext servletContext = null;
 private static Logger log = Logger.getLogger(TisConfig.class.getName());
 private static javax.mail.Session mailSession = null;
 private static HttpSession session = null;
 private static String appName = null;
 private static String appLocation = null;
 private static String tomcatVersion = null;
 private static String dbText = null;
 private static String databaseURL = null;
 private static boolean aAppLocked = false;