Hi Ayhan,
Thank for helping.
I noticed the memory used for Hud, RC, and browsers were never above 110M and so now I rule out the memory problem guess I made earlier.
Having said that, this problem still remained. However, I noticed whenever this problem happened, no error log seen on Selenium RC nor the console where I run my test, but when I traced back the Hub log, I did see the following two types of error appeared on the Selenium Hub log:
Error 1:
[java] WARNING: SimpleHttpConnectionManager being used incorrectly. Be sure that HttpMethod.releaseConnection() is always called and that only one thread and/or method is using this connection manager at a time.
[java] May 25, 2010 12:29:29 PM com.thoughtworks.selenium.grid.hub.HubServlet forward
[java] INFO: Responding with 200 / OK
[java] May 25, 2010 12:29:29 PM com.thoughtworks.selenium.grid.hub.HubServlet forward
[java] INFO: Processing '1 => "kwmToolForm_source1", cmd => "click", sessionId => "6f6e48b983b9416dbcf40a5a543a59a2"'
[java] May 25, 2010 12:29:29 PM com.thoughtworks.selenium.grid.hub.HubServlet forward
[java] INFO: Responding with 403 / <html>
[java] <head>
[java] <title>Error 403 Forbidden for Proxy</title>
[java] </head>
[java] <body>
[java] <h2>HTTP ERROR: 403</h2><pre>Forbidden for Proxy</pre>...[1140 characters truncated]
[java] May 25, 2010 12:29:29 PM com.thoughtworks.selenium.grid.hub.HubServlet forward
The following two lines caught my attention,
[java] WARNING: SimpleHttpConnectionManager being used incorrectly. Be sure that HttpMethod.releaseConnection() is always called and that only one thread and/or method is using this connection manager at a time.
[java] WARNING: Remote Control at
10.44.46.128:5000 is unresponsive
Error 2:
[java] May 25, 2010 11:03:18 AM org.apache.commons.httpclient.SimpleHttpConnectionManager getConnectionWithTimeout
[java] WARNING: SimpleHttpConnectionManager being used incorrectly. Be sure that HttpMethod.releaseConnection() is always called and that only one thread and/or method is using this connection manager at a time.
[java] May 25, 2010 11:04:18 AM org.apache.commons.httpclient.HttpMethodBase readResponseBody
[java] INFO: Response content length is not known
[java] May 25, 2010 11:04:18 AM org.apache.commons.httpclient.HttpMethodBase getResponseBody
[java] WARNING: Going to buffer response body of large or unknown size. Using getResponseBodyAsStream instead is recommended.
[java] May 25, 2010 11:04:18 AM com.thoughtworks.selenium.grid.hub.remotecontrol.RemoteControlProxy unreliable
[java] WARNING: Remote Control at
10.44.46.128:5000 is unresponsive
[java] May 25, 2010 11:04:18 AM com.thoughtworks.selenium.grid.hub.remotecontrol.GlobalRemoteControlPool unregisterRemoteControlIfUnreliable
[java] WARNING: Unregistering unreliable RC [RemoteControlProxy
10.44.46.128:5000#true]
[java] 2010-05-25 11:04:18.120::WARN: /selenium-server/driver/
[java] java.io.IOException: Attempted read on closed stream.
[java] at org.apache.commons.httpclient.AutoCloseInputStream.isReadAllowed(AutoCloseInputStream.java:165)
[java] at org.apache.commons.httpclient.AutoCloseInputStream.read(AutoCloseInputStream.java:125)
[java] at org.apache.commons.httpclient.HttpMethodBase.getResponseBody(HttpMethodBase.java:684)
[java] at com.thoughtworks.selenium.grid.HttpClient.request(HttpClient.java:55)
[java] at
com.thoughtworks.selenium.grid.HttpClient.post(HttpClient.java:35)
[java] at com.thoughtworks.selenium.grid.hub.remotecontrol.RemoteControlProxy.forward(RemoteControlProxy.java:65)
[java] at com.thoughtworks.selenium.grid.hub.remotecontrol.commands.SeleneseCommand.execute(SeleneseCommand.java:40)
[java] at com.thoughtworks.selenium.grid.hub.HubServlet.forward(HubServlet.java:52)
[java] at com.thoughtworks.selenium.grid.hub.HubServlet.process(HubServlet.java:41)
[java] at com.thoughtworks.selenium.grid.hub.HubServlet.doPost(HubServlet.java:31)
[java] at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
[java] at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
[java] at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:459)
[java] at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:360)
[java] at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:231)
[java] at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:629)
[java] at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:149)
[java] at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:141)
[java] at org.mortbay.jetty.Server.handle(Server.java:303)
[java] at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:452)
[java] at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:735)
[java] at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:636)
[java] at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:209)
[java] at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:349)
[java] at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:217)
[java] at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:475)
[java] May 25, 2010 11:04:19 AM com.thoughtworks.selenium.grid.hub.HubServlet forward
[java] INFO: Processing '1 => "Keywords Mining Tool - Results", cmd => "isTextPresent", sessionId => "7001c54a9b95481ea6a592bc77a1cb4c"'
[java] 2010-05-25 11:04:19.170::WARN: /selenium-server/driver/:
[java] java.lang.NullPointerException
[java] at com.thoughtworks.selenium.grid.hub.remotecontrol.GlobalRemoteControlPool.updateSessionLastActiveAt(GlobalRemoteControlPool.java:212)
[java] at com.thoughtworks.selenium.grid.hub.remotecontrol.commands.SeleneseCommand.execute(SeleneseCommand.java:39)
[java] at com.thoughtworks.selenium.grid.hub.HubServlet.forward(HubServlet.java:52)
[java] at com.thoughtworks.selenium.grid.hub.HubServlet.process(HubServlet.java:41)
[java] at com.thoughtworks.selenium.grid.hub.HubServlet.doPost(HubServlet.java:31)
[java] at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
[java] at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
[java] at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:459)
[java] at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:360)
[java] at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:231)
[java] at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:629)
[java] at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:149)
[java] at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:141)
[java] at org.mortbay.jetty.Server.handle(Server.java:303)
[java] at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:452)
[java] at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:735)
[java] at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:636)
[java] at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:209)