Issue 7765 in selenium: java.util.ConcurrentModificationException under heavy load

[sele...@googlecode.com]

Status: New
Owner: ----
Labels: Type-Defect Priority-Medium Status-Untriaged

New issue 7765 by gregory....@gmail.com:
java.util.ConcurrentModificationException under heavy load
http://code.google.com/p/selenium/issues/detail?id=7765

What steps will reproduce the problem?
1. Have 50+ slaves connected to the server
2. Drive each slave to perform testing tasks
3. Periodically pull console page: http://ip_of_the_server:4444/grid/console

What is the expected output? What do you see instead?
Expected: server runs reliably.
Actual: console page takes more than 2 hours to return (did not wait
longer) and see these exceptions in the log
SEVERE: cannot forward the request null
java.util.ConcurrentModificationException
at java.util.HashMap$HashIterator.nextEntry(HashMap.java:810)
at java.util.HashMap$EntryIterator.next(HashMap.java:851)
at java.util.HashMap$EntryIterator.next(HashMap.java:849)
at
org.apache.http.pool.AbstractConnPool.purgePoolMap(AbstractConnPool.java:469)
at
org.apache.http.pool.AbstractConnPool.closeIdle(AbstractConnPool.java:500)
at
org.apache.http.impl.conn.PoolingHttpClientConnectionManager.closeIdleConnections(PoolingHttpClientConnectionManager.java:358)
at
org.openqa.selenium.remote.internal.HttpClientFactory.getGridHttpClient(HttpClientFactory.java:97)
at org.openqa.grid.internal.TestSession.getClient(TestSession.java:203)
at
org.openqa.grid.internal.TestSession.sendRequestToNode(TestSession.java:366)
at org.openqa.grid.internal.TestSession.forward(TestSession.java:227)
at
org.openqa.grid.web.servlet.handler.RequestHandler.forwardRequest(RequestHandler.java:96)
at
org.openqa.grid.web.servlet.handler.RequestHandler.process(RequestHandler.java:129)
at org.openqa.grid.web.servlet.DriverServlet.process(DriverServlet.java:83)
at org.openqa.grid.web.servlet.DriverServlet.doGet(DriverServlet.java:61)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at
org.seleniumhq.jetty7.servlet.ServletHolder.handle(ServletHolder.java:565)
at
org.seleniumhq.jetty7.servlet.ServletHandler.doHandle(ServletHandler.java:479)
at
org.seleniumhq.jetty7.server.session.SessionHandler.doHandle(SessionHandler.java:225)
at
org.seleniumhq.jetty7.server.handler.ContextHandler.doHandle(ContextHandler.java:1031)
at
org.seleniumhq.jetty7.servlet.ServletHandler.doScope(ServletHandler.java:406)
at
org.seleniumhq.jetty7.server.session.SessionHandler.doScope(SessionHandler.java:186)
at
org.seleniumhq.jetty7.server.handler.ContextHandler.doScope(ContextHandler.java:965)
at
org.seleniumhq.jetty7.server.handler.ScopedHandler.handle(ScopedHandler.java:117)
at
org.seleniumhq.jetty7.server.handler.HandlerWrapper.handle(HandlerWrapper.java:111)
at org.seleniumhq.jetty7.server.Server.handle(Server.java:349)
at
org.seleniumhq.jetty7.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:452)
at
org.seleniumhq.jetty7.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:47)
at
org.seleniumhq.jetty7.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:884)
at
org.seleniumhq.jetty7.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:938)
at org.seleniumhq.jetty7.http.HttpParser.parseNext(HttpParser.java:634)
at
org.seleniumhq.jetty7.http.HttpParser.parseAvailable(HttpParser.java:230)
at
org.seleniumhq.jetty7.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:66)
at
org.seleniumhq.jetty7.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:254)
at
org.seleniumhq.jetty7.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
at
org.seleniumhq.jetty7.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
at java.lang.Thread.run(Thread.java:679)


Selenium version: 2.42.2
OS: Ubuntu 12.04 LTS
Browser: Chrome

--
You received this message because this project is configured to send all
issue notifications to this address.
You may adjust your notification preferences at:
https://code.google.com/hosting/settings

[sele...@googlecode.com]

Comment #1 on issue 7765 by gregory....@gmail.com:

java.util.ConcurrentModificationException under heavy load
http://code.google.com/p/selenium/issues/detail?id=7765

Also see these exceptions in the log:

Aug 18, 2014 5:37:06 PM org.openqa.grid.web.servlet.handler.RequestHandler
process
SEVERE: cannot forward the request Entry
[id:174400][route:{}->http://10.200.178.146:5555][state:null] has not been
leased from this pool
java.lang.IllegalStateException: Entry
[id:174400][route:{}->http://10.200.178.146:5555][state:null] has not been
leased from this pool
at org.apache.http.util.Asserts.check(Asserts.java:40)
at org.apache.http.pool.RouteSpecificPool.free(RouteSpecificPool.java:123)
at org.apache.http.pool.AbstractConnPool.release(AbstractConnPool.java:308)
at
org.apache.http.impl.conn.PoolingHttpClientConnectionManager.releaseConnection(PoolingHttpClientConnectionManager.java:280)
at
org.apache.http.impl.execchain.ConnectionHolder.releaseConnection(ConnectionHolder.java:100)
at
org.apache.http.impl.execchain.ResponseEntityWrapper.releaseConnection(ResponseEntityWrapper.java:68)
at
org.apache.http.impl.execchain.ResponseEntityWrapper.eofDetected(ResponseEntityWrapper.java:107)
at
org.apache.http.conn.EofSensorInputStream.checkEOF(EofSensorInputStream.java:199)
at
org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:138)
at
org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:150)
at com.google.common.io.ByteStreams.copy(ByteStreams.java:169)
at com.google.common.io.ByteStreams.toByteArray(ByteStreams.java:214)
at
org.openqa.grid.internal.TestSession.drainInputStream(TestSession.java:445)
at org.openqa.grid.internal.TestSession.forward(TestSession.java:265)

[sele...@googlecode.com]

Updates:
Cc: francois...@gmail.com nirvd...@gmail.com
Labels: Component-Grid

Comment #2 on issue 7765 by barancev:

java.util.ConcurrentModificationException under heavy load
http://code.google.com/p/selenium/issues/detail?id=7765

(No comment was entered for this change.)

[sele...@googlecode.com]

Comment #3 on issue 7765 by flaviu.c...@gmail.com:
java.util.ConcurrentModificationException under heavy load
https://code.google.com/p/selenium/issues/detail?id=7765

I am seeing the same thing, occasionally. I run a grid with 10 nodes, each
running up to 5 tests at once. The only part of my stack trace that is
different is instead of

at java.util.HashMap$EntryIterator.next(HashMap.java:851)
at java.util.HashMap$EntryIterator.next(HashMap.java:849)

I see

at java.util.HashMap$HashIterator.remove<Unknown Srouce>

Grid seems to hang indefinitely after this point.

[sele...@googlecode.com]

Comment #4 on issue 7765 by flaviu.c...@gmail.com:

java.util.ConcurrentModificationException under heavy load
https://code.google.com/p/selenium/issues/detail?id=7765

I am seeing the same thing, occasionally. I run a grid with 10 nodes, each
running up to 5 tests at once. The only part of my stack trace that is
different is instead of

at java.util.HashMap$EntryIterator.next(HashMap.java:851)
at java.util.HashMap$EntryIterator.next(HashMap.java:849)

I see

at java.util.HashMap$HashIterator.remove<Unknown Srouce>

[sele...@googlecode.com]

Comment #5 on issue 7765 by AndyUM...@gmail.com:

java.util.ConcurrentModificationException under heavy load
https://code.google.com/p/selenium/issues/detail?id=7765

I have been experiencing the ConcurrentModificationException described in
#1 above with version 2.42.2. My environment is the same as the steps
listed above. I reverted my hub to version 2.41.0 and no longer have the
exception issue being logged, however, my grid/console viewer still takes a
VERY (>5min) long time to load. It eventually succeeds, but it didn't use
to take that long to load.

[sele...@googlecode.com]

Updates:
Status: NeedsClarification

Comment #6 on issue 7765 by barancev:

java.util.ConcurrentModificationException under heavy load
https://code.google.com/p/selenium/issues/detail?id=7765

Apache HttpClient was updated in Selenium version 2.43, please retest with
this version.

[sele...@googlecode.com]

Comment #7 on issue 7765 by gregory....@gmail.com:

java.util.ConcurrentModificationException under heavy load
https://code.google.com/p/selenium/issues/detail?id=7765

The issue is fixed in 2.43 :)

[sele...@googlecode.com]

Updates:
Status: Fixed
Labels: -Status-Untriaged

Comment #8 on issue 7765 by barancev:

java.util.ConcurrentModificationException under heavy load
https://code.google.com/p/selenium/issues/detail?id=7765

(No comment was entered for this change.)