java.util.ConcurrentModificationException under Heavy Load

[Kenan Malik]

Hi,

Under heavy load we're seeing the following
java.util.ConcurrentModificationException:

Dec 8, 2009 5:55:02 PM org.apache.catalina.connector.CoyoteAdapter
service
SEVERE: An exception or error occurred in the container during the
request processing
java.util.ConcurrentModificationException
       at java.util.ArrayList.writeObject(ArrayList.java:572)
       at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown
Source)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke
(DelegatingMethodAccessorImpl.java:25)
       at java.lang.reflect.Method.invoke(Method.java:585)
       at java.io.ObjectStreamClass.invokeWriteObject
(ObjectStreamClass.java:917)
       at java.io.ObjectOutputStream.writeSerialData
(ObjectOutputStream.java:1339)
       at java.io.ObjectOutputStream.writeOrdinaryObject
(ObjectOutputStream.java:1290)
       at java.io.ObjectOutputStream.writeObject0
(ObjectOutputStream.java:1079)
       at java.io.ObjectOutputStream.defaultWriteFields
(ObjectOutputStream.java:1375)
       at java.io.ObjectOutputStream.defaultWriteObject
(ObjectOutputStream.java:391)
       at java.util.Collections$SynchronizedCollection.writeObject
(Collections.java:1606)
       at sun.reflect.GeneratedMethodAccessor83.invoke(Unknown
Source)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke
(DelegatingMethodAccessorImpl.java:25)
       at java.lang.reflect.Method.invoke(Method.java:585)
       at java.io.ObjectStreamClass.invokeWriteObject
(ObjectStreamClass.java:917)
       at java.io.ObjectOutputStream.writeSerialData
(ObjectOutputStream.java:1339)
       at java.io.ObjectOutputStream.writeOrdinaryObject
(ObjectOutputStream.java:1290)
       at java.io.ObjectOutputStream.writeObject0
(ObjectOutputStream.java:1079)
       at java.io.ObjectOutputStream.defaultWriteFields
(ObjectOutputStream.java:1375)
       at java.io.ObjectOutputStream.writeSerialData
(ObjectOutputStream.java:1347)
       at java.io.ObjectOutputStream.writeOrdinaryObject
(ObjectOutputStream.java:1290)
       at java.io.ObjectOutputStream.writeObject0
(ObjectOutputStream.java:1079)
       at java.io.ObjectOutputStream.writeObject
(ObjectOutputStream.java:302)
       at org.apache.catalina.session.StandardSession.writeObject
(StandardSession.java:1517)
       at org.apache.catalina.session.StandardSession.writeObjectData
(StandardSession.java:959)
       at de.javakaffee.web.msm.SessionSerializingTranscoder.serialize
(SessionSerializingTranscoder.java:76)
       at net.spy.memcached.transcoders.SerializingTranscoder.encode
(SerializingTranscoder.java:135)
       at net.spy.memcached.MemcachedClient.asyncStore
(MemcachedClient.java:274)
       at net.spy.memcached.MemcachedClient.set(MemcachedClient.java:
631)
       at
de.javakaffee.web.msm.MemcachedBackupSessionManager.storeSessionInMemcached
(MemcachedBackupSessionManager.java:692)
       at
de.javakaffee.web.msm.MemcachedBackupSessionManager.backupSession
(MemcachedBackupSessionManager.java:466)
       at de.javakaffee.web.msm.SessionTrackerValve.backupSession
(SessionTrackerValve.java:117)
       at de.javakaffee.web.msm.SessionTrackerValve.invoke
(SessionTrackerValve.java:107)
       at org.apache.catalina.core.StandardHostValve.invoke
(StandardHostValve.java:128)
       at org.apache.catalina.valves.ErrorReportValve.invoke
(ErrorReportValve.java:102)
       at org.apache.catalina.valves.AccessLogValve.invoke
(AccessLogValve.java:567)
       at org.apache.catalina.core.StandardEngineValve.invoke
(StandardEngineValve.java:109)
       at org.apache.catalina.connector.CoyoteAdapter.service
(CoyoteAdapter.java:293)
       at org.apache.jk.server.JkCoyoteHandler.invoke
(JkCoyoteHandler.java:190)
       at org.apache.jk.common.HandlerRequest.invoke
(HandlerRequest.java:291)
       at org.apache.jk.common.ChannelSocket.invoke
(ChannelSocket.java:769)
       at org.apache.jk.common.ChannelSocket.processConnection
(ChannelSocket.java:698)
       at org.apache.jk.common.ChannelSocket$SocketConnection.runIt
(ChannelSocket.java:891)
       at org.apache.tomcat.util.threads.ThreadPool
$ControlRunnable.run(ThreadPool.java:690)
       at java.lang.Thread.run(Thread.java:613)

Any thoughts on what may be causing this and how best to deal with
this?
BTW, this is with the latest 1.0.1 release of memcached-session-
manager
and Tomcat/6.0.20.

Thanks much.

Best,

-kenan

[Martin Grotzke]

Hi Kenan,

the obvious problem is that the serialization code for the array list is not thread safe.

I see two solutions:

1) session backup (using java serialization) is synchronized on the session

2) use another serialization strategy that does not impose this problem

For 1) I'd need some time to implement and (performance) test this.

For 2) there's not yet a final release of memcached-session-manager. However, I've already implemented pluggable serialization, you can have a look at [1]. I can send you a msm jar with this feature included, and also a build of your favorite serialization strategy (I'd suggest msm-javolution-serializer). One thing you have to be aware of: Even if I have run performance tests ([2]) to compare the serialization strategies I don't have any of the others (than java serialization) running in production - therefore you should do the normal testing...

Another thing to consider: The reason for the problem seems to be that there are multiple requests running in parallel for the same session. This might be caused by ajax requests or by requests for static resources (.gif, .css etc.) which are served by tomcat. You might check which resources are served by tomcat which should not trigger a session backup and tune the requestUriIgnorePattern setting in your configuration (see also [3]).

Cheers,

Martin

[1] http://code.google.com/p/memcached-session-manager/wiki/SerializationStrategies

[2] http://code.google.com/p/memcached-session-manager/wiki/Performance

[3] http://code.google.com/p/memcached-session-manager/wiki/SetupAndConfiguration

--
Martin Grotzke
http://www.javakaffee.de/blog/

[Martin Grotzke]

Thinking more about this: this issue should be caused by the application changing the arraylist at the same time when it is serialized (iterating over it).

Therefore solution 1) even wouldn't solve this issue.

And for 2) you would need to select a strategy that handles concurrent array modifications by s.th. like cloning/copying etc.

I'm not sure how xstream handles this, for the javolution implementation I can say that there's currently no copying done (I have written the binding part via reflection by myself, javolution is just used as xml writer/reader). However this should be implemented, so if you'd like to take this route I could do this the next days.

Cheers,
Martin

[Kenan Malik]

Hi Martin,

Thank you so much for your response.

Our app uses extensive use of RESTful (JSON) service calls from web
and flash clients
and will have multiple requests in parallel for the same session id.
Unfortunately we don't
have a way around this.

Also under high load, we needed to up the sessionBackupTimeout value
( default of 100ms )
which was too low. We're currently testing with a value of between
1000ms and 3000ms.

BTW, when you have the javolution serialization strategy ready
( Option 2), we would certainly
be interested in running our load tests against it to see if it helps.

Thanks again!

Best,

-kenan

[Martin Grotzke]

Hi Kenan,

On Sat, Dec 12, 2009 at 11:35 PM, Kenan Malik <kenan...@gmail.com> wrote:

Hi Martin,

Thank you so much for your response.

Our app uses extensive use of RESTful (JSON) service calls from web
and flash clients
and will have multiple requests in parallel for the same session id.
Unfortunately we don't
have a way around this.

Ok. Do you already handle concurrent session access (e.g. synchronize on the session)?

Mostly this is not done simply because most session access is readonly, and writing to the session occurs less frequently.

 

Also under high load, we needed to up the sessionBackupTimeout value
( default of 100ms )
which was too low. We're currently testing with a value of  between
1000ms and 3000ms.

I'd also consider switching to async mode (sessionBackupAsync="true") and let spymemcached do the rest. The drawback of course is, that you are not sure if/when session is stored in memcached (and you don't have the guarantee anymore that the latest version of the session is available in memcached in the case of a tomcat failure). However, requests don't take longer to be processed just because the session was not already sent to memcached, and therefore you'll reduce load because you'll have less concurrent requests being processed.

Can you tell what's the number of requests/second you have during high load (where only requests with a session are interesting for memcached session backup)? Do you know the number of requests sent to memcached?

 

BTW, when you have the javolution serialization strategy ready
( Option 2), we would certainly
be interested in running our load tests against it to see if it helps.

Yes, I'll let you know. Some tests are still missing.

Btw: what are you using for load tests, how are you testing your app?

One thing that might be interesting to you: I want to send sessions to memcached only in the case the session data has changed (based on a hash over the serialized bytes). To be able to this I also need a change in spymemcached, which is the reason why I cannot do this right away. I expect this feature to reduce the requests to memcached significantly and that remaining requests will be done in shorter time. Unfortunately I cannot say when this will be available, but Q1 2010 should be possible.

Cheers,

Martin

 

Thanks again!

Best,

-kenan

[Martin Grotzke]

Hi Kenan,

I thought more about the concurrency / concurrentmodification issue. The solution, that the memcached-session-manager would copy collections before they get serialized can only _reduce_ the probability, that a concurrent modification occurs. It cannot _prevent_ a concurrentmodification fully, as another thread still might change the collection when it is copied.

Preventing a concurrent modification can only be done by the application. But that's not an easy thing to do. The solution that I can image right now is (and that's more or less the way how spring-mvc solves this problem) a servlet-filter that synchronizes requests on the session (or a special object stored in the session). The drawback of this solution is that concurrent requests will get processed one after the other. To optimize this, the filter could use a read/write lock, so that requests the are considered as readonly requests (e.g. GET requests) obtain a read lock and requests that are considered as session-modifications obtain a write lock (e.g. PUT/POST/DELETE requests). Of course the question then is how to differentiate between read/write requests.

And finally the question is, if it's worth the effort to fix this (and test and check consequences of the solution etc.)...

Cheers,

Martin

[Kenan Malik]

Hi Martin,

Thanks for your reply. Please see my comments below.

> Ok. Do you already handle concurrent session access (e.g. synchronize on the
> session)?
>
> Mostly this is not done simply because most session access is readonly, and
> writing to the session occurs less frequently.

We don't currently synchronize on the session in our code.

> > Also under high load, we needed to up the sessionBackupTimeout value
> > ( default of 100ms )
> > which was too low. We're currently testing with a value of  between
> > 1000ms and 3000ms.
>
> I'd also consider switching to async mode (sessionBackupAsync="true") and
> let spymemcached do the rest. The drawback of course is, that you are not
> sure if/when session is stored in memcached (and you don't have the
> guarantee anymore that the latest version of the session is available in
> memcached in the case of a tomcat failure). However, requests don't take
> longer to be processed just because the session was not already sent to
> memcached, and therefore you'll reduce load because you'll have less
> concurrent requests being processed.

Thanks for the tip in case we need it. For our requirements, we'll
probably
stick with sessionBackupAsync="false" just so we do know that the
session
got stored successfully and the latest session is available.

> Can you tell what's the number of requests/second you have during high load
> (where only requests with a session are interesting for memcached session
> backup)? Do you know the number of requests sent to memcached?

We created a test JSP page which would write about 4k of random string
data to the session. A JMeter test was written to hit the page with
20 threads
with each thread maintaining it's own session for the life of the
test.

The requests/second sustained an average of around 10 sessions/second.
Since each request wrote to the session we saw a write to memcached
for every request.

The tests were run on the following hardware:

Platform:
Mac OS X - 10.4.11

Hardware:
Dual Processor 2.66 Ghz Dual-Core Intel Xeon
3GB RAM

JVM:
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_16-
b06-275)

Tomcat:
6.0.20 running JVM options '-Xmx1024m -XX:MaxPermSize=128m -server'

JMeter:
2.3.4 running JVM options '-Xmx256m'

> One thing that might be interesting to you: I want to send sessions to
> memcached only in the case the session data has changed (based on a hash
> over the serialized bytes). To be able to this I also need a change in
> spymemcached, which is the reason why I cannot do this right away. I expect
> this feature to reduce the requests to memcached significantly and that
> remaining requests will be done in shorter time. Unfortunately I cannot say
> when this will be available, but Q1 2010 should be possible.

This sounds like a very useful scalability feature.

Thanks again,

-kenan

[Martin Grotzke]

Hi Kenan,

On Thu, Dec 17, 2009 at 3:13 AM, Kenan Malik <kenan...@gmail.com> wrote:

> Can you tell what's the number of requests/second you have during high load
> (where only requests with a session are interesting for memcached session
> backup)? Do you know the number of requests sent to memcached?

We created a test JSP page which would write about 4k of random string
data to the session.  A JMeter test was written to hit the page with
20 threads
with each thread maintaining it's own session for the life of the
test.

Ok. Do you also know the numbers of your production app (especially when load is high)? Take the sessions in your production app ~4k?

Cheers,

Martin

[Martin Grotzke]

Hi Kenan,

On Sat, Dec 12, 2009 at 11:35 PM, Kenan Malik <kenan...@gmail.com> wrote:

BTW, when you have the javolution serialization strategy ready
( Option 2), we would certainly
be interested in running our load tests against it to see if it helps.

Yesterday I released msm 1.1 together with the javolution serializer and an option to copy collections before serialization. The setup page tells you what you need to get javolution based serialization running:

http://code.google.com/p/memcached-session-manager/wiki/SetupAndConfiguration

To copy collections before serialization you need to set copyCollectionsForSerialization="true".

Please let me know how it works for you.

Have a happy new year,

cheers,

Martin

 

Thanks again!

Best,

-kenan