[Sakai Jira] Created: (KERN-655) possible memory leak posting lots of user creates
Paul Bristow (JIRA)
-------------------------------------------------
Key: KERN-655
URL: http://jira.sakaiproject.org/browse/KERN-655
Project: Nakamura
Issue Type: Bug
Components: System - memory
Affects Versions: 0.3
Environment: Windows XP SP3 x86, java (and javac) 1.5.0_15
Reporter: Paul Bristow
Have had problems with nakamura running out of heap space after adding approx 1000 users via /system/userManager/user.create.html POSTs with HTTP Basic Authn.
Most recently added 2000 users without exhausting heap but memory footprint grew to 732Mb and had not dropped 10 minutes after posts ceased.
On earlier test was seeing unclosed session warnings:
See post in http://groups.google.com/group/sakai-kernel/browse_thread/thread/ef0f37665e6aa381
have tried tests as suggested by Ian Boston
use File::Find;
my $dir = '.';
my $i = 0;
for ($i = 0; $i < 100; $i++) {
find(\&process_file, $dir);
}
sub process_file
{
my $file = $_;
if (-d $file) {
; #print $file, " is dir\n";
}
else { if (-l $file) {
; #print $file, " is link\n";
}
else {if (-f $file) {
print $file, " is file\n";
### test 1###
#open(IN, "<$file");
#while (<IN>) {
# ;#print;
#}
#close IN;
### test 2###
$url = $File::Find::name; #gets relative path
$url =~ s/^\.\///; #directories get ./ prepended
$url = 'http://localhost:8080/' . $url;
print $url, "\n";
`curl $url`;
}}}
No growth to same level as posting users but slowdown on test 2.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://jira.sakaiproject.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
Paul Bristow (JIRA)
[ http://jira.sakaiproject.org/browse/KERN-655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Paul Bristow updated KERN-655:
------------------------------
Attachment: heap1.PNG
tenured1.PNG
jconsole screens for heap first 1600 and tenured first 2000 user adds this morning from clean startup (deleted load and sling dirs). Build from 23 Feb.
Job still running. will post again on completion or failure
> possible memory leak posting lots of user creates
> -------------------------------------------------
>
> Key: KERN-655
> URL: http://jira.sakaiproject.org/browse/KERN-655
> Project: Nakamura
> Issue Type: Bug
> Components: System - memory
> Affects Versions: 0.3
> Environment: Windows XP SP3 x86, java (and javac) 1.5.0_15
> Reporter: Paul Bristow
> Attachments: heap1.PNG, tenured1.PNG
Paul Bristow (JIRA)
[ http://jira.sakaiproject.org/browse/KERN-655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Paul Bristow updated KERN-655:
------------------------------
Attachment: jconsole.zip
zipped screen dumps of jconsole
*2.PNG files at around 4000 users added
*3.PNG files at around 5000 users added
*4.PNG files at around 6000 users added
I also checked for FINAL_WAIT connections in case they were building up. My guess is about a minutes worth of connections are in FINAL_WAIT at any time. the change parallels the drop in transaction processing speed
bash-3.1$ netstat -a | grep 8080 | grep localhost | grep -c WAIT
180
bash-3.1$ netstat -a | grep 8080 | grep localhost | grep -c WAIT
138
bash-3.1$ netstat -a | grep 8080 | grep localhost | grep -c WAIT
88
> possible memory leak posting lots of user creates
> -------------------------------------------------
>
> Key: KERN-655
> URL: http://jira.sakaiproject.org/browse/KERN-655
> Project: Nakamura
> Issue Type: Bug
> Components: System - memory
> Affects Versions: 0.3
> Environment: Windows XP SP3 x86, java (and javac) 1.5.0_15
> Reporter: Paul Bristow
> Attachments: heap1.PNG, jconsole.zip, tenured1.PNG
Paul Bristow (JIRA)
[ http://jira.sakaiproject.org/browse/KERN-655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Paul Bristow updated KERN-655:
------------------------------
Attachment: jc2.zip
tenured_java6.PNG
jc2.zip contains graphs from java 5 instance at 9248 users added. By now each addition takes up to 15 seconds. Still no actual OOM or other errors.
At this point I killed the job - tenured6.jpg is from 15 minutes later
tenured_java6.png is from a quick test with java 6 (both built and ran) on same 23 Feb codebase. Run up to 2500 users and stopped (at 15:15:~20)
> possible memory leak posting lots of user creates
> -------------------------------------------------
>
> Key: KERN-655
> URL: http://jira.sakaiproject.org/browse/KERN-655
> Project: Nakamura
> Issue Type: Bug
> Components: System - memory
> Affects Versions: 0.3
> Environment: Windows XP SP3 x86, java (and javac) 1.5.0_15
> Reporter: Paul Bristow
> Attachments: heap1.PNG, jc2.zip, jconsole.zip, tenured1.PNG, tenured_java6.PNG
Ian Boston (JIRA)
[ http://jira.sakaiproject.org/browse/KERN-655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=94086#action_94086 ]
Ian Boston commented on KERN-655:
---------------------------------
The fact that you are getting maxswwp GC's every 2 minutes and there is slow growth on the tenured space shows that if there is a leak its slow. I also sense that rate of rise is declining. A much longer test will verify (12h).
However, each addition taking 15s is totally unacceptable. Something is very wrong.
We did some tests in July and it took about 8h to load 100K users over http with < 1G of heap. I still think thats really slow.
> possible memory leak posting lots of user creates
> -------------------------------------------------
>
> Key: KERN-655
> URL: http://jira.sakaiproject.org/browse/KERN-655
> Project: Nakamura
> Issue Type: Bug
> Components: System - memory
> Affects Versions: 0.3
> Environment: Windows XP SP3 x86, java (and javac) 1.5.0_15
> Reporter: Paul Bristow
> Attachments: heap1.PNG, jc2.zip, jconsole.zip, tenured1.PNG, tenured_java6.PNG
>
>
Ian Boston (JIRA)
[ http://jira.sakaiproject.org/browse/KERN-655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ian Boston updated KERN-655:
----------------------------
Fix Version/s: 0.4
Priority: Critical (was: Major)
Scheduling
> possible memory leak posting lots of user creates
> -------------------------------------------------
>
> Key: KERN-655
> URL: http://jira.sakaiproject.org/browse/KERN-655
> Project: Nakamura
> Issue Type: Bug
> Components: System - memory
> Affects Versions: 0.3
> Environment: Windows XP SP3 x86, java (and javac) 1.5.0_15
> Reporter: Paul Bristow
> Priority: Critical
> Fix For: 0.4
>
> Attachments: heap1.PNG, jc2.zip, jconsole.zip, tenured1.PNG, tenured_java6.PNG
>
>
Ian Boston (JIRA)
[ http://jira.sakaiproject.org/browse/KERN-655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ian Boston updated KERN-655:
----------------------------
Attachment: Screen shot 2010-03-03 at 15.30.29.png
Screen shot 2010-03-03 at 15.30.38.png
Able to reproduce the problem.
java -d32 -Xmx512m -server -Dcom.sun.management.jmxremote -Djava.awt.headless=true -XX:MaxPermSize=256m -jar app/target/org.sakaiproject.nakamura.app-0.3.jar -f -
and then run
sh testCreateUsers.sh
...
created testuser20100303142037-3857
Should be able to try the same with raw sling.
> possible memory leak posting lots of user creates
> -------------------------------------------------
>
> Key: KERN-655
> URL: http://jira.sakaiproject.org/browse/KERN-655
> Project: Nakamura
> Issue Type: Bug
> Components: System - memory
> Affects Versions: 0.3
> Environment: Windows XP SP3 x86, java (and javac) 1.5.0_15
> Reporter: Paul Bristow
> Priority: Critical
> Fix For: 0.4
>
> Attachments: heap1.PNG, jc2.zip, jconsole.zip, Screen shot 2010-03-03 at 15.30.29.png, Screen shot 2010-03-03 at 15.30.38.png, tenured1.PNG, tenured_java6.PNG
Ian Boston (JIRA)
[ http://jira.sakaiproject.org/browse/KERN-655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ian Boston updated KERN-655:
----------------------------
Attachment: Screen shot 2010-03-03 at 16.43.35.png
Ran with Sling as it was when we released 0.3,
Creates users in < 50ms per user and does 4K users in about 50MB of heap.
Clearly we have added something to the code base that is very hungry.
> possible memory leak posting lots of user creates
> -------------------------------------------------
>
> Key: KERN-655
> URL: http://jira.sakaiproject.org/browse/KERN-655
> Project: Nakamura
> Issue Type: Bug
> Components: System - memory
> Affects Versions: 0.3
> Environment: Windows XP SP3 x86, java (and javac) 1.5.0_15
> Reporter: Paul Bristow
> Priority: Critical
> Fix For: 0.4
>
> Attachments: heap1.PNG, jc2.zip, jconsole.zip, Screen shot 2010-03-03 at 15.30.29.png, Screen shot 2010-03-03 at 15.30.38.png, Screen shot 2010-03-03 at 16.43.35.png, tenured1.PNG, tenured_java6.PNG
Ian Boston (JIRA)
[ http://jira.sakaiproject.org/browse/KERN-655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ian Boston reassigned KERN-655:
-------------------------------
Assignee: Ian Boston
> possible memory leak posting lots of user creates
> -------------------------------------------------
>
> Key: KERN-655
> URL: http://jira.sakaiproject.org/browse/KERN-655
> Project: Nakamura
> Issue Type: Bug
> Components: System - memory
> Affects Versions: 0.3
> Environment: Windows XP SP3 x86, java (and javac) 1.5.0_15
> Reporter: Paul Bristow
> Assignee: Ian Boston
> Priority: Critical
> Fix For: 0.4
>
> Attachments: heap1.PNG, jc2.zip, jconsole.zip, Screen shot 2010-03-03 at 15.30.29.png, Screen shot 2010-03-03 at 15.30.38.png, Screen shot 2010-03-03 at 16.43.35.png, tenured1.PNG, tenured_java6.PNG
>
>
Ian Boston (JIRA)
[ http://jira.sakaiproject.org/browse/KERN-655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=94458#action_94458 ]
Ian Boston commented on KERN-655:
---------------------------------
Slingon its own does not appear to be showing any signs of a memory issue, at 10K users still using 50MB of heap.
> possible memory leak posting lots of user creates
> -------------------------------------------------
>
> Key: KERN-655
> URL: http://jira.sakaiproject.org/browse/KERN-655
> Project: Nakamura
> Issue Type: Bug
> Components: System - memory
> Affects Versions: 0.3
> Environment: Windows XP SP3 x86, java (and javac) 1.5.0_15
> Reporter: Paul Bristow
> Assignee: Ian Boston
> Priority: Critical
> Fix For: 0.4
>
> Attachments: heap1.PNG, jc2.zip, jconsole.zip, Screen shot 2010-03-03 at 15.30.29.png, Screen shot 2010-03-03 at 15.30.38.png, Screen shot 2010-03-03 at 16.43.35.png, tenured1.PNG, tenured_java6.PNG
>
>
Ian Boston (JIRA)
[ http://jira.sakaiproject.org/browse/KERN-655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=94463#action_94463 ]
Ian Boston commented on KERN-655:
---------------------------------
The leak appears to be in the DynamicSecurityManager that was introduced to enable, group deny and dynamic groups.
The slowness is a number of factors:
Group deny means that its not possible to optimize the whole security sub system for read, since we must test the ACL's for deny.
We create lots more nodes in Nakumara on user creation that in Sling.
First thing to locate and fix is the leak.
> possible memory leak posting lots of user creates
> -------------------------------------------------
>
> Key: KERN-655
> URL: http://jira.sakaiproject.org/browse/KERN-655
> Project: Nakamura
> Issue Type: Bug
> Components: System - memory
> Affects Versions: 0.3
> Environment: Windows XP SP3 x86, java (and javac) 1.5.0_15
> Reporter: Paul Bristow
> Assignee: Ian Boston
> Priority: Critical
> Fix For: 0.4
>
> Attachments: heap1.PNG, jc2.zip, jconsole.zip, Screen shot 2010-03-03 at 15.30.29.png, Screen shot 2010-03-03 at 15.30.38.png, Screen shot 2010-03-03 at 16.43.35.png, tenured1.PNG, tenured_java6.PNG
>
>
Ian Boston (JIRA)
[ http://jira.sakaiproject.org/browse/KERN-655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=94465#action_94465 ]
Ian Boston commented on KERN-655:
---------------------------------
Loaded 25K users, in about 30 min, no sign of a leak, requests running at 40ms per user create towards the end.
> possible memory leak posting lots of user creates
> -------------------------------------------------
>
> Key: KERN-655
> URL: http://jira.sakaiproject.org/browse/KERN-655
> Project: Nakamura
> Issue Type: Bug
> Components: System - memory
> Affects Versions: 0.3
> Environment: Windows XP SP3 x86, java (and javac) 1.5.0_15
> Reporter: Paul Bristow
> Assignee: Ian Boston
> Priority: Critical
> Fix For: 0.4
>
> Attachments: heap1.PNG, jc2.zip, jconsole.zip, Screen shot 2010-03-03 at 15.30.29.png, Screen shot 2010-03-03 at 15.30.38.png, Screen shot 2010-03-03 at 16.43.35.png, tenured1.PNG, tenured_java6.PNG
>
>
Ian Boston (JIRA)
[ http://jira.sakaiproject.org/browse/KERN-655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=94472#action_94472 ]
Ian Boston commented on KERN-655:
---------------------------------
I think we have a leak in the JMS Sessions,
a javax.jms.Session is not thread safe and will consume resources if commit is not called, these were missing from the OSGiJMSBridge resulting in a single session being used by multiple threads and being held open. Even with the standard security manager there was a noticeable leak, mainly on the message objects and stats.
> possible memory leak posting lots of user creates
> -------------------------------------------------
>
> Key: KERN-655
> URL: http://jira.sakaiproject.org/browse/KERN-655
> Project: Nakamura
> Issue Type: Bug
> Components: System - memory
> Affects Versions: 0.3
> Environment: Windows XP SP3 x86, java (and javac) 1.5.0_15
> Reporter: Paul Bristow
> Assignee: Ian Boston
> Priority: Critical
> Fix For: 0.4
>
> Attachments: heap1.PNG, jc2.zip, jconsole.zip, Screen shot 2010-03-03 at 15.30.29.png, Screen shot 2010-03-03 at 15.30.38.png, Screen shot 2010-03-03 at 16.43.35.png, tenured1.PNG, tenured_java6.PNG
>
>
Ian Boston (JIRA)
[ http://jira.sakaiproject.org/browse/KERN-655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=94473#action_94473 ]
Ian Boston commented on KERN-655:
---------------------------------
Looks like there may also be some unclosed JCR sessions
3.03.2010 21:44:33.986 *WARN* [Finalizer] org.apache.jackrabbit.core.SessionImpl Unclosed session detected. The session was opened here: java.lang.Exception: Stack Trace
at org.apache.jackrabbit.core.SessionImpl.<init>(SessionImpl.java:249)
at org.apache.jackrabbit.core.SessionImpl.<init>(SessionImpl.java:271)
at org.apache.jackrabbit.core.XASessionImpl.<init>(XASessionImpl.java:105)
at org.apache.jackrabbit.core.RepositoryImpl.createSessionInstance(RepositoryImpl.java:1517)
at org.apache.jackrabbit.core.RepositoryImpl.createSession(RepositoryImpl.java:964)
at org.apache.jackrabbit.core.SessionImpl.createSession(SessionImpl.java:443)
at org.apache.jackrabbit.core.DynamicSecurityManager.getUserManager(DynamicSecurityManager.java:338)
at org.apache.jackrabbit.core.SessionImpl.getUserManager(SessionImpl.java:785)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.sling.jcr.base.SessionProxyHandler$SessionProxyInvocationHandler.invoke(SessionProxyHandler.java:109)
at $Proxy11.getUserManager(Unknown Source)
at org.apache.sling.jcr.base.util.AccessControlUtil.getUserManager(AccessControlUtil.java:116)
at org.sakaiproject.nakamura.securityloader.Loader.createGroup(Loader.java:643)
at org.sakaiproject.nakamura.securityloader.Loader.installFromPath(Loader.java:389)
at org.sakaiproject.nakamura.securityloader.Loader.install(Loader.java:324)
at org.sakaiproject.nakamura.securityloader.Loader.registerBundleInternal(Loader.java:200)
at org.sakaiproject.nakamura.securityloader.Loader.registerBundle(Loader.java:129)
at org.sakaiproject.nakamura.securityloader.SecurityLoaderService.activate(SecurityLoaderService.java:202)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.felix.scr.impl.helper.BaseMethod.invokeMethod(BaseMethod.java:213)
at org.apache.felix.scr.impl.helper.BaseMethod.access$500(BaseMethod.java:38)
at org.apache.felix.scr.impl.helper.BaseMethod$Resolved.invoke(BaseMethod.java:542)
at org.apache.felix.scr.impl.helper.BaseMethod.invoke(BaseMethod.java:434)
at org.apache.felix.scr.impl.helper.ActivateMethod.invoke(ActivateMethod.java:138)
at org.apache.felix.scr.impl.manager.ImmediateComponentManager.createImplementationObject(ImmediateComponentManager.java:226)
at org.apache.felix.scr.impl.manager.ImmediateComponentManager.createComponent(ImmediateComponentManager.java:118)
at org.apache.felix.scr.impl.manager.AbstractComponentManager$Unsatisfied.activate(AbstractComponentManager.java:991)
at org.apache.felix.scr.impl.manager.AbstractComponentManager.activateInternal(AbstractComponentManager.java:298)
at org.apache.felix.scr.impl.manager.AbstractComponentManager$3.doRun(AbstractComponentManager.java:505)
at org.apache.felix.scr.impl.ComponentActivatorTask.run(ComponentActivatorTask.java:67)
at org.apache.felix.scr.impl.ComponentActorThread.run(ComponentActorThread.java:96)
at java.lang.Thread.run(Thread.java:637)
> possible memory leak posting lots of user creates
> -------------------------------------------------
>
> Key: KERN-655
> URL: http://jira.sakaiproject.org/browse/KERN-655
> Project: Nakamura
> Issue Type: Bug
> Components: System - memory
> Affects Versions: 0.3
> Environment: Windows XP SP3 x86, java (and javac) 1.5.0_15
> Reporter: Paul Bristow
> Assignee: Ian Boston
> Priority: Critical
> Fix For: 0.4
>
> Attachments: heap1.PNG, jc2.zip, jconsole.zip, Screen shot 2010-03-03 at 15.30.29.png, Screen shot 2010-03-03 at 15.30.38.png, Screen shot 2010-03-03 at 16.43.35.png, tenured1.PNG, tenured_java6.PNG
>
>
Ian Boston (JIRA)
[ http://jira.sakaiproject.org/browse/KERN-655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=94474#action_94474 ]
Ian Boston commented on KERN-655:
---------------------------------
The stack trace appears to be happening a lot when I switch from the PluggableDefaultAccessManager to the standard DefaultAccessManager.
It looks like that might have been holding sessions open so they didnt get finalized, now that they are getting finalized, they are showing up as being left open.
> possible memory leak posting lots of user creates
> -------------------------------------------------
>
> Key: KERN-655
> URL: http://jira.sakaiproject.org/browse/KERN-655
> Project: Nakamura
> Issue Type: Bug
> Components: System - memory
> Affects Versions: 0.3
> Environment: Windows XP SP3 x86, java (and javac) 1.5.0_15
> Reporter: Paul Bristow
> Assignee: Ian Boston
> Priority: Critical
> Fix For: 0.4
>
> Attachments: heap1.PNG, jc2.zip, jconsole.zip, Screen shot 2010-03-03 at 15.30.29.png, Screen shot 2010-03-03 at 15.30.38.png, Screen shot 2010-03-03 at 16.43.35.png, tenured1.PNG, tenured_java6.PNG
>
>
Ian Boston (JIRA)
[ http://jira.sakaiproject.org/browse/KERN-655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=94475#action_94475 ]
Ian Boston commented on KERN-655:
---------------------------------
Making that change eliminated the session leak, we still need to a) fix the unclosed sessions b) find out why this doesnt happen in Sling. Will pick up tomorrow.
> possible memory leak posting lots of user creates
> -------------------------------------------------
>
> Key: KERN-655
> URL: http://jira.sakaiproject.org/browse/KERN-655
> Project: Nakamura
> Issue Type: Bug
> Components: System - memory
> Affects Versions: 0.3
> Environment: Windows XP SP3 x86, java (and javac) 1.5.0_15
> Reporter: Paul Bristow
> Assignee: Ian Boston
> Priority: Critical
> Fix For: 0.4
>
> Attachments: heap1.PNG, jc2.zip, jconsole.zip, Screen shot 2010-03-03 at 15.30.29.png, Screen shot 2010-03-03 at 15.30.38.png, Screen shot 2010-03-03 at 16.43.35.png, tenured1.PNG, tenured_java6.PNG
>
>
Ian Boston (JIRA)
[ http://jira.sakaiproject.org/browse/KERN-655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=94509#action_94509 ]
Ian Boston commented on KERN-655:
---------------------------------
The stack trace 3 comments back was due to a subtle change in the way in which the Jackrabbit DefaultSecurityManager manages the session used for the UserManager in 1.6. Since we have a patched DynamicSecurityManager based on the Jackrabbit 1.5 DefaulSecurityManager, we missed the change in the upgrade done in December.
I have now reverted all but the essential changes to our version so that we dont miss anything when moving from 1.6 to 2.0 in the next few weeks.
I am not certain that this fixes the memory, leak but at least now we are not seeing sessions that have not been logged out being finalized, I will run another batch of tests to confirm sessions are not leaking.
> possible memory leak posting lots of user creates
> -------------------------------------------------
>
> Key: KERN-655
> URL: http://jira.sakaiproject.org/browse/KERN-655
> Project: Nakamura
> Issue Type: Bug
> Components: System - memory
> Affects Versions: 0.3
> Environment: Windows XP SP3 x86, java (and javac) 1.5.0_15
> Reporter: Paul Bristow
> Assignee: Ian Boston
> Priority: Critical
> Fix For: 0.4
>
> Attachments: heap1.PNG, jc2.zip, jconsole.zip, Screen shot 2010-03-03 at 15.30.29.png, Screen shot 2010-03-03 at 15.30.38.png, Screen shot 2010-03-03 at 16.43.35.png, tenured1.PNG, tenured_java6.PNG
>
>
Ian Boston (JIRA)
[ http://jira.sakaiproject.org/browse/KERN-655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=94512#action_94512 ]
Ian Boston commented on KERN-655:
---------------------------------
That appears to have eliminated sessions not being closed, I will put the pluggable DefaultAccessManager back in to see what happens now.
> possible memory leak posting lots of user creates
> -------------------------------------------------
>
> Key: KERN-655
> URL: http://jira.sakaiproject.org/browse/KERN-655
> Project: Nakamura
> Issue Type: Bug
> Components: System - memory
> Affects Versions: 0.3
> Environment: Windows XP SP3 x86, java (and javac) 1.5.0_15
> Reporter: Paul Bristow
> Assignee: Ian Boston
> Priority: Critical
> Fix For: 0.4
>
> Attachments: heap1.PNG, jc2.zip, jconsole.zip, Screen shot 2010-03-03 at 15.30.29.png, Screen shot 2010-03-03 at 15.30.38.png, Screen shot 2010-03-03 at 16.43.35.png, tenured1.PNG, tenured_java6.PNG
>
>
Ian Boston (JIRA)
[ http://jira.sakaiproject.org/browse/KERN-655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ian Boston updated KERN-655:
----------------------------
Attachment: Screen shot 2010-03-04 at 12.45.21.png
Screenshot (ending 12.45.21) is of Nakamura as fixed, after about 4K users.
Memory appears to be at steady state.
There are some issues remaining.
1. Addition rate has droped to about 5/s which is not acceptable IMHO anything < 10/s is not acceptable.
2. We appear to have some other smaller leak on ActiveMQ. Looking at a memory snapshot we have
139K ActiveMQMessageProducers retaining 105MB after a GC.
I think we are not doing something right in our use of ActiveMQ. I will disable the OSGiToJMS bridge to see if that is also the cause of the slowness.
> possible memory leak posting lots of user creates
> -------------------------------------------------
>
> Key: KERN-655
> URL: http://jira.sakaiproject.org/browse/KERN-655
> Project: Nakamura
> Issue Type: Bug
> Components: System - memory
> Affects Versions: 0.3
> Environment: Windows XP SP3 x86, java (and javac) 1.5.0_15
> Reporter: Paul Bristow
> Assignee: Ian Boston
> Priority: Critical
> Fix For: 0.4
>
> Attachments: heap1.PNG, jc2.zip, jconsole.zip, Screen shot 2010-03-03 at 15.30.29.png, Screen shot 2010-03-03 at 15.30.38.png, Screen shot 2010-03-03 at 16.43.35.png, Screen shot 2010-03-04 at 12.45.21.png, tenured1.PNG, tenured_java6.PNG
Ian Boston (JIRA)
[ http://jira.sakaiproject.org/browse/KERN-655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ian Boston updated KERN-655:
----------------------------
Attachment: Screen shot 2010-03-04 at 13.31.41.png
This screenshot is after adding 3800 users with the OSGi to JMS bridge disabled.
Something about the way we use JMS is causing a leak, confirmed by a Memory Snapshot in Yourkit which reports About 50MB of the 100MB is due to Derby buffers.
Speed is a little better but still of concern.
> possible memory leak posting lots of user creates
> -------------------------------------------------
>
> Key: KERN-655
> URL: http://jira.sakaiproject.org/browse/KERN-655
> Project: Nakamura
> Issue Type: Bug
> Components: System - memory
> Affects Versions: 0.3
> Environment: Windows XP SP3 x86, java (and javac) 1.5.0_15
> Reporter: Paul Bristow
> Assignee: Ian Boston
> Priority: Critical
> Fix For: 0.4
>
> Attachments: heap1.PNG, jc2.zip, jconsole.zip, Screen shot 2010-03-03 at 15.30.29.png, Screen shot 2010-03-03 at 15.30.38.png, Screen shot 2010-03-03 at 16.43.35.png, Screen shot 2010-03-04 at 12.45.21.png, Screen shot 2010-03-04 at 13.31.41.png, tenured1.PNG, tenured_java6.PNG
Ian Boston (JIRA)
[ http://jira.sakaiproject.org/browse/KERN-655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ian Boston resolved KERN-655.
-----------------------------
Resolution: Fixed
Fixed, but need to open 2 more, one for JMS message persistence and one for performance.
> possible memory leak posting lots of user creates
> -------------------------------------------------
>
> Key: KERN-655
> URL: http://jira.sakaiproject.org/browse/KERN-655
> Project: Nakamura
> Issue Type: Bug
> Components: System - memory
> Affects Versions: 0.3
> Environment: Windows XP SP3 x86, java (and javac) 1.5.0_15
> Reporter: Paul Bristow
> Assignee: Ian Boston
> Priority: Critical
> Fix For: 0.4
>
> Attachments: heap1.PNG, jc2.zip, jconsole.zip, Screen shot 2010-03-03 at 15.30.29.png, Screen shot 2010-03-03 at 15.30.38.png, Screen shot 2010-03-03 at 16.43.35.png, Screen shot 2010-03-04 at 12.45.21.png, Screen shot 2010-03-04 at 13.31.41.png, tenured1.PNG, tenured_java6.PNG
>
>
Ian Boston (JIRA)
[ http://jira.sakaiproject.org/browse/KERN-655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=94580#action_94580 ]
Ian Boston commented on KERN-655:
---------------------------------
Fixed the ActiveMQ issues by making the connections poolable and the messages non persistent.
I think we need to review the JMS messages since a proportion need to be persistent, but only a proportion.
> possible memory leak posting lots of user creates
> -------------------------------------------------
>
> Key: KERN-655
> URL: http://jira.sakaiproject.org/browse/KERN-655
> Project: Nakamura
> Issue Type: Bug
> Components: System - memory
> Affects Versions: 0.3
> Environment: Windows XP SP3 x86, java (and javac) 1.5.0_15
> Reporter: Paul Bristow
> Assignee: Ian Boston
> Priority: Critical
> Fix For: 0.4
>
> Attachments: heap1.PNG, jc2.zip, jconsole.zip, Screen shot 2010-03-03 at 15.30.29.png, Screen shot 2010-03-03 at 15.30.38.png, Screen shot 2010-03-03 at 16.43.35.png, Screen shot 2010-03-04 at 12.45.21.png, Screen shot 2010-03-04 at 13.31.41.png, tenured1.PNG, tenured_java6.PNG
>
>