Issue 133885 in chromium: GPU memory leak on the mac with Flash

[chro...@googlecode.com]

Status: Assigned
Owner: vang...@chromium.org
CC: k...@chromium.org, stuart...@chromium.org, tha...@chromium.org
Labels: Type-Bug Pri-2 Area-Internals Feature-GPU Feature-Plugins OS-Mac
Mstone-21 ReleaseBlock-Stable

New issue 133885 by vange...@google.com: GPU memory leak on the mac with
Flash
http://code.google.com/p/chromium/issues/detail?id=133885

Version: 20+
OS: MacOSX

What steps will reproduce the problem?
1. Open the OpenGL driver monitor and turn on the graph for "Current Video
Memory In Use"
2. Open a tab and navigate to youtube.com
3. In the same tab, click on one of the videos
4. Keep going back and forth between the youtube front page and the video
page.

What is the expected output? What do you see instead?
GPU memory graph should stay fairly flat over time. Instead it keeps
climbing.

It doesn't seem to be happening with just plain composited pages. Something
seems to be leaking in our handling of CA plugins.

[chro...@googlecode.com]

Comment #1 on issue 133885 by stuart...@chromium.org: GPU memory leak on

the mac with Flash
http://code.google.com/p/chromium/issues/detail?id=133885

Have we tested with another CA plugin (e.g., Unity or Silverlight) to make
sure it's not on the Flash side?

[chro...@googlecode.com]

Updates:
Cc: jbau...@chromium.org jba...@chromium.org sa...@chromium.org

Comment #2 on issue 133885 by k...@chromium.org: GPU memory leak on the mac
with Flash
http://code.google.com/p/chromium/issues/detail?id=133885

That's a really good question. I just tested on a MacBook Pro (15",
mid-2010, GeForce GT 330M, 10.7.4), and it actually does look like it might
be a problem just with Flash.

Attached are two graphs from the OpenGL Driver Monitor tracking the free
and used VRAM on the system from just before when Chrome is launched
through several back-and-forth cycles with both the Flash and Unity
plugins. Tested 21.0.1180.0 (Official Build 142910) canary.

For the Flash test, I alternated between the following two URLs:

http://www.youtube.com/
http://www.youtube.com/watch?v=jozTK-MqEXQ&feature=g-logo-xit

For the Unity test, I alternated between the following two URLs:

http://unity3d.com/unity/
http://unity3d.com/gallery/demos/live-demos#tropical-paradise

The Flash graph shows a steady increase in VRAM consumption that the Unity
graph doesn't.

One possible difference is that I was implicitly alternating between two
pages that both contained Flash, while only one contained a Unity plugin
instance. I'll see whether that seems to matter.


Attachments:
Flash-in-OpenGL-Driver-Monitor.png 62.1 KB
Unity-3D-in-OpenGL-Driver-Monitor.png 70.3 KB

[chro...@googlecode.com]

Comment #3 on issue 133885 by k...@chromium.org: GPU memory leak on the mac
with Flash
http://code.google.com/p/chromium/issues/detail?id=133885

I reran the experiment so that in both cases I was clicking a link to
alternate between the two pages. For the two Flash pages I alternated
between:

http://www.youtube.com/
http://www.youtube.com/watch?v=jozTK-MqEXQ&feature=g-logo-xit

and for the two Unity pages I alternated between:

http://unity3d.com/gallery/demos/live-demos#tropical-paradise
http://unity3d.com/gallery/demos/live-demos#butterfly

Attached are the two new graphs. Flash shows a clear increase in VRAM
consumption over time.


Attachments:
flash-alternating.png 62.2 KB
unity-alternating.png 71.7 KB

[chro...@googlecode.com]

Comment #4 on issue 133885 by k...@chromium.org: GPU memory leak on the mac
with Flash
http://code.google.com/p/chromium/issues/detail?id=133885

Here's the same graph when running Flash 11.3.300.257 in Firefox Aurora
15.0a2 (2012-06-21). VRAM consumption doesn't grow over time. So it's
either a bug in Chrome, a bug in Chrome's bundled Flash, or a bug in Flash
that is triggered by user agent detection.

I had to disable automatic graphics switching in the Energy Saver
preferences to force the system on to the discrete GPU; Firefox stays on
the integrated GPU by default on this system. (Chrome doesn't do this
because of system stability issues with this GPU combination.)


Attachments:
flash-in-firefox-aurora.png 60.3 KB

[chro...@googlecode.com]

Comment #5 on issue 133885 by k...@chromium.org: GPU memory leak on the mac
with Flash
http://code.google.com/p/chromium/issues/detail?id=133885

I can confirm that disabling the "Enable hardware acceleration" under
Flash's "Settings..." causes the VRAM leak to go away when running in
Chrome. I don't know exactly what that setting does though. It doesn't
*seem* to disable the use of the Core Animation drawing model.

[chro...@googlecode.com]

Updates:
Owner: k...@chromium.org
Cc: vang...@chromium.org

Comment #8 on issue 133885 by k...@chromium.org: GPU memory leak on the mac
with Flash
http://code.google.com/p/chromium/issues/detail?id=133885

I've set breakpoints in CGLCreateContext and CGLDestroyContext in the Flash
plugin process and can see very clearly that a CGLContextObj created
underneath Chrome's call into the plugin to fetch its CALayer is not
destroyed.

I've tried adding a retain/release pair to that CALayer instance, but it
didn't help. Going to look through Firefox's sources to see how it manages
the plugin instance and see if there's some difference in Chrome's lifetime
management.

[chro...@googlecode.com]

Comment #9 on issue 133885 by k...@chromium.org: GPU memory leak on the mac
with Flash
http://code.google.com/p/chromium/issues/detail?id=133885

One site which allocates multiple Flash plugins per page and which might
reproduce the leak more visibly is
http://www.oursportscentral.com/sports/?m_id=Baseball .

[chro...@googlecode.com]

Comment #10 on issue 133885 by k...@chromium.org: GPU memory leak on the mac
with Flash
http://code.google.com/p/chromium/issues/detail?id=133885

Ran across this interesting Mozilla bug indicating that CARenderers might
be holding on to their CALayers longer than expected:

https://bugzilla.mozilla.org/show_bug.cgi?id=556453

However, it doesn't appear to be the problem in Chrome at least -- applying
a similar fix to src/webkit/plugins/npapi/webplugin_delegate_impl_mac.mm
didn't help.

[chro...@googlecode.com]

Updates:
Status: Started
Cc: apatr...@chromium.org

Comment #11 on issue 133885 by k...@chromium.org: GPU memory leak on the mac
with Flash
http://code.google.com/p/chromium/issues/detail?id=133885

Thanks to help from jbauman, thakis and apatrick, did extensive
experimentation today and now know one code change which avoids the
resource leak.

I built Firefox Aurora from sources in order to be able to compare its
behavior to Chrome. Setting breakpoints in CGLCreateContext and
CGLDestroyContext indicated that the same OpenGL context leak existed in
Firefox! However, jbauman pointed out that
CGLRetainContext/CGLReleaseContext might be being used, which was an
excellent point, and in fact those are used (by Core Animation) to maintain
its contexts. However, many calls to these functions are made when
navigating to and away from a YouTube video; it would be tedious to figure
out which contexts they're being called against.

With help from thakis I tried to instrument the dealloc call on the class
of the CALayer returned by the plugin, using the trick in
chrome/browser/ui/cocoa/custom_frame_view.mm, in order to see if the layer
was being deallocated in one browser but not the other. The reference
counts looked identical in the two browsers. However, this instrumentation
didn't work well. We saw various CALayers being deallocated at various
points, including during plugin resizing in Chrome, but not the root one.

There is one obvious difference in behavior between Chrome's and Firefox's
plugin hosts. Firefox forcibly invalidates and deletes all NPObject
instances vended by the plugin instance immediately after calling
NPP_Destroy. Chrome only does this for the plugin's scriptable object, but
not other objects. See Issue 119414 and
https://chromiumcodereview.appspot.com/9979022 . It turns out that when
visiting a YouTube page and then navigating away from it, Firefox forcibly
deletes three NPObjects. It didn't look simple to implement Firefox's
behavior in Chrome, but I tried commenting out the code which does the
cleanup in Firefox. This didn't cause the resource leak to start happening
in Firefox, indicating the behavioral difference was coming from elsewhere.

After a few failed experiments I tried -- on recommendation from thakis
based on feedback from jbauman -- manually calling release against the
CALayer instance vended by the plugin. Releasing it twice after the plugin
instance is destroyed fixes the GPU resource leak, apparently without
introducing plugin process crashes.

Tomorrow I'll track down exactly where in the Firefox code that CALayer is
being released. There must be a difference between Firefox's management of
this layer's lifetime and Chrome's. Regardless, it seems clear that Flash
is not following the specification in
https://wiki.mozilla.org/NPAPI:CoreAnimationDrawingModel for the management
of this layer's lifetime.

[chro...@googlecode.com]

Updates:
Cc: w...@chromium.org

Comment #12 on issue 133885 by k...@chromium.org: GPU memory leak on the mac
with Flash
http://code.google.com/p/chromium/issues/detail?id=133885

(No comment was entered for this change.)

[chro...@googlecode.com]

Updates:
Cc: viettrun...@chromium.org

Comment #13 on issue 133885 by k...@chromium.org: GPU memory leak on the mac

[chro...@googlecode.com]

Comment #14 on issue 133885 by jbau...@chromium.org: GPU memory leak on the

mac with Flash
http://code.google.com/p/chromium/issues/detail?id=133885

Somewhat relevant: https://bugs.webkit.org/show_bug.cgi?id=58282

[chro...@googlecode.com]

Comment #15 on issue 133885 by jbau...@chromium.org: GPU memory leak on the

mac with Flash
http://code.google.com/p/chromium/issues/detail?id=133885

Also
http://code.google.com/searchframe#OAMlx_jo-ck/src/third_party/WebKit/Source/WebKit2/Shared/Plugins/Netscape/mac/NetscapePluginModuleMac.mm&exact_package=chromium&q=ReturnsRetainedCoreAnimationLayer&ct=rc&cd=4&sq=&l=475
and WKNVExpectsNonretainedLayer in WebKit2.

[chro...@googlecode.com]

Comment #16 on issue 133885 by stuart...@chromium.org: GPU memory leak

on the mac with Flash
http://code.google.com/p/chromium/issues/detail?id=133885

> Somewhat relevant: https://bugs.webkit.org/show_bug.cgi?id=58282

Yes, I brought this up with Adobe when I first discovered that plugins were
using the pre-spec behavior and leaking, so it should have been fixed, but
perhaps they have regressed. We should talk to Adobe again to make
absolutely sure they aren't using the incorrect retain model in Chrome.

[chro...@googlecode.com]

Updates:
Status: ExternalDependency
Owner: viettrun...@chromium.org
Cc: dhar...@chromium.org

Comment #19 on issue 133885 by k...@chromium.org: GPU memory leak on the mac
with Flash
http://code.google.com/p/chromium/issues/detail?id=133885

viettrungluu@ has confirmed via code changes to the built-in Flash plugin
for Chrome that this is purely a bug in Flash.

He has code changes in hand that will be submitted to Adobe which fix this
leak and improve correctness when running in all of Chrome, Safari and
Firefox.

I'm going to assign this bug to Trung and mark it ExternalDependency. Once
we have a new build of Flash integrated into Chrome on Mac OS this bug will
be fixed.

[chro...@googlecode.com]

Comment #20 on issue 133885 by rn...@chromium.org: GPU memory leak on the

mac with Flash
http://code.google.com/p/chromium/issues/detail?id=133885

Is there anyway to disable the GPU process for Flash (or disable the GPU
acceleration entirely) until the bug is fixed (especially on machines with
limited RAM; e.g. less than 4GB)? I find myself manually killing GPU
process (and the Flash process because otherwise Flash never paints) every
time I visit youtube.com because it uses up all the RAM on my machine.

I suspect most of ordinary users don't even know how to recover from the
law memory state, forced into rebooting their machines.

[chro...@googlecode.com]

Comment #21 on issue 133885 by stuart...@chromium.org: GPU memory leak

on the mac with Flash
http://code.google.com/p/chromium/issues/detail?id=133885

> He has code changes in hand that will be submitted to Adobe which fix
> this leak and
> improve correctness when running in all of Chrome, Safari and Firefox.

Trung, I'm not sure how you ended up fixing this in a way that works with
both WebKit.framework and everything else, but in case it's helpful I had a
thought about that this morning:
https://bugs.webkit.org/show_bug.cgi?id=58282#c7

[chro...@googlecode.com]

Updates:
Owner: jeffr...@chromium.org

Comment #22 on issue 133885 by kar...@google.com: GPU memory leak on the

mac with Flash
http://code.google.com/p/chromium/issues/detail?id=133885

jeff trung gave them the code, so this is on you to push with adobe folk so
we can get it out.

[chro...@googlecode.com]

Comment #23 on issue 133885 by jeff...@google.com: GPU memory leak on the

mac with Flash
http://code.google.com/p/chromium/issues/detail?id=133885

Indeed. Email thread in progress.

[chro...@googlecode.com]

Comment #24 on issue 133885 by kar...@google.com: GPU memory leak on the

mac with Flash
http://code.google.com/p/chromium/issues/detail?id=133885

ping?

[chro...@googlecode.com]

Comment #25 on issue 133885 by jeff...@google.com: GPU memory leak on the

mac with Flash
http://code.google.com/p/chromium/issues/detail?id=133885

Adobe said:

"You will see a version of the fix in the next Flash Player beta going out
on Adobe Labs. We've already locked the beta sources and the release will
be available in a few days."

So it sounds like the fix will be in Flash 11.4.

[chro...@googlecode.com]

Comment #26 on issue 133885 by kar...@google.com: GPU memory leak on the

mac with Flash
http://code.google.com/p/chromium/issues/detail?id=133885

so if this comes post stable are we blocking stable on this?

[chro...@googlecode.com]

Comment #27 on issue 133885 by jeff...@google.com: GPU memory leak on the

mac with Flash
http://code.google.com/p/chromium/issues/detail?id=133885

I'm not sure what the original rationale for making this
ReleaseBlock-Stable was (maybe the engineers can chime in?). My impression
is just that we want Adobe to fix this ASAP, which it sounds like they're
doing (ish).

[chro...@googlecode.com]

Updates:
Labels: -ReleaseBlock-Stable

Comment #28 on issue 133885 by kar...@google.com: GPU memory leak on the

mac with Flash
http://code.google.com/p/chromium/issues/detail?id=133885

[chro...@googlecode.com]

Comment #29 on issue 133885 by wiltz...@chromium.org: GPU memory leak on

the mac with Flash
http://code.google.com/p/chromium/issues/detail?id=133885

@kbr or others, we've witnessed this on stable right? If so then getting
rid of the release block stable for Chrome is fine. If not, we almost
certainly want to keep it and push Adobe further, because this will cause
lots and lots of problems (pages won't be visible) as we run out of VRAM.