Understanding Web developer needs for memory-leak detection tools
Robert O'Callahan
track down a memory leak in GMail. The process gave us some insights into
what Web developers need from tools. Here's roughly what I think is needed:
1) Ability to identify objects that may be leaking. We were able to do a
series of actions that should have returned the app to a stable state, but
we could see (via cycle collector dumps) that there were ever-increasing
numbers of certain kinds of objects. A good tool would let the developer
take snapshots and track changes in the numbers of objects of different
types, perhaps highlighting the fastest-growing kinds of objects.
2) Inspect possibly-leaked objects. Once we realized that certain DOM
objects kept growing in number, we wanted to find out more about them. A
tool should offer inspection of the contents of the new DOM and JS objects.
3) Find paths from roots to suspicious objects. Once we identified
suspicious object instances --- objects that we thought should be dead, but
weren't collected --- we needed to explain why the objects couldn't be
collected. This meant finding paths in the heap from roots to the suspicious
objects. We worked backwards from the suspicious objects, identifying
objects with references to suspicious objects and manually trying to decide
whether those frontier objects were suspicious or not. With that approach we
had to be careful to ignore back-references from the descendants of
suspicious objects back to the suspicious objects we were looking at.
Perhaps a shortest-path analysis would have been better. We would probably
need to work hard to give good explanations of why objects are rooted
For the particular bug we looked at, stack traces for object allocations
would not have been useful. The underlying bug was a failure to unregister a
listener callback, there was probably nothing wrong at any allocation site.
Rob
--
"Now the Bereans were of more noble character than the Thessalonians, for
they received the message with great eagerness and examined the Scriptures
every day to see if what Paul said was true." [Acts 17:11]
johnjbarton
> David Baron, Andrew McCreight and I went over to Google today to try to
> track down a memory leak in GMail. The process gave us some insights into
> what Web developers need from tools. Here's roughly what I think is needed:
One of the biggest problems is getting developers to provide test cases.
I have lots of complaints but no one can come up with a case.
> 1) Ability to identify objects that may be leaking. We were able to do a
> series of actions that should have returned the app to a stable state, but
> we could see (via cycle collector dumps) that there were ever-increasing
> numbers of certain kinds of objects.
What do you mean here by "certain kinds"? That is how could you classify
them? Or maybe I am thinking about JS objects where classification is
hard but you are talking about eg HTML elements?
Where any of the accumulating objects functions?
> A good tool would let the developer
> take snapshots
Taking snapshots by itself would enable lots of cool tools. We could
then do diffs and we could do replays accurately.
and track changes in the numbers of objects of different
> types, perhaps highlighting the fastest-growing kinds of objects.
> 2) Inspect possibly-leaked objects. Once we realized that certain DOM
> objects kept growing in number, we wanted to find out more about them. A
> tool should offer inspection of the contents of the new DOM and JS objects.
I know a tool that does that ;-).
> 3) Find paths from roots to suspicious objects. Once we identified
> suspicious object instances --- objects that we thought should be dead, but
> weren't collected --- we needed to explain why the objects couldn't be
> collected. This meant finding paths in the heap from roots to the suspicious
> objects. We worked backwards from the suspicious objects, identifying
> objects with references to suspicious objects and manually trying to decide
> whether those frontier objects were suspicious or not. With that approach we
> had to be careful to ignore back-references from the descendants of
> suspicious objects back to the suspicious objects we were looking at.
> Perhaps a shortest-path analysis would have been better. We would probably
> need to work hard to give good explanations of why objects are rooted
>
> For the particular bug we looked at, stack traces for object allocations
> would not have been useful. The underlying bug was a failure to unregister a
> listener callback, there was probably nothing wrong at any allocation site.
More information would be great, since we have the ability to track
event listeners (in eventbug), though not their add and remove
(https://bugzilla.mozilla.org/show_bug.cgi?id=524674). If we knew the
test case sequence we could consider whether additions to eventbug could
help.
But why then would object allocation traces not help? The listener was
creating objects that were not collected right?
jjb
Robert O'Callahan
> On 4/14/2011 7:42 PM, Robert O'Callahan wrote:
>
> 1) Ability to identify objects that may be leaking. We were able to do a
>> series of actions that should have returned the app to a stable state, but
>> we could see (via cycle collector dumps) that there were ever-increasing
>> numbers of certain kinds of objects.
>>
>
> What do you mean here by "certain kinds"? That is how could you classify
> them? Or maybe I am thinking about JS objects where classification is hard
> but you are talking about eg HTML elements?
> Where any of the accumulating objects functions?
For DOM and other host objects it's easy, we can classify by the C++ class.
For JS objects it's harder, maybe we can classify by prototype or allocation
site. Just doing host objects would be a fine start.
and track changes in the numbers of objects of different
>
>> types, perhaps highlighting the fastest-growing kinds of objects.
>> 2) Inspect possibly-leaked objects. Once we realized that certain DOM
>> objects kept growing in number, we wanted to find out more about them. A
>> tool should offer inspection of the contents of the new DOM and JS
>> objects.
>>
>
> I know a tool that does that ;-).
Yeah but it's hard to do off a snapshot since our normal DOM APIs won't let
you inspect the state of the object as it was when the snapshot was taken.
> 3) Find paths from roots to suspicious objects. Once we identified
>> suspicious object instances --- objects that we thought should be dead,
>> but
>> weren't collected --- we needed to explain why the objects couldn't be
>> collected. This meant finding paths in the heap from roots to the
>> suspicious
>> objects. We worked backwards from the suspicious objects, identifying
>> objects with references to suspicious objects and manually trying to
>> decide
>> whether those frontier objects were suspicious or not. With that approach
>> we
>> had to be careful to ignore back-references from the descendants of
>> suspicious objects back to the suspicious objects we were looking at.
>> Perhaps a shortest-path analysis would have been better. We would probably
>> need to work hard to give good explanations of why objects are rooted
>>
>> For the particular bug we looked at, stack traces for object allocations
>> would not have been useful. The underlying bug was a failure to unregister
>> a
>> listener callback, there was probably nothing wrong at any allocation
>> site.
>>
>
> More information would be great, since we have the ability to track event
> listeners (in eventbug), though not their add and remove (
> https://bugzilla.mozilla.org/show_bug.cgi?id=524674). If we knew the test
> case sequence we could consider whether additions to eventbug could help.
>
Actually in the GMail case these were not DOM event listeners, but a custom
listener callback pattern implemented entirely in JS.
But why then would object allocation traces not help? The listener was
> creating objects that were not collected right?
>
Yes, but the reason they're not collected has nothing at all to do with how
they are allocated.
johnjbarton
> On Fri, Apr 15, 2011 at 3:46 PM, johnjbarton<johnj...@johnjbarton.com>wrote:
>
>> On 4/14/2011 7:42 PM, Robert O'Callahan wrote:
>>
>
> For DOM and other host objects it's easy, we can classify by the C++ class.
> For JS objects it's harder, maybe we can classify by prototype or allocation
> site. Just doing host objects would be a fine start.
Just FYI: the UI for classification by prototype would be more effective
with names for constructors and Salman Mirghasemi just completed some
work on a static analysis approach naming anonymous functions. His paper
should be available in a on-line in week or so.
>
> and track changes in the numbers of objects of different
>>
>>> types, perhaps highlighting the fastest-growing kinds of objects.
>>> 2) Inspect possibly-leaked objects. Once we realized that certain DOM
>>> objects kept growing in number, we wanted to find out more about them. A
>>> tool should offer inspection of the contents of the new DOM and JS
>>> objects.
>>>
>>
>> I know a tool that does that ;-).
>
> Yeah but it's hard to do off a snapshot since our normal DOM APIs won't let
> you inspect the state of the object as it was when the snapshot was taken.
I'd like to learn more about what prevents us from re-constituting a
window state from a stored checkpoint. JS and in particular the DOM APIs
dramatically accelerates our ability to develop tools and to experiement
with ideas. Checkpointing would open a lot of opportunities in tool
development.
If anyone has a testcase showing memory-leaks in JS, Honza and I will
develop tools to diagnosis such a leak. We just can't get anyone with
these problems to report them.
Robert O'Callahan
> If anyone has a testcase showing memory-leaks in JS, Honza and I will
> develop tools to diagnosis such a leak. We just can't get anyone with these
> problems to report them.
>
I obviously can't give you the real test-case but you can write your own
according to the following pattern:
1) Create a global EventBroadcaster object with methods
registerListenerForEvent(eventName /* string */, callback);
unregisterListenerForEvent(eventName /* string */, callback);
runListenersForEvent(eventName /* string */);
Implement it using a hashtable mapping to a list of callbacks.
2) Create a fake "edit" action which adds bunch of callbacks to the global
object, then removes those callbacks. Make each callback a closure which
hangs onto a big data structure, e.g. a DOM document.
3) Introduce a bug where you forget to remove one or more of those
callbacks.
4) Run that edit action over and over again. Presto, memory leak!
Really this is just having a global data structure which you keep adding
stuff to and sometimes forget to remove the stuff you added.
John J Barton
> On Sun, Apr 17, 2011 at 4:56 AM, johnjbarton <johnj...@johnjbarton.com>wrote:
>
>> If anyone has a testcase showing memory-leaks in JS, Honza and I will
>> develop tools to diagnosis such a leak. We just can't get anyone with these
>> problems to report them.
>>
>
> I obviously can't give you the real test-case but you can write your own
> according to the following pattern:
>
> 1) Create a global EventBroadcaster object with methods
> registerListenerForEvent(eventName /* string */, callback);
> unregisterListenerForEvent(eventName /* string */, callback);
> runListenersForEvent(eventName /* string */);
> Implement it using a hashtable mapping to a list of callbacks.
>
> 2) Create a fake "edit" action which adds bunch of callbacks to the global
> object, then removes those callbacks. Make each callback a closure which
> hangs onto a big data structure, e.g. a DOM document.
But at this step we need to do more than just point to |document|
correct? We really need to point to some data that otherwise would be
collected right?
jjb
Robert O'Callahan
<johnj...@johnjbarton.com>wrote:
> Robert O'Callahan wrote:
>
>> On Sun, Apr 17, 2011 at 4:56 AM, johnjbarton <johnj...@johnjbarton.com
>> >wrote:
>>
>>
>> If anyone has a testcase showing memory-leaks in JS, Honza and I will
>>> develop tools to diagnosis such a leak. We just can't get anyone with
>>> these
>>> problems to report them.
>>>
>>>
>> I obviously can't give you the real test-case but you can write your own
>>
>> according to the following pattern:
>>
>> 1) Create a global EventBroadcaster object with methods
>> registerListenerForEvent(eventName /* string */, callback);
>> unregisterListenerForEvent(eventName /* string */, callback);
>> runListenersForEvent(eventName /* string */);
>> Implement it using a hashtable mapping to a list of callbacks.
>>
>> 2) Create a fake "edit" action which adds bunch of callbacks to the global
>> object, then removes those callbacks. Make each callback a closure which
>> hangs onto a big data structure, e.g. a DOM document.
>>
>
> But at this step we need to do more than just point to |document| correct?
> We really need to point to some data that otherwise would be collected
> right?
>
In this example a tool would
a) report the kinds of objects whose counts were growing over time; in this
case, various objects including DOM documents
b) let the user inspect the DOM document objects that were live after some
leaks had occurred
c) when the user selects a DOM document object that should have been
collected, let the user explore the paths leading backwards from that object
to a root (in this case, the global object). These paths would all lead back
to the EventBroadcaster object. In fact, reporting which objects are common
to all the paths from roots to the leaked object would be a very useful
feature.
Boris Zbarsky
> But at this step we need to do more than just point to |document|
> correct? We really need to point to some data that otherwise would be
> collected right?
Yes. Like say the responseXML from an XMLHttpRequest.
-Boris
johnjbarton
Jan 'Honza' Odvarko and I created three different simple test cases,
strings, images, and 'edit':
http://getfirebug.com/tests/issues/memory/test.html
We are able to detect the leaks in these simple cases. We have no idea
about potential false positives, missing cases, performance in real
apps, etc. We also hit another wrapper bug that we need to create test
case for. Honza plans to put this out in 1.8a2 in a few days for feedback.
jjb
Jan Honza Odvarko
I have appended detailed description of what to do and expect when
going
through all individual test-cases on that page.
http://getfirebug.com/tests/issues/memory/test.html
1.8a2 will be available on Friday.
Honza
Jan Honza Odvarko
http://blog.getfirebug.com/2011/04/22/firebug-1-8a2/
Honza