Severe memory leak on YouTube live streams

[Alistair]

Hi,

There is a memory leak issue with YouTube live streams.  Not quite a traditional memory leak but Video DownloadHelper will use up more and more memory the longer a live stream on YouTube is left on.  This problem is likely to be on other video streaming sites as well.

To replicate the problem:

1. Have VideoDownloadHelper enabled

2. Go to a YouTube live stream and leave it open.

That's it, if you leave it long enough Firefox will use all the available system memory and then the computers hard drive will start thrashing.  I confirmed it was this addon by going to about:memory, creating a memory report, seeing the addon UUID listed on top of addons part of the memory display (from memory it was about 8GB used at the time) and going to the moz-extension page associated with that.

I believe the problem is that the media processing part of the UI where you choose what to download gets all these little video segments that can be downloaded the list, and that list keeps growing and growing.  Perhaps if each tab had a limit of the last X segments displayed for download it would fix it, I don't know, but that seems like an obvious thing that would keep using more memory.

Because of this memory usage problem I now keep the addon disabled until I need to use it.  This is a huge problem for anyone using this addon, even if they only have it enabled and aren't using it to download anything, it needs to be looked at.

But otherwise thanks for this addon, I do not regret paying for a license, I am actually very happy with it.

[Wild Willy]

It may be worth it to try the 7.6.3a1 beta. Michel made it specifically in response to
some problems I & a few others were having with livestreams. You can read about it here:

https://groups.google.com/g/video-downloadhelper-q-and-a/c/S4K1QgvT-K4

The beta has improved the situation, for me at least with Golf Channel livestreams. I'd
say the improvement is about 95%. Every once in a while, the problem the beta is meant
to solve pops up. But mostly it's good. The problem was one of VDH erroneously
believing the livestream was finished when it wasn't. Even though you aren't seeing that
specific problem, the fact is that VDH 7.6.3a1 beta contains changes in the processing of
livestreams so it couldn't hurt to try it.

There is also an issue that I believe causes thrashing on the pagefile. You can read
about that here:

https://groups.google.com/g/video-downloadhelper-q-and-a/c/uBknY74Q1SI

I'm not sure this second thread actually applies to you. I can say that when I record
Golf Channel livestreams, I don't see a proliferation of small entries in the VDH menu.
Mind you, I don't leave the livestream page open while I'm recording the livestream.
Although I have enough bandwidth to waste by doing such a thing, it offends my sense of
aesthetics to do that. I see it as the livestream competing for bandwidth with the
recording of the livestream. So I launch the VDH recording of the livestream then close
the page I'm recording from. As long as there is at least one other browser window open,
you can do that. That may stop VDH from littering its menu with small items. In any
case, I don't believe Golf Channel causes the VDH menu to get littered with little
entries. That may be yet another annoying thing only YouTube does. Just guessing here.
I haven't found a reason to record a livestream from YouTube. That may happen one day
but it hasn't yet. So I'd like to verify what you say but I can't.

It occurs to me that I have listened to live operas on XM. The Metopera Radio channel on
XM switched to online only about a year ago. VDH sees the sort of proliferation of
little entries you describe on the XM opera page. The difference is VDH doesn't seem to
be able to record the operas. I think this is a DRM issue, which is why I haven't
reported it. But I've listened to at least one opera recently that lasted 6 hours. I
don't know how many hundreds of entries there were in the VDH menu by the end of that.
But it certainly has never caused a resource problem on my Windows 7 system.

[Alistair]

I have done a very rough test with the new beta.  The problem is still there.  I must stress that this occurs when the extension is active, it doesn't have to be doing any downloading at all, and that this occurred while I wasn't downloading anything.

After putting on a YT stream.  I used Firefox's about:memory measurement tool every few minutes, and I am just showing the one really relevant line on it that pertains to the extension.

This is shown from the extensions part (about:memory#start1) of the about memory page.  Each line has a few minutes of time between them.

68.12 MB (25.54%) ++ top(moz-extension://4751cad8-afa0-45c9-af72-6fcb587b3cc4/_generated_background_page.html, id=77)

115.87 MB (12.42%) ++ top(moz-extension://4751cad8-afa0-45c9-af72-6fcb587b3cc4/_generated_background_page.html, id=77)

568.63 MB (27.47%) ++ top(moz-extension://4751cad8-afa0-45c9-af72-6fcb587b3cc4/_generated_background_page.html, id=77)

1,174.31 MB (52.30%) -- top(moz-extension://4751cad8-afa0-45c9-af72-6fcb587b3cc4/_generated_background_page.html, id=77)

As the time between the measurements are not immensely different then each other you can notice that the memory consumption increase starts out slow, but then increases faster as time goes by.

This test was about as dodgy as can be, but I'm not sure if it is really wanted, and I can do a test in a more controlled way if anyone wants it.

Also, thanks to Michel for writing this fantastic extension.

[Wild Willy]

I found this livestream:

https://www.youtube.com/watch?v=FOd3-IVVZ9o

I suppose it's one that Michel could use if he wanted to investigate this. It appears to
be a permanent stream. I don't know if there are any country restrictions on it & I
don't know if the URL changes from day to day. If you're searching for it, it's under
the title "partypoker TV LIVE 24/7." I muted the audio & I have to say I wasn't watching
it. I just left it on, running in the background while I did other things.

I took a look at about:memory. I've never looked at that before & it made my eyes spin.
I have no idea how to filter out the noise from the relevant information. Perhaps you
could enlighten me a bit on that score.

I do know how to look at the Windows 7 Resource Monitor. It has a Memory tab. On that,
the memory usage appeared to be quite stable in the 5G-6G range . . . at first. Yes, the
VDH menu got an ever growing proliferation of entries. No, I was not actually using VDH
to download anything. According to the good old Task Manager, my CPU usage was hovering
in the 20%-25% range, enough to keep the fans in my system case blowing at a rate that
was noticeably audible.

I left this running for about 90 minutes or so while I did other things. I was using VLC
to listen to a podcast I had downloaded hours before (not with VDH) & I wasn't even in
front of my computer. Everything seemed to be running fine. But then the podcast
started skipping. Task Manager was showing CPU still in the same 20%-25% range. The
Resource Monitor did seem to indicate memory was being chewed up. I have 8G real RAM &
Resource Monitor was saying most of it was in use. My system became rather sluggish. I
discovered that the live feed had actually stopped, I have no idea when, so I closed it.
I probably should have looked in the VDH menu to see how many variants had accumulated
there but it didn't occur to me until just now. Now that I have thought of it, I find
that the VDH menu won't drop down so I can't look at the orphan count from the YouTube
feed. One side of the menu outline begins to appear but the whole menu does not open &
populate. In fact, Firefox has gone into the "not responding" state. I'm going to have
to kill it from the Task Manager. My entire system is essentially being handcuffed by
this. Oops. No. Firefox did wake up long enough for me to close the last window, which
was a page displaying this discussion thread. It took about a minute but Firefox did
eventually close normally. My CPU usage is back down to negligible. Resource Monitor
shows under 2G in use. The rest of the system has resumed normal responsiveness. I
probably should have checked the Disk tab of Resource Monitor to see whether the system
was thrashing. But things were so unresponsive, I'm not sure it would have responded.

Bottom line: I can confirm there is a problem here. But it seems to be unique to YouTube
because, as I've said, I have streamed operas from XM (will be doing one here in a few
minutes), & the problem does not occur. Mind you, that's audio-only. It is XM
<i>radio</i> after all. Still, it's a livestream & VDH does keep growing its menu on XM.
I have also watched Golf Channel livestreams (definitely video with that audio) while not
recording them in VDH & the problem does not occur. The VDH menu does not continuously
acquire new variants from Golf Channel. So this appears to be a problem just with
YouTube . . . pending other people reporting the same thing on other sites.

[Alistair]

We absolutely agree, I said 'This problem is likely to be on other video streaming sites as well' in the opening comment because I because I thought it could be a general problem, I shouldn't have said that.  The problem has only shown on YouTube for me and saying it was likely to be more general was complete speculation, I apologize for that.

I had the VDH menu not drop down as well after some time with the new beta.   I don't remember this happened before I updated to the beta.

To look at the about:memory part.  What I did was look at it when Firefox had reserved 12GB from the OS and the page file was thrashing and I was trying to find what specifically used the memory.  Noticed that the allocations under the extension part was huge.  I'll show what is under that in my browser at the moment (there is no problem with anything on this one, everything is running smoothly):

----------

extension (pid 22752)
Explicit Allocations

241.11 MB (100.0%) -- explicit
├───99.15 MB (41.12%) -- window-objects
│   ├──53.48 MB (22.18%) -- top(moz-extension://4751cad8-afa0-45c9-af72-6fcb587b3cc4/_generated_background_page.html, id=77)
│   │  ├──38.94 MB (16.15%) ++ js-zone(0x20918515000)
│   │  └──14.54 MB (06.03%) -- active/window(moz-extension://4751cad8-afa0-45c9-af72-6fcb587b3cc4/_generated_background_page.html)
│   │     ├──14.47 MB (06.00%) -- js-realm(moz-extension://4751cad8-afa0-45c9-af72-6fcb587b3cc4/_generated_background_page.html)
│   │     │  ├──13.75 MB (05.70%) -- classes
│   │     │  │  ├───6.77 MB (02.81%) -- class(Object)/objects
│   │     │  │  │   ├──4.73 MB (01.96%) -- malloc-heap
│   │     │  │  │   │  ├──4.53 MB (01.88%) ── slots
│   │     │  │  │   │  └──0.20 MB (00.08%) ── elements/normal
│   │     │  │  │   └──2.04 MB (00.85%) ── gc-heap
│   │     │  │  ├───4.39 MB (01.82%) -- class(Function)/objects
│   │     │  │  │   ├──4.38 MB (01.82%) ── gc-heap
│   │     │  │  │   └──0.01 MB (00.00%) ── malloc-heap/slots
│   │     │  │  └───2.60 MB (01.08%) ++ (5 tiny)
│   │     │  └───0.71 MB (00.30%) ++ (5 tiny)
│   │     └───0.08 MB (00.03%) ++ (3 tiny)
│   ├──19.29 MB (08.00%) ++ (34 tiny)
│   ├──18.47 MB (07.66%) -- top(moz-extension://dd3d4b2a-3b27-4bd0-80cd-7c6602197bdf/background.html, id=65)
│   │  ├──14.18 MB (05.88%) -- active/window(moz-extension://dd3d4b2a-3b27-4bd0-80cd-7c6602197bdf/background.html)
│   │  │  ├──14.10 MB (05.85%) -- js-realm(moz-extension://dd3d4b2a-3b27-4bd0-80cd-7c6602197bdf/background.html)
│   │  │  │  ├──12.92 MB (05.36%) -- classes
│   │  │  │  │  ├───5.33 MB (02.21%) -- class(ArrayBuffer)/objects
│   │  │  │  │  │   ├──5.25 MB (02.18%) ── non-heap/elements/wasm
│   │  │  │  │  │   └──0.08 MB (00.03%) ++ (2 tiny)
│   │  │  │  │  ├───5.15 MB (02.14%) ++ (11 tiny)
│   │  │  │  │  └───2.44 MB (01.01%) ++ class(Array)/objects
│   │  │  │  └───1.18 MB (00.49%) ++ (7 tiny)
│   │  │  └───0.08 MB (00.03%) ++ (3 tiny)
│   │  └───4.29 MB (01.78%) ++ js-zone(0x2091b15a000)

----------

The above is showing the memory usage of 2 extensions here (I am only showing the first part of the extensions report here, not the complete thing).  As you can see it is a tree, the total usage for one node includes everything below it on the tree.  For this case I only care about what is used by the whole extension, not anything more specific about what is allocated inside it (that is more for the extension author), so I only care about the topmost node for an extension.  For the two extensions listed here the top nodes are:

53.48 MB (22.18%) -- top(moz-extension://4751cad8-afa0-45c9-af72-6fcb587b3cc4/_generated_background_page.html, id=77)

18.47 MB (07.66%) -- top(moz-extension://dd3d4b2a-3b27-4bd0-80cd-7c6602197bdf/background.html, id=65)

One of this  lines can be used to identify the extension that is using the memory, it has the URI of the extension of it right in it:

moz-extension://4751cad8-afa0-45c9-af72-6fcb587b3cc4/

You can actually copy that and put it into the URL bar and see some more details on the extension.  I specifically look at the file name of it, the line shows at the top for me and starts with '300: jar:file:///'.  You can open the file referred to there in a program that can open zip files, and then open the manifest.json file in that and it should identify the extension.  It looks like you can also just browse the file in the extension in FireFox as well.  So if you want to open the manifest.json file in browser, just append manifest.json to the end of the moz-extension URI.  Which gives the the URI below:

moz-extension://4751cad8-afa0-45c9-af72-6fcb587b3cc4/manifest.json

This can be used to identify the extension and is slightly easier than opening the extension's archive.

So to recap how I did this, it is just go through the memory report at about:memory just looking at the top level nodes to see where most of the memory is being used.  Find the webpage or extension that is using that memory in that list.  If it is an extension, identify it by going to it's moz-extension:// URI.

This is just what I did to work out why my browser was suddenly using a lot of memory (there may be better ways), not something I do usually, but hopefully it helps.  There is a bit more info here https://firefox-source-docs.mozilla.org/performance/memory/about_colon_memory.html .  That may help as well.

[Wild Willy]

Interesting. Thanks for the clue. Now I'm no longer completely clueless.

Maybe there's some things that I can add to the mix here to make things a bit easier.
First off, we can get the ID for VDH from the VDH Settings dialog. Look in the attached
image VDH Settings. I suspect this ID is different on everybody's system. I don't know
that for certain. The ID I have on my system isn't in the fragment you've posted so
that's where my suspicion comes from.

We can identify the VDH process ID, actually the CoApp's, as well. Look in the attached
image Resource Monitor. Like I say in the image, I'm not doing any VDH downloads so the
about:memory listing doesn't show that particular process ID.

I'm also attaching an image of my about:memory display. No, not the whole thing, just
one screenload.

Next time I'm having one of these system freezes, I'll try to identify what's going on
using these tools. That is, if I can get my system to respond enough to get to them.
Firefox was doing the "not responding" trick when I was testing this yesterday so it
might be a tough ask.

In any case, we can look at these things all day & all night & it won't do anybody any
good. We're not developing VDH. Michel would be in a position to interpret these things
& actually do something about them.

[gridsleep]

Thanks for all the above information. I, too, am having the apparent memory leakage during YT stream play. I have simply taken to disabling VDH when not in use. I have seen up to 30% or more RAM being swallowed and there is 32GB RAM in my machine so it is not a trivial problem. I will try the beta and report back whether it changes anything. Also I will check all other streaming sites that I access to confirm that this is a YT related problem only. YT's highest speed streams come down in .MP4a or .OGV, though VDH has been saving them as either .webm, .WMV, or .MP4. I am wondering if that has any bearing.

[gridsleep]

The following screen cap may have a bearing on where the memory is going. It keeps racking up every segment of the stream.

[Setsu]

Can confirm this issue is still not fixed in the latest release of the Firefox version ( 7.6.6 production build).

HLS Live history is unchecked in the settings tab.

Watching a live stream causes the number of video segments to grow uncontrollably until memory is exhausted and the system begins paging.

This makes the addon unusable even when idle and not actively downloading anything.

If this bug is difficult to fix can we at least have a setting that limits the total number of video segments a tab could have?

It seems unlikely that anyone would need hundreds or even thousands of segments for a single video shown to them.

gridsleep 在 2022年7月7日 星期四晚上7:08:14 [UTC-7] 的信中寫道：

[mjs]

> HLS Live history is unchecked in the settings tab.

YouTube does not use the HLS streaming format therefore that setting has no relevance to this issue. The best thing to do is just disable the

extension for live streams. Another idea is to run the live stream in a private browser window  where the extension is not enabled.

It isn't in my browser but I understand some people still have it enabled for private windows.

I'll throw in one more thing and this only works currently in Edge and Chrome.

https://groups.google.com/g/video-downloadhelper-q-and-a/c/DeVMK_5ees8

[spiral...@gmail.com]

I have not seen this issue, but I will confirm I noticed a memory leak when watching long term streams using HLS,  The only way to stop it so far that's makes sense is to uncheck "HLS Enable" and enabling when needed.   

[spiral...@gmail.com]

One Thing i have notice recently in Firefox when you uncheck HLS enabled and you open an additional tab to play the stream in the background.  The stream actually pauses, and doesn't continue until you switch back to the video as an active tab.  But when HLS enable is checked,  this behavior is contentious playing in the background. This may be the actual key VHD even not in use causes the stream to be saved (Buffered).

[Cheeses K. Riced]

Out of nowhere, 7.7.0a4 appears to be working again, just like that. I don't know what changed. It was unable to download anything from YT last week, the usual and well documented aggregation problem. Now, It just works. Don't know what happened. Did YT rescind some of its Draconianism? Or are they gearing up for a new assault on the senseless? Or did 7.7.0a4 change without notice? I was keeping VDH for non YT work and Easy Youtube Video Downloader Express (limited to 720p because i am not going to send them any money) for YT work, but things appear back more toward normal for now. Wait and see, eh? Wait and see.

[Cheeses K. Riced]

I take it back. Yes, VDH works, but only when one is signed into a YT account. When not logged into Google in YT, VDH is grayed out and unavailable. Is that only if the aggregation helper is purchased? I don't know, but it cramps my style of Internet anonymity. I have stopped using all social media and VDH is making me stayed logged in to YT just to use VDH. I do not see why that has to be the case. VDH can verify my ID without having to be logged into YT. They should not be mutually inclusive.