Multiple concurrent large downloads + relatively restricted bandwidth = system seizure when one download completes

[Wild Willy]

I apologize for the vague nature of this problem report but I can't be any more specific.  Here's my environment:

Windows 7 64-bit, VDH 7.3.6a1, CoApp 1.2.4, license verified, Firefox 62.0 64-bit, VLC 3.0.4 Vetinari

This has happened to me a few times today & the pattern is always the same.  I think part of the way to recreate this problem requires a relatively slow Internet connection.  Mine is about 530KB, what Windows Resource Monitor reports as 550,000 bytes per second.

Here's how the pattern has gone.  I'll launch several simultaneous downloads, say 6, using the CoApp.  Sorry, these are from a pay site so I can't share URLs.  Each one will be an MP4 file in the 1G-2G range.  VDH will report, after all 6 downloads are safely in progress, that times to complete will be in the range of 2-8 hours, sometimes longer.  Things will run fine for a couple of hours until one of the downloads completes.  During that time, I will periodically drop down the VDH menu to check how things are going & each download will have progressed some.  I will also go to the directory using Windows Explorer (File Manager) into which I'm doing these downloads & refresh the display.  The file sizes will show increases on all the files.

But when a download completes, my entire system comes to its knees.  When I'm downloading only a single file, I usually notice my whole system becomes unresponsive for 3 or 4 seconds, then the VDH notification pops up & things go back to normal.  But in this case, the VDH notification pops up & things do not go back to normal.  Task Manager gives me the "task is not reesponding" message.  You know things are bad when even Task Manager refuses to take focus.  I have the Task Manager window positioned so that I can see all the Firefox threads as well as the CoApp thread.  The CoApp, which until then will have been showing a memory usage in the 1G range, suddenly is showing 3G & grows.  I've seen it go over 4G.  I have 8G of real RAM.  The Task Manager, when it shows anything at all, says my memory usage is over 90%.  But with Task Manager flipping in & out of unresponsive state, that display comes & goes.

Meanwhile, Firefox windows become extremely sluggish.  The disk activity light on the computer case is on solid, not flickering.  Firefox windows respond eventually, but in times measured in minutes.  It's all I can do to get logged off the pay site & close the browser windows.  Things aren't helped at all by the fact that Task Manager, even though it's in the "not responding" state & refuses to take focus, still sits on top of all open windows obscuring whatever is behind it.  I can't get any window to display on top of Task Manager.  Eventually I get all Firefox windows to close & the system sits there with the disk activity light still on solid, not flickering.  After waiting a couple of minutes, Firfox generates a crash report.  Fortunately, the button that says "Quit Firefox" is visible peeking around from behind the Task Manager window so I can click it to get Firefox to send its crash report to Mozilla & close.  When the crash report finally gets sent, all the Firefox threads disappear from the Task Manager display, taking the CoApp with them, Task Manager finally starts responding again, & I get my system back to running normally.

Until everything happens again.  I do want to download those files, which I am managing to do one at a time.  For the time being, it seems like I am stuck downloading only 1 file at a time.  I know I can set VDH to download only 1 file at a time & let its queueing take care of things.  But this is not supposed to be necessary.  I have had lots of downloads running concurrently in the past & never had this problem.

I am speculating that there is a memory leak in the CoApp.  I think the fact that the entire system becomes unresponsive while the CoApp is finalizing a download, even when it's the only download I'm doing, is also a clue to . . . something.

I wish I could be more specific.  If there is something I can do to take some diagnostics for you, let me know.  They would have to be things that get launched early because when this starts happening, there isn't much chance I'll be able to launch a diagnostic since my system is essentially unusable until I can clear the condition.

It also occurs to me that the CoApp is holding things in RAM until it's time to write them to disk.  Unfortunately, when you've got a bunch of large downloads all running at the same time, this can make the CoApp's memory demand grow prohibitively large.  I suggest this design needs some rethinking.  Perhaps you can hold less transient stuff in RAM & write more of it to disk every so often instead.  I know, I'm guessing & maybe that's not how things actually work.  But I'm trying to think of reasons for the CoApp to need such a large memory footprint.  I think that footprint is the root cause of the problem.  At the least, I'm hoping to maybe trigger some ideas, ones I wouldn't know enough to come up with myself.

[Wild Willy]

Actually, I should revise one thing I just said.  CoApp memory usage does not remain stable.  It gradually grows over the course of all my downloads.  By the time the first download is ready to complete, perhaps 2 or more hours into this process, the CoApp memory footprint is up to 4G.  But overall system memory usage has also gradually grown & is over 90% on my system.  I think this pushes the system into thrashing & that's why everything gets so unresponsive.  So I suggest again that a rethink of the CoApp design might be in order here.

[mig]

Maybe i missed this information from your post but what type of download are you performing, plain, HLS, DASH ? This is very important because the coapp does very different things depending on the situation.

For instance if it is a plain download, VDH relies on an external library linked into the coapp. If the problem is here, that library cannot be fixed by us and should be replaced with another one.

If it is HLS or DASH, the coapp is only used to save blocks of data to the disk.

[Wild Willy]

Plain download.  Sorry, forgot to mention that.  What's the name of this external library?  Where is it supposed to be located?  I guess once I know what it is, I'll be able to figure out where I can get a new one from & what to do to install it.

[mig]

It might not be easy, depending on your javascript development skills, but here it is:

- the download library is available from https://github.com/kevva/download

- the coapp code using this library is at https://github.com/mi-g/vdhcoapp

It's all open source stuff.

[Wild Willy]

OK Michel, that's just a bit too cryptic.  You're the boss of an application.  If your application uses an internal tool that I'm supposed to separately install, you have to help me a bit.  First off, I've looked at that kevva download page & I am entirely mystified.  What am I supposed to download?

I did click that button & got a small ZIP file.  And I do mean small.  It's only 6K.  There is a readme file in there but it is way too cryptic for me.  I'm on Windows 7 & the install instruction (there is exactly one) looks very UNIX-y to me.  Which is to say, it's gibberish to me.

VDH is your product, Michel.  Tell me what I'm supposed to do.  What files from this ZIP file do I need to place in which directories on my system?  Will I be replacing any files?  Are there some commands I'm supposed to execute -- remember now, ON WINDOWS -- to tell VDH to use this?  I assume whatever I do, Firefox should not be running at the time.

Come on, man.  Help me.  This isn't supposed to be a doctorate-level oral defense or a criminal cross-examination.  Help me.

[mig]

Well, you asked for details: " What's the name of this external library?  Where is it supposed to be located?  I guess once I know what it is, I'll be able to figure out where I can get a new one from & what to do to install it." so i gave the details :) but i also wrote"it might not be easy, depending on your javascript development skills"

I'm afraid i cannot explain how the whole thing works, but at some point, i will experiment replacing that download library with one that better handles download errors, bandwidth control, multi-connection downloads. Hopefully, this will also fix any possible leaks.

[Wild Willy]

Oh I get it.  You're trying to conscript slave labor to fix problems.  Nice try.  Actually, if I spoke Javascript, I might have actually tried to work on this.  But . . .

When I asked where is it located, I meant where on my system is the library located.  I thought you were saying it was a DLL or something like that & all I needed to do was get the new version & put it in the right place.  But asking me to get inside this & try to understand it & make changes is just begging for trouble.  All we need is some newbie like me to come in & add a few hundred new bugs to a program that was just invented a few months ago & has been through all this testing by us users.  I would be very upset.  I think we'd have more than a few upset users on our hands as well.  No, I trust you to come up with something far better than I ever could.  My role is to describe how it doesn't work right & hope you can connect that to something inside that needs changing.  I'm happy to be a user.  You would not be happy if I were a coder.

[anuhea55 00]

WildW....wow, this is a Read thread....for me, I do many 6 simultaneous downloads using the CoApp to stack (did a 20 stack yesterday).  My files are strictly one URL, mp4 videos that can be viewed during the download in my chosen folder.  I did see the TaskManager Memory get to 70% (I've got 8gig, but use Win10).  My files were in the order of 500-600mb, and about 20-40min downloads.

Today, with only FireFox open with this site, my taskmanager is at 42% memory with Firefox(5 entries) using a total of 915mb.  Under the Performance tab of TaskM, I show 3.4gb in use and 4.5gb available...SO,  with no downloads, FF and the rest of Win does not leave a whole lot of memory open for use by any VDH downloading.  Chrome may use less space, but the Win memory stuff is pretty fixed.

As to downloading large 1-2gb files for 2-8hrs....man, these are large and I could not wait that long.  My largest download was 3gb mp4, but these was not done using VDH, just a website straight download.  I was surprised that this went in less than 30min.  I was doing 3 of these together, but they were .MKV format.  I will see if I can trigger a VDH read.

As to a memory leak in CoApp....well I have had the computer freeze but cannot attribute it to VDH.  I've had window freezes when background system scans are made, like virus, malwarbytes, etc run.  During my extensive downloads, I get as high as 70-80%mem.  WildW, maybe you need to reduce to 4 active downloads  (I used to use 4 but now have it at 6).  With such huge files being done at one time and for so long, maybe that's the best you can get...at most, any tweaks here will just get a little more memory...I'll keep following.

[Wild Willy]

I've given up trying to fight with it.  Given my limited bandwidth, there's really no advantage to trying to push 5, 10, 20 simultaneous downloads through my pipe.  The total time it takes to download them simultaneously adds up to about the same time it takes to download them one at a time.  So I've changed my VDH setting to Max concurrent downloads =1 & Max concurrent stream downloads = 1.  I can still queue up a bunch of downloads & they'll kick off one after the other.

But even the queuing gives me a problem, but it's not a VDH problem.  See, this pay site puts expiration times on their download URLs.  I believe this is a common practice with pay sites.  As long as you've got an active connection to a URL, it will stay valid no matter how long it takes you to download it.  But if the URL goes unused for some period, it times out & becomes invalid.  With my downloads taking so long, I might be able to queue up 2, maybe 3 downloads behind the active one.  But if I queue up more, the ones in positions 3, 4, 5 etc. all time out while the first 1 or 2 are downloading.  If I had more bandwidth, I might be able to download 10, 20, 30 files at a time in a reasonable amount of time.  I could also do 1 download at a time & queue up 10, 20, 30 downloads & they would all go through within the timeout period.  In fact, if I wanted that many downloads today, even with my limited bandwidth, I could tell VDH to download them all simultaneously.  None of them would time out.  But then I'd have to deal with this issue of VDH suddenly causing my system to freeze.  So I'm doing single-threaded downloads.  My system still freezes up when VDH is finalizing a download, but it's for only a few seconds & things quickly get back to normal.

While I have been messing with all this, I have observed the occasional request to download that doesn't take.  I tell VDH to download something & on the source page for the video VDH reports that the download is queued.  But the blue square does not increment the number by 1, and when I click the blue dot in another browser window or tab, it does not list the latest download as being queued.  When the download in progress ends, the queued download does not start.  Solution: execute the "Reload extension" function.  That's in the VDH settings under the "More..." button.  This seems to clear whatever condition is bothering VDH & I can go back to download the one it said was queued but actually was not.  Unsurprisingly, this also drops the CoApp's memory footprint to a more reasonable level.

[anuhea55 00]

WildW...Your 8gb PC should handle normal files but since you had 1-2gb mp4, that took 4-6hrs+....I can see the potential slowdown.  My single URL downloads of which I do 6 and stack the rest, are about 300-600mb and go about 20-45min.  My largest file download (using a straight download button on a site) was 5.5gb.  I have also done 2-3gb in about 1hr.

The underlying server that houses the files could also restrict, and the number of connections on the net.  Also, Your PC's hard drive could be maxing out, especially if you are at some capacity.  Multiple downloads will store the file in fragmented order...maybe a defrag of the disk...or if you are storing directly to a USB connected device.  We've discussed about the MB/sec speeds that we've used for tracking, and yours seemed fairly good.  Hardware could also be a problem.  Mine is an older HP that went from Win7 to Win10 too.

The time out on a site, have never experienced....but is curious.  I do have a site that will stop the video from playing in its window after maybe 20-30sec, however VDH has already captured the URL and I just download, and start playing the partial *.mp4 after a few minutes.

Finally, the "Reload extension"...is that the same function as Disable/Enable of the Add-on under the 3bars.  Can you do this while VDH is actually doing downloads in the Blue Button?

[Wild Willy]

> WildW...Your 8gb PC should handle normal files but since you had 1-2gb mp4, that took 4-6hrs+....I
> can see the potential slowdown.  My single URL downloads of which I do 6 and stack the rest, are
> about 300-600mb and go about 20-45min.  My largest file download (using a straight download button
> on a site) was 5.5gb.  I have also done 2-3gb in about 1hr.

Maybe I'm not being clear.  While the many multiple downloads are in progress, my system runs fine.  I have an Intel Core i7 Quad CPU running at 4GHz with 8G of RAM.  I have 4 HDDs of varying sizes (500G containing my system boot partition, 150G that I kept from an older system, 6T that I have only recently installed & which is still over 90% empty, 6T also recently installed on which I have nothing yet), all of which have acres of unused space.  The size of any individual download has no impact on my system performance.  The only impact from a large download is the length of time it takes to download.  The performance problem I observe is transient & happens only when VDH finalizes a download.  It's transient & short lived when I'm single threading my downloads.  It's transient & long lived when I'm running multiple simultaneous downloads.  When I am trying to do multi-threaded downloads, the transient problem causes the remaining in-flight downloads to all silently terminate, as I can observe in Resource Monitor.  VDH claims the downloads are still in progress but the CoApp threads normally displayed in Resource Monitor all disappear & it reports my usage of bandwidth has dropped to 0.  I didn't think to try "Reload extension" but that might have cleared the problem.  The handful of times I hit the problem, I just patiently logged off the site & closed all my browser windows, which took several minutes but did eventually complete, ending with a Firefox crash report after I closed the last browser window.  It appears Firefox crashed while trying to terminate.  It didn't crash until after I successfully closed all the browser windows normally.  At least I didn't have to reboot when this happened.

> The underlying server that houses the files could also restrict, and the number of connections on
> the net.  Also, Your PC's hard drive could be maxing out, especially if you are at some capacity.

My observation of this server farm is that they apply no throttling.  Windows Resource Monitor consistently shows that whether I have 1 or 20 downloads running simultaneously, my full bandwidth is being used.  As for my system capacity, it is not being taxed . . . except when a download ends.

> Multiple downloads will store the file in fragmented order...maybe a defrag of the disk...or if you
> are storing directly to a USB connected device.  We've discussed about the MB/sec speeds that we've
> used for tracking, and yours seemed fairly good.  Hardware could also be a problem.  Mine is an
> older HP that went from Win7 to Win10 too.

I'm probably going to stick with W7 until MSft stops maintaining it.  I'm not using a USB HDD.  My 4 drives are all SATA internal drives.  Defrag is an interesting idea but I don't think it should be necessary since all my partitions are NTFS.  In any case, I don't think I'm hitting any hardware limits.

> The time out on a site, have never experienced....but is curious.  I do have a site that will stop
> the video from playing in its window after maybe 20-30sec, however VDH has already captured the URL
> and I just download, and start playing the partial *.mp4 after a few minutes.

Your thing where the site plays only 30 seconds sounds like another issue entirely.  URL timeout is, I believe, a common thing on pay sites.  They'll let you stream or download a file for however long it takes.  A video might be an hour or two long.  With my bandwidth, a download can take an hour or two or even longer.  In both those cases, the site happily lets me stay connected to the URL in question for however long it takes.  But if I load up a page with a video on it & I just leave the page sitting idle, at some point the page times out.  Usually a simple page reload reconnects me & then I can download off the page.  Sometimes the site will make me log on again, supplying ID & password.  But after that, I can do whatever I want.  The problem I have is that since my downloads take relatively long amounts of time, things I may have queued up in VDH may time out in the interim.  So I can't use the VDH queuing the way I'd like.  I am clear that this is not a VDH problem or even something VDH should or could be concerned with.  The site simply has a rule that I have to launch either a stream or a download within a certain amount of time or else the page times out.  That's just how it works.

> Finally, the "Reload extension"...is that the same function as Disable/Enable of the Add-on under
> the 3bars.  Can you do this while VDH is actually doing downloads in the Blue Button?

No no.  I mentioned it in my previous post.  Go into VDH Settings, click the "More..." button, and it's one of the selections in the list that drops down.  I would suspect that doing a "Reload extension" while you have any downloads in progress will abort the downloads immediately.  I haven't verified that but it just makes sense to me.  It would make sense to me that an attempt to "Reload extension" while VDH has downloads in progress should pop up a prompt asking if you really want to abort all your in-flight downloads.  Again, I haven't tried it so I don't know.