Resuming interrupted downloads

[Wild Willy]

Michel, I know we closed the book on being able to resume interrupted downloads automatically in VDH.  But I just discovered this technique while I was actually looking for something else.  It seems like there's perhaps the kernel of an idea for maybe giving us resumptions at least in some cases.  Besides, this is something that might interest folks here.  VDH downloads media off the web.  My topic here deals with downloading.  I think I've convinced myself this thread is allowed here.

Here's the situation.  I've posted a number of times about how slow my Internet connection is.  I regularly download stuff from a web site behind a pay wall.  The typical download is between 2G & 4G.  I do not use VDH for my downloading primarily because VDH does not offer resumptions.  At least by using the download button provided by the web site, I get Firefox's download manager, which does permit resumptions . . . a lot of the time.  Not always, but frequently.

This pay web site has a kind of time out in place.  If, as happens all too often, one of my downloads fails, I can always get FF to successfully resume the download if it's been less than an hour since I started the download.  It's like the web site expects you to complete your download within an hour.  If a download has passed its 1 hour window, resumptions are guaranteed to fail.  Well, it's a rare item that takes less than an hour on my connection.  I can't tell you how many times I've had a download fail on me with 30 minutes, 10 minutes, 1 minute left & the resumption fails as well.  All I've known to do is to let FF delete the partial download, which might be over 3G at the moment of failure, and start the download over from the beginning.

I was feeling the pain of this particularly deeply recently so I decided to look for a FF extension that at least automatically recognizes when a download has failed & clicks the resume button for me.  As it works (or rather, works badly) now, I have to basically babysit my downloads to see when I have to click the resume button.  Surely there's something that will do the clicking for me.

I found one extension that claimed it was exactly what I was looking for.  Of course, before I got too excited, I read the reviews.  There were 32 of them & every single one said, "This does not work."  Wow.  The definition of vaporware.

So I kept looking.  I found something else that's a bit fancier.  It's more of a scheduler.  You start your download but the extension doesn't let it start until your specified start time.  The extension holds your download in paused state until your download time arrives, then it starts the download.  Later, the extension pauses the download when the clock gets to the time (specified by you) that the download window closes.  Evidently, the author of this thing has an ISP that allows free bandwidth but only between midnight & 5:00 AM so he built a tool that would fire up at midnight & suspend at 5:00.  But once again, the reviews said the thing has a bad habit of cancelling downloads when it should just pause them.

I was determined so I kept looking.  That's when I came across a description of how to deal with failed downloads.  My first reaction when I read it was to smack my forehead & say this is so simple, so obvious, why didn't I think of it?  My second reaction was that it was too shady to work.  I tried it anyway & I'm here to tell you, this works.  Let me lay it out.  I repeat, this is not an idea I had independently.  I found it searching the web.

1.  Launch your download.  I repeat, this is NOT using VDH.  This is a straight download in FF that gets an entry in the FF download manager, which is available via Ctrl+j or Tools -> Downloads.

2.  Watch your download progress in the FFdm with eager intensity or go about doing other things.  Your choice.

3.  At some point, you notice that your download has failed.  If you are absolutely certain that clicking the resume button in the FFdm will work, go ahead & do it.  If it succeeds in resuming, you'll be back at step 2.  Now, there is a possibility that the serving web site does not support resumptions.  In which case, you are totally out of luck.  Nothing I describe here is going to work on that site.  All you can do is start your download over from scratch & hope it doesn't fail again.  But from what I read, I gather most sites do support resumptions so everything I describe here will work on most sites.

4.  If you have a failed download & you click the resume button in the FFdm, there is a chance that this resumption will fail as well.  You'll know right away.  The hoped-for outcome is that the download resumes & the progress bar continues to grow from where it was before the failure.  But if instead FF suddenly tells you your download is going to be of size 110 bytes, then it's just in the process of failing a second time.  In the process of doing this second failure, FF deletes your partially downloaded file.  You are officially out of luck at this point.  You have to start over from the beginning of the file.  Eventually, the FFdm shows the second consecutive failure (it usually takes a few seconds while it thinks things over) & the only thing you can do is delete the entry from the FFdm & start over.

5.  But there is something else you should do at this point.  When FFdm first shows that your download failed, stop.  Don't touch the FFdm window.  Instead, go to the directory where you are doing your download.  Use Windows Explorer (the file manager, not the web browser) to go to that directory.  The way FF works is it puts 2 files in that directory.  One file has the name you chose when you first launched your download.  Let's say you called it Stuff.MP4.  This file is size 0 for as long as FF has not completed the download.  The other file you should find is named Stuff.MP4.part.  This part file is where FF writes the data as it downloads it.  It is your partially completed download.

6.  Copy those 2 files to another directory.  This is insurance against one of those resumptions that fails.  (I suspect you don't really have to copy Stuff.MP4 but hey, its size 0.  What does it hurt?  Do it.)

7.  Go back to FFdm.  You have a choice between 2 things you can do at this point.  One thing is click the resume button & pray it works.  If it does, no harm no foul.  Step 6 ends up being unnecessary & you can delete the copied files.  NOT THE ORIGINALS.  The copies.  You will be back to step 2.

Or you can just delete the entry from FFdm.  This erases the 2 originals whose copies you made in step 6.  Use the UpArrow & DownArrow keys on your keyboard to get to the entry in question in FFdm, then just hit the Delete key.  You should do this step from the keyboard.  Avoid the mouse.  It's much easier from the keyboard.  Also faster, more accurate, & safer.

8.  At this point, I go to the web page I'm downloading from, which I always leave open in FF while the download is in progress.  I reload the page.  This pay web site tends to expire its pages so I go the cautious route & reload the page.  I think this is a safe thing to do.  You can choose to do this or skip it.  Your experience will tell you whether this is a good thing for you to do on any given site.

9.  Start your download again.  Yes, it will start from the beginning of the file.  Patience.  Read on.

10.  Switch back to FFdm & pause the download.  You need to let this run only a couple of seconds.  The time it takes you to switch from the web page to FFdm should be long enough.  Pause the download.  You do this by once again navigating to the entry in question via the UpArrow & DownArrow keys.  Then hit Spacebar.  Spacebar pauses the download.

11.  Switch from FFdm to the directory where you stored the copied files in step 6.

12.  Copy the 2 files from there to the directory FF is downloading into.  Drag'n'drop.  Keyboard shortcuts.  Even a command line.  Your choice.  Any way that works.  Yes, copy with replace.  Both files.  This fools FF into thinking it is not 2 seconds into your download but an hour, 2 hours, 3 hours, whatever it was when the download failed.

13.  Go back to FFdm & resume the download.  Since your desired entry will still be selected, just hit Spacebar again.  Spacebar acts as a toggle to pause then resume a download.  So resume your download & clap your hands in glee that you managed to save a few hours of download time & you don't have to start over from scratch after all.

-----------------------

I believe, correct me if I'm wrong, FF cannot download streams like HLS, ADP, Dash, and the like.  This is why VDH exists.  Also, again correct me if I'm wrong, if you do use VDH to download HLS, ADP, or Dash streams, you must use the CoApp.  I do know that VDH creates a part file for some of its downloads.  I don't know if a programmatic way of doing the steps I describe above would work for an interrupted stream download.

However, if you're using VDH with the browser instead of the CoApp, I wonder if the procedure I've described here could be programmed into VDH/CoApp so we could get automatic resumptions from the point of failure.  I know, not everything can be resumed.  FF itself regularly fails to resume downloads, although this procedure might decrease that frequency.

A thought worth pondering?  At least I shared something I learned.  That's got to be worthwhile, right?

[mig]

The problem of resuming an interrupted download is very different depending on whether you are downloading a plain http document or a stream like HLS or DASH. In the first case, you just need to remember a starting point, like the size in bytes of the remain file of a previous aborted download, and request the download of this document just specifying the index where to start.

In the streams case, this is much more tricking. While the video is being downloaded, chunk by chunk, some content payload is being written to a file, but at the same time, a lot of information, mostly related to indexing and timing, is kept into the memory space and only written at the very end of the download process. If you lose that in-memory information, the data that have been written to the file are simply useless.

For theses reasons the fact Firefox is able to resume simple downloads (1 URL => 1 file) just from a file does not really help implementing a reliable recovery mechanism into VDH.

On top of this, there is the fact that very often, video URLs are time-limited, so when trying to resume a download, the initial URL may simply not be accepted by the server any more.

[Wild Willy]

Thanks for looking in here, Michel.

I kind of figured streams wouldn't work.  And I do know about URLs for videos changing.  The instructions I posted above deal with that in step 8.  I reload the page before I restart the download.  After that, it's just a matter of fooling FF into thinking it's the same download, which it actually is, even if the URL it's downloading from is different.

Oh well.  It was worth a try.  I'm just happy I now have a way of never losing hours of download time again.

[Wild Willy]

I had reason to hunt up this thread so I decided to take the opportunity to update the technique described above.  Everything still works the same as I described except there's an extra dodge you have to be aware of.  Firefox has changed its naming convention.  I say above that the part file would include the name of the target file, something like Stuff.mp4.part.  This is no longer true.  The part file has a name that is a collection of random letters & digits.  You still save the length-0 target file & the part file to a second directory.  But when you restart the download from the beginning & pause it after a couple of seconds (step 10 above), the name of the part file will be something you've never seen before.  It will be a new collection of random letters & digits.  At that point, you need to copy/paste that new part file name over to your saved part file.  You must rename your saved part file so it matches the name of the new part file.  Then you can follow the above steps from that point forward as I laid them out 2 & 1/2 years ago.