Why am I only able to download 20 minutes of a 40-minute video?

[Adam]

So I am trying to use Video DownloadHelper on Firefox to download a Gutfeld! episode...  But, for some reason, the download will stop at about 50% and tells me it's finished.  Then I check on it, and it's only 20 minutes or so in length, which is about half of the 40-ish-minutes-long episode.

I'm using Windows 10 Home.  I'm using Firefox.  I'm clicking Download (to my specific file folder) on the "HLS streaming - 1280x720 - 42:22 - ENGLISH - 5.3 Mbps - MP4" option.  The url is https://www.fox.com/watch/892f8f65d2fe727bc018e345a6da6fcd/

Is there anyone out there who can help me?

Again, I'd just like to be able to download the video in its entirety.

Thanks in advance.

[Wild Willy]

Yes, very odd.  I downloaded that one just to see.  I also got a resulting MP4 of duration 21:56.  I just skimmed the thing, I didn't sit & watch it.  But in quickly skimming through it, it seemed to end before 21:56.  Looking more carefully at the end of the file, VLC reported it still had 30 seconds to play when it stopped.  So even though it claimed to be 21:56, it actually stopped at 21:26.  Strange, to say the least.

So I went to Plan B.  You can read all about what I did by searching this forum for a thread with this title:

VDH can't download it? Maybe ffmpeg can.

You should also know about this thread:

https://groups.google.com/g/video-downloadhelper-q-and-a/c/BzPLK2YyL-s

I want you to be sure to look for certain references you can find in that thread by doing string searches within the text of that web page.  The particular strings I want you to find are:

stealth quote
short tutorial

Don't skip this.  It's important.  Otherwise, I wouldn't mention it.

Now, back to Plan B.  I opened the Network Monitor & sorted it on the Type column.  That makes the listing far easier to understand.  My first choice is to look for something of type x-mpegurl.  There was nothing like that.  My second choice is to look for something of type vnd.apple.something.  That got me what you can see in this image:

I ran that URL through ffprobe & got the results you can see in attached file ffprobe.txt.  You'll see that each Program includes a Stream identified as Timed_ID3 data.  This mysterious content has been a problem for VDH for a while.  You can search this forum with "timed_id3" as your search key & see all those other occurrences.  In any case, you should not be surprised that I then tried to download this content with ffmpeg.  See the attached file Gutfeldmp4 Log,txt.  You'll see that I got the dreaded 403 Forbidden error.  I did try this twice, just to make sure I hadn't hit one of those cases where the manifest expired faster than I could use it.  There's just no way around this.  It seems like the web site has some stealth DRM in place, meaning they don't support the commands that would be needed to do this download with ffmpeg.

I would say you are officially out of luck.  I don't know how VDH managed to download as much of this content as it did when ffmpeg can't even access it.  You're going to have to find some screen recording software if you really want your own copy of this content.  Downloading seems out of the question.

For the record, and to follow my own advice of what constitutes a proper problem report, I am running Windows 7 64-bit, Firefox 113.0.1 64-bit, licensed VDH 7.6.5a3 beta (same as 7.6.6), CoApp 1.6.3.

[Wild Willy]

I was thinking about this.  I know.  A dangerous activity.  Our fellow member mjs mentioned something the other day & it was like an airplane.  It took a while to land.  But it did finally land.  And the first passenger off the plane was . . .

VLC

That's your solution.  I think.  I'm in the process of verifying that, but it looks promising so far.  Anyway, here's what I did in VLC.

Open the VLC preferences dialog with Ctrl+p.  Then select the radio button at the bottom for ALL.  There is a setting somewhere in the multitude of VLC settings that tells the preferences dialog to automatically open on ALL.  I leave it as an exercise for you to find it.

Then I went to the VLC wiki.  You can find the link to that by pressing F1 in VLC.  I looked for instructions on how to record something.  I did find those instructions.  That's good, because sometimes their documentation can leave a little something to be desired.  But this time, the advice is to navigate to the Input/Codecs section of preferences.  That gives you a long page of settings on the right side of the dialog.  Scroll it all the way to the bottom.  There you will see one setting labelled Record directory.  There is a blank entry field on the right side of the dialog & a Browse button.  Click the Browse button.  This opens a standard operating system file selection dialog.  Navigate your disk storage to the directory where you want VLC to write recordings.  Select it, then click the Select Folder button in the dialog (or just hit Enter).  Then close the VLC preferences dialog by clicking Save in the bottom right corner of the dialog.

You have to click Save.

Did I mention that you should click the Save button?

Sorry.  Maybe other people are swifter than I am.  I am always forgetting to do this step, not just in VLC but in other software as well.

Click Save.

So once you click Save (you have no excuses now if you forget), go visit the video page on fox.com again.  There is some kind of "preview pass" on a timer on that page.  It's good for an hour.  Since I was there earlier today, I figured I better clear out all the cookies for that site before I bother visiting it again.  I assumed my pass had expired & it wouldn't let me see the content again.  I have a Firefox add-on called Cookie Manager.  Such an inventive name.  It lets you delete cookies without the added overhead of also deleting cache.  The tool for clearing cookies built into Firefox does not simply delete cookies.  It also deletes cache associated with those cookies.  But any sort of large-scale activity like deleting all the cookies for a site will involve a potentially time-consuming step of deleting some untold number of Gigs of files from your browser disk cache.  So I use the add-on instead of the built-in tool.

Anyway, after removing the old cookies, I went to the site & did what I described earlier to get the master manifest.  A word about those.  You can't use any of the URLs I posted earlier, nor will you be able to use the ones I'll be posting below.  You'll note these are ugly URLs with long strings of gibberish in them.  The gibberish associates the content of the page with your individual web session.  These things are not only unique to a visit by each user, but they time out.  The URLs that worked for me earlier probably fail for me now.  (I haven't bothered to check but it wouldn't surprise me.)  They almost certainly won't work for you.  You have to perform the STEPS I describe here.  You can't just copy my URLs.  You have to go through the process I describe to generate the files & URLs that will work for you.

Once again, I generated a report with ffprobe.  I won't attach a new copy of that.  It's pretty much the same as what I attached above.  Just the URLs are different.  The report lists 10 Programs: Program 0 - Program 9.  I settled upon Program 2 as the one I wanted to get.  That's the third of the 10 programs.  The way ffprobe works is it lists what it is reading from the master manifest as it is constructing its report.  Usually, you would just skip to the bottom of the report & use what's there.  But this is one time when you have to look at that preliminary stuff.

There are simply dozens upon dozens of lines that ffrpobe reports it is skipping.  Fortunately, all of the stuff in the preliminaries is in the same order as the Programs listed at the bottom of the report.  Since I wanted Program 2 (the third one), I looked for the text string "opening" & stopped on the third one.  I didn't read every line, of course.  I did a string search in my text editor to find it.  Here's an extract of that region of the ffprobe report on my system:

That is the URL of the stream manifest for Program 2.  I copied that URL to my system clipboard.

Then I went over to VLC & hit Ctrl+n.  This opens the Open Network Stream dialog.  I pasted the URL into the text box & hit Enter.

Quick like a bunny & clicked the Record button in VLC before the stream actually launched & started playing in fullscreen.  I then switched tasks to the Windows file manager & found the directory where I had told VLC to record things.  I did not do this quick like a bunny.  It took an agonizing several seconds during which I wondered whether this was working.  But eventually, oops, here was a file with an outlandish name that was obviously being written by VLC.

At this point, as I was looking at the ffprobe report to explain everything here, I noticed I had actually made a mistake.  Look at this bit taken from near the bottom of the ffprobe report:

There are 2 variants of resolution 1280x720, which appears to be the highest resolution on offer here.  But I didn't notice Program 1 when I first looked at this.  Program 1 has a higher value for variant_bitrate than Program 2.  That means Program 1 is of a higher quality than Program 2.  I would have chosen Program 1 to record if I had noticed that.  Oh well.  It's not like I'm going to actually watch this.  What I've done is good enough for the purposes of this illustration.

Now, this is decidedly a sub-optimal solution to getting a file on your system containing this video.  The optimal solution would be for either VDH or ffmpeg to get this.  But we've already discovered that VDH gets tripped up by something.  My speculation is that the Timed_ID3 data is the culprit.  And ffmpeg is blocked by the dreaded 403 Forbidden error I reported earlier.  So the next choice in the hierarchy of choices is VLC.

VLC is sub-optimal because you have to record the content.  Record, not download.  That means you can't take advantage of all that lovely bandwidth you are paying your ISP for.  You have to let the item play for, in this case, the 40-something minutes of its duration.  At least you can just mute the sound & go about doing whatever else you'd like to do while the thing is recording.

Oh look.  VLC isn't recording anything.  I've been babbling on here so long, it's finished.  Let's see what I have . . .

Oh stink.  Look at the Date created time & the Date modified time.  It looks like this was recording for 42 minutes.  But the duration of the thing says it's only about 14 minutes long.  I verified that in VLC.  There is only about 14 minutes of the show here.  Why didn't it record the whole time?  I wasn't looking at the VLC window while this was going on, plus I had muted the audio.  I was here writing this.  Did it play for the whole 42 minutes?  Why would it play for 42 minutes but quit recording after 14?  There's nothing in the VLC message log.  I can't explain this.  It looked promising.  Maybe this approach would work for other content on this web site, or on another web site.  Maybe it would actually work for you.  You should try it & report your results here.

Well, it looks like you need to go for the very least optimal solution, a screen recorder.  This is a step below VLC.  Like VLC, you have to let the item play in the screen recorder.  But unlike VLC, you basically can't do anything else on your computer while you're making the recording.  Your desktop activity will get recorded by the screen recorder, as will anything else you do that generates sound.  At least, that is my experience with OBS on Windows 7.  They say the issues I have with it have been solved in versions of OBS for more modern versions of Windows.  I'm not going to Windows 10 or 11 just to make OBS work.  Plus, there may be better tools out there than OBS.  I haven't bothered to look because it's not a priority for me.  I use OBS maybe once every 6 months.  It isn't worth it to me to go look.  Maybe it is for you.

Well, there you have it.  Use whatever parts of this you can.  Good luck.

[mjs]

Was that recording or downloading ? Because I did what was shown in a video on YouTube by opening media > covert&save>network.

If downloading then looks like it was half a success and half a failure.

[Wild Willy]

Last night I did Ctrl+n, which is Open Network Stream. I pasted in the URL of the show &
hit Enter. There was a short hesitation in VLC while it digested the command, during
which I clicked the Record button.

Tonight, I did something different. I still had to clear out my cookies before the site
would let me access the video. It gave me a master manifest, same as last night.
However, the Programs in there were a lot different. Tonight, there were 15 programs,
0-14. Also there were 6 Programs with 1280x720 resolution video Streams, but they
appeared to be only 3 distinct such variants. There were 2 occurrences of each of 3
distinct bit rates. It turned out once again that I picked Program 2. This time, I made
sure I was getting the one with the highest bit rate. Just for yucks, I tried to get it
once again with ffmpeg. 403 Forbidden. It's so weird that they let ffprobe go through
but they block ffmpeg. In any case, I did the same trick as last night to fish the URL
for Program 2 out of the preliminary listing part of the ffprobe report. I even ran
another ffprobe, this time on this 1280x720 variant, just to make sure it was what I
wanted.

This time, I did what you said. Media -> Convert./Save -> Network -> specify a target
file. Let her rip. I went on about other things. Before I did, I made sure VLC was
writing something into the output file. It's size was increasing. But here it is almost
an hour later. The web page is telling me that my preview pass has expired. But VLC is
still doing something. The file size is not increasing but Resource Monitor is telling
me VLC is using between 120,000 & 150,000 bytes per second of bandwidth, not very heavy
usage. I don't know what it's doing but it doesn't look like much. I tried opening the
file I have, which is 493K, not very big, in a second instance of VLC. It wouldn't play.

I think I'll just leave the alleged Convert/Save operation run for a few more hours.
Maybe it will eventually complete. Maybe it will eventually time out. I think if I
interrupt it, I will have this 493K unplayable file, which is pretty useless. So I'll
let it run & report later what happened.

[Wild Willy]

Such a waste of time. After I let it run for another long time, I decided it was just
doing nothing. So I unplugged my network connection from electricity. That's a trick
I've used to get VDH to correctly terminate a download in progress. Didn't do anything.
So I stopped the Convert/Save. I clicked the Stop button in VLC. That got it to
finalize the file. It played just fine . . . all 3 seconds of, "We are experiencing
technical difficulties. We'll be back soon."

This content is doomed. I am giving up now.

[Adam]

Sorry this caused so many headaches for you both.  I just want to say "thanks," especially to Wild Willy, for going the extra mile to try to reach a solution.

Guess it wasn't meant to be...

[Wild Willy]

I love experimenting. The takeaway here should be that just because this particular
content was a problem, doesn't mean all content is this much trouble. The various things
I tried, the ffmpeg & the 2 methods of recording with VLC, can & will work elsewhere.
They might even work on other content on this site. You can't know until you try. Just
keep these techniques in your toolkit & try them out next time VDH doesn't just work.

[Wild Willy]

This is just nagging at the back of my mind. I went back to that episode of the show.
It took several reloads of the page before it stopped telling me oops we're having
technical difficulties. Eventually, I was able to launch playback in the web page. That
got me the information I described above. I had to get a new master manifest because
this site, like nearly every other site on the web, generates new objects for every web
visit. I will get new URLs every time I visit this page. And the URLs I get will not
work for anybody else. The gibberish strings in the URLs are meant to tie my specific
visit to that web page with the content there. When you visit the exact same page to
view the exact same content, your URLs will all be different & they will not work for
anybody else. Also, after a time, those URLs will stop working, even for you.

Due to Google's limit of 8M on a single post, I have attached the first 10 images to this
post. I will attach the rest to a second post immediately following this one.

So, I did the same thing as last night, Convert/Save in VLC. But tonight, I sat here &
watched it while it ran. Fortunately, this site gives pretty good service, something on
the order of 6 million bytes per second, so it didn't take but a couple of minutes to
record the whole show. The first 15 . . . Wait. I should reword that. There is no
predicting what order Google will put my image attachments in. So. The attached images
numbered #01 - #15 (look at them in numerical order, no matter how Google might scramble
them) are screenshots I took over the course of the recording of this content. The time
indexes displayed in the VLC window are as I explain in captions in the first image.
There was a very strange pattern. The recording would proceed apparently normally for a
time. Then it would revert to time 0:00, as if it were starting over. After a few
seconds, the recording would appear to skip forward to where it left off before reverting
to 0:00. Then a bit later, it would jump back to 0:00 again. After a few seconds, once
again it would skip forward to the next section of the program. It did this several
times. It did it once at the very end of the show for a short duration but I didn't
catch that one in an image.

After the recording was done & I had only the first 14:41 of the show, I went back to the
web page & hovered my mouse over each of the little boxes on the timeline in the video
player. This is what you can see in attached images #16 - #20. You can see the time
indexes (more or less, the mouse is a terribly inaccurate device) of those markers. It
seems like whenever the recording reached one of those markers, VLC dropped back to the
beginning & started over.

I speculate that each of the segments of this show has its own URL. But I don't know how
to find the URLs for those segments. The master manifest gives me the URL for only the
start of the show. For reasons I can't totally explain, VLC repeatedly records the first
segment of the show even when it claims to be recording the second through sixth
segments. This web site is feeding the segments as separate URLs, one after the other,
when it is played in the player on the web page. It must be using some weird protocol
that VLC is unaware of. I don't know how to discover those second & subsequent URLs. It
may be that they are hidden in things in the Network Monitor that appear as you go along.
As you can see in my images, I only started playback in the web page, just enough to get
the master manifest to show up in the Firefox Network monitor. I stopped playback after
6 seconds. Maybe if you actually play a bit of each segment of the show, its master
manifest will show up in the Network Monitor & that segment can be recorded (probably
repeatedly) by VLC. I am not sufficiently motivated to go through that exercise. Plus,
this is all speculation anyway. I'm trying to fit my observations into a theory that may
or may not be accurate.

It is very annoying that I can ffprobe the master manifest, & it shows the content as
being of duration 43:15, the full show. But the site rejects ffmpeg. If they would
implement the support on their site that would allow ffmpeg to work, I'm quite confident
it would get a proper recording of the full 40+ minutes of the show. But they probably
don't want anybody to record it. Wankers. I don't know why they don't just put it
behind DRM protection. Then we all could have gone away without running through this
rigamarole only to end up in the same failing place.

[Wild Willy]

Part deux.

[Wild Willy]

I have done a little more investigating here. The little boxes in the timeline are the
places where advertising is inserted into the online stream. After the ad plays, there
is no new manifest that appears in the Network Monitor. So I was wrong when I guessed
that. It seems that however they put the ads into the stream is derailing VLC.

There are objects of type octet-stream showing up in the Network Monitor. These appear
to be segments of the video with an extension of .ts. Unfortunately, there is nothing in
those URLs like &range=. We have encountered many other cases of chunks of video whose
URLs have this &range= qualifier in them. By removing the &range=, we have been able to
discover the full video. But this case resists that approach. In the other cases, the
items were not octet-stream but mp4 or webm. So this site is doing something that seems
specifically intended to thwart us.

I really need to give up on this for real.

[mjs]

I know this has been looked at great depth , but two things I've been wondering.

1. Does ffmpeg use a user agent ? And perhaps it needs the same one as Firefox.

2. Can ffmpeg use cookies ? Does the master manifest use cookies on fox ?

[Wild Willy]

It is possible to use a -user_agent parameter on your invocation of ffmpeg. At some
point after I investigated this case, I started using that on all my ffmpeg downloads.
My decision had nothing to do with this case. I'm not sure it's made any difference but
it doesn't seem to harm anything either.

There is some mention of cookies in the ffmpeg documentation. I invite you to read it &
see if you are more successful than I am at figuring out how to actually use a cookie on
an fffmpeg invocation. But it's not simply a matter of providing a parameter on an
invocation of ffmpeg. That's the easy part. The hard part is constructing the cookie.
I have no idea how to do that, nor how to actually decide which of the several dozen
cookies from this web site that were in my browser might be helpful on an invocation of
ffmpeg. If you manage to figure it out, do post about it.

I don't think I've ever seen a reference to cookies in a manifest. I've seen references
to key information in manifests, but never cookies.

[mjs]

I'd check it out on fox if their master manifests use cookies but it's for those residing in the US & I don't have a vpn.

You might be able to verify that, whenever you give a left click on an item in the network another part of the network opens on the right.

There is a cookies tab, all you'd have to do check if the manifest has cookies. Not inside the manifest but rather does it use cookies in that tab.

[mjs]

Well, well, well. It works with the latest beta and coapp 2.0.1 , I tried bob's burgers on the fox site and it got the whole thing. 

Be sure to use quick side download or side download.

[John Kennedy]

Tell me to print where is this side menu?

Em sáb., 7 de out. de 2023 06:04, mjs <matt...@gmail.com> escreveu:

Well, well, well. It works with the latest beta and coapp 2.0.1 , I tried bob's burgers on the fox site and it got the whole thing. 

Be sure to use quick side download or side download.

--
You received this message because you are subscribed to the Google Groups "Video DownloadHelper Q&A" group.
To unsubscribe from this group and stop receiving emails from it, send an email to video-downloadhelper...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/video-downloadhelper-q-and-a/ab1ada32-343b-49b3-bba5-51cab5f6a588n%40googlegroups.com.

[mjs]

It's in the secondary menu and for now it only works for HLS video.

Mig shows more on it https://groups.google.com/g/video-downloadhelper-q-and-a/c/JZ3HSPh8bL8

[John Kennedy]

My video now has such low audio. It doesn't show the video

John Kennedy Suporte, Perfil Sistemas (31)3488-6538 | (31) 9.8856-1070 | joh...@gmail.com www.perfilsistemas.com.br

Em sáb., 7 de out. de 2023 às 20:50, mjs <matt...@gmail.com> escreveu:

It's in the secondary menu and for now it only works for HLS video.

Mig shows more on it https://groups.google.com/g/video-downloadhelper-q-and-a/c/JZ3HSPh8bL8

--

You received this message because you are subscribed to the Google Groups "Video DownloadHelper Q&A" group.
To unsubscribe from this group and stop receiving emails from it, send an email to video-downloadhelper...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/video-downloadhelper-q-and-a/CAEKJYCE0Kq8q%3DZ1JQjOFEq1dvtDFwg_B_K5sXxMAmbQMjqQeqw%40mail.gmail.com.