How to download from YouTube

[Wild Willy]

It has been pretty sad around here lately, what with VDH being unable to download YouTube content since about the end of May, beginning of June.

https://groups.google.com/g/video-downloadhelper-q-and-a/c/orAHFdaLbUE

And now Michel may be softening us up for a defeat.

https://groups.google.com/g/video-downloadhelper-q-and-a/c/wvQp_W2G8pA

Well, I believe any defeat should not be total.  I'm hoping this thread encourages Michel to at least fix VDH to the extent that it continues to download the YouTube content that it still can.  If it encounters the content that is now inscrutable, at least generate an error message up front so we don't waste our time even attempting the download.

What you see below is the result a bunch of hours of improvising, trial & error, guessing.  It is an application of the approach you can read about by starting here:

https://groups.google.com/g/video-downloadhelper-q-and-a/c/BzPLK2YyL-s

First, you need to carefully read the first 2 paragraphs there.  Then look for "evasive maneuvers" within the text of that web page.  That will give you a link to a discussion we've been having about trying to deal with web sites that VDH doesn't process correctly.  It gives either bad results or no results on those sites.  So this thread is, in a sense, a continuation of that discussion with the problematic web site being YouTube.

Before we get started, I should tell you about my environment, as EVERY problem report here should.

Windows 7 64-bit, Firefox 114.0.1 64-bit, licensed VDH 7.6.5a3 beta (same as 7.6.6), CoApp 1.6.3

Actually, my VDH & CoApp are irrelevant here since I won't be using them.

I encourage somebody who is on another platform, Mac, Linux, Chrome, Edge, whatever, to add a tutorial here about how to do things differently on your platform.  I'm sure a lot of what I'm doing is the same regardless of the operating system & browser.  But I have to believe there are large variances of certain steps for other platforms.  Please add screenshots & narrative, both text AND pictures, showing how to do those things on the other platforms.  Text is NEVER enough.  Include screenshots.

Also, before we get started you need to go read this thread:

https://groups.google.com/g/video-downloadhelper-q-and-a/c/O_bRwxrfqS8

In there, you will find a post from Michel in which he gives a link to a Bugzilla report that he opened.  You need to go read that.  There is a circumvention in there for the problem he describes.  I have found that that circumvention affects what you get from YouTube.  So you have to implement the circumvention described in the Bugzilla report before you try any of what I'm posting below.

Why don't I just tell you here what that circumvention is?  Because I neither encourage nor condone laziness.  I'm doing some work here.  You do a little work, too.  There's only so much spoonfeeding I'm willing to do.

[Wild Willy]

Let's take this as our first example.

https://www.youtube.com/ @ HollyRandallUnfiltered

I've had to surround the @ with spaces to prevent Google from interpreting it as an E-mail address.  But you can see below that the URL doesn't have any spaces in it.

It is important to have the Network Monitor open before you navigate to the page containing the content you want to download.  There are 3 ways to open the Network Monitor.

Mouse click around the menus Tools -> Browser Tools -> Web Developer Tools.
Use the keyboard shortcut Ctrl+Shift+i, which appears on the menu in the first choice.
Use the keyboard shortcut F12.

Any one of those ways will give you the Network Monitor as shown in that image.  If what you see is not exactly the same as that image, you need to click the 2 buttons indicated: Network, All.  I also strongly recommend you click on the indicated column heading Type.  If you leave the Network Monitor to its own devices, it presents data in the order in which the serving web site sends it to your browser.  And YouTube is particularly active in sending stuff to your browser even when you just have a page open sitting idle.  That is pretty much equivalent to random order.  To bring some organization to the chaos, click the Type column header to sort the information by object type.  This groups together all the bits of web pages (html), cascading style sheets (css), images of various types (png, jpeg, webp, etc.), javascripts (js), and so on.  The one we will be focusing on is the json group.

This page is continually evolving.  So its content may not look exactly like that image when you go visit it.  That's why I'm giving this URL.

https://www.youtube.com/watch?v=ECe4fjDtWf4

That is the URL of the link we will click.

For normal web sites, meaning pretty much every web site except YouTube, you will be looking in the Network Monitor for a manifest.  HLS manifests normally have names that include a file extension of .m3u8, & they are of type x-mpegurl.  DASH manifests normally have names that include a file extension of .mpd, & they are of type dash.  We've got none of that here.  We are looking for a json.  Why?  Experience has shown us that failing easier alternatives, information for downloading is within an object called a json.

For YouTube, we are looking for a json whose name starts with player?key=, the one highlighted in that image.  There are several other json objects both above & below that one.  You have to hunt for it.

How did I know that?  Like I said, hours of guessing, trial & error, improvising.  Lots of, "Hmm.... I wonder what's in here?"  So what is in there?  Here's how you find out.

Pop up the context menu on that object, cascade out the Copy Value submenu, & select the Copy Response action.  This places the contents of that object in your system clipboard.  Then go to your favorite text editor, like Notepad.  Not word processor.  Not Wordpad.  Not Acrobat Reader.  Text editor.  My favorite text editor is Notepad++.  It would be too far off-topic for me to explain Notepad++, or Notepad++ plugins.  I will offer this reference & after that, you're on your own.

https://notepad-plus-plus.org/

There are quite a few Notepad replacements out there.  That tells you there's much about Notepad that could be improved.  Pick your favorite.

So here's what that player json looks like:

Dog's breakfast.  Fortunately, Notepad++ offers a plugin to help bring some structure to this junk so that a human can make heads or tails of this.

So much better.  It's astonishing to see that a file that is just a single line of almost 100,000 characters can actually be hiding some useful information that not just a computer but also a human can understand.

Now we want to look for the character string mimeType.  How did I know that?  You should be ready for this answer.  Hours of looking at these things & trying to figure out what patterns are in there to help me download this content.  I have attached that json for you to look at yourself.  I've named it raw json.txt  I highly recommend you scroll through it a bit before you start doing the searches I'm showing here.  Of course, I've had to change the file name to .txt because of Google restrictions.  When you download it, just rename it back to raw.json.

This seems to be a descriptor for an mp4 of resolution 640x360, pretty poor.  I'm not guessing at that.  Finally, something that doesn't involve guessing.  You can read that right there in that image.  Look for it.  This mp4 appears to be a unified video+audio file.  Again, look for the clues.  They are there.  That seems like a horribly low quality version of this podcast.  So you keep looking.

What you will find is alternating videos in mp4 & webm format.  The resolution is the same in each member of each pair.  There are several resolutions being offered here.  Of course, I want the highest resolution I can find.  So after searching the entire file to find all the videos, I settled on this one.

Comparing this one to the earlier one, we can see that this one doesn't show any audio information.  So this is a video track without audio.  This is something we've been told for years: that YouTube changed their practice a few years ago & started presenting their video & audio in separate files.  Here is evidence of that.

What is that horrifically ugly URL right above all the neatly formatted information?  Let's find out.

When I hover the mouse over any part of that URL, Notepad++ conveniently highlights the whole thing.  Better still, if you double click anywhere inside a URL, Notepad++ opens the URL in a new browser window.  I assume it uses system settings to determine that it should open your system default browser.  I do not have & have never used Chrome, Edge, Opera, Safari, Waterfox, or any other browser.  I know.  I'm so closed-minded.  Deal with it.  Anyway, here's what I get when I double click that URL.

This page can appear like just a black hole.  Hover your mouse over the page, anywhere over the page.  It will make the player controls appear, as in that image.  But don't just hover your mouse there.  Pop up the context menu.  Click Mouse Button 2.  Then execute Save Video As...

That's such a lousy default file name.  Change it to something meaningful.

Then launch the download.

You'll want to open that in its own tab so you can magnify it & actually read it.  I can't help that everything in that one is so small.  It's a double wide image of my 2 monitors.

This is YouTube's usual throttling in action.  However, our experience with downloads with VDH has been that it would take approximately half the duration of the video.  This one is a little over an hour long.  So this one taking a mere 9 minutes is still about THREE TIMES FASTER than what VDH was giving us.  I'm not really criticizing VDH for this.  VDH must not be using the same API call as Firefox.  Maybe this will encourage Michel to figure out how to get this stuff as fast as Firefox does.  For now, just be happy that this is so "fast."  It's actually pitifully slow.  The capacity of my Internet connection, what I am paying my ISP for, is 62.5 MILLION bytes per second.  Good web sites give anywhere from 10 to 15 million bytes per second.  Half decent web sites give maybe 5 million bytes per second.  Admittedly, this is way better than what we've become accustomed to with VDH.  I'm getting about 1% usage of my bandwidth.  I'm not impressed.  Glad it's faster, but not impressed.

I have seen cases in which you can cancel the download & start it over from scratch, and that will give you much faster downloads.  Three cancel/retries on this one didn't speed things up any.  This was the best I got.

So here's the result.

Yay.  Now we've got a silent video of this podcast.  What good is a silent podcast?

[Wild Willy]

Fear not.  Go back to the json & keep searching.  You will find this audio mp4.

I chose to demonstrate this with mp4 because it seems like most people prefer mp4 over webm.  Since we got a video mp4, we ought to get a matching audio mp4.  I don't know why there's only 1 audio mp4 but 3 audio webms.  But that's what's there.  It seems to be a pattern.  Other videos I've looked at show the same pattern: 1 audio mp4, 3 audio webms.

Once again, just double click that URL.  I won't show all the steps of the download again.  It's just the same as for the video.  But I will show this one.

This time, of course, you want to execute Save Audio As...

And here's the results.

Audio tracks always contain much less information that video tracks.  Even though the Windows properties claim this took about 1 minute to download, it was actually only a few seconds.

And now this can be played synchronously in VLC.  You don't need to put the video & audio together into a single file.  VLC is quite happy to play these as separate files.

You can see the video was fine.  You'll have to trust me that the audio was just as fine.

I've posted the player json I got from that web page.  It's full of URLs.  They worked great, just like I'm showing in the images.  But they won't work for you.  That's because the URLs contain information that tied my web session with that video on YouTube.  In addition, the URLs expire.  In fact, all YouTube pages expire.  If you open any YouTube page & let it sit idle for, I don't know, a couple of hours, maybe less, maybe more, then come back & click the play button in the player, it won't play.  I can't remember now for certain but I think it throws up some kind of error message.  You have to visit the web page yourself.  That gets you content specific to you, including a player json with URLs in it tailored for your web session.  So you can't use the json I posted.  But you can just follow my steps & get your own player json.  It will work as I've shown here.

One other thing I want to point out.  Go back upthread & look at the images of the YouTube page.  Look in the upper right corner.  I was not logged on while I was doing all of this.  I leave it as an exercise for you to try this while you are logged on.  That would be logged on with a generic free Google ID, & logged on with a paid ID, on either Google or YouTube.  I would like to hear feedback from somebody who tries this while you are logged on.  Tell us your experience.

Now here is one difference I have discovered between mp4 & webm.  Playing synchronous mp4 files does play fine in VLC but skipping forward & backward is a bit glitchy.  The left & right arrows are supposed to skip a small interval.  I believe the default is 5 seconds.  That's what I have it set to & I think that's the default but I honestly can't remember.  I have found that I have to hit the arrow a couple of times to make it actually work.  I could fix that by merging the two tracks into a single file.

But things are much worse with webm.  My experience has been that hitting one of the arrow keys causes the video to restart from the beginning.  If I do it a few times, the audio continues to play but the video freezes.  There is no way out of that except to use the Next function in VLC to skip to the next video, which is the same video, assuming you have only 1 video in the VLC playlist.  That starts playback over from scratch.  With synchronous webm playback, you have to just sit & watch the thing, no skipping allowed.  So I would strongly recommend you merge the 2 tracks together if you have retrieved the content as webm files.  Unified webms with both video & audio in the same file don't have any problems skipping around.

I am attaching a small Windows command script, a .bat file, to this post.  The script is actually named Merge Video with Audio.Bat.  I've changed the file name to .txt because I don't want anybody clicking the attachment & launching execution of the script.  Download the file & just rename it to .bat.

That script calls another one named M:\ffmpeg\Setffmpeg.Bat.  This is a script consisting of a single line:

Set ffmpeg=M:\ffmpeg\ffmpeg-2023-03-05-git-912ac82a3c-full_build\bin

This shows that I have installed ffmpeg on my M partition in directory ffmpeg.  When I downloaded the ffmpeg package from ffmpeg.org & unzipped it, it created the subdirectory with the ugly name.  Within that subdirectory, there is another subdirectory named bin.  That is the location of ffmepg & ffprobe.

You can install ffmpeg wherever you want on your system.  You would have to change that one-line script named Setffmpeg.Bat to reflect wherever you unzip ffmpeg. You can save the script anywhere on your system.  Wherever you save it, change the one line in Merge Video with Audio.Bat to reflect wherever you put Setffmpeg.Bat.  The attached script Merge Video with Audio.Bat can also reside anywhere on your system.  I do not have the 2 scripts in the same directory on my system.  Merge Video with Audio.Bat will prompt you for the input tracks, the destination directory, & the destination file name.  Those can actually be 3 different directories on your system, although you will normally do everything in one directory.  But this script does not have to be in the same directory with any files you might download from YouTube.

These scripts use very elementary Windows concepts.  Don't ask me to explain them.  Go read up on Windows command scripts if you don't already understand what the scripts do & how they do it.  Ditto for ffmpeg.  I have posted a tutorial on ffmpeg elsewhere in this forum.  Go find that if you want to understand ffmpeg.

I am not finished with this tutorial.  But I need to take a break.  I will post more when I'm ready.

[Wild Willy]

I meant to mention the last step of Merge Video with Audio.Bat.  It opens the log file in Notepad++.  That's another line you would have to change depending on where you install Notepad++.  If you install Notepad++.  If you use another Notepad replacement, recode that line so it works on your system.  You can even recode it to use Notepad if that's all you have.  Or you can just remove the line & manually open the log file.  It's up to you.

[Wild Willy]

Sometimes, a video doesn't have a player json file.  I don't know why.  Also, if you reload a page, the player json disappears from the Network Monitor.  I have not discovered a way to make it reappear, short of closing all browser windows, starting the browser again, then surfing to the same page again.  I have done a reload on our sample page from upthread here to simulate a situation in which a player json is not shown in the Network Monitor.

If you're really eagle-eyed, you'll notice I was logged on for this sequence.  That's my free generic Google account that gives me a GMail address, lets me post in this group, & all the other things a free ID gives me.  Which isn't anything noteworthy.  But I was logged on & it had no effect on anything I was doing.  Everything acted the same as when I wasn't logged on.  Now we need to hear from people who have paid Google IDs, & from people who have paid YouTube IDs.  I believe those are different things.  Do correct me if I'm wrong.

So.  Back to our story.  In the absence of a player json, the fallback trick is to look at an object whose name starts with log_event?alt=json.  As with the player json earlier, do a Copy Response to get it onto your system.  The file I got from this one was this:

-------------------------------

{
  "responseContext": {}
}

-------------------------------

That's totally useless.  But don't give up.  Execute a different function.

Copy All As HAR also puts something into your system clipboard that you can paste into your text editor.

What's a HAR?  HAR stands for HTML Archive.  But being a smartypants & knowing what the acronym stands for doesn't mean I know what it is.  I've attached this one to the next post.  It turns out I went over the Google limit for size of post with that one attached here.  So it's on the next post.  That shows you what one looks like.  But what is it for?  I don't really know.  All I know is what I'm about to say.

How did I know to do this?  Right.  I guessed.  I tried some of the other entries on that menu & Copy All As HAR gave me something I could work with.

Unlike the player json earlier, this one appears to be nicely formatted.  Let's see what we get by searching once again for mimeType.

Hmmm......  This seems to be information that looks a lot like what we got in the player json earlier.  Going by the line number, it appears that inside the text/html thing, we have something that looks a whole lot like it might be our player json.   This last search hit looks like that 640x360 object we saw before.  Could this be the player json we're looking for?  I tried running the Notepad++ plugin to format this as a json.  It just threw up an error & did nothing.  No help there.  If I knew more, I might have been able to extract just the part of this file that might be player json.  But I don't know more so we're stuck with this.  Maybe somebody reading this can instruct us.  But if this is player json, let's test that theory.  Keep looking for mimeType.

A couple of search hits further down, & still within that same line, we've got something promising.  So, hovering the mouse in the suspected right area of text . . .

Well, what have we got to lose?  Let's double click this bad boy.

Ewww.....  And no amount of mouse hovering made anything appear in the window.  On a hunch, let's look at the URL in the URL bar.  Position it all the way at the right end of it.

My hunch is based on the fact that I've seen this before.  For some reason, Notepad++ picked up the " at the end of the URL & included it in what it gave to the browser.  OK.  Let's just remove the trailing ", which we can see is not actually part of the URL.

Made no difference.  Still got a blank web page.

At this point, I started really inspecting this URL closely.  Here's a paste of that monstrosity from the HAR:

https://rr5---sn-vgqsrnl6.googlevideo.com/videoplayback?expire=1687276259\\u0026ei=g3aRZJGNAtGklu8P14KfwA0\\u0026ip=2603%3A6011%3Ac306%3Aa341%3Aa428%3A503e%3Aac95%3A3af9\\u0026id=o-AHuZ_21CSNbWhtK0I3YZGJB6YKpZFqJezB01XWXR4neG\\u0026itag=137\\u0026aitags=133%2C134%2C135%2C136%2C137%2C160%2C242%2C243%2C244%2C247%2C248%2C278\\u0026source=youtube\\u0026requiressl=yes\\u0026mh=wc\\u0026mm=31%2C29\\u0026mn=sn-vgqsrnl6%2Csn-vgqsknlz\\u0026ms=au%2Crdu\\u0026mv=m\\u0026mvi=5\\u0026pl=35\\u0026initcwndbps=1705000\\u0026siu=1\\u0026spc=qEK7B52gqPuKrXwPKJq4AFIkwlVeu853uJvmq6xnDy77aU8ahc87EYQ\\u0026vprv=1\\u0026svpuc=1\\u0026mime=video%2Fmp4\\u0026ns=23LsNII3Ok4pFLTjVjNSCxQN\\u0026gir=yes\\u0026clen=385352673\\u0026dur=3943.874\\u0026lmt=1687152563732115\\u0026mt=1687254325\\u0026fvip=3\\u0026keepalive=yes\\u0026fexp=24007246%2C24362686%2C51000024\\u0026c=WEB\\u0026txp=5432434\\u0026n=SNG9tozxoPa2SCt\\u0026sparams=expire%2Cei%2Cip%2Cid%2Caitags%2Csource%2Crequiressl%2Csiu%2Cspc%2Cvprv%2Csvpuc%2Cmime%2Cns%2Cgir%2Cclen%2Cdur%2Clmt\\u0026sig=AOq0QJ8wRQIgPonbgvXX1E-cQ6XcGEUO3-FEon_ap3DYKIpzG89ZqsgCIQC6R-EKlhEW5DLBeXAd6pE8oEZF31wQNweZGfwYrMsUIw%3D%3D\\u0026lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps\\u0026lsig=AG3C_xAwRQIgFnt8LxNfIoeDOJFtQ8i9Vcae87wDdPsdg90je_johuECIQDmaSisRboQkBT26Us1JbZG9G4FXisM9DfE8rgrXfFqPA%3D%3D

I noticed something really strange in there.  You can see it if you look really carefully.

\\u0026

\ is not a valid character in URLs.  / is but \ is not.  What's going on there?  I have seen in these files that \ seems to be some kind of escape character.  In the case of this URL, it is not terminated by ".  It is terminated by \".  By the same token, the start of the URL is also not ".  It is \".  So why is there a \ in the middle of this URL?  In fact, there's a whole bunch of them.  What is it escaping?

At this point, I just Googled u0026.  I didn't have much hope that I would find anything.  I shouldn't be so pessimistic.  You do the same web search.  What do you find?  That should make you go back up a couple of images, to where you can see this:

"mimeType": "text/html; charset=utf-8"

When I was still in diapers, 2+2 always equalled 4.  So let's try this Replace All:

Wow!  You're kidding, right?  36 occurrences?  Holy cow.  Well, let's double click on this:

Gee.  That looks awfully familiar.  I think we all know what to do with this.

And yes, the audio mp4 that mates with this is further down in the HAR.

[Wild Willy]

I should have noticed that by itself, it is over 12M.  But zipping it into a single zip file still didn't please Google.  So I've split the file more or less in the middle & put it into 2 zip files.  Even though I've split the file so each half has about the same number of lines, they aren't the same size.  One zip file is about twice the size of the other.  But each one is under the 8M limit so this works.  The first half is attached to this post.

[Wild Willy]

And here's the rest of it.  Just unzip the 2 files, then paste them into a single file in your text editor.

[Wild Willy]

So that's it problem solved.  No more to worry about, right?

Sadly, no.  Let's look at this one:

https://www.youtube.com/watch?v=sD3eveWBEUU

There's a player json.  Looks promising.  But start looking for mimeType again.

This looks a lot different from the player json we looked at upthread.  In that one, the URL appeared first & the metadata about it appeared after.  Here, the metadata appears first & the URL appears after.  Not only that, but it's not a simple URL.  There's this signatureCipher thing.

There's a little prefix, then &url=, then what looks like a URL that is quite similar to the ones we've been dealing with.  But when you double click on that URL, you get another one of those empty white browser windows.  This time, there's no spurious " at the end of the URL.  There's no \\u0026.  Nothing I can see that is wrong with this URL.  Except it doesn't work.  Its structure looks a lot like the URLs that worked upthread here.  But for reasons I cannot fathom, it just doesn't work.

The occurrence of the word Cipher here makes me think this has something to do with either a password or encryption.  But the clip is ordinary.  I've downloaded it using one of those echo download sites.  The download was pretty fast, too.  There's nothing remarkable about this video in any way.  What is going on with this one?

I even tried Save All As HAR.  Still, nothing but URLs that don't work.

Maybe this is what Michel is talking about when he says VDH may never again work on YouTube.  I don't accept that.  There is content on YouTube that is amenable to downloading using the approach I've laid out upthread here.  Surely he can program that into VDH.  But there is also this kind of content on YouTube, that doesn't just give you a URL that works.  Maybe Michel can figure out how to deal with these.  At the very least, we should get a VDH that can deal with at least some YouTube content.  Some is better than none.  Still, we've got users coming on here right & left saying they can download YouTube content with this or that application, with this or that echo download web site.  If they can do it, VDH should be able to do it as well.  I have a hard time, a very hard time, accepting that VDH will simply not support YouTube any more.

[mjs]

Those player json urls can be opened in a new tab and Firefox presents it nicely as an alternative to copying onto the computer.

On the Yuja Wang encore video, if other sites or software can get the video then they must be altering something in the url.

You can still get it , using the hit details open the audio and video links in a tab. A prompt will come up, give it a name you want.

The important part then is to give it a .mp4 extension after the file name.

 

The files will play but there is a problem if you want the files merged.There is no information in the file properties and using ffprobe on the files :

[mov,mp4,m4a,3gp,3g2,mj2 @ 000001e6a036ac40] Format mov,mp4,m4a,3gp,3g2,mj2 detected only with low score of 1, misdetection possible!
[mov,mp4,m4a,3gp,3g2,mj2 @ 000001e6a036ac40] moov atom not found
video.mp4: Invalid data found when processing input

[Wild Willy]

I tried what you said, mjs, on the Yuja video. It downloaded files that did not play for
me. Not separately, not synchronously. The absence of a moov atom gave me the idea to
rename the video to .h264. But it wouldn't play & as you said, it wouldn't merge. I'm
surprised you got any results from there. I've been able to download that one only by
using 9convert.com. I suppose that proves there's nothing wrong with the video. Neither
VDH nor my attempts to get it from the player json have all failed. I've tried that page
perhaps a dozen times over several days during the past week or so & I have never gotten
that one directly, only through the echo download site.

Interesting idea, Jay. I'm not entirely sure I follow what you're proposing. Perhaps a
few mockups of screenshots would help Michel understand what you're proposing. He's
really the one who would need to understand it. We're just users.

[mjs]

Actually the file will only play for about 30 seconds then some error occurs, so what I did doesn't solve anything. I didn't think about this earlier when I said the files can't be merged , since if ffmpeg can't then neither can VDH.  Even if VDH somehow could the file would be missing the moov atom thing, I don't know what it is but must be important. And Mig has said before that VDH will only detect what it sees directly in the network.

I'd be surprised if this is somehow fixed but this looks like YouTube has delivered the knock out punch to VDH.

[Wild Willy]

Oh, I'm not that pessimistic. There is still content on there that can be downloaded
using the player json technique. It's just that it appears there are 2 formats to the
player json. Typical, they can't just pick a standard & adhere to it. If the URL is in
front of the metadata in the json, the URL will work. You can open it in a browser tab &
then Download Video/Audio As... But if the metadata in the json is in front of the URL,
the URL is complete trash. I think the signatureCipher thing is some sort of decryption
key, but I have no idea how to use it. It seems the other tools do. But I don't. I can
only hope Michel can figure it out.

My understanding is that the moov atom contains information like the resolution, frame
rate, bit rates, & other stuff needed to play it back. It's strictly a video thing. I
believe it can be located at either the front or the back of the file. I would think it
belongs at the front but that's just me trying to apply logic, a dangerous pursuit at all
times. Seems to me I saw mention at some point of tools that will relocate the moov atom
from one end of an mp4 to the other, but I can't remember which end is the preferred end.

I'm not sure what a moov atom would contain for an audio mp4. In addition, there is no
such thing as a moov atom for mkv & webm. It's something only mp4s have.

It's interesting that you got even a partial file that would play. What I got wouldn't
even start to play.

The ffprobe stuff I saw on the Yuja encore video made me think there was some sort of
weirdness going on in both the video & the audio. But that was in the files downloaded
onto my system by VDH, not the source streams. I think at some point a week ago I did
try to ffprobe the URLs in the Hit Details & they just said they encountered invalid data
in the input. If YouTube can stump ffprobe, they're doing something really weird. But
then why can all those other tools & web sites get the videos no problem? What are they
doing?

[Wild Willy]

I've been forgetting to comment on something else you said. I know that you can just
double click on a json in the Network Monitor & Firefox will give you a pretty, formatted
display of the json. But the URLs in that display aren't links. You can't click them.
You still have to copy/paste them to another browser tab. By importing the json into
Notepad++, you can get the pretty format & the URLs are links. You can double click on a
URL in Notepad++ & that launches Firefox on that web page. More convenient. Plus, if
there's any manipulation of the URL you need to perform, you've already got it in your
text editor.

[Wild Willy]

When you get to the browser window that contains just the player of either the video
track or the audio track, there are 3 options.

You could try to download the track with VDH. It might recognize the track, it might
not. I imagine it's like everything else on YouTube: try it & see whether it works. You
might need to let the track play in the browser page for a couple of seconds. That might
prod VDH to recognize it if it hasn't already. But even if VDH sees the track, you will
most likely be held back by YouTube's usual throttling. We have observed, when VDH used
to work on You Tube, that VDH downloaded everything in 50% of the duration of the clip.
A 5-minute clip would download in 2 & 1/2 minutes. A 5-hour clip would download in 2 &
1/2 hours. I have seen YouTube give such putrid download speeds as 30,000 bytes per
second, dial-up modem speeds. I have also seen it give over 6 million bytes per second.
So it's not that they don't have the capacity to give us decent download speeds. They
just refuse to. I imagine this is an inducement to get people to pay for their service.
I know there's fellow user's here who have paid. I have never been able to cost justify
such an outlay.

You could try to download the track with ffmpeg. This is probably always going to work.
But my experience has been that ffmpeg downloads from YouTube run at one of those putrid
speeds. I had the rather upsetting experience of leaving an ffmpeg download running for
NINE HOURS. After that much time, the URL finally timed out & the download terminated
prematurely. At that point, I had about 70% of the file, which did play fine, but like I
say, it wasn't the whole clip. And the full clip's duration was only about 20 minutes.
It couldn't download a 20-minute clip in 9 hours. How disgusting is that? I don't know
how YouTube can detect that it's ffmpeg. I've even started using the -user_agent switch
on my ffmpeg invocations to make it look like it's just Firefox making the request.
Somehow, YouTube figures it out & really sticks it to me. So this would be an option of
last resort. Or maybe not even that feasible. I avoid this one.

The last option is the one I show in the tutorial upthread: using Firefox as the
downloader. YouTube seems to be willing -- SOMETIMES -- to actually give decent download
speeds, faster than what VDH has traditionally been able to achieve. There have been
times when I launch the download, then immediately Ctrl+j over to the Firefox download
manager & the download has already completed. And I'm not talking about a 30-second
audio track. I'm talking about a video track of about 800M. Sadly, there's no
predicting when you'll get this lucky. You might try the same download again & get the
usual 50% duration speed. I have found that it is possible -- SOMETIMES -- to cancel the
download, which erases the partially downloaded file, & start the download over, which
obviously at that point is from the beginning. On the retry, I have seen the speed
improve by a couple of orders of magnitude. I have also cancelled & retried 3 or 4 times
& I've gotten the same speed each time. So it's a crap shoot. But I do always try it
just to see if I might be lucky this time.

So on balance, I would lean towards using Firefox itself as the downloader all the time
with this technique.

[Wild Willy]

So I now have the new VDH 7.7.01a beta. Here are some early results.

VDH now actually detects all the MP4 variants that it never used to. And they actually
work, which they never used to. I'm curious about the MKV variants, though. In my
research upthread here, I saw lots of WEBMs, no MKVs. For the 2 tests I ran below, VDH
was showing me MKVs. What I saw in the jsons was WEBMs. Do enlighten me on this
subject, Michel.

I am no longer getting the 2 spurious short content errors. Don't tell me you didn't do
anything about those, Michel. You must have done something.

1920x1080 content now downloads as 1920x1080, not 1912x1088. I have been plagued by
1912x1088 resolution results for at least a year, perhaps longer. This appears to be
fixed. Once again, Michel, don't tell me you don't know what you might have done to
affect this.

VDH downloaded the Lilly Bell interview just fine as an MP4. Based on the fact that I
was able to download this one using my player json technique, I'm not surprised by this
success. The down side is it still took half the duration of the clip to download. I've
got lots of discussion above about how Firefox seems able to download the content with a
much higher download speed. What are you doing different from what Firefox does? You
need to look into doing whatever it is that Firefox does.

The Yuja Wang encore video downloaded. I was not expecting that to work because of the
signatureCipher thing. What did you do to make it work? That one is too short for me to
properly measure whether the download speed was throttled to force it to download in half
its duration. The temporary intermediate files that VDH creates are in a work directory
(and deleted). So the result file in my target directory shows a creation date/time & a
last modified date/time that are equal. I can't tell from those properties how long a
download takes.

Here's one that I haven't mentioned before. It was one of the test cases I used to
develop my player json technique.

https://www.youtube.com/watch?v=ZkH6jz1gy3Y

This one has a 4K variant that I was able to download with the player json technique.
But VDH recognizes nothing on this page even after I changed resolutions a couple of
times & let it play for a couple of seconds at both 3840x2160 & 1920x1080. I thought VDH
would recognize the variants that are 1920x1080 & lower resolutions & just fail to
recognize the higher ones. Or maybe recognize the higher ones & just fail when you try
to download them. So the state of affairs is that if there is a 4K variant available
along with several variants of regular HD & lower, VDH recognizes nothing. This is not
entirely clear on the beta download page.

On balance, this looks like a winner. Darn fine work, Michel! I'm looking forward to
the rest of the fixes. And, of course, your answers to my questions above.

[Jaymo]

Anystream added YT recently, if you have that your good!

[Wild Willy]

I am having trouble posting my comments.  They keep getting hidden by ...  I am going to post each paragraph one at a time & see how that goes.  Stay tuned.

[Wild Willy]

What is it with this?  It is not a signature . . .  So I now have the new VDH 7.7.01a beta.  Here are some early results.

[Wild Willy]

VDH now actually detects all the MP4 variants that it never used to.

And they actually work, which they never used to.

I'm curious about the MKV variants, though.

In my research upthread here, I saw lots of WEBMs, no MKVs.

For the 2 tests I ran below, VDH was showing me MKVs.

What I saw in the jsons was WEBMs.

Do enlighten me on this subject, Michel.

.

[Wild Willy]

I am no longer getting the 2 spurious short content errors.

Don't tell me you didn't do anything about those, Michel.

You must have done something.

1920x1080 content now downloads as 1920x1080, not 1912x1088.

I have been plagued by 1912x1088 resolution results for at least a year, perhaps longer.

This appears to be fixed.

Once again, Michel, don't tell me you don't know what you might have done to affect this.

.

[Wild Willy]

VDH downloaded the Lilly Bell interview just fine as an MP4.

Based on the fact that I was able to download this one using my player json technique, I'm not surprised by this success.

The down side is it still took half the duration of the clip to download.

I've got lots of discussion above about how Firefox seems able to download the content with a much higher download speed.

What are you doing different from what Firefox does?

You need to look into doing whatever it is that Firefox does.

.

[Wild Willy]

The Yuja Wang encore video downloaded.

I was not expecting that to work because of the signatureCipher thing.

What did you do to make it work?

That one is too short for me to properly measure whether the download speed was throttled to force it to download in half its duration.

The temporary intermediate files that VDH creates are in a work directory (and deleted).

So the result file in my target directory shows a creation date/time & a last modified date/time that are equal.

I can't tell from those properties how long a download takes.

.

[Wild Willy]

Here's one that I haven't mentioned before.

It was one of the test cases I used to develop my player json technique.

https://www.youtube.com/watch?v=ZkH6jz1gy3Y

This one has a 4K variant that I was able to download with the player json technique.

But VDH recognizes nothing on this page even after I changed resolutions a couple of times & let it play for a couple of seconds at both 3840x2160 & 1920x1080.

I thought VDH would recognize the variants that are 1920x1080 & lower resolutions & just fail to recognize the higher ones.

Or maybe recognize the higher ones & just fail when you try to download them.

So the state of affairs is that if there is a 4K variant available along with several variants of regular HD & lower, VDH recognizes nothing.

This is not entirely clear on the beta download page.

.

[Wild Willy]

On balance, this looks like a winner.

Darn fine work, Michel!

I'm looking forward to the rest of the fixes.

And, of course, your answers to my questions above.

.

[Wild Willy]

Sorry folks.  That seems to be what I have to do tonight to get stuff to post.  To post & not be hidden by ...  I've never seen anything like this.  Let's hope it's temporary.

[Wild Willy]

And let's hope this doesn't get deleted.  It's not spam.  It's not NSFW.  Why would it be deleted?  I don't get it.

[Wild Willy]

And now my original post has reappeared.  I'm not deleting my multi-part repeat of it.  Some of it might get deleted again.  It's a safety net.  Just ignore the duplicate.

[mig]

Willy, can you check your spambox for a mail from me ?

[Wild Willy]

Nothing. I'm sending this as a post to the forum & an E-mail directly to you. That way,
you'll have my E-mail address to use in a reply.

[mig]

i resent the mail

[Wild Willy]

Michel, I have received nothing from you.  I don't have things secretly fall into my Spam folder.  I have defined a filter that moves anything that lands in my Spam folder over to my Inbox.  I don't trust Google to correctly recognize spam.  I want to verify for myself whether something is spam.  So your messages, even if they did go to my Spam folder, I wouldn't know it because they immediately bounce into my Inbox.

I sent you a couple of E-mails using the address you use to post here.  It's an address I've used before to communicate with you & it has worked in the past.  I did that so you would have an E-mail with my address on it that you could just use directly in a Reply.  But I haven't heard from you.  Come to think of it, maybe you haven't heard from me either.  Something weird is going on.

Let me try this.  I'm going to do a Reply to author on your last post.  Let's see if that goes through.  Worst comes to worst, you could do the same to this post.

[mig]

I got your mail and replied to author from here

[Wild Willy]

I got it. So weird that it was so hard to make it work.

[mjs]

Just a bit of testing on various videos the links are clickable after opening the player json in a new tab, the ones that are not have the signature

cypher & these seem protected anyway. On anything music related it has that cypher, not sure what other content has it.

[Wild Willy]

I have learned some things in the past couple of days that I believe will be useful for others.  When I say learned, I mean it was all discovery on my part.  I didn't read any documentation on this.  I gather there is some.  But I decided to do my usual thing.  What happens if I click this?  What's inside here?  What does this thing do?  Guessing.  Improvising.  Trial & error.  You really should try it some time.  It's the best way to learn.  Don't be afraid.  What's the worst that could happen?  Something just doesn't do anything?  Is that so terrible?  This is really what everybody should be doing.  Then you would answer your own questions more often than not.

In this thread over here:

https://groups.google.com/g/video-downloadhelper-q-and-a/c/ppuouJJcK4M

A fellow user was having a problem getting VDH to download something.  We weren't communicating until he posted some screenshots of what he was doing.  It turned out he was trying to record a YouTube livestream.  VDH is known to have problems with YouTube livestreams.  You can find a handful of discussions in this forum on the subject.  They are not recent at all.  The problems were that VDH didn't record the livestreams properly, and worse, your system would go into a coma.  If you were super patient, you could wait it out, after reloading the extension (VDH) then disabling it.  Your system would come out of its coma after maybe an hour.

But that's a bit of a detour.  The main point here is that the livestream mentioned in that other thread is a perfect vehicle for showing some new ideas here.

This story begins with a preliminary step.  Go to the YouTube home page.  This has to be the first time you've visited this page since you opened your first browser window.  If you've been to this page in this browser session before, close all your browser windows & start over.  This is important, & I will eventually get to why.  Click in the search field at the top.  Move the mouse cursor to the bottom of the screen, off the browser window.  This is important.  The mouse cursor must not be positioned over any of the YouTube content.  Type in "le media" as the search key.  Press Enter.  When I did it, I got this.  YouTube being what it is, every page changes all the time.  So when you do this search, what you find will likely be similar to what I'm showing here but not identical.  The differences will be unimportant.

Now scroll the page down a short way.  You should find this livestream.  When I generated this search results page, the livestream was the fourth clickable item on the page, after one of those "+8 more" things you see all the time on YouTube search results pages.  Next, open our best friend the Network Monitor, same as what I did upthread here.

It's empty.  That's what you should expect at this point.  It is critically important that up to this point, the mouse cursor should be parked off the browser window.

At this point, I was going to click that link for the livestream.  So I moved my mouse up into position . . . and stopped cold.

Ignore the blue arrows for the moment.  I'll get to those.

Holy cow!  You can see I was still hovering my mouse over the link.  You can see the tooltip that hovered up.  I never did actually click the link at this point, or at any other point in the rest of this story.  I just sat there & marveled at the extremely annoying behavior of YouTube.  I hadn't done anything yet.  Why are they inundating me with all this junk?  I might not click that link.  Maybe I just want to read the tooltip.  Some of these titles are long & the tooltip shows you what isn't immediately visible.  I'm amazed that even at this late date, I am still discovering brand new reasons to hate YouTube.

But on a hunch, I figured YouTube was preloading the web page I was about to possibly or possibly not go visit.  That means that maybe -- I was still guessing at this point --  maybe that json with the name player was already here in my Network Monitor.

Now pay attention to the blue arrows.  Drag the top edge of the Network Monitor up as far as it will go.  Yes, completely cover the YouTube web page.  We're not going to be interested in anything there for the rest of this little story.  So just make the Network Monitor as big as it can get so we can get a good view of what's in it.

At the same time, click the column heading for Type, to get the objects in the Network Monitor sorted into some kind of rational sequence.

Now scroll around in the Network Monitor until you bring the group of json objects into view.

Ignore the blue arrow there for the moment.

Well look at that!  A player json.  Maybe this is why so many YouTube pages are missing their player json.  When you click a link on a search results page, it's like visiting the page for a SECOND time.  You might have thought it was your first time.  But YouTube, in its infinite wisdom, knows so much better than any of we stupid users do about how to browse web pages.  Maybe this is why I so often have to fall back to the log_event approach I describe upthread.  I have explained that if you reload a YouTube web page, it loses its player json.  The same thing happens if you close a given YouTube page, & you're now revisiting it DURING THIS BROWSER SESSION.  This is why I said above that you have to be doing all of this for the first time in THIS browser session.  It's all about the elusive player json.  This is the file we want the most.  The log_event thing is a suboptimal, if workable, second choice, a fallback approach.  The first choice is always the player json.  So you have to be careful that you're doing things for the first time in THIS session.  Otherwise, your player json will disappear & I have not discovered a reliable way to get it back short of cycling the browser.

Now pay attention to the blue arrow there.  You want to drag the right side of the Network Monitor window into the middle of the display, more or less.

This brings a whole new area of the Network Monitor into view.  Some of you may be snickering at me for being so dumb.  You may have already known about this area.  So sue me.  I didn't.  I'm slow.  Senility, in my case, isn't creeping, it's galloping.  I am discovering things that have always been there but I've never known about & never realized how useful they could be.  Once you have that right side pulled into view, click the Response heading indicated by the blue arrow.

Now, this stuff looks interesting.  What if we start poking around in here?

I'm just hovering my mouse over that area.  The tooltip -- the HUGE tooltip -- that comes up is just an echo of the URL under the mouse cursor.  The cursor does change as if that's a clickable link.  But I tried it.  It's not.  So switch mouse buttons.

Ignore the black circle for the moment.

Now we've got a context menu popping up.  I've tried the Copy All option.  It gives a lot of stuff, more than we need.  Copy Value is as much as we need.

The two items I've put in black boxes should be about jumping off the page at you.  These are THE MOST USEFUL things you can get from a web page from which you want to download a video.  In other words, we just hit the jackpot.

Back to Copy Value.  It's not too much to expect that you already guessed that this puts something into your system clipboard.  Now swap tasks over to your text editor & do a paste.

All the way at the bottom of what you pasted you'll find our 2 manifest URLs ripe for the picking.  What do you do with master manifests?  You should know from the many other threads I've posted in here.  You run a master manifest through ffprobe.  And most unusually, we have our choice of DASH & HLS master manifests here.  Like I said.  Jackpot.

I've got the output from both ffprobe invocations captured in attached file ffprobe French Livestream Manifests.txt.  Two ffprobes, one output file.  The first ffprobe report, the one for the DASH master manifest, is rather interesting.  It shows 6 video Streams for mp4 video tracks of various resolutions.  The h264 & avc1 labels tell us that these are of type mp4.  Then it shows 6 video Streams for mkv video tracks of the same set of resolutions.  The vp9 label tells us that these are of type mkv.  Remember this.  This says these are of type mkv.  I'll mention this again below.  Finally, we've got 2 Audio streams, one of bitrate 64,000, the other of bit rate 144,000.  That means the second one is of a higher quality.  Maybe that will be noticeable in a download.  This is a livestream of a TV news channel.  The emphasis is on talk, not music.  Quality of audio content is not of the highest importance here.  Since these are the only audio tracks here, I guess they are equally compatible with either mp4 or mkv.  The mp4a label says they are audio mp4, which is equivalent to mp3.  I don't know what exactly HE-AAC & LC mean.  I suppose I should look that up at some point.  Not knowing that, however, doesn't stop us from using this stuff.

I must point out that this structure of multiple video tracks sharing common audio tracks is something that has historically tripped up VDH.  At the minimum, this structure causes VDH to offer separate video & audio tracks on its menu.  Worst case is VDH recognizes nothing or downloads one of those too-fast things that needs to be repaired.  But this is YouTube & Michel has written special code into VDH to handle this particular variation on the DASH theme.  DASH as it occurs on other sites is currently beyond VDH's capabilities.  Currently.  I believe Michel is working on this.  I don't know it for a fact.  It's just that we've discussed it & it seems to be in his consciousness now, so that's a step.  Just don't hold your breath waiting for proper support for DASH streams on other sites to appear in a new VDH.

The second ffprobe report, the one for the HLS master manifest, is kind of boring.  Useful, but as these things go, boring.  It's a simple set of 6 paired audio & video Streams of type mp4.  This is the kind of master manifest that VDH has always handled properly.

So let's download this sucker.  How?  With ffmpeg, of course.  Why not with VDH?  Because, like I said earlier, VDH historically has problems with YouTube livestreams.  VDH also does not reliably stop a download from the blue dot status menu.  Sometimes it works, and you get a playable file.  Sometimes it fails, & gives you a junk file that might be repairable, but that's still an untested possibility.  Besides, this is a livestream.  As a test, I didn't want to record hours upon hours of it.  I just wanted 5 minutes.  You can't tell VDH to record a livestream for any given duration.  You can't tell it to download something that isn't a livestream for any duration, either.  But it's a standard feature of ffmpeg.

Since the HLS master manifest contains only mp4 videos, I decided to use the DASH manifest to download one of its mkv videos with ffmpeg.  I used a second invocation of ffmpeg to download one of the mp4 videos from the HLS master manifest.  Those results are in attached files French Livestream via DASHmkv Log.txt and French Livestream via HLSmp4 Log Filtered.txt.  What I filtered out of the full HLS log was a bunch of lines showing ffmpeg skipping stuff.  Manifests, especially manifests for livestreams, regularly contain lines that ffmpeg doesn't need.  But it dutifully reports each such line as it's skipping it.  Bloat.  I've filtered those out.

You'll note that the HLS download ran from 16:18 through 16:23, during which it recorded 5 minutes of the livestream.  The DASH download ran from 16:20  through 16:22, during which it also recorded 5 minutes of what anybody would assume is the same live stream.  So logic would dictate that the two result files show at least some of the same content.

Why oh why do I still at this stage of my life continue to make the egregious error of applying logic to a situation?  The 2 files I recorded have NOTHING in common.  They show completely different people in the 2 videos discussing completely different topics.  The only things they had in common were that they played fine with video & audio, and they lasted 5 minutes.

Now how could that be?  It's a livestream & I was recording the same livestream 2 different ways.  Well, it must not have been the same livestream.  At least, I must not have been recording the same part of the livestream in each operation.  I don't know why that is.  If somebody can enlighten me, I'd welcome it.  I have always been under the impression that when ffmpeg got thrown into a livestream, it recorded from that moment forward, barring other things you could do.  These 2 downloads overlapped.  Why didn't they get the same thing?

I will point out something that might be relevant, although I still can't say these things totally explain my results.  In each ffprobe report as well as each ffmpeg download there is a line like this:

  Duration: N/A, start: 17131642.927667, bitrate: N/A

Duration: N/A is exactly what you expect with a livestream.  You don't know how long it's going to last.  So nobody can say what its duration might be.  But the interesting thing is the start value.  I believe that is a number expressed in milliseconds.  It is my understanding that it expresses the current location of the measurement relative to the start of the livestream.  Each ffprobe & each ffmpeg shows a different value for start.  The number here is taken from the ffprobe of he DASH manifest.  If I've got this right, that tells us that the stream had been in progress for about 4 hours & 45 minutes when I poked my nose in.

But the start reported in the ffprobe of the HLS manifest is 62312.450278, barely over an hour after the start of the livestream.  That is way, way, way different.

The start values in the 2 ffmpegs are in the ballpark with their respective ffprobes, a bit later in each case.  That at least makes some sense.  What doesn't make sense is why the HLS manifest points so far back in the livestream.  I think the DASH stream was probably much closer to "now," meaning around 16:20 my time in the afternoon.  I guess that partly explains why the 2 videos were completely different.  They were extracted from different moments in the livestream.  But it doesn't explain why the HLS manifest didn't also point to "now."

The 2 ffmpegs ran at 2 wildly different download speeds.  The DASH recording . . . It was a recording, not a download.  It was getting what was being broadcast at the moment.  That's a bit of a guess because I never did actually look in on the livestream.  It got 5 minutes' worth of the program in under 2 minutes, at a speed of a little under 330,000 bytes per second.  That's a bit faster than the usual throttling to 50% of the duration of the content.  Not a lot faster, just a bit.  But if it was recording live content, why didn't it take 5 minutes to get 5 minutes of content?  I don't have an answer.

On the other hand, the HLS download . . .  It was a download, not a recording.  It was downloading something that was being held as an archival record of what had been broadcast about 3 hours before I came along.  But it got 5 minutes of the livestream in 4:52.  That would lead you to conclude it has to be a live recording.  But the start value says otherwise.  I have to say I'm stumped.  Its speed was only about 140,000 bytes per second.  But that's all it takes to broadcast a livestream.  Look at the bitrates for the HLS mp4.  They're pretty low.  It doesn't take much bandwidth to broadcast such relatively low quality video.  The Golf Channel stuff I have streamed usually requires on the order of 700,000 to 900,000 bytes per second.  I'm not knocking this French broadcast.  I'm just making observations.  The picture quality of the livestream was perfectly acceptable.  The DASH mkv reports only one bitrate, but it's, frankly, pitiful.  Nevertheless, it looked OK when I played it in VLC.  Maybe bitrates don't measure quality the same way for mkv as for mp4.  I honestly don't know.  Then again, we're not talking about a feature movie in a theater.  It's a news broadcast.  The expectations, the standards against which to measure it, are different.

The mystery remains why the HLS manifest pointed into the past while the DASH manifest pointed at the present.  Also, why the DASH recording acted like a download while the HLS download acted like a recording.  Mystery upon mystery . . .

For the sharp-eyed among you, you might notice a couple of ffmpeg parameters on the HLS download that don't appear on the DASH download.  Initially, I coded both -m3u8_hold_counters 25 and -max_reload 25 on the DASH recording.  But ffmpeg said it didn't recognize them & refused to execute, so I pulled them off & it worked.  I'm not entirely sure why.  Those parameters are meant to control how long a livestream recording lingers after the livestream ends.  I don't know why they didn't work on the DASH recording.  I don't understand why they were rejected.  They were accepted for the HLS download, which doesn't make sense.  It was a download so I can see how they would not be necessary.  On the other hand, they were superfluous there since I was limiting the length of the recording to 5 minutes with -t 00:05:00.  Maybe those 2 parameters are not permitted with mkv.  I honestly don't know.  And the ffmpeg documentation is so execrably bad, I'm not even going to look for an answer in there.

There is one ffmpeg parameter that would level the playing field.  -live_start_index 0 would tell the download to begin from whenever the livestream began.  If I had coded that on both invocations of ffmpeg, I'm sure I would have gotten the same 5 minutes of video twice, both from the start of the livestream.  It's a tricky thing with livestreams.  When do they begin?  In this particular case of a TV station in France, maybe it began 10 or 15 years ago, whenever they first went on the air.  They appear to be a 24x7 service.  Obviously, their server doesn't hold an archival copy of every second they have ever broadcast.  It looks like they do hold at least the most recent 4 hours.  But how do they manage that?  Do they incrementally throw away stuff so they only ever keep 4 hours into the past?  Do they maybe keep up to 6 hours & then throw all of that away & start a new stream?  Some other approach?  I ask these questions just to make you aware that rewinding a livestream to the start (-live_start_index 0) should not be taken for granted.

Also, because they do broadcast 24x7, their stream never ends.  With VDH being so unreliable about terminating an inflight download, I always hesitate to use VDH for something like this.  If you can use another method, like for example ffmpeg, I think that is preferable.

I speak French.  But I was able to make out only 1 word in 10 or 15 of what these people were saying.  But these are people in France.  They just don't pronounce the language like I'm used to hearing it.  I'm used to Martin McGuire & Dany Dubé who are the French language radio announcers for the Montreal Canadiens hockey games.  That's my hometown team.  That's the kind of French I can understand.

[Wild Willy]

Erratum:  The ffprobe of the HLS manifest shows it is taking its starting point from 62 SECONDS, not 62 minutes, after the start of the livestream.  I did pass first grade arithmetic . . .

[Wild Willy]

You'd think at this point that it's case closed.  But there's more gold to mine from the Network Monitor.

I mentioned above that one of the images has a black circle around a small right-pointing triangle.  Go back to that one & look at it again.  This is very much like what Michel does in the Hit Details available in VDH.  He borrowed his idea from elsewhere.  I'm sure the Network Monitor & the VDH Hit Details are not the only places you'll encounter this sort of interface.  Back to the story.  You want to click the little triangle.  That makes some stuff expand from inside that object.

As an alternative to a mouse click, you can use keyboard equivalents.  The UpArrow, DownArrow, PageUp, & PageDown keys also navigate that right half of the Network Monitor.  But it's a bit inconsistent.  The highlight for the current line often goes out of sight off the top or bottom of the display & you can't tell where you're at.  I find I am constantly mixing up mouse clicks with keystrokes.  In any case, RightArrow, when you can get the proper line highlighted, does the same thing as mouse clicking the little right-pointing arrow.  You can also click the down-pointing arrow or type LeftArrow to collapse something that's expanded.

Here, I've expanded the streamingData object.  Why?  You know the answer.  Trial & error.  Guessing.  I expanded everything that could be expanded to see what's there.  I recommend you do the same.  But that's a lot of extraneous, somewhat off-topic material.  I'm going to show only what I think is pertinent to what we're doing here.

Below the 2 URLs that have already been so useful to us, the streamingData contains a wealth of other useful information.

That adaptiveFormats thing looks intriguing.  There's a list there with a bunch of ellipses within brace brackets.  There must be a bunch of things hiding inside there.  Expand it.

I discovered there were 13 things labelled Object, numbered 0-12.  I did expand them all & look at them all.  It turned out that Object 0 was the interesting one.

That isn't where the interesting stuff was.  Scrolling that down a bit . . .

This might explain why VDH sometimes downloads something that turns out to be only 5 seconds long.  In fact, in that thread I mentioned above that led me to this livestream, I encountered exactly that just the other day.

https://groups.google.com/g/video-downloadhelper-q-and-a/c/ppuouJJcK4M

These objects were in pairs: video/mp4 then video/webm.  I'm not showing every object here but I did look at them all.  Each mp4 of a given resolution was followed by a webm of the same resolution.  So you have a choice of mp4 or webm to download at any given resolution.  Except they were all only 5 seconds long.  I don't know why YouTube does this.  It seems like just so much bloat when all the same content is contained in the DASH master manifest we got earlier.  Then there's the HLS master manifest that duplicates only the mp4 variants.

Wait.  What?  Didn't I say earlier that the DASH master manifest described mkv videos?  But here they're webm.  What gives?  I didn't know so I Googled it.  Apparently webm, that is the webm video format is a subset of the mkv video format.  If you've been paying close attention, the new VDH 7.7.0a1 beta lists YouTube content as mkv, not webm, despite all this stuff in the player json saying it's webm.  My sense is this is largely a cosmetic difference.  Don't dwell on it.  I think if Michel is choosing to give this content to us as mkv, that's my cue to do the same thing.  I think mkv might be a more general file type, more flexible, so I would go with that.

Here's the lucky 13th Object, Object 12.  It's too big to show in one image, so here's the whole thing in 2 images.

It's an audio mp4 of duration 5 seconds, to match all the videos above it.  This was the only audio Object.

[Wild Willy]

You have to understand that this stuff doesn't work on all YouTube content.  Not all player json objects contain the same data.  Even taking my latest discovery of preloading into account, some YouTube pages just don't have a player json.  Plus there's the issue of the signatureCipher thing.  If you see one of those, the adjacent URL is not going to work as is.

I did a web search on the subject of signatureCipher.  I discovered there's an underground subculture, almost a cottage industry surrounding the cracking of the signatureCipher.  Some of the things I read go back 10 years or more.  It's not a new thing YouTube just started using.

There seems to be no pattern to which YouTube content is protected with this thing & which isn't.  It seems like just another of Google's capricious tricks it can get away with because it's a monopoly.

I found one guy who referred to Google employees as googies.  I love it.  Googies.  I'm going to use it.  Allegedly, there's a team of googies whose responsibility it is to change the algorithm for decoding the signatureCipher.  The algorithm never stays the same for very long.

A number of posts I found claimed to have solved the mystery of the signatureCipher.  But the explanations were couched in Linux-speak & python code fragments & several other things that I am utterly ignorant of.  If anybody can come in here & explain it in terms a non-insider can understand, I'd be most grateful.

I believe the solution goes something like this.  I don't have a great deal of confidence that I've grasped the ideas.  But this is my current collection of misconceptions.

The signatureCipher is sliced up into substrings.  The various substrings are fed into the constantly changing algorithms.  There are several of them.  These algorithms are written in javascript.  These things are actually sent to your browser among the js objects you can see in the Network Monitor.  Using magic spells I didn't fathom, these js things are fed the various parts of the signatureCipher.  The results are then used to append certain strings of the form &this=that to the URL that everybody can see in the player json objects or the log_event json objects I have identified above.  Once these magic strings are added to the URL, then you can just visit that address & do the Save Video/Audio As... function I describe upthread here.

Your mission, should you choose to accept it . . .

[Wild Willy]

There's an article in Wikipedia on HE-AAC. It mentions LC in passing. Knowing anything
about this doesn't seem to have added any significant information to my existence.

[Wild Willy]

I have the new VDH 7.7.0a4 beta now. With this new beta, 2 of the videos cited above as
failing in VDH now work:

https://www.youtube.com/watch?v=sD3eveWBEUU
https://www.youtube.com/watch?v=ZkH6jz1gy3Y

[Wild Willy]

We've had plenty of activity here lately.  The 8.0.0.0a2 beta has been released that cures pretty much all the problems on YouTube.  Pretty much.  Not yet absolutely everything.  But pretty much.  When it doesn't work even now, you have to resort to the techniques I've laid out upthread here.  And as I note above, there is still content that even those techniques can't retrieve.

The official 8.0.0.0 has also now been approved & is available from the standard Mozilla add-ons site.  It is identical to 8.0.0.0a2 beta.  I wonder why we now have an extra digit in the release number.  Whatever.  A name is a name.

It seems that VDH now includes support to a certain extent for DASH content on other web sites.  This is most welcome.  On YouTube, we are now getting the full array of mkv & mp4 variants that we didn't used to get.  Used to be the mp4 variants never worked.  Now they do.  Plus we're seeing all the resolutions, including those we didn't used to see.

YouTube itself seems to prefer mkv.  I have seen YouTube content for which the highest resolutions are provided only in mkv format.  It's available in both mp4 & mkv format for lower resolutions but occasionally, the top resolution is available only as mkv.  So what?  Download mkv.  VLC plays mkv exactly the same as mp4.  There is no need to convert mkv to some other format.

I have recently learned that the webm format is a subset of the mkv format.  If you're wondering where some webm content has disappeared to, fret not.  Michel has made the wise decision to offer what used to be webm content as mkv.  Mkv is a more general format & probably safer to use.  Like I just said, VLC handles mkv just fine.  Go with it.

Vimeo is another problem.  For content on web pages directly at vimeo.com, VDH has always been able to download that just fine.  But a lot of other sites use Vimeo to host their A/V content, what we call embedded Vimeo content.  Those items have always been a problem for VDH.  The new 8.0.0.0a2 beta/8.0.0.0 still can't get embedded Vimeo content properly.  When VLC plays what VDH downloads, the entire video content flashes by in a second then freezes.  The audio, on the other hand, is fine.  For embedded Vimeo downloads, you need to go read this thread:

https://groups.google.com/g/video-downloadhelper-q-and-a/c/sNfTCMYfiTU

The high level summary is to either:

1. Download the content with VDH then repair it.

2. Download the content with ffmpeg.

If you want details, go read that other thread.

But we still have nervous nellies whimpering that this new VDH works for a bit & then quits working on YouTube.  This is . . .

FALSE!!!!

It has long been known that on some web pages, you need to reload the page.

RELOAD THE PAGE!!!!!

Yes, a lot of YouTube pages initially give you an empty VDH menu.  You should know from long experience that you need to . . .

RELOAD THE PAGE!!!!!

Sometimes it's not enough to launch playback in the YouTube media player on the web page.  But when you see that even that doesn't make anything appear on the VDH menu you have to . . . all together now . . .

RELOAD THE PAGE!!!!!

[Wild Willy]

There is one extra step you need to perform when you upgrade to the latest release of VDH, either the official 8.0.0.0 or the 8.0.0.0a2 beta, which are, as I've already said, identical.  These instructions are on the congratulations page that you get when you update to the latest beta.  But if you don't read that page carefully, you won't know about this.  You shouldn't just see the congratulations page & think yeah yeah yeah OK fine & dismiss it.  Read what it says.  I'm not sure where, if anywhere, this is documented for folks who never used the beta but just installed the official release from the Mozilla add-ons repository.  I have not installed the official release because I already have the beta.  Maybe when you install from the Mozilla repository, you get the same congratulations page.  Perhaps someone can add a comment here about this.  This is a one-time operation you need to do after you go to this release of VDH.  Do it once & then you can forget about it.

There is some discussion of this in this thread:

https://groups.google.com/g/video-downloadhelper-q-and-a/c/qLM5Pv_rbsQ

The specific posts are here:

https://groups.google.com/g/video-downloadhelper-q-and-a/c/qLM5Pv_rbsQ/m/ByIn26WoBgAJ
https://groups.google.com/g/video-downloadhelper-q-and-a/c/qLM5Pv_rbsQ/m/6hc81JerBgAJ

Although those posts mention a slightly earlier beta, the instructions I give here applied to every beta release between that beta & the latest beta as well as the latest official release.  As you can read there, this little process should be going away at some future date.  But for now, & perhaps for the next unknown number of betas, you will have to go through the steps I've just outlined.  You must do them once after you install until we get official notice that this is no longer necessary.  If you fail to perform these steps, VDH will continue to fail to recognize certain variants on YouTube, & perhaps on other sites as well.

[Wild Willy]

With the release of 8.0.0.0a3 beta for Firefox, is the above step for resetting settings still necessary?  I performed it because I took my own advice & did it.  We do not have any information saying yes or no whether this is still necessary.  It doesn't seem to hurt anything to do it again with the new beta.  But I'd like a clear answer on this question.

[mig]

No, you do not have to reset those settings again.

[Wild Willy]

I was having trouble with this one as well. I visited the page & VDH just errored out
immediately on the 1920x1080 variants for both MP4 & MKV. Then I thought I'd try my
evasive maneuvers detailed extensively upthread here. That wasn't working either. Then
I decided to give it one last try by clicking the link on the search results page. For
some reason, that time it worked. I double checked & the URL given by the search results
was the same as the one you've given here. You never know. A search might give a
different URL for the same video. But it didn't in this case. The download worked fine
& the video played fine all the way to the end, although I didn't sit & watch it, just
sampled it at intervals. But it appeared to be all there with both video & audio right
to the end.

I don't know what this means. Maybe you still have to reload some pages a few times even
with 8.0.0.0a3 beta. There may be strange effects having to do with timeouts. All
YouTube pages time out. You'd think you were getting to the download pretty fast. But
nobody knows how YouTube really works. They might time something out really fast here &
there. I suppose that means it's always a good idea to retry anything after a page
reload. I would recommend at least 3 attempts before I gave up. Whatever. I did
eventually get VDH to download this one after a couple of reloads.

By the way, I've seen a number of people posting that they're getting errors during
aggregation. I saw that when this video failed, too. But that error is the SECOND
error. The FIRST error is that VDH couldn't get information about the video. That's the
important error. The second error is just a result of the first, which is to say that it
is noise & completely uninformative. The first error is the only one worth paying
attention to. If the download isn't even attempted, of course the aggregation is going
to fail. There isn't anything to aggregate. Seems a bit silly that VDH attempted to
aggregate anything when the download itself failed. Maybe that's something worth looking
into.

[Wild Willy]

There are several other threads updated today that appear to be reporting the same thing.
What I am seeing as a pattern is this:

1. Go to the YouTube page.
2. Attempt to get VDH to download the video.
3. This either succeeds or fails.
4. When it fails, you get 2 errors. The first is that VDH couldn't get information on
the video & the download immediately aborts. This is followed by a spurious error
claiming there was a problem aggregating the video. Duh. The download failed. Of
course aggregating will also fail. There's nothing to aggregate. Ignore this second
error. It is the result of the first error.
5. RELOAD THE PAGE.
6. Retry the VDH download.
7. If step 6 succeeds, and it usually does, smile & go away happy.
8. If step 6 fails, go back to step 5. Keep doing this until the download works. You
need to fail at least 3 times before you give up & post about it here.

It seems that the claimed benefit of 8.0.0.0a3 has not occurred. Reloading seems to
still be required. The new behavior seems to be the 2 errors. Before, VDH would just
not show any variants for the video & a reload would make them appear. Usually,
something would then download. Now, the variants are appearing but VDH is generating an
error on any attempted download. But reloading is still the solution.

On the subject of my post upthread dated July 17, what has changed? Is it still
necessary to have done that once? Or is not no longer necessary at all? Does 8.0.0.0a3
require that you have done that at some point in the past? Or is 8.0.0.0a3 no longer
reliant on that ever having been done? In other words, have you eliminated the silly
list of resolutions? I have just exported my VDH settings & the silly list appears to
still be there. So I believe that you (Michel) still rely on people having done the
thing in the July 17 post at least once ever. Do correct me if I am wrong. And if I am
wrong, you need to put code into VDH to clean that junk out of the settings. You don't
want that cluttering things up forever.

[jcv...@gmail.com]

This one worked for me, but yes, there is an issue.

jerome

[ALAN FERNLEY]

it doesnt work but you are still taking peoples money? seriously?????

On Tuesday, 20 June 2023 at 06:44:46 UTC+1 Wild Willy wrote:

It has been pretty sad around here lately, what with VDH being unable to download YouTube content since about the end of May, beginning of June.

https://groups.google.com/g/video-downloadhelper-q-and-a/c/orAHFdaLbUE

And now Michel may be softening us up for a defeat.

https://groups.google.com/g/video-downloadhelper-q-and-a/c/wvQp_W2G8pA

Well, I believe any defeat should not be total.  I'm hoping this thread encourages Michel to at least fix VDH to the extent that it continues to download the YouTube content that it still can.  If it encounters the content that is now inscrutable, at least generate an error message up front so we don't waste our time even attempting the download.

What you see below is the result a bunch of hours of improvising, trial & error, guessing.  It is an application of the approach you can read about by starting here:

https://groups.google.com/g/video-downloadhelper-q-and-a/c/BzPLK2YyL-s

First, you need to carefully read the first 2 paragraphs there.  Then look for "evasive maneuvers" within the text of that web page.  That will give you a link to a discussion we've been having about trying to deal with web sites that VDH doesn't process correctly.  It gives either bad results or no results on those sites.  So this thread is, in a sense, a continuation of that discussion with the problematic web site being YouTube.

Before we get started, I should tell you about my environment, as EVERY problem report here should.

Windows 7 64-bit, Firefox 114.0.1 64-bit, licensed VDH 7.6.5a3 beta (same as 7.6.6), CoApp 1.6.3

Actually, my VDH & CoApp are irrelevant here since I won't be using them.

I encourage somebody who is on another platform, Mac, Linux, Chrome, Edge, whatever, to add a tutorial here about how to do things differently on your platform.  I'm sure a lot of what I'm doing is the same regardless of the operating system & browser.  But I have to believe there are large variances of certain steps for other platforms.  Please add screenshots & narrative, both text AND pictures, showing how to do those things on the other platforms.  Text is NEVER enough.  Include screenshots.

Also, before we get started you need to go read this thread:

https://groups.google.com/g/video-downloadhelper-q-and-a/c/O_bRwxrfqS8

In there, you will find a post from Michel in which he gives a link to a Bugzilla report that he opened.  You need to go read that.  There is a circumvention in there for the problem he describes.  I have found that that circumvention affects what you get from YouTube.  So you have to implement the circumvention described in the Bugzilla report before you try any of what I'm posting below.

Why don't I just tell you here what that circumvention is?  Because I neither encourage nor condone laziness.  I'm doing some work here.  You do a little work, too.  There's only so much spoonfeeding I'm willing to do.

[jcv...@gmail.com]

It doesn't work on Youtube for the moment. It has been fixed, then a new issue appeared. The dev team is working hard on it to make it available again.

We do not "take" money, people buy licenses, and if some users are not satisfied, they are refunded.

jerome

[Wild Willy]

I need to ask again, Jérôme. I see the list of resolutions in my settings when I export
them to a json. This tells me that even though Michel said you don't have to do the
reset settings thing I describe upthread (July 17 post), VDH still relies on your having
done it at least once at some point. Yes? No? And if no, VDH needs to clean that junk
out of the settings. It's a huge amount of silly data if you're not using it any longer.

Whenever this next beta comes out, you know I'm going to ask what the latest change is.
So let's save time & have Michel answer that question without being asked whenever the
next beta is released.

[mig]

Resetting the "Video qualities" was necessary to detect high-quality variants, but it should be a one time thing. You should not have to redo it.
If you do not detect 4K (on videos where a 4K variant is available on YT), this is another problem. Can you share the link to the video page ?

[Wild Willy]

So this advice is still that you have to do the reset thing I describe above in my July
17 post at least once since 8.0.0.0 & all the betas numbered 8.0.0.0ax. Doing it once in
any of those releases is enough. But you have to have done it once. I repeat, it is
still required. You just don't ever have to do it more than once & it will be good for
any subsequent betas for the time being. Thank you for clarifying.

The issue is not restricted to 4K. Honestly Michel, just pick anything at all off the
YouTube home page. Nothing is working. You get the unable to get information error.
Then you get the failure to aggregate error which should not even occur. If the download
failed, VDH should not be attempting to aggregate anything. There is nothing to
aggregate. Of course it will fail. It seems wasteful to have even attempted an
aggregation. Surely you can detect the earlier failure & skip the attempted aggregation.

Fortunately, my alternative solution detailed upthread here still works. Here's what I
do:

1. Visit any YouTube page & suffer the VDH failure.
2. Reload the page a few times in the forlorn hope it will fix anything.
3. It does not.
4. Formulate a search key that will cause the present page to show up as a search result.
5. Park the mouse cursor off the web page & execute the search. Type the search key into
the search field, park the mouse off the page, then hit Enter.
6. Being careful to keep the mouse parked off the web page, scroll the search results
until you see the link to the desired page. Use the keyboard scrolling keys ONLY.
Position the desired search result above the middle of the display.
7. Open the Network Monitor via F12. It is important to NOT USE the mouse for this.
8. Adjust the height of the Network Monitor using the mouse. Be careful not to
accidentally swipe the mouse over any search results. Let the Network Monitor occupy the
bottom half of the display.
9. If you need to reposition the search results, do it WITHOUT the mouse. Keep the mouse
parked off the search results. F6 swaps focus between the URL bar & the page body. With
focus on the page body, you can scroll the page using the keyboard until the desired
search result is just peeking out over the top of the Network Monitor.
10. YouTube likes to send noise to the Network Monitor when you're not doing anything.
If there is anything in the Network Monitor at this point, click the little trash can in
the upper left corner of the Network Monitor. This is another unavoidable mouse
operation, so make sure you don't accidentally move the mouse over anything in the body
of the search results.
11. With the Network Monitor now empty, finally you can move the mouse over the web page.
Briefly hover over your desired search result.
12. This will fill up the Network Monitor.
13. Follow my instructions upthread here for finding the json with the name player. Then
follow the rest of the instructions upthread to download the video track & the audio
track of the desired item. Note that in this process, you never actually navigate back
to the YouTube page that contains the video. You never actually click the search result.
14. I have discovered a little trick that seems to speed up the downloads. After you
launch the download of a track via Save Video/Audio As..., close the browser tab showing
Firefox's built in media player. It may or may not immediately cause the download speed
to increase. If it does not speed things up, just wait a bit. I have found that after a
few minutes, suddenly the download speeds up & completes much sooner than advertised in
the Firefox Download Manager, which you get by typing Ctrl+j in any browser tab.
Patience is advised.