Downloading Vidalytics hosted videos

[Louis]

Hi

I'm running VDH on Firefox.  When a site hosts their video's using Vidalytics, the stream is broken up in to thousands of short video segments.  Clicking on the VDH toolbar icon shows a long list of them, each about 400 kB.

A sample video can be found on the page: https://series.welloflife.com/grb/part1-live/

From Firefox's "inspector" view

And this is what the VDH Icon shows:

Thanks!!

[jc vdh]

yes, when you get a long list of small items it means VDH didn't find the aggregation info and won't be able to make it, sorry.

jerome

[Wild Willy]

I visited that page & did the usual thing of opening the Network Monitor of Firefox & looking for a manifest.  I didn't find a manifest but I did find the attached file.  I've named it stream.txt here for ease of viewing but the real name of the file displayed by the Network Monitor was stream.mpd.  It seems to have a number of things I have come to recognize in m3u8 files but it's not quite the piece of cake an m3u8 file would be.  This mpd file appears to be a recipe for constructing a manifest.  It looks like an XML file instead of a manifest, but there are "manifesty" things in it.  The <Representation> tags appear to tell us the resolution of the video in question.  The highest resolution here appears to be 1280x720 & there's a choice of getting that as either an MP4 or a WEBM.  There's 2 choices for MP4 that appear to differ in their "Bandwidth" parameter, one being 1 million, the other 2 million.  I suspect this is an indication of the bit rates, meaning the video quality of the downloaded result.  There appear to be 3 choices of resolution for the first MP4, 3 choices of resolution for the WEBM, & 3 choices of resolution for the second MP4.  There is also what appears to be an audio MP4.  Does this indicate there's separate audio & video?  Or is it just that the videos all have audio but you can download an audio-only version as well?  I can't tell.  In any case, it appears that the structure here is

<AdaptationSet><Representation><SegmentTemplate></Representation></AdaptationSet>

with multiple <Representation> tags within each <AdaptationSet>.

Anomalously, the <SegmentTemplate> tag is not terminated by a </SegmentTemplate> tag but simply a /> at the end.  In any case, the <SegmentTemplate> tags appear to describe the formula for constructing the URLs of the individual chunks of each downloadable stream.  If you were really ambitious, much more ambitious than I am, I believe you could probably construct a proper manifest (m3u8) using the information you can see in this XML file.

Why do people keep coming up with these weird ways of presenting media on the web?  Is downloading videos really something they don't want you to do?  Or does everybody just want to be a rugged individualist?  I do truly, fervently wish the ISO would promulgate standards that everyone would adhere to for this.  I rather like the simplicity of m3u8.  My vote would go to that.

[jc vdh]

thanks for the feedback Willy.

Yes, they just don't want people to download :)

jerome

[Paul B]

Hi all

I have a same experience with a video flexability course I was Interested in.

https://hyperbolicstretching.com/men/

But after making the purchase I discovered I did not read all on the sales page

I contacted the seller to ask if a download link could be given but he refused and put it down to copywrite infringment

all I wante to do was download the videos so I could watch on my mobile whilst I was on the move

Is there any software out there that can download online videos?

Paul B

[Wild Willy]

In the time since I first posted in this thread, I've learned some things.  One of them is that a .mpd file is a manifest for DASH streams.  DASH stands for Dynamic Adaptive Streaming over HTTP.  DASH is an alternative to HLS, HTTP Live Streaming.  It is possible to feed a .mpd file into ffprobe the same as you can do with a .m3u8.  So here's the information I've gotten from the URL the original poster gave above.  First, the URL of the .mpd file is this:

https://quick.vidalytics.com/video/o8U49vKp/jyXxbwOZnoHv2EI8/15032/10315/stream.mpd

When I double clicked on that entry in the Firefox Network Monitor . . .

. . . it gave me a file that I have attached below as streammpd. txt.  I can't post something called stream.mpd, which is the name as you can see it in the URL.  Google does not accept a file with .mpd extension as an attachment on a post.  So I adjusted the name to suit Google.

Then I executed this command:

ffprobe -protocol_whitelist file,crypto,data,http,https,tls,tcp "https://quick.vidalytics.com/video/o8U49vKp/jyXxbwOZnoHv2EI8/15032/10315/stream.mpd"

That's 1 line.  Google has wrapped it to fit on the page here.  I executed it as a single line.

There is no need to try to understand the content of the .mpd file, something I tried to do the first time I posted in this thread.  You can get ffprobe to do it for you.  This ffprobe command generated the output you can see in the attached file ffprobe.txt.

It is much easier to figure out from the ffprobe output what is being offered by this web site.  Apparently, what was on this web site a year ago is still there.  You can see in the ffprobe output that there are 9 video-only streams being offered & a single audio-only stream.  Each video stream shows a resolution, the highest being 1280x720.  But there are 3 of them at that resolution.  How do you choose which one to download?  Look in the metadata section for each stream.  There is a value given for variant_bitrate.  The first 1280x720 has the highest variant_bitrate, 2000000.  Let me put some commas into that: 2,000,000.  The other 2 1280x720 streams show variant_bitrate of only 1 million.  In addition, the first 1280x720 stream shows 1958 kb/s, the second one 954 kb/s, & the third one doesn't show that.information.  So I chose the first one.  It appears to be the highest quality stream among the 3 1280x720 choices.  One resolution, 3 levels of quality.  I keep mentioning that resolution & quality are not the same thing.  Here's an example to illustrate that.  The confusion comes from YouTube using the word "quality" to refer to the resolution.  Gee, thanks so much.  NOT!!!

Then I executed the commands below.  The first one got the video, the second one got the audio.  Again, Google has wrapped the 2 commands onto multiple lines but the originals were single lines each.

ffmpeg -protocol_whitelist file,crypto,data,http,https,tls,tcp -hwaccel auto -i "https://quick.vidalytics.com/video/o8U49vKp/jyXxbwOZnoHv2EI8/15032/10315/stream.mpd" -codec: copy  -map 0:0 "Q:\VDH Testing\Health Video.mp4"

ffmpeg -protocol_whitelist file,crypto,data,http,https,tls,tcp -hwaccel auto -i "https://quick.vidalytics.com/video/o8U49vKp/jyXxbwOZnoHv2EI8/15032/10315/stream.mpd" -codec: copy  -map 0:9 "Q:\VDH Testing\Health Audio.mp4"

The important bits in there are the -map parameters.  The value of each -map parameter is taken from the stream definition shown in the ffprobe output.  You don't have to figure anything out.  ffprobe does it for you.

Here's my results.  This shows the files I downloaded.  Each one took about 1 minute to download.

Here's the Windows Properties of the 2 files.

Curiously, ffmpeg showed this error just as it was completing each download.

[https @ 000000000a4b3400] HTTP error 403 Forbidden
[dash @ 00000000004172c0] Failed to open fragment of playlist
[dash @ 00000000004172c0] Error when loading first fragment of playlist

Very curious, indeed.  Despite these errors, the downloads did complete successfully.  I don't know where these errors came from.  I have no idea what it might have been trying to open when the whole file was already downloaded.  So strange.  I do note that the URL given by the original poster is not the URL that showed up when I clicked that link.  The URL I actually got was this:

https://welloflife.com/gut-renewal-new

But you'll notice that the original poster's URL ends with /part1-live/.  I'm guessing this is actually just the first of several parts.  Maybe the streams offered by the web site are chained together & ffmpeg was trying to read a link to the next part of the series.  Maybe if you let it play long enough, another .mpd file shows up for part 2.  And then later for part 3, & so on as you get to each part.  I'm not going to try playing the video in the web page to see how long it is.  It's advertising.  I don't need to dig into that.

Whatever.  I left the 2 files I did get separate & played them synchronously in VLC, as you can see here.

I didn't actually sit & watch it, just sampled it.  You'll have to trust me that I could hear the audio just fine.  I skipped forward to about 6:15 & let it play from there to the end.  The video & audio were all there.

[Wild Willy]

You can even combine the 2 ffmpeg commands like this:

ffmpeg -protocol_whitelist file,crypto,data,http,https,tls,tcp -hwaccel auto -i "https://quick.vidalytics.com/video/o8U49vKp/jyXxbwOZnoHv2EI8/15032/10315/stream.mpd" -codec: copy  -map 0:0  -map 0:9 "Q:\VDH Testing\Health Video.mp4"

This maps both the video stream & the audio stream into the same output file, resulting in a single output file that contains both video & audio.

[Fred]

can't get ffprobe to work with the -protocol-whitelist option. it fails with

Failed to set value 'file,crypto,data,http,https,tls,tcp' for option 'protocol_whitelist': Option not found

The documentation also seems to be missing the -protocol-whitelist option

[Wild Willy]

For what it's worth . . .

Singularly uninformative, as usual with the ffmpeg documentation, but this at least confirms that it exists.  I note that in the text of your post, you call it -protocol-whitelist with a hyphen in the middle.  That should be -protocol_whitelist with an underscore in the middle.  But the error message you quoted has the underscore so I'll assume you didn't execute what you said but what the error message claims.  I last used ffprobe myself just a few minutes ago for reasons unrelated to this thread & I have exactly this coded:

ffprobe -protocol_whitelist file,crypto,data,http,https,tls,tcp

Hyphen in front of the parameter name.  Underscore in the middle.  A single space after the parameter name.  Parameter value not surrounded by apostrophes, quotes, or anything.  No embedded blanks.  Separated by commas.  I came up with this list by trial & error based on experience over time.  Ffmpeg kept giving me errors about protocol x not being in the whitelist.  So I would add protocol x to the list & I would get one step further.  This is a list I've had for rather a while now.  Ffmpeg hasn't thrown me any new protocols to add to the list lately.

Maybe the problem is your version of ffmpeg/ffprobe.  When did you get it?  This information should be in the opening output generated by ffprobe, the output that you would normally ignore.  Just to satisfy my curiosity, please post those first 2 lines of output.  For comparison, here's what mine says:

ffprobe version 2021-07-11-git-79ebdbb9b9-full_build-www.gyan.dev Copyright (c) 2007-2021 the FFmpeg developers
  built with gcc 10.3.0 (Rev2, Built by MSYS2 project)

July 11, 2021.

It occurs to me that there are "skinny" & "fat" versions of ffmpeg.  They are built with fewer or more options enabled.  The ffmpeg I got was the fattest one I could find on ffmpeg.org.  In contrast, the ffmpeg Michel has included with VDH is a skinny one he built himself.   I discovered a long time ago that Michel's skinny version does not handle all the cases I have encountered in the wild, hence my reliance on an ffmpeg I got myself from ffmpeg.org.  Perhaps your version is too skinny.