Video DownloadHelper Q&A

[Garen 5566]

Recently I have issue on recording live stream , it will auto stop after few min, after upgrading to latest version. Previously, i am able to record until the show end. Anyway to solve it?

[Wild Willy]

I've seen something like this when I've recorded a round of a golf tournament off Golf
Channel. This is a problem that is new for me. A few months ago, I used to have a
pitifully slow Internet connection that masqueraded as "broadband." My ISP labeled it as
5Mbps. Of course, this masks the true nature of the connection. First of all, the M
didn't stand for Meg but for Million. One million is 1,000,000 but 1 Meg is
1Kx1K=1,024x1,024=1,048,576. 1M is MORE THAN 1 million. You should understand that 1K
is more one thousand, 1Meg is more than one million, & 1Gig is more than one billion. Be
informed. And the b in my 5Mbps didn't stand for bytes, the unit that EVERYBODY thinks
in, but bits. So my bandwidth was actually (allegedly) 5,000,000 bits per second. To
turn that into units that real humans really understand, that's 5,000,000/8=625,000 bytes
per second. Problem is I never observed line speeds that high. My maximum bandwidth, as
observed in the Windows Resource Monitor, never exceeded about 550,000 bytes per second &
was more often between 530,000 & 540,000 on cooperating web sites.

Back to the topic at hand. When I had my slow connection, my recording of a live golf
tournament would last typically between 6 & 8 hours. The broadcast itself would last
only between 2 & 4 hours, but my connection was so slow that it took way longer than that
to record with VDH. It was a live event but VDH would still be recording it hours after
the broadcast had ended. I've mentioned this before. My suspicion is that NBC (the
owner of Golf Channel) doesn't give VDH the information it needs to record the live
stream. Rather, NBC gives VDH the location on a server of a file that is being
chase-recorded & VDH dutifully downloads that. It's too bad that the VDH guys, who are
in France if you didn't happen to know that, can't look at this because in order to even
connect to the NBC live streams, you have to log into your cable TV provider. You have
to be a subscriber to a cable TV system in order to look at the NBC live streams. You
can't subscribe to a North American cable provider when you live in France. You also
can't subscribe to French cable when you're in the USA. And they say we live in a free
world. Hah! Nothing is free.

I do so love to digress. So, when I had a slow connection, I would launch the recording
of the allegedly live stream a little before the broadcast actually started & go about
doing something else, usually out of the house, for the next few hours. Typically, the
NBC Sports web site makes the live stream page available 15 minutes before the actual
broadcast is scheduled to start. What I ended up with always began (still does) with
about an hour of the NBC logo behind the message, "Coverage will begin shortly." Then I
get however many hours there are of the broadcast, followed by a few minutes of the NBC
logo behind the message, "Coverage has ended." A three-hour broadcast ends up being an
MP4 file of duration a bit over 4 hours, with junk at the start & the end. In a way, I
suppose this is good because it means you can connect to the live stream late & still get
the whole thing from the beginning. But it isn't really recording the live stream.

But now I have a proper Internet connection after changing ISPs. Now I have what my new
provider calls a 400Mbps connection. This actually gives me nominal bandwidth of
50,000,000 bytes per second. I have yet to find a web site that will actually stream
anything to me at anything approaching that speed. For the Golf Channel live streams, I
observe a bandwidth usage in the range of 650,000 to 900,000 bytes per second. That's
when I sit & watch the live streams. When I record them with VDH, I appear to get
between 2 million & 3 million bytes per second. NBC simply throttles my connection.
That's probably fair enough given that there are untold thousands if not tens of
thousands of viewers streaming at the same time. But at that speed, the VDH download
catches up with the chase-recording. I have found that if I launch the recording during
the first hour or two of the broadcast, it catches up AND STOPS. VDH believes the stream
has ended & stops the download. This is despite the fact that I have these VDH settings:

Download retries = 100
Retry delay = 10,000

The 10,000 is milliseconds, so that works out to 10 seconds. Multiply by the 100 retries
& I should be getting a total timeout window of 1,000 seconds, which is 16 & 2/3 minutes.
That should be plenty of time to allow NBC to think about giving me the next segment of
the chase-recording. Nevertheless, NBC appears to be sending something to VDH to make it
think that the broadcast is over before it really has ended.

My solution has been to wait until about the last 15 minutes or less of the broadcast
window before launching the VDH recording. This typically gives me my three-hour
broadcast in a four-hour MP4 in a VDH download that lasts about an hour, giving me a file
that is anywhere between 8G & 10G. The downside of this is that I have to physically be
present in front of my computer to log into NBC, launch the live stream so VDH can detect
it, and then launch the VDH download. Situations like this would greatly benefit from
some sort of VDH API that would allow me to automate this process in some scripting
language. On the other hand, it would also take some sort of Firefox API that would
allow me to do the necessary mouse clicks under control of a script & I don't know that
such a thing exists. Then again, the URL of the live stream is not known until 15
minutes beforehand so maybe I'm dreaming of something that simply isn't possible.

So, my suggested workaround is to wait until nearly the end of the broadcast window
before you start your VDH recording. That is, if you are getting something that
resembles the NBC technology. If you're really getting a live stream & VDH is seeing the
real live stream, then this approach is obviously not a good one for you.

That brings me to a more pertinent question. What things will cause VDH to terminate a
live recording? What sort of signal from the stream server will tell VDH to stop now,
preempting the VDH retry mechanism? How can one of us users out here put some
instrumentation on our connection to look for that signal? Such instrumentation would
help diagnose this issue & perhaps result in a change to VDH.

[Wild Willy]

Today I had a perfect example.  I wanted to record today's round of a golf tournament
that is being played in Ireland for the next 4 days.  I wanted a file on my system so I
could skip the ads.  I also wanted to be able to skip the large segments of the broadcast
in which I have no interest.  This is a first-ever tournament in which both men & women
are playing, not against each other, but in alternating groups on each of 2 courses.
Since I am interested only in the women, I want to be able to skip the parts showing the
men, no doubt at least half the air time.  The broadcast window for this happened to be 5
hours.  So I waited until about half an hour before the end & launched my VDH recording.
Everything was going fine for about half an hour.  Then suddenly it stopped.  There was
an error message in the VDH log that led me to believe there had been some sort of line
error.  Or maybe there was some sort of momentary fault on the NBC server, which wouldn't
be all that surprising given they are carrying the delayed 2020 Tokyo Olympics at the
moment.  When I went back to the live stream page, they were still showing the "Coverage
has concluded" message, which is to say that the live stream was still transmitting even
though the broadcast had ended.  So I launched my VDH recording again, which of course
started from the beginning.  This time VDH happily downloaded for a little over an hour
and a half.  I got a file of size 11.3G & duration 6 hours & 4 minutes.  That's the 1
hour of junk at the start & a few more minutes of junk at the end, bracketing 5 hours of
actual content.

My purpose in posting here today is not simply to add an interesting (maybe) data point.
My purpose is to draw attention back to this thread.  There are questions asked here that
have not been answered.  Do please answer.  I want to know what causes VDH to prematurely
terminate a live stream without going through the timeout specified by the user in the
VDH settings.  Don't tell me there's endless possibilities like line noise, server error,
the user's computer had a transient error.  I know all of that.  I'm asking you to read
the code & tell me what conditions you look for.  Tell me what gets VDH to terminate a
live stream.  Tell me your IF statements.  There isn't an infinite number of those.
Explain the program behavior to me.  I'm not asking you to post the code.  I'm asking you
to explain the program behavior in user-oriented terms.  This is a reasonable request.
We've got this software.  Explain to us how it works.

[Wild Willy]

There are some legitimate questions here that need answers. So I'm posting here in the
hopes I catch Michel's attention.

[jcv...@gmail.com]

Hi Willy,

I'll poke Michel, I think it's a bit holiday time for him, but who knows...

jerome

[Wild Willy]

This issue is quite annoying. I've been watching the Olympic Ladies' Golf this week.
NBCOlympics.com has been offering the live streams every night in 2 parts. They include
about 3.5 hours in part 1 & the rest in part 2. It has totaled about 8 to 9 hours of
live streams each night. Here's what I have done. I start the download in VDH of part 1
a couple of hours after it has started. In general, this download has ended when there
has been between an hour & half an hour left in the live feed. Clearly, this is wrong
but at least the live feed is still in progress. So I launch the download of the same
live feed a second time. This time, VDH downloads the full length of the feed. (Yes, it
downloads a large chunk of the same data again but VDH is unable to start a download
beginning at a certain offset, meaning resuming an interrupted download from the point of
failure.) By then, part 2 has been running for a while so I put VDH on that download.
It also ends prematurely but a second try, which starts when there is under an hour left
in that live feed, does get the entire remainder of the stream. With my downloads
staggered like this, I am able to start watching a bit sooner than I would if VDH
actually captured the entirety of each feed. I am able to see a silver lining to this.
These feeds do not play in VLC while they are in progress. I have downloaded things,
live feeds & archived files, which I have been able to play in VLC while VDH was still
downloading them. I don't know what makes it possible to do that kind of chase playback.
But these Olympic golf streams have not been the type I could watch from the front while
VDH was still writing on the back.

I've tried to analyze this situation the best I can as someone who does not have access
to the code (and it's doubtful I'd be able to figure it out even if I did). My first
possible conclusion is that the NBC web site is sending some kind of signal that tells
VDH that the feed has ended. It would be a lie because the feed in fact is still
running. But it could be that NBC is sending a termination signal. I don't know what
that signal would be. But in this case, I would have to say there's nothing to be done
about it. What is VDH supposed to do if the web site lies & says a stream has ended when
it hasn't?

My next possible conclusion is that VDH has a bug in its logic for processing live
streams. In this case, I must repeat my question from above: What conditions cause VDH
to close a live stream that is still in progress? My experience with these Olympic golf
feeds is too consistent day to day for me to say I experience an Internet connection
failure every time. Although this is one possible reason VDH might prematurely end a
download, live feed or other type, I simply cannot accept that particular explanation in
this case. This did NOT happen FOUR DAYS in a row. Similarly, NBC didn't have some sort
of server error the exact same way on 8 live feeds over 4 consecutive days. So it is
totally relevant to ask what are the exact conditions that cause VDH to close a download
before it's finished. I think I am being entirely reasonable to expect an answer to this
question.

My next possible conclusion is that VDH has a bug in its logic for processing the
timeout. I already explained upthread here that my timeout settings in VDH ought to be
telling VDH to continue to monitor the download for a bit more than 15 minutes after the
download appears to stall. In contrast, here is what I have observed over the past few
days, as well as numerous other times previously. When VDH is getting to the point where
the blue dot active status screen says the download is getting near the end, I start
watching it closely. With a live stream, it is typical for VDH to display 100% complete
0:00 remaining, and a few seconds later oops here it cycles back to maybe 95% or 96%
complete & a few minutes remaining. I've also seen it drop back to 80% or less but in
the case of these Olympic golf feeds it's been 95% or more. This is entirely not
surprising. Clearly, the VDH download has caught up with the live feed. It has recorded
the live stream from the beginning (remember, this is the weird NBC live stream
technology) & it has caught up to the present. VDH then has to wait for NBC to . . . I
don't know . . . send it another manifest describing the next 5 minutes or whatever of
the live stream. With live streams, I would expect that a web site would send a
manifest, VDH would download the appropriate stream according to what the manifest tells
it, and when that completes, the web site sends a new manifest for the next time interval
of the live stream. Do correct me if I'm wrong. So, I've seen VDH reach 100% complete &
cycle back a bit maybe 2 or 3 times with these Olympic golf live streams, and with other
live streams I've downloaded in the past. Then it reaches the point where VDH just sits
there on 100% for several minutes. I monitor the file activity using Windows Explorer
(the file manager) & I see the file size continuing to grow. Even though VDH has said it
has reached 100%, it does continue to download the stream. But then it quits, as I've
described above. The problem is that the delay between the last time I see the file size
increase & the time VDH ends the download is only a few seconds. VDH seems to be
ignoring the timeout value I've specified in the VDH settings.

My next possible conclusion is one I can't offer. I don't have the imagination. I'm
sure there's more possibilities & I'd be happy to hear about them. But this is my limit.

Since these streams are all behind my cable TV subscription, you over in France can't
test them out. A pity. That means you need to give me some tools for collecting the
information you need to diagnose this situation. I would imagine you would need to give
me something that logs the packets as VDH receives them, something along the lines of
what ffmpeg does. Maybe that would show whether the serving web site has sent VDH a lie
about the stream ending. I don't know whether this sort of instrumentation would be a
separate bit of software I would run or a special debug version of VDH/CoApp that
includes added logic to create the necessary progress log. I have suggested before, and
it seems to me a few other people have as well, that VDH ought to be generating some sort
of progress logging. I know it takes computer power to generate such things, and such
extra logic would make the product larger. Maybe you could include a setting that turns
the logging on & off so people don't get the log unless they want to try to debug a
situation. But I see it as a severe design oversight that you have not had such debug
logging built into your product from the start. Surely you use such instrumentation in
your development activities. Make it available to us so we can debug situations like
this . . . and like every other problem that's reported here. In the worst case, send me
a debug version so I can collect this data for you. I've been a programmer. You can
trust me with something like this.

[Wild Willy]

And another week goes by with no response. Time to bump this again.

[Wild Willy]

I am greatly disappointed by the absence of a response here. I've asked a great number
of questions here. And don't give me the excuse that you don't have time to read such a
lengthy thread. I'm a user with a license. I went to the trouble to write all of this.
There had to be a reason. You have to have faith in me that I wrote this because I
thought I was conveying facts that I thought were relevant. You should make the effort
to read what I have to say & answer my questions. The questions aren't all in one place.
You have to read the whole thing to find them. Please do me the courtesy of responding
here.

[jcv...@gmail.com]

Hi Willy,

I have a call with Mig this morning, I'll talk about this.

jerome

[mig]

Sorry for the delay in my response.

I assume the issue here concerns only HLS-streamed videos.

HLS can be used to stream live media (like a sport event) or static content (like a movie). In both situations, a video manifest describes a set of segments corresponding to smaller audio-video chunks. By playing those audio video segments consecutively, the user sees a continuous flow of media.

In the case of a movie, the manifest contains the description of all segments and if this manifest is downloaded several times, it always returns the same content.

In the "live" situation, the manifest only describes a number of audio-video chunks from the recent past to the present moment. Downloading the manifest several times will show new chunks appearing (last broadcasted content) and old ones disappearing (too old to keep). When the live event is over, the manifest may be unavailable from the server (with various error response codes), or just simply stop updating its content.

Since there is nothing in HLS that allows telling apart live or static videos, VDH must use some algorithms to determine the download job has been completed. Unfortunately, this may lead to some errors when deciding that the download is over.

In the current VDH version, the manifest is reloaded every time all chunks described in the previous manifest have been downloaded.  If no new chunk has been added to the manifest for twice the duration of the longest chunk (or at the minimum, 10 seconds), the download of the  video is considered over and the operation is finalized (i.e. the synchronisation metadata kept in memory are written and the file is closed).

After reading the code again, maybe some configurable parameter could be added to accommodate contexts where the video manifest is updated irregularly, or if network issues cause the timeout to be reached while new data are available.

[Wild Willy]

Better late than never. Thanks for dropping in here, Michel.

My impression is that the combination of the existing VDH settings Download retries &
Retry delay should also be applied. My observation is that it is not. Regardless of the
duration of the longest chunk received so far, this timeout interval in the VDH settings
should be respected. It looks to me like in my case of the chase download, the duration
of the longest chunk changes with each new download of the manifest. The longest chunks
in the manifests found near the end of the stream are quite short. I am speculating that
VDH resets this longest duration chunk value with each new retrieval of a manifest.
Perhaps you should determine a longest chunk with a manifest & change this value only
when the longest chunk gets longer with a subsequent manifest. If a subsequent manifest
has a shorter longest chunk, do not reset the longest chunk value. In other words, make
the longest chunk duration a persistent value for the duration of the stream, not just a
most-recent-manifest calculation. Plus, make sure you respect the 2 VDH settings that
set a timeout. You should use whichever is longer, the longest duration chunk or the
user-specified timeout. Alternatively, you should wait for the duration of the longest
chunk & ADD the user-specified delay to that interval. Maybe that would be excessive.
In the case of the NBC streams, it may be that certain manifests have a longest chunk
that lasts an hour or more. Waiting that long would be rather annoying. I think if the
manifest becomes unavailable, that is a good signal that the stream is ended. But I
would like VDH to wait at least the user-specified timeout even after that. Perhaps a
new manifest would again become available after a few attempts to retrieve one fail. Is
that even a possible architecture? I wish I had some way of determining what NBC is
sending me. Could you create a "manifest log" in the same directory as the download &
let that be a file that persists after the download ends? Just write each manifest into
this log as you receive it & don't erase the file when the download completes. Write a
time stamp on each log entry. Also write into that log the determination that VDH makes
of what the duration of the longest chunk was per manifest. In addition, if you attempt
to read a manifest & that fails, write a log entry with a time stamp noting that event.
This would be controlled by a new setting in VDH that the user could toggle on & off.
Each write would first open the file then append the new manifest, & you would close the
file after each write. As you calculate a new longest duration chunk, you would do
another open/append/close operation to insert the longest duration chunk value. This
would make it possible for me to read the log file as the download progresses. I would
just like to be able to launch the VDH download of one of these NBC streams at any point
during its availability & feel confident it will capture the entirety of the stream. At
the moment, I have to time when I launch the download so it's within the last half hour
or so of the broadcast window, so it lasts past the end of the broadcast window. That
seems to be the only way VDH doesn't terminate the download prematurely. For cases in
which VDH is capturing the real live stream, this may not be an issue. But if you can't
tell the difference between truly live & retrieving from a file, as appears to be the
case with NBC, then it shouldn't matter.

What do you think of these ideas?

[Wild Willy]

Oh Michel. I asked some further questions in response to your last post here. I ask
again, what do you think?

[Wild Willy]

I've got another source of live streams this week. ESPN is covering the US Open tennis
tournament. They have live streams of all the matches. I think. I haven't actually
verified that but it looks like it. They have what they call ESPN3 online. The ESPN3
matches appear to cover the 2 show courts as well as selected other matches on other
courts involving players from the US. The rest of the matches are available on something
they call ESPN+. ESPN3 is available to me by virtue of my having a cable TV subscription
that includes ESPN. I sign in for ESPN3 streams the same way I sign in on the NBC
Sports/Golf Channel site. In fact, signing in on either of those or Tennis Channel
grants me access to the other 2 sites without a further sign in as long as I'm still in
the same browser session. Closing all browser windows appears to clear the sign in.
ESPN+ is behind a secondary paywall & I have not paid so I can't watch those matches,
which unfortunately covers a large percentage of the action. In past years, they offered
pretty much every match through ESPN3 but I guess the salad days are over.

In any case, I picked a live match at random & looked at what VDH was recognizing. It
didn't seem to be working well so I did the usual Network Monitor thing in Firefox &
discovered a manifest. This manifest appeared to be offering 7 different streams at
various resolutions and frame rates. Yes, frame rates. I had not seen that in a
manifest before. Apparently, some of these streams were at 30fps & some were at 60fps.
The only thing VDH recognizes is the BANDWIDTH parameter. According to this manifest,
FRAME-RATE is a valid parameter name as well. It would be nice if VDH would put that
information (when available) in the list of variants on the menu. It already does show
us the BANDWIDTH values recalculated from what's in the manifest to units of Mbps.
FRAME-RATE would be a welcome addition. In any case, after 2 or 3 times clearing hits in
the VDH menu & reloading the web page, VDH finally came up with a range of variants that
looked like it was finally recognizing the manifest. I picked what I thought was the
highest resolution variant & started recording that. It seems that 1280x720 is the best
they've got. Nothing appears to be 1920x1080 or 3840x2160. After a few minutes, VDH
simply stopped recording. I had an MP4 of about 434M whose properties you can see in the
attached image Live Stream Properties. The VLC properties are on the left, the Windows
properties on the right. That's a very weird frame rate in the VLC properties.
Nevertheless, the file played fine in VLC with both video & audio. The problem is that
its duration was not the advertised 2:16:25 you can see in the image. The file stopped
playing at about 12:26. This was reinforced by the date created/date modified attributes
of the file, which showed a span of 12 minutes. The match was not over. The live stream
was still in progress. But VDH abruptly stopped recording it. Very annoying.

As part of this ESPN coverage, they have an archive of completed matches. Only the
matches that were originally on ESPN3 are available for me to view. The ESPN+ matches in
the archive are behind the same paywall as the ESPN+ live streams. I don't know what the
delay is between the end of a match & its appearance in the archive, but it seems to be
short. I also don't know how long these will all be kept available on their site. No
matter. I picked a completed match at random & downloaded it. VDH got an MP4 that was
about 1.71G. In a suspicious coincidence, this download also lasted just 12 minutes.
You can see the interesting properties in attached image Archived Match Properties. The
problem here is that the data is accurate. The download stopped abruptly at the point
where the score was 4-3 in the first set. According to the stats available on the
tournament web site, this match lasted 2 hours & 19 minutes. It ended in straight sets
but both sets were decided in tiebreaks. So VDH once again did something annoying &
didn't download everything.

We need . . . at least, I need a debug version of VDH from you. I described it above.
Let me collect some relevant information for you. There's probably different reasons
between the NBC Sports live streams & the ESPN live streams why they stopped. The NBC
sports live streams aren't live but rather archived files on a server that are being
chase recorded. The ESPN site has both live streams & archived files. VDH is really
getting live streams off ESPN, unlike NBC. I say this because the ESPN recording starts
at the point of whatever you can see in the browser at the moment you launch the VDH
download. NBC gives us the whole broadcast from the beginning no matter when you start
recording within the broadcast window. That's actually kind of a nice feature of NBC,
but those recordings stop prematurely as well. More surprising is that the archived
broadcasts on ESPN don't download the complete files either.

So I'm here again. Begging. Give me a debug version of VDH. My suspicion is that
whatever we discover about the NBC & ESPN streams will transfer to other sites. Please.
Let's do this.

[Wild Willy]

Just for yucks, I went back to that archived tennis match at ESPN. Instead of trying to
use VDH, I decided to see what would happen if I tried to use ffmpeg directly. I opened
the Network Monitor, filtered on m3u8, then clicked the play button on the video. Once I
had a manifest in the monitor, I paused playback of the match. Curiously, I had an ever
growing list in the Network Monitor, but they were all GIFs except for the first entry,
which I have attached here as Tennis Manifest.txt. Very shady that a bunch of GIFs all
have the string m3u8 in their file names. You'll note there is a
#EXT-X-MEDIA:TYPE=AUDIO,GROUP-ID="aac" entry in the manifest. Normally, this would be an
audio-only stream. But look closely. There is no URL with this entry. I don't know why
it's here. All the video entries in the manifest refer back to this one via the
parameter AUDIO="aac" but that doesn't seem necessary. There is an I-FRAME entry, but
only one. In the other instances where I have encountered these I-FRAME entries, there's
been one for each regular video entry. Curious there's only one here. So many
mysteries.

After carefully looking at the entries, I picked the one that had these parameters:

RESOLUTION=1280x720,BANDWIDTH=9624718,FRAME-RATE=60.000

I took that combination as an indication that it was the highest resolution video being
offered here. So I used the URL from that entry as my input to ffmpeg. I'm attaching
that URL here as a separate item just because it is so ugly. 2,688 characters. For a
simple URL! Good grief! You can marvel at it in attachment Tennis URL.txt.

The download via ffmpeg eventually completed after 52 minutes. You can see its size &
all that stuff in attached image Tennis Directory. I've also attached a screenshot of
the properties similar to the properties attachments in my preceding post. This data
reflects what I really retrieved. It played fine in VLC, with both video & audio, all
the way to the end, for the entire 2 and a half+ hours. Both the aforementioned
tiebreaks were there. Much as I love Garbiñe, I didn't actually sit & watch the match.
I already knew the result so there wasn't much point in spending the 2-plus hours to
watch it. I did notice, though, that the ads were in here as well. Generally, with the
NBC/Golf Channel streams, the ad breaks are replaced with "Coverage will resume shortly."
It's as if NBC doesn't want online viewers to see the ads. Maybe they think the
advertisers should pay extra to get NBC to include them in their live streams. This is
clearly not an issue for ESPN because all the ads are here. Of course, if I were to
watch one of these things, I would be using the VLC keyboard shortcuts for skipping over
the ads.

Just to be complete, I've attached the ffmpeg log file. It may be of some interest.

Clearly, you should be wondering why ffmpeg can download the whole file but VDH can't
download for any longer than 12 minutes. The necessary URL is in the manifest. What is
VDH doing wrong? I'm sure a debug version of VDH would help us get to the bottom of this
mystery.

[mig]

I just pushed a new beta for Firefox. Version 7.6.3a1 from https://www.downloadhelper.net/firefox/betas

In this version:

- the reference for chunk done is the longest one for the video and not only for the chunks currently described in the manifest

- the minimum timeout is 40 seconds and no longer 10 secs

- additional traces in the console log to show 1) whenever a chunk is found to be the longest one in the video 2) the number of new chunks discovered in the latest manifest download

Does this improve the VDH download in your case ?

[Wild Willy]

I'll test this a bit in the next day or two. When you say "console log," are you talking
about Tools -> Browser Tools -> Browser Console? I've never thought to look there. Has
that always been something that has been useful with VDH? If yes, I'll add this idea to
the Table of Contents thread. Wait. I mean I'll add it if it's useful. Whether it's
always been useful doesn't affect whether I add an entry to the other thread. But if VDH
has always put things in there, I wish that had been common knowledge all along.

[Wild Willy]

For my first test, I went to the ESPN archive of completed US Open tennis matches &
picked one at random to download. VDH 7.6.3a1 beta spent 26 minutes downloading a file
that ended up being 3.16G, of duration 1:22:01, & resolution 1280x720. The file played
fine until the score was 4-5 in the second set on serve. In the middle of a commercial
break, the download abruptly ended. Problem is the match actually went 3 sets & lasted 2
hours & 15 minutes. I suppose this is progress, given that my last attempt to download
one of these archived matches quit after 12 minutes. But it's still a failed VDH
download.

You haven't had a chance yet to answer my question about the Browser Console but I went
ahead & left it open for the duration of the download. Then I selected the contents of
the window & copy/pasted it into a text file, which I have attached. I don't know if
maybe I should select only certain types of records or maybe filter on something, so what
is attached is every type of record the window can contain. I figured since it's fairly
small, there's no harm in attaching it here. If it's useless, oh well. No harm done.

Also, during that last ad where the download cut off, the video suddenly sped up while
the audio remained fine. The entire recording up to that point seems fine, although I
must confess I only sampled it, not watched it all the way through. I did play the very
end of the file. It turns out that even though the file properties said the duration was
1:22:01, it actually stopped playing at 1:21:28. At 1:20:58, play stopped for a
changeover & they went to a commercial. At the instant the ad started, the video went to
double speed while the audio stayed standard. I'm wondering now if there isn't
something VDH is doing to cause this problem after all. I thought it was a problem at
the source but now I'm not so sure. I saw it again with one of the 6 downloads I've done
over the weekend from NBC Sports. (I'm actually watching the Solheim Cup coverage today
on TV, not the online feed, nor downloading it.) The file in question was perfect for
about 3 hours & 20 minutes, then the last 2+ hours had perfect audio playing against
double speed video. To watch the last chunk of the file, I had to do what I describe
over here:

https://groups.google.com/g/video-downloadhelper-q-and-a/c/w9993gUmxr4

The other 5 live streams I downloaded from NBC Sports this weekend were fine. It's
certainly an intermittent problem but it's kind of suspicious that it happens on
downloaded streams that play fine if you just sit & watch them in a browser window. So
in just a couple of days, I've seen this mismatched speed problem once on a recorded live
stream from NBC Sports & once on a static file download from ESPN. I don't know if you
can think of any sort of debug logging information you could generate that might help
figure out whether this is an issue with VDH. I have to say I'm just suspicious at this
point, not certain.

I will pick a tennis match at some point in the next few days to try to make a live
recording from ESPN & see what happens. After rhe Solheim Cup returns to Europe today,
there is no LPGA tournament this coming weekend. So I will next try to record a live
stream of an LPGA tournament from NBC Sports/Golf Channel the following weekend.

[Wild Willy]

I decided to watch a live tennis match tonight & I took advantage of the opportunity to
record it with VDH at the same time as I watched the live stream online. Just my luck,
it ended up being the latest time a women's match has ended in the history of the US
Open, about 2:15 AM. You seem to have fixed the problem as I observed it previously on
ESPN live streams. I ended up with an MP4 of size 7.08G, duration 3:44:12, resolution
1280x720. Such a coincidence, it took 3:44 to record, too. Funny how that works. I
played the MP4 back in VLC & sampled it at 1 minute intervals. I wasn't about to sit &
watch the recording all the way through since I had just spent nearly 4 hours watching
the match live. It looked like the video & audio were just fine throughout. I left the
VDH active download status menu open for most of the time I was watching the live stream.
Easy to do with 2 monitors. Every time I looked, VDH was saying that the download was
100% complete and 0:00 time remained to completion. I think this is entirely as
expected. My bandwidth is such that the download is permanently caught up with the live
stream. I would imagine that the appearance of the manifest for the next block of the
stream would keep VDH connected. I guess their chunks are so small that each segment of
the live stream got downloaded essentially as fast as it appeared. I just think the
100%/0:00 were amusing.

They didn't play many ads during this stream, unusual but most welcome. The players sit
down for a 90-second break after every other game. This is just the way pro tennis
works. This is the perfect opportunity for the broadcaster to run ads, and normally
that's what you get. But for some reason, they kept the cameras rolling on court during
most of the rest periods. Maybe the actual broadcast on TV had commercials, but I was
watching the live stream online. Under normal circumstances, I would probably have just
watched ESPN through my cable TV service, but I wanted to run this test. So I'll never
know if the regular cable broadcast benefited from the absence of ads. I suspect they
ran ads on TV & it was just the online stream that was nearly ad-free.

But I did notice something rather odd during the few ads I happened to hit during my
sampling. The video of the ads was fine but the audio was bad. It pulsed, 2 seconds on,
2 seconds off. This is not what I heard when I was watching the live stream. When they
came out of the ads, thankfully, the audio reverted to normal during the tennis, and of
course the video continued to be fine.

The recording ended normally after the copyright notice at the end & after 3 or 4 ads
that appeared after the program ended. These ads played fine, with good video & good
audio, no pulsing. Until they got to the last one. About a minute or so from the end,
the ad was playing with normal audio but double-speed video. I suppose if I were
recording this for real, I would be thankful that the dual-speed problem waited until
after the match was over before it appeared. Out of curiosity, I kept the live stream
playing after VDH had closed the download. The ads played with good video & good audio.
There were several more ads on the live stream than got recorded, too. Eventually, the
ads stopped & they streamed an image saying, "Your event has ended. Thanks for
watching." I left that playing for over half an hour & it would probably still be
playing in Firefox if I hadn't finally closed it.

That raises the question, how did VDH recognize that the stream had ended? As far as I
could tell, they streamed several more ads in the live stream that VDH did not record.
Not that I'm desperate to watch ads but the live stream seemed to still be running. In
fact, it looked like the live stream never did really end. Maybe the manifests stopped
updating. Is there some way VDH generates a log of this activity? And then there's this
dual-speed problem again. I'm beginning to get more suspicious about this. I had never
seen this before about 3 weeks ago, when I first posted about it. Now here I've seen it
3 times in the past couple of days. I'm still not totally convinced this is a VDH
problem, but then I'm not totally convinced that it isn't, either. Some diagnostic
logging from VDH could end the mystery.

[mig]

When you say "console log," are you talking

about Tools -> Browser Tools -> Browser Console?

From the Firefox menu (top right), pick Addons and Themes, in the add-ons tab, click the gear icon at the top right and pick Debug add-ons. Note that you can skip this and open directly about:debugging#/runtime/this-firefox

Locate VDH and click the corresponding Inspect button. You should now have have access to the console where the extension log appears.

In normal situations, this log is silent (at least for traces coming from VDH). Traces have been added to the 7.6.3a1 only version to help understanding this particular issue. But in general, if VDH is acting weird, it's worth giving it a look in case an error message is displayed there.

[mig]

Unfortunately, debugging this kind of issue without having a reproducible test case during development is more than hard. Adding traces with you reporting the resulting trace won't work. If we can find a test case that is runnable from France, either directly or through a VPN, then we will have a chance to debug it.

[Wild Willy]

OK. Found it. It sure looks a lot like the regular Network Monitor you can get from
F12. I'll be sure I have that open next time I download a live stream. This whole
business of international restrictions & cable TV subscriptions is rather getting in the
way here. I'll keep reporting whatever problems I see. The dual-speed videos are
particularly annoying. Maybe one time you'll stumble upon something in the code while
you're looking for something else & you'll figure it out.

What will you want from that console log? Just copy/paste whatever is in there? Any
special filtering or additional clicks I ought to do?

[mig]

The traces i added were intended to understand the early download ending. They probably won't help for the speed issue.

The logs i am looking for are the latest messages chunkDuration XXX and chunksAddedCount XXX before the download stops too early.

[Wild Willy]

For anybody else who might want to participate, this is how to do what Michel is talking
about. Some of the wording is a bit different. This could be down to minor variations
in Firefox between Linux & Windows. Also, I just got Firefox 92.0 this morning, which
may also have caused some minor changes in wording.

[Wild Willy]

OK. Sounds like I can filter on chunks & post that. Will those messages come out under
the Network tab or should I look in one of the others, like Inspector,
Console, Debugger . . ?

[Wild Willy]

Is there any way you can change the title of this thread to something more meaningful?
Like

Debugging premature live stream termination

[mig]

Is there any way you can change the title of this thread to something more meaningful?

Apparently not. The more it goes, the less features we have in Google Groups :(

[mig]

Will those messages come out under
the Network tab or should I look in one of the others, like Inspector,
Console, Debugger . .

Messages will show in the console.

[Wild Willy]

A tennis match completed earlier this afternoon happens to be one I'd like to see. So
first, I opened the VDH toolbox console like you explained above. Then I went to the
page at ESPN on which you could play the match. Since I expected the VDH download to
fail, I did the usual trick with the Firefox Network Monitor to find the manifest &
interpret it to get the URL of the highest quality stream they were offering. No
manifest showed up until I launched playback for a couple of seconds. Once I did that, I
had a manifest that showed the best stream was resolution 1280x720. Curiously, the
manifest claimed this stream was 60fps but ffmpeg detected 24fps. Until the download
completed, I reserved my judgement. I extracted the relevant URL & launched my usual
script that downloads a stream using ffmpeg.

Once I had that running, I closed the Firefox Network Monitor & looked at the VDH menu.
I was surprised that there was nothing there that looked like the video I wanted. I had
already launched playback but VDH wasn't detecting anything. So I refreshed the page.
That made exactly one variant appear even without launching playback again. That
surprised me because the manifest showed 7 streams available. Whatever. I told VDH to
download the one it found. After only 11 minutes, VDH said it was done. The MP4 file
was 1.27G, duration 1:52, resolution 960x540, frame rate 30fps. According to ffmpeg,
this match lasted 2:50. So VDH came up 1 hour short. After the VDH download completed,
I looked in the debug console. The contents of it were so short, I'll just paste it
directly into this post:

> 17:46:15.143
> Unknown localization message weh_prefs_label_downloadcompletedelay main.js:1
> 17:46:15.143
> Unknown localization message weh_prefs_description_downloadcompletedelay main.js:1
> 17:46:15.143
> Unknown localization message weh_prefs_label_tbvwsgrabdelay main.js:1
> 17:46:15.144
> Unknown localization message weh_prefs_description_tbvwsgrabdelay main.js:1
> 17:46:15.144
> Unknown localization message weh_prefs_label_forcedcoappversion main.js:1
> 17:46:15.144
> Unknown localization message weh_prefs_description_forcedcoappversion main.js:1
> 17:49:09.804 chunkDuration 2048 main.js:1:137107
> 17:49:09.804 chunkDuration 4096 main.js:1:137107
> 17:49:09.809 chunksAddedCount 2586 main.js:1:137447
> 17:49:12.917
> Unknown localization message quality_unspecified main.js:1
> 17:49:45.326 pagedata-script execution error Missing host permission for the tab main.js:1:172458
> 17:49:54.304 chunksAddedCount 2586 main.js:1:137447
> 17:49:54.307 pagedata-script execution error Missing host permission for the tab main.js:1:172458
> 17:57:58.341 chunkDuration 2048 main.js:1:137107
> 17:57:58.342 chunkDuration 4096 main.js:1:137107
> 17:57:58.352 chunksAddedCount 2586 main.js:1:137447
> 17:59:12.356
> Error: Please use $(ref:runtime.getURL). main.js:1
> 17:59:12.358
> Error: Please use $(ref:runtime.getURL). main.js:1
> 18:10:02.715
> Error: Please use $(ref:runtime.getURL). main.js:1

I hope this is meaningful.

Out of curiosity, after the VDH download was complete, I launched playback of the video
in the browser & let it play a couple of seconds. This did make VDH recognize the 7
streams I saw in the manifest. It's weird that launching playback before that didn't
make VDH recognize all the variants. I did an initial page load, a launch of the stream
for a couple of seconds then pause, and VDH recognized nothing. Then I did page reload,
and that got a low-res variant to appear. Then I launched the stream for a couple of
seconds a second time, and VDH finally recognized all the variants. I can only assume
that ESPN does something particularly weird & it just takes persistence to get it to tell
VDH whatever it is that you need. Like I said, a manifest I opened became available
after the first launch of playback.

While my ffmpeg download continued, I played back the abbreviated match as VDH had
retrieved it. I only sampled it because I didn't want to spoil my viewing of the match
from the other download. It seemed to be fine with both video & audio & no speed
problems. This appeared to be another one of those mostly ad-free streams. My sampling
interval was 1 minute so I should have run into whatever ads were there. In the slightly
less than 2 hours I had, I hit only 1 ad. It had the pulsing audio problem. The match
play all seemed to have perfectly fine audio.

The ffmpeg download completed in 52 minutes. The MP4 file was 5.83G, duration 2:49:57,
resolution 1280x720, frame rate 58fps. Odd that ffmpeg detected 24fps at the start of
the download but the file ended up as 58fps. I did ffprobe on the file & it showed
58fps, the Windows properties showed 58fps, & the VLC properties showed 58.753804fps. So
they all agreed on the frame rate. At this point, I've only sampled part of the ffmpeg
download, which does appear to play fine. I have to go out to run an errand so I won't
actually sit & watch this until later. But I did find the one ad in the ffmpeg download
that I had found in the VDH download. The pulsing audio is not in the ffmpeg version. I
did skip to the end of the file & found the closing copyright notice (without
accidentally seeing the final score). There were a couple of ads tacked on after that &
they also played fine, no pulsing audio, no speed problems. I think there is something
going on with VDH & the ads here. I don't know if ESPN might be inserting the ads in
such a way that the stream has different properties while an ad is playing. Whatever the
case, ffmpeg seems impervious to it while VDH has a bit of a problem with it.

I was curious to know how many chunks ffmpeg retrieved. Usually, the ffmpeg log shows
that the web site has used a naming convention for the URLs that includes a sequentially
incrementing number somewhere within the URL. But in this case, the chunks seem to be
coming from 2 different sites, and each one has its own naming convention. So I couldn't
just read off the number from the last ffmpeg log entry the way I can with the opera
downloads. I used the search facility of my text editor (Notepad++) to count the
occurrences of the 2 site names. This may not be correct. For all I know, somewhere in
this huge log it may have gotten chunks from more sites than just the 2 I noticed. My
best guess is there were something like 2600 chunks in this download. I can say that the
download consisted of 599,142 frames. That's a number you can read straight off the last
ffmpeg log line before the completion of the download.

I do so wish you had access to this. I'm sure you'd figure this out without too much
trouble. It's too bad both Golf Channel & ESPN are inside a cable subscription that
isn't available outside this country.

[Martin]

Ive been having similar trouble with non-subscription ESPN downloads of tennis.  The video stops after about an hour right as the first commercial starts.  You can try this one.  (It will only work for about a week then will probably get locked down.)

https://www.espn.com/watch/player/_/id/6bb052e1-3775-4a56-9f97-bbef4decafd7

[Wild Willy]

> ** Reply to message from Martin <feld...@gmail.com> on Sun, 12 Sep 2021 16:10:20 -0700 (PDT)

I'm very glad to hear another voice in this conversation.

I'm not sure what you mean by "non-subscription." I clicked that link & it gave me the
Medvedev/Auger-Aliassime match, but before I could even press play on it, it was
prompting me to log in with my cable provider. In other words, when I visited the link,
it verified that I did indeed have a subscription. In any case, I'm not going to mess
with it. I've done several tests & now it's up to Michel to react. He asked for some
debug output & I posted it above. Maybe he's quietly working behind the scenes on
something suggested by that debug output. Maybe it didn't tell him anything. We haven't
heard back from him since I posted that so it's all a mystery at this point. I'll be
trying some things with next weekend's LPGA tournament from Golf Channel/NBC Sports.
Their content seems to be provided differently from how ESPN does it. If Michel
specifically posts another VDH beta or CoApp beta, and asks for another test on ESPN,
I'll do that. But for now, I'm just sitting tight & using my bandwidth for other things.

[Wild Willy]

Golf day.

I've explained before, but it's worth repeating as a reminder, that NBC Sports/Golf
Channel makes their live stream pages available 15 minutes in advance of their actual
broadcast window. Today's tournament is being played in Portland, Oregon. The broadcast
window today starts at noon local time there, 15:00 my time. The broadcast window is 3
hours today. So at 14:45, I launched their live stream & started the recording via VDH.

At the same time, I tried to use the manifest offered by the live stream. Sadly, when I
tried to use one of the URLs out of the m3u8 file, ffmpeg got a 403 Forbidden error. I
tried it twice using 2 different manifests I got a couple of minutes apart & it did the
same thing both times. Looks like NBC's security is a bit stronger than ESPN's. I do
wish I could have done a parallel recording with ffmpeg to compare with what VDH does.
Oh well. Win some, lose some.

The download went fine for about 2 hours & 7 minutes. Pretty much the whole time, VDH
reported 100% complete, 0:00 remaining to download. This is entirely as expected,
entirely normal. It's a live stream. I have a fast enough connection that I am
retrieving the stream about as fast as it's being put on the line. Then the download
abruptly stopped. I had an MP4 of size 3.86G, duration 2:51:27, resolution 1280x720.
This is not up to standard for resolution but it was the only variant offered at the time
I launched the download. This is not the usual situation. The file played fine, both
video & audio, for the full length of the file. It started with a lengthy section of
"Coverage will begin shortly," as it always does, & then there was not quite 2 hours of
actual program broadcast. Fortunately, plenty of time was left in the broadcast window
so I was able to launch the VDH download again. This time, VDH offered the full array of
available resolutions so I got the 1920x1080 one. Maybe during that 15 minutes of
pre-broadcast, VDH isn't recognizing all the resolutions. No, that can't be it. I got 2
different manifests during that time, hoping to use them with ffmpeg as I said above, &
they both showed a bunch of resolutions. I should have counted. But there were at least
6 different resolutions, yet VDH had offered me only one, and that one menu entry did not
show a resolution. I think there's a problem there. In any case, the download I didn't
launch until 17:30 ran for 1 hour & 12 minutes & resulted in an MP4 of size 8.79G,
duration 4:19:54, resolution 1920x1080, frame rate 30fps. It contained the entire 3
hours of what went on the air, preceded & followed by the junk they attach to the file.

Bottom line, this didn't work. But it looked so promising for 2 hours. Oops. Sorry. I
forgot to open the debugging console on the first attempt. Smack me. There's the second
round of the tournament tomorrow. There was no point in opening the debug console for
the later recording since that always works. I'll try again tomorrow.

[Wild Willy]

I've launched the recording of today's round of golf at 14:49. It's running now. I
remembered to open the debug console. Yay! I expect this will fail in a couple of hours.

In the meantime, I've noticed some things about the manifest NBC gives us here. I've
attached it for you to analyze along with me. I've also attached 2 screenshots of the
VDH menu this is giving me. It took 2 screenshots because I had to scroll it.

At the top of the manifest there's #EXT-X-MEDIA:TYPE=CLOSED-CAPTIONS. VDH doesn't
process captions, subtitles, call them what you will. That's fine. This entry in the
manifest does NOT contain a URL so VDH couldn't do anything with it anyway. Which is
moot because these are closed captions. In other words, the captions are embedded in the
media. They are not a separately downloadable entity. In ffmpeg's notation, a stream
(let's say for program 2) would consist of stream 2:0 of video, 2:1 of audio, & 2:2 of
closed captions. VDH would simply see this as part of the stream it's downloading. VDH
wouldn't process this in any way. It would just include that data in whatever it is
saving to my hard drive. I wouldn't have to do anything about it. When it comes time to
play the resulting MP4 in VLC, I could turn the captions on & off at will. They would
not be a separate file. Since the captions are not their own stream, I question why
there is even an entry in the manifest for them. Maybe the video player embedded in the
Golf Channel web page uses it in some way. Nevertheless, this entry contains the
identifier GROUP-ID="cc". Every subsequent entry in the manifest refers back to these
captions with the reference CLOSED-CAPTIONS="cc". This is not information that VDH can
act upon. It must be something used by the embedded video player.

There's a number of entries in the manifest for #EXT-X-I-FRAME-STREAM-INF. In the big
opera thread

https://groups.google.com/g/video-downloadhelper-q-and-a/c/8V2cRB-bcK4

I did find out what these I-FRAME things are. There's a link in there to an external
location where I had a conversation with somebody who seemed to know what they're for. I
refer you to that. VDH should ignore these I-FRAME things, and it does. They are not
streams usable by the end user. Unfortunately, the manifest is littered with these
things so you need to concentrate on blocking them out of your consciousness.

Then we come to the information VDH can use. I see entries for the following resolutions:

1920x1080, 1280x720, 1024x576, 896x504, 640x360, 512x288, 384x216, 256x144

That's 8 resolutions. But look at the VDH menu. There's 16 variants being offered.
This is double what's correct. Why?

Here's my speculation on what the problem is. If you look carefully, for each resolution
there's actually TWO #EXT-X-STREAM-INF entries. You can link them together by both their
RESOLUTION parameter values as well as their BANDWIDTH parameter values. But look
closely at the first entry of each pair. It does NOT contain a URL. It is NOT a stream
descriptor. The second entry of each pair is a stream descriptor. I believe VDH is
mistakenly interpreting the first #EXT-X-STREAM-INF descriptor in each pair and
ERRONEOUSLY placing an entry for it in the VDH menu. It's noise. It's a duplicate. The
value in the exp parameter of the first entry in each pair appears to be the same as the
URL that appears as part of the second entry of each pair. Well, not quite the same.
The exp value in the first of each pair doesn't start with https:// like the second of
each pair does. But I say these are duplicates & shouldn't be cluttering up the VDH
menu. As a test, I launched the download of the duplicate of the 1920x1080 stream I had
started about an hour earlier. I was a bit surprised to discover that it actually
launched. VDH must do something with the exp parameter value to make it work. This
second download is no doubt cluttering up the debug console. Sorry about that.

But this reminds me of something I see with nearly every video on YouTube. I typically
see a bunch of entries on YouTube that VDH puts in its menu as simply MP4. There's no
resolution, no duration, no file size. The VDH menu entry says simply MP4. Often as
not, attempting to download one of these things results in an immediate error. I'm
wondering if that's another manifestation of what I'm seeing here with NBC. I can't tell
because I haven't found any m3u8 manifests on YouTube so I don't know how you figure out
what to do on YouTube. This is why I'm guessing.

I'd be most interested to hear your take on this. Am I analyzing this right or am I
completely wrong?

Oh. My second download of what I believed was a duplicate stream just completed. I
guess what I thought would be clutter in the debug console is not clutter after all.
It's probably helpful to note that this download terminated at 15:59. There's an error
in the debug console right then. I don't understand it. Maybe it means something to
you. Anyway, I have an MP4 that took 21 minutes to download, is 2.18G, duration 1:34:57,
resolution 1920x1080, frame rate 30fps. It starts with about 54 minutes of the usual
"Coverage will begin shortly" & then there's actual program content, which lasts only
about 40 minutes. Obviously, this is wrong. My first download is still in progress,
which is encouraging for the moment. Also, this is once again, same as yesterday, a 3
hour broadcast window. This clip of only 40 minutes clearly isn't what I should be
getting. It's playable, looks & sounds just fine, but it's only a fragment of what I'm
trying to get. It will be interesting to see what happens with the download that
continues as I write this. It's approaching 90 minutes & continues normally . . . so
far. These streams are of the type that won't play in VLC while the download is in
progress. Some downloads can play in VLC while the download is still running, but this
one happens to not be like that. I'd be curious to know if there's some way you can tell
without trying it whether you'll be able to chase play a download during the download.
In any case, since I can't play it yet, I'm assuming the download still in progress is
going along normally.

[Wild Willy]

I just noticed that the failed download I reported earlier generated this in the VDH
error log:

Requested download of chunked stream, but no chunkset found.

Sounds like a transient error at the other end.

Meanwhile, here's the status of the other download I mentioned above. It ran from 14:49,
until 18:02, when it terminated without error. Today's broadcast window ran from 15:00 -
18:00, 3 hours. So VDH recorded for 3 hours & 13 minutes. I have an MP4 of size 8.48G,
duration 3:57:41, resolution 1920x1080, frame rate 30fps. The approximately 58 minutes
of extra duration in the file is due to the approximately 54 minutes of "Coverage will
begin shortly" at the beginning, and about 2 minutes of "Coverage has ended" at the end.
(I skipped around in the file using VLC specifically looking for these things. I'll
watch the whole thing all the way through in a bit.) The existence of these stretches of
junk says the full broadcast is there as it is supposed to be. But 54+2 doesn't add up
to the 58 minutes of excess content. Here's the rest of the story. At about the 54
minute mark, there is a short snippet of coverage of the tournament. Apparently, they
ran a short teaser just before the broadcast window. It lasted maybe 30 seconds, & then
there's a bunch of ads. Not all of the teaser is there. Apparently somebody didn't push
the button in time to put the beginning of the teaser into the online live stream. So
the live stream joins the teaser "already in progress" as they like to say. The actual
broadcast follows. Unlike their streams in the past, they are putting the full content
of the ads into the live stream, both yesterday & today. In the past, the ad breaks were
covered by the message, "Coverage will resume shortly." This caused most of the returns
from ad breaks to cut off a few seconds of the tournament play. I skip over the ads
either way but I think I prefer the ads being included. This way, there isn't the
truncation between the ad break & the resumption of the coverage of play. It seems like
it would be easier for them to just always let the ads play in the online stream. I
don't know why they ever did it the other way.

Curiously, the debug console log (attached) ends with another error, the same one that
occurred when the other download failed. But this one was NOT accompanied by an error in
the VDH error log. Seems obvious to me that VDH was able to determine differences in the
circumstances. What might those be?

In other words, today it worked fine. I did have that failure I accidentally captured,
as I described earlier. But my main download worked. Yesterday it failed. Today it
worked. I hate that. If something is going to work, it should work all the time. If
something is going to fail, it should fail all the time. The present situation does not
engender the highest level of confidence in me. Can it really all be down to transient
errors from the server? Maybe there's a little more fault tolerance you can build into
VDH so that one of these server errors doesn't necessarily terminate the recording of a
live stream. When one of these errors occurs, isn't it possible to keep prodding the
server a few times to see if it will send another manifest? In this context, manifest &
chunkset are synonymous, right? When this error occurs, does the server actually
terminate the connection from its end? Is this something you could log? Since there
seem to be different types of errors, I should think you'd want to log the reasons for
the errors. You'd also want to log when the connection is terminated & which end that
termination came from, the server or VDH. Are servers really so unreliable that every
time a live stream recording gets dropped prematurely, it's because the server suffered
an error? Let's log it & find out.

I've just copy/pasted the whole debug console contents into the attachment. If there's
things I should filter on, let me know.

Jerome, I know you're reading this. It looks like you need to go poke The Boss again. I
posted stuff here a week ago & he hasn't said boo about it. I posted stuff yesterday &
I'm posting more stuff now. Do please get him to come back here & pay this some more
attention. Thanks.

[jcv...@gmail.com]

I'll poke him, but I know is working on this.

jerome

[mig]

Sorry but debugging the issue with HLS on VDH without being able to test it by myself is very hard.

Instead, to start with, I am currently working on a feature for Firefox beta using the ffmpeg capabilities to reconstruct a video file directly from the HLS media manifest. I'll let you know here when it can be tested.

[Wild Willy]

That sounds intriguing. You may also want to take a peek at this thread:

https://groups.google.com/g/video-downloadhelper-q-and-a/c/uBknY74Q1SI

There's a discussion of VDH's use of ffmpeg over there. A number of erroneous statements
might have been made there, by me & others.

[Wild Willy]

I have some thoughts on the design of this new feature.

You need to add a new setting to the Behavior tab of VDH Settings. This field will be
just like the Default download directory. It will be where you have the user enter the
location of ffmpeg, complete with a Change button so the user can go through a file
selection dialog to find his ffmpeg. Yes, this implies that the user has gone to
ffmpeg.org & gotten her own copy of standard ffmpeg. Just to make it as simple as
possible, it should result in the full path to ffmpeg.exe, not just the directory
containing it. In case you missed it, I am suggesting that VDH use the standard ffmpeg
for the download. Whether to use standard ffmpeg for the download might be a new VDH
setting that the user can toggle. If you continue to include a customized ffmpeg within
VDH/CoApp, you can use it for whatever functions you want to use it for. But for
downloads using ffmpeg, I say you should use the standard ffmpeg. Leave it to the user
to maintain this standard ffmpeg at whatever version level may be desired. Any errors in
this area are on the user to correct. Don't make any attempt to validate this.

Please capture the ffmpeg log into a file in the download directory. For example, if I
download into a particular directory a file named:

LPGA Round #1.mp4

Then VDH should let ffmpeg write its output log into the same directory in a file named:

LPGA Round #1 ffmpeg log.txt

If the log file already exists, that means the user has attempted to download something
into the same mp4 file name previously. Then either of 2 things is possible. (1) You
already prompted me to replace the mp4 file & I said YES. (2) I deleted the mp4 file
myself before telling VDH to download it again, thus avoiding the dialog prompt in VDH,
but I didn't also delete the earlier ffmpeg log. Either way, the existing ffmpeg log is
no longer useful & you should erase it & replace it without prompting the user about it.
The important idea is to keep the ffmpeg logging feature. It is useful. If necessary,
let the user turn the logging on & off via a new VDH setting.

Provide 3 new VDH settings, one for supplying global ffmpeg switches, one for supplying
ffmpeg switches for the input file, one for supplying ffmpeg switches for the output
file. I am assuming VDH will use ffmpeg with a single input file & a single output file,
even though ffmpeg supports more complex processing. Your goal is to download videos,
not provide a generalized wrapper for ffmpeg. If the user wants to get more complex,
it's on the user to do that outside the context of VDH. I have not been able to find an
explanation in the ffmpeg documentation of what happens if the same switch is supplied
twice. I've looked. I don't know if the first instance applies & the second is ignored,
or vice versa. Whichever way it works, VDH should place the user's specified switches in
the position that makes them take precedence over the same switches that you might code
internally in VDH or the CoApp. Do not attempt to validate or otherwise process these
switches & their values. It is on the user to get them right. The ffmpeg log will be
crucial in the user's attempt to figure out if & when he gets any of these switches
wrong. In case you're reading this, Maro, this is how you would get your -hwaccel
option. Please document all ffmpeg switches & values you do code inside VDH/CoApp.

[mig]

Thanks for the suggestions but there are good reasons why we don't rely on custom-installed ffmpeg (nor offer GPU-enabled ffmpeg versions). One of them being providing support on our product as we have limited resources.

We will probably offer what you suggest at some point but it will be limited to Linux (for which we don't have to offer support as we don't sell the product).

Sorry for that.

[Wild Willy]

I don't understand. Didn't you post on September 27 that you are working on using ffmpeg
to download videos? I'm confused now.

And Linux only? I think I indicated that even if you do it for Windows, a great deal of
the burden of support would be on the user.

[mig]

Yes, i'm working on being able to download HLS streams using ffmpeg directly but with the ffmpeg version embedded into the coapp. Using another release of ffmpeg that would have been installed directly by the user is not on the agenda.

I said we may provide, on Linux only, the ability to tell the coapp which ffmpeg binary to use. This is because Linux users are in average much more tech-savvy than Windows ones (i said "in average" !) and since we give the Linux version for free, people cannot get too angry after us if it does not work after they experiment exotic configurations.

[Wild Willy]

You said above that you don't want to rely on "custom-installed ffmpeg." But you rely on
custom-installed Firefox, custom-installed Chrome, custom-installed Edge. For that
matter, you rely on custom-installed Windows, custom-installed Mac, custom-installed
Linux. There's no big deal to another custom-installed piece of software. Besides,
installing ffmpeg consists of downloading the package from ffmpeg.org & unzipping it.
That installs it. There isn't even an installer. You just unzip it.

I'm envisioning this positioned as a third download processor alternative: alternative
#1=browser, alternative #2=CoApp, alternative #3=ffmpeg. You could say ffmpeg is only
for advanced users & there's not going to be a whole lot of support. Come on, you have
to permit some support if we find that you didn't do something right, like pick up the
parameters correctly or use the specified ffmpeg location correctly. But when it comes
to actually using it, well, you've got a whole bunch of unpaid support personnel right
here in the support forum. You needn't be concerned about supporting it, even on
Windows. You've demonstrated that VDH is portable across platforms. Don't unnecessarily
create a fork just for this. Personally, I would use the CoApp most of the time. I
would resort to ffmpeg only if the CoApp seems to be having trouble with a particular
download.

As an aside, I did something you might not approve of & I certainly wouldn't expect you
to support me in this. But a few days ago I replaced your custom ffmpeg in the CoApp
directory with the standard ffmpeg. It seems to be working just fine. I suspect I've
simply not used any particular feature of VDH that requires your customizations to ffmpeg
found in the version you supply with the CoApp. What features of VDH would simply not
work if I were to try them in my current configuration? I didn't delete your custom
ffmpeg, just renamed it in case I hit a situation in which I would need it. With the
standard ffmpeg in place in the CoApp directory, I have downloaded from YouTube,
downloaded simple MP4 files, & downloaded Golf Channel streams (I'm getting one right now
as I write this, it's about to complete within the next 10 minutes). I'm guessing you
don't even use your customizations in ffmpeg in these cases. So what did I lose by
replacing the custom ffmpeg with the standard one?

[mig]

Not sure we understand each other.

Anyway, feel free to hack the system to replace the coapp ffmpeg with your own flavour. You're on your own. Just understand that we don't want to provide tools to help in the hack (configuring the ffmpeg path) because we would also have to provide support for fixing problems when it won't work. And in this area, it gets complicated very quickly. We just won't be able to do so. This is why we won't provide the "flexible-ffmpeg" feature you are asking for.

So for the time being, the plan is to offer the ability to download HLS streams using the coapp ffmpeg to do some of the work currently done by the extension. According to your tests (verified by myself), this should solve the issue you are having with the recording ending too soon. This will be one more blade to the Swiss army knife VDH is.

[Wild Willy]

Of course. I'm not looking for help with any hack. I knew I was on my own the moment I
swapped the 2 ffmpegs. I guess I'll discover on my own over time what I've lost by not
using your customized ffmpeg.

In that other thread I referred to on September 27, we've had some speculative discussion
about why VDH or the CoApp causes excessive page file activity to the point of freezing
my system for 15-20 minutes. My speculation is that it has to do with the data you keep
in storage until a download completes. When that download does complete, you have to
write all that data into the downloaded file. My claim is that it is that operation that
causes my system to thrash. Will your use of ffmpeg eliminate this thrashing? If so,
may I suggest you use ffmpeg in more cases than recording live streams. No matter what,
I'd really like to have you address this thrashing issue.

[mig]

If the load issue is related to the data being kept in memory and written at the end of the download, then probably the pure coapp download method should solve the problem.

I don't see anything obvious that would be missing if you replace internally the ffmpeg binary into the coapp, but ffmpeg build contains hundreds of parameters and it will take a lot of tests in many use cases to make sure everything is running as expected.

Back to the hack/support issue, don't get me wrong, the problem is not supporting you as an individual. It's that VDH is a popular product that has been installed over 500 million times (half a billion). So if we put in place a feature where 5% of the users require our help to make it work, this will be the end of VDH :/ Maintainability and supportability are important criteria when deciding what features to put into the product.

[Wild Willy]

> If the load issue is related to the data being kept in memory and written
> at the end of the download, then probably the pure coapp download method
> should solve the problem.

This is good news. I await the new beta eagerly.

> I don't see anything obvious that would be missing if you replace
> internally the ffmpeg binary into the coapp, but ffmpeg build contains
> hundreds of parameters and it will take a lot of tests in many use cases to
> make sure everything is running as expected.

Oh. I thought there would be features of VDH that would not work because they require
the customizations you've made to ffmpeg. By swapping in the standard ffmpeg, I thought
I wouldn't have your customizations so certain features of VDH would simply fail if I
tried to use them. If the only customization of ffmpeg you've done is to build it
"light" meaning with a lot of options not selected, then my assumption is wrong. In any
case, I swapped the standard ffmpeg into the CoApp directory to see if maybe the freezes
I was seeing would go away. They do not. So I'm going to go back to using the ffmpeg
you distribute. I have to say I was a bit surprised to see that everything I tried,
which I admit was not a wide variety of things, still worked normally with standard
ffmpeg. I thought it was because I wasn't requesting any VDH functions that used your
customizations to ffmpeg. If you have not actually added any source code that you have
written to the open source of ffmpeg, that explains some things. By swapping in the
standard ffmpeg, I was just providing a lot of extra ffmpeg code that you never invoke.
There is no code in your ffmpeg that is different from what's in standard ffmpeg.
There's just more code in standard ffmpeg & VDH never uses any of that extra code. Do I
have this right now?

> Back to the hack/support issue,

> . . .
> half a billion
> . . .

> 5% of the users require our help to make it work,
> this will be the end of VDH

What? You can't talk to half a billion users all by yourself? What could you possibly
have to do that would be more important? Besides, you're immortal so you have infinite
time to get around to everybody. Not only that but you're clairvoyant & omniscient. You
solve all bugs before they've been created. I am humbled & honored that you allow me to
use VDH.

Just as an aside, I've downloaded several Golf Channel live streams with the beta you
posted in the course of this thread & it appears that your clairvoyant omniscience has
made VDH work pretty reliably for these streams. So, finally dropping the silliness, you
did good work with that.

[mig]

> Oh. I thought there would be features of VDH that would not work because they require

> the customizations you've made to ffmpeg.

No, the idea in providing our own ffmpeg into the coapp is to have a base we know and trust so we can support it. On top of that, the rest of the coapp provides additional services that are required by VDH but not provided by the browser extension API, plus all the glue to make those services callable from the extension.

[Wild Willy]

I just had a rather annoying event occur. I was recording the live stream of today's
round of golf. The broadcast window was from noon until 15:00. It got to be nearly
16:00 & VDH was still allegedly recording. VDH said it had recorded 100% of the stream &
0:00 remained. But it's like that from an early point in the recording so that didn't
really mean anything. I looked in the Resource Monitor & the rate of line usage led me
to believe VDH was just retrieving the usual "Coverage has ended" screen that NBC likes
to tack on the end of their streams. So I told VDH to stop recording. Bad idea. I had
over 8G of a corrupt MP4. I tried playing it in VLC but it wouldn't play. ffprobe told
me the moov atom was missing. I Googled this symptom & found advice to try ffmpeg with
the -movflags faststart option. This apparently is indicative of the inclusion of an
older ffmpeg utility called qt-faststart.exe within the standard ffmpeg. I also found a
tool called Wondershare that has a basic repair function & an advanced repair function.
Both ffmpeg & Wondershare told me it couldn't find a moov atom. I think it really wasn't
in the file. With some sadness I deleted the file. Fortunately, Golf Channel will be
rebroadcasting this show so I will have a second chance at it. I think I will just watch
it this time. Obviously, they won't have a live stream of it. They may offer the show
online on their web site but I think I will just watch it via regular cable instead of
trying to record it.

My speculation is this. This moov atom is the stuff you keep in storage that I've been
complaining causes my system to thrash when you write it at the completion of a download.
I'm thinking that when I tell VDH to stop a recording, it ought to write what part of
this stuff it has up to that point. Instead, VDH simply discards the data, leaving me
with an irreparably corrupt file. When your new version of VDH is available, the one you
say will use ffmpeg as the download mechanism, I believe this will not be a problem. I
gather from your comments above that ffmpeg doesn't hold this data in storage but
continuously writes it into the target MP4 file. Given that, I'm speculating that
stopping a live stream recording will leave me with a playable MP4 instead of a corrupt
one. In addition, I am guessing that after VDH with ffmpeg has completed a certain
duration of the recording, I will be able to start watching the partially recorded file
while VDH continues to record it. I'll try this out to test my hypothesis once this new
VDH is available.

[Wild Willy]

I just tried to record this livestream:

https://twitter.com/CanadiensMTL/status/1465350026356473859

Well, it's not a livestream any more but when I first launched the VDH recording of this,
it was still live.  VDH chewed on it for a few minutes then said the file was ready.  I
discovered I had a file of duration about 5 minutes.  Not the whole thing by far.  I
deleted it without even attempting to watch it.

After the stream was no longer live, I reloaded VDH & then reloaded the web page.  I
reloaded VDH to remove the clutter in the VDH menu from my original visit to the page.
This time, VDH downloaded the full video of duration just over an hour.  At least Twitter
gives decent download speed so it took something less than 10 minutes to download.  The
same thing is apparently available on YouTube.  In my experience, that download would
have taken at least half an hour.  I have scanned through the file quickly to make sure
it's all there so I could post this.  It looks good.  I'll be watching it here in a few
minutes.  If I don't post here again about this, it means the file was indeed good.

So when you get to testing whatever it is that you are doing with ffmpeg, you should hunt
up some Twitter livestreams & test with them.