Video DownloadHelper new beta 7.6.3a5 for Firefox

[mig]

A new VDH beta version 7.6.3a5 has been released for Firefox: https://www.downloadhelper.net/firefox/betas

It integrates a number of locale updates (sv, uk, ko, hu, el, pl, id, nl), plus a new parameter "HLS end timeout" to help fine tuning situations where VDH stop recording live HLS streams too early.

Note that VDH betas are only available for Firefox. Improvements from the beta versions will then be integrated to the next stable release for all supported browsers (Chrome, Edge, Firefox).

[Wild Willy]

Thanks for this, Michel, both for posting the beta AND for announcing it. I have changed
my value for the new parameter to 120. I think 2 minutes is a good delay to allow for a
livestream to resume. I will probably test this out later this week on the Women's
British Open Golf Championship. I'll let you know how it goes. I assume this is the
"other" change that was in 7.6.3a1 that you had not previously ported forward. I also
assume the rest of the livestream changes in 7.6.3a1 are now in 7.6.3a5. I'm eager to
test this.

[mig]

To make things clear about that timer: VDH reads periodically the HLS manifest file. When the manifest hasn't shown any new chunk for a period of time that is the maximum between this HLS end timeout and twice the duration of the longest chunk, VDH considers the download as completed.

[Wild Willy]

Twice the duration of the longest chunk . . . Do you have any sense of what web sites
typically choose as their chunk size? 20 seconds? 5 minutes? Half an hour? Is there
no trend at all? If I've been recording a site whose chunks are half an hour long, this
could make VDH sit there for an hour before it times out. I'll have to use this beta for
a while, assuming it performs as well as 7.6.3a1 for me, before I firm up a conclusion.
My sense is that you should just take the user's specified value without reference to the
duration of the longest chunk. But let me use this on a few things before we make any
further program changes.

I happen to have some ffmpeg logs here from stuff I recorded off Medici over the past
week. Three of them were livestream recordings, the other was a simple download of an
archived concert. The livestreams all show ffmpeg logging something every 100 frames.
This seems to be mostly intervals of 4 or 2 seconds duration. Mostly. Not uniformly,
but mostly. I'm not sure if that actually correlates to stream chunks. The repetitive
100 frames per log line seems too regular to be a coincidence. Trouble is I don't know
if this is an ffmpeg policy or simply a reflection of what Medici was feeding it. The
download of the archived concert was much less regular. The numbers of frames logged per
line varied quite a bit, & not surprisingly, so did the durations. I would not be
surprised to learn chunks don't have to all be the same size.

I did notice that the logs for the livestreams were showing ffmpeg periodically reading
the manifest, then reading some number of chunks. It seemed like the number of chunks
read after each manifest read varied between 2 & 5 chunks. There doesn't seem to be a
good pattern here that lets me conclude anything.

I think it's fair to say that the Medici chunks in their livestreams are quite small.
This means that in practice, my timeout value would prevail. For their archived content,
the situation is quite different. But then, I would hope you do not sit there for 20
minutes after downloading the last chunk of an archived HLS stream. You probably can't
tell just by looking at a manifest whether you're dealing with a livestream or a simple
file. But I sure would hope you don't treat them the same. It would be most educational
& helpful if VDH offered the kind of logging ffmpeg does. Plus, I'm looking at Medici,
where I have the possibility of comparing ffmpeg & VDH. On Golf Channel, all I've got is
VDH. I still don't know how you manage to get that to work when I consistently get 403
Forbidden - access denied when I try to just ffprobe a manifest on NBC Sports/Golf
Channel. I would dearly love to compare VDH & ffmpeg for a golf tournament. Oh well.
It is what it is.

[mig]

Keep in mind we are dealing with live broadcasts, so audio/video chunks are unlikely to be very long. But if they really use 30 minutes chunks, we will have to wait a long time, like an hour, before deciding we shouldn't expect any more piece. in practice chunks are a few seconds long but the extension must support any duration. If we had the information in the manifest that the livestream is over, we wouldn't go through that algorithm to decide when to call the recording complete.

[Wild Willy]

OK. That makes sense. Now here's my experience with 7.6.3a5. I watched it do this
twice. When the blue dot status menu says the download has reached 100%, in other words
when the recording has caught up, it stops recording. These golf tournament livestreams
start one hour before the broadcast window. Typically you get a minute or two of
whatever programming they had on at that time. Then you get a screen that says,
"Coverage will begin shortly." So VDH dutifully rewinds to the beginning of the
livestream & records an hour of "Coverage will begin shortly." What I observed, twice, I
want to emphasize this, I watched this happen twice, VDH caught up & stopped recording.
During the catch up phase, the Resource Monitor showed line usage bouncing around between
2 & 4 million bytes per second. It also showed the rate of data writes to my output file
was, not surprisingly, in that range as well. But as soon as the status menu went to
100%, writes to the file stopped & line usage went to about 1800 bytes per second. The
CoApp entry in the Resource Monitor appeared to be reading something from the Internet
connection, just enough to keep the recording connected. But it wasn't reading the
livestream. Maybe it was reading the manifest & deciding erroneously that there was no
more livestream to record. But the last update time on the output file was not
continuing to update & the file size was not increasing after I left it idling for about
5 minutes. And I did this TWICE. So it wasn't a coincidence.

I have reverted to 7.6.3a1. It did the catch up thing for a few minutes. Then its
behavior differed from 7.6.3a5. After 7.6.3a1 caught up, it reverted to 85% complete &
continued to download at the 2-4 million bytes per second rate. After it caught up
again, it reverted to 95% & continued to download at the elevated line load. When it
caught up to 100% for the third time, it pretty much stayed caught up. The blue dot
reports 100% complete permanently. I expect this to remain the case for the entire
expected 7 hours of this broadcast. The line usage is now fluctuating between about
600,000 & 800,000 bytes per second. This is what I would expect if I were just sitting &
watching the broadcast in a Firefox window. It occasionally drops to about 300,000 bytes
per second. This indicates that they are in an advertising break. During the
advertising breaks, the online livestream replaces the actual advertising with a banner
that says, "Coverage will resume shortly." When coverage resumes shortly, the line load
bounces back up into the 600,000-800,000 bytes per second range.

7.6.3a5 does not work. 7.6.3a1 works. You need to port the 7.6.3a1 changes forward.
Unchanged. No more trying to do something that you think is more general, more elegant,
more . . . I don't know what. 7.6.3a1 works. Don't try to fix it. Bring those changes
forward unchanged. Quit trying to make these piecemeal changes that you've been doing in
the last couple of betas. Don't add the user setting for the livestream timeout. I told
you before that 7.6.3a1 works. I'm telling you again that 7.6.3a1 works. Go with that.
Drop whatever it is you did in the a4 & a5 betas. They don't work. 7.6.3a1 works.

[Wild Willy]

This past weekend, actually Thursday through Sunday, the ladies held their final major of
the golf season near Edinburgh, Scotland. The livestreams Thursday & Friday ran 7 hours.
The livestreams on the weekend were supposed to be split, with the early portion lasting
4 hours & the later portion 3 hours. But they ended up needing a playoff so part 2
Sunday went 4 & 1/2 hours. VDH 7.6.3a1 recorded all 6 livestreams, a total of 29.5
hours, flawlessly. I didn't even have that annoying intermittent problem that people
have started reporting more frequently lately, where the audio is fine but the video is
running at about 6 times normal speed.

When can we expect the 7.6.3a1 changes, which I have tested extensively now since last
September & proven they work perfectly, when can we expect those changes to appear in a
future beta?

[Wild Willy]

Actually, come to think of it, the total recording time was longer. Because of the
prefix of "Coverage will begin shortly," the total recording time was more on the order
of 33-34 hours. Whatever. 7.6.3a1 handled it perfectly. I tried 7.6.3a5 twice on
Thursday & it failed both times, so I ditched it. I'm hoping you will follow my example
& ditch it as well. You got it right in 7.6.3a1. That's what you need to port forward.

[Wild Willy]

Here's some program design thoughts for you.

I originally thought the NBC Sports/Golf Channel livestreams were weird because they
always seemed to go back to the beginning & VDH would record stuff I had missed. I
thought NBC was actually feeding me something that had already been recorded & saved on a
hard drive in their storage farm & I was getting a store-and-forward image of a chase
recording of the livestream My experiments on Medici TV with ffmpeg have taught me
otherwise. It seems that it is a standard feature of livestreams that you can rewind
them to the start. The livestream probably has been & is being recorded in the storage
farm. But it's not as weird as I thought. In my experiments with Medici TV informed by
things I've found via web searches, I've been rewinding Medici livestreams & recording
them with ffmpeg. I now understand it is something you've been doing with VDH. At a
certain point, the recording, whether with ffmpeg or VDH, catches up. You can see this
in the Network Monitor with both downloaders. The line load will hold in the millions of
bytes per second for however long it takes to record the part of the livestream you have
missed. The line load will then drop into a range between 500,000 & 1 million bytes per
second, indicating the livestream is now actually live because the recording has caught
up. The ffmpeg log shows the speed factor dropping from some multiple in the 5-10x range
down closer to 1. The VDH blue dot status menu will show a steady 100% complete without
dropping below that level.

In light of this, we can identify 2 distinct phases of a livestream recording: the
catch-up phase, the live phase. The likelihood is that the largest chunk you read during
the catch-up phase is probably a rather substantial chunk of minutes, while the largest
chunk you read during the live phase is more likely to be in the seconds range. So you
should have a graduated decision tree for determining when to terminate a livestream. If
the largest chunk is 5 minutes or longer, it should be enough to wait just 5 minutes, not
double the length of the largest chunk. If the largest chunk is in the 1-5 minute range,
double the length is probably sensible. If the largest chunk is under 1 minute, I'd say
5 retries should be enough. Maybe 10. No more.

Still, 7.6.3a1 continues to operate flawlessly on my Golf Channel livestreams. I'm
patiently waiting for you to figure out why 7.6.3a5 can't seem to properly manage the
transition from the catch-up phase to the live phase. I would accept if you don't try to
figure it out but simply port the 7.6.3a1 changes forward to a new beta. Either way, I
believe a new VDH setting is not necessary. Just make the decision to terminate the
recording as I've just described.

[dave madsen]

Having designed and written lots of software, may I suggest that at some point, a reasonable design decision may be to allow the user to intervene in some way if they desire.  There's nothing worse than software that thinks it knows what it's doing but has corner cases that don't work AND that don't allow the user to make a decision, leaving him up the river and helpless to do anything about it.  Sure, some users will complain that the software isn't as smart as they want, but at least they can do *something*.

The software can always be made smarter in a later release, but right  now I'd rather have something that works even if I have to poke at it. 

I'm also thinking that you may never find a heuristic that covers all the cases.  If the user can *gracefully* terminate a recording AND have whatever is already recorded be playable, that may be good enough (for now?).

Another lovely thing would be to provide enough information about a recording in progress so that a user can make an "informed guess" about what to do.

But I think that whatever you do, rather than extending development time, do the best you can for now and shove out another beta.  Probably even a couple more as you refine the code.  The problem is exacerbated given the long release times of VDH.  Do you think Willy's blood pressure can stand another month of waiting?? :-) :-) 

BTW, I've had FF and Chrome licenses for quite a while.  Using 'doze 10 currently.  Too lazy to put it on my Linux boxes. :-)

[Wild Willy]

Your concern for my blood pressure is touching, Dave. I can allay your worries, though.
My blood pressure remains quite good. I get it checked regularly. Despite appearances,
I'm not so invested in VDH that there is any sort of connection between it & my blood
pressure. I am not Borg & assimilation is not on the agenda.

The reason I brought up the conditions for terminating a livestream is that it appears
Michel recently put some effort into tweaking that in VDH, including adding a user option
for controlling it. But as I understand it, the user option would actually not be very
useful. I've seen livestreams that refuse to end because somebody at the server didn't
do the proper thing to end them. The proper thing is to remove the manifest from the
server. When that happens, I have observed ffmpeg doing a couple of failed retries to
read the manifest & then it terminates normally. VDH most likely does the same, but the
absence of a VDH log leaves me assuming instead of knowing. On the other hand, when the
server folks forget to remove the manifest, ffmpeg will go on nearly forever trying to
read it. Since the manifest is still there, ffmpeg always succeeds, so it thinks the
stream is still live. Fortunately, I discovered a couple of ffmpeg parameters that can
limit its rereading. The rereading is useless but it's not ffmpeg's fault. The server
is still serving the manifest when it shouldn't. I don't know how VDH handles this. The
livestream looks like it's still pumping content onto the connection so I can't blame VDH
if it goes on & on. In the past, when I have hit the Stop button in VDH, it has ended
the recording but I have been left with a multi-meg file of hours duration that will not
play. This is due to the unfortunate design decision, which Michel has explained on here
somewhere if you look hard enough for it, to keep the moov atom in program storage. When
I told VDH to end the download, it did not write the moov atom, thus leaving behind a
file that would not play, and worse, could not be repaired. I have not read anything
from Michel on here that he addressed this issue, although I have used the Stop function
recently & it seemed to work properly, and another user commented in another thread
within the past few days that he thought the a5 beta fixed the problem. I'm still uneasy
about using the feature in VDH. Until Michel says he did something to fix the issue, I
will remain skeptical.

But terminating a livestream recording is not the primary problem with the more recent
betas. The problem is that they prematurely decide the livestream has ended. In the
case of the most recent a5 beta, it did not actually stop the recording. But it did stop
writing data into the target file. The livestream was still streaming but VDH had
stopped listening. This is the case I described above where the a5 beta reports 100%
complete but does not continue recording and it does not terminate. The a1 beta reaches
100% complete but recognizes that all it has done is catch up. It realizes the stream is
still live & continues to record. After the catch-up phase, the a1 beta continuously
reports 100% complete, never wavering, until the livestream finally ends. This is the
primary issue that I have been repeatedly trying to get Michel to address. The a1 beta
got it right. Michel posted that beta last September & I have been using it successfully
ever since, most recently today. The a5 beta simply catches up & thinks it's done.
There's some difference between the successful a1 beta & the failing a5 beta that needs
addressing. A new beta with this issue addressed would be most welcome. I believe the
path of least resistance is to simply port forward the tested & successful a1 beta
changes. If Michel believes he can do an even better job than what he did with a1, OK,
but make it work.

I appreciate the idea of letting the user tell the software what to do in certain cases.
I was a software developer for decades until I began to enjoy my low blood pressure
retirement. I know what you're talking about. There is a concept that software should
do only so much & certain decisions should be left to the user. I believe you are
proposing that VDH let the user tell it when to stop recording a livestream. The stop
button on the blue dot active status menu would be such a mechanism. Perhaps another
mechanism should be to allow the user to tell VDH that no, the livestream may be at 100%
but it has not stopped. Keep recording. But I'm not sure that is really feasible. If
VDH has done its 100% thing, I might not notice that until a while later. If I tell VDH
to get back on the horse, I will have missed a few minutes, half an hour, 2 hours of the
livestream. What is VDH going to do about that? No, the proper response to this
situation is not a new knob or dial the user can twist or poke. The proper response is
to fix the erroneous decision in the software to not continue recording. That decision
is properly coded in the a1 beta. It is not properly coded in the a5 beta.

[Wild Willy]

Here's an example to reinforce just how tricky this end-of-livestream issue can be. As I
mentioned above, I recorded a livestream today. It was, as you might have guessed,
another round of golf from Golf Channel. The broadcast window ended & usually, VDH ends
the recording & pops up the notification within 5 minutes after the broadcast window
ends. This is because Golf Channel hangs a couple of minutes of "Coverage has ended" on
the end of the broadcast of play. But today, it was taking longer. I thought that's odd
but checking the Resource Monitor showed that the CoApp was reading data at a rate that
indicated the livestream was still live. Resource Monitor also showed that the target
MP4.part file was getting data written to it at something between 200,000 & 500,000 bytes
per second. So I was hesitant to just stop the recording. But after about an hour & a
half, it was still recording. Final rounds of golf tournaments sometimes run long
because there can be a playoff. There was a playoff in the last round last week & it did
run on the order of 90 minutes long. But today is only the second round. There's no way
they were broadcasting golf this late. There aren't enough players in the tournament
that any would have still been on the course. So I unplugged my network connection to
make VDH terminate the recording, which it promptly did. Like I said before, I don't
trust the Stop button on the blue dot status menu in VDH. When I played the MP4 back,
the entire program of golf was there, no glitches, no too-fast video, no problems.
Perfect recording. But when I got to the end, there was an hour and a half of "Coverage
has ended." In other words, the server was still spewing spam into the livestream long
after the livestream had actually ended. I cannot in any way fault VDH for this. It
looked like the livestream was still on the air. I knew there was a problem because of
things only a human could possibly know. No software could have been able to distinguish
what was happening from a normal livestream. So even though it was junk & the livestream
had in fact ended, VDH correctly recorded the extra 90 minutes of junk.

Conclusion: things aren't always simple & VDH does the best it possibly can with what it
finds.

[Wild Willy]

Just to show the difficulties we have to deal with . . .

This week's golf tournament is unusual in that for the first 2 days, they ran 2
simultaneous full-field tournaments on 2 golf courses, one for the men, one for the
women. Men & women were playing in alternating groups around both courses. As is
typical for nearly all tournaments, there was a cut after the second round yesterday. It
happened to be low 60 scores & ties for both the men & the women. Today's third round
was still alternating groups of men & women but they were all on one golf course because
the cut reduced the number of players to a level that could fit on one course. What is
very rare for golf tournaments is that they made a second cut today, to low 35 scores &
ties for both the men & the women. Today's livestream began with the usual 1 hour & 9
minutes of "Coverage will begin shortly." Then we were supposed to get 4 & 1/2 hours of
golf. They ran the broadcast 13 minutes long in an apparent attempt to show the last 2
groups finish their rounds. But then they just cut off the broadcast without showing
them after all. There is no explaining the logic of TV programming. Then we had the
usual "Coverage has ended" tacked on the end. Yesterday, that ran to 90 minutes & would
probably have run for a few hours more if I hadn't cut it off. Today, after just 7
minutes & 35 seconds, the livestream ended on its own without my intervention. That is
the usual case. Yesterday was an anomaly. Also, the recording was completely without
problems once again. I live in perpetual dread of the too-fast video syndrome, but so
far this week, I've had 3 good rounds. May that continue tomorrow.

This all just goes to show that this whole business of recording livestreams -- even
regular old downloads, really -- is fraught with pitfalls & traps. I say again, VDH does
the best it possibly can with what it finds. It does an excellent job of it, too. It's
not perfect, but it's certainly good enough that I bought a license & will probably use
it forever. Or until I go senile, whichever comes first.

[Wild Willy]

I mentioned all the other rounds so I might as well complete the story. Today's final
round of the golf tournament recorded perfectly with VDH 7.6.3a1 beta & ended without my
intervention. There was no too-fast video. The audio kept switching from being out of
synch with the video by being ahead of it & behind it, back & forth. The offset kept
changing, too, from -150ms to -300ms to -750ms to +150ms to +300ms to +750ms. It was
rather annoying. But as I've said elsewhere, the audio is out of synch with the video at
the source. It is observable when I am watching the golf through my cable box instead of
on my computer. They also decided to stop the livestream before the golf was finished.
It turned out that the winners in both the women's & the men's tournaments were in the
next to last groups. So they just stopped broadcasting before the last women's & men's
groups, who had no chance of catching the winners, finished their final holes. The last
3 women had hit onto the final green & the last 3 men had teed off on the final hole.
But we didn't get to see them finish. TV networks. There's no explaining them.

[Wild Willy]

I see there is now a 7.6.3a6 beta posted. I have installed it just based on the faith I
have in you, Michel. But do please tell us what we should expect from this beta. I am
hoping you have abandoned your other line of investigation & simply ported forward the
7.6.3a1 changes along with the other fixes you've been including in the other recent
betas. Either that or you've attempted to correct whatever wasn't working in the 7.6.3a5
beta. Do please enlighten us.

[Oleksandr Bohoslavets]

Please can you unsubscribe me from your mailing list? 

--
You received this message because you are subscribed to the Google Groups "Video DownloadHelper Q&A" group.
To unsubscribe from this group and stop receiving emails from it, send an email to video-downloadhelper...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/video-downloadhelper-q-and-a/62fbc1dc.c80a0220.5f545.5612%40mx.google.com.

[Wild Willy]

No, but you can. You are subscribed because you subscribed yourself to this thread &
possibly others. You can stop the E-mails yourself. Nobody else can do it for you. You
have to do it. You understand that these E-mails are not coming from the individual
people posting here. They are automatically generated by Google because you asked to
receive them. Here's how you can stop them. Log into Google & visit the threads you
have subscribed to. Apparently this is one of them. There is a checkbox at the top
marked Subscribe. Uncheck it. In addition, there is a link in the left panel of any
page of this group labeled My Membership Settings. You have to be logged in to see it.
Click that link. Look carefully at all the settings. There are some that control how
you receive E-mails from Google on behalf of the group. You'll want to change them.
You'll figure it out. It's pretty obvious what you have to do.

Good luck!