VDH not detecting youtube videos

[Stewart Andreason]

Over the last 2-4 weeks, several youtube videos have escaped detection, I can no longer download.
Here are 2 examples, the second being of the Music type, it seems most music videos are no longer downloadable.
http://www.youtube.com/watch?v=pNo5BlrhtZo
http://www.youtube.com/watch?v=nlVNYdTl_Lo

[Stewart Andreason]

Forgot to mention browser is Firefox in linux, and VDH is up to date at 5.6.1.

[M. Koch]

Same issue here. A lot of Youtube vieos don't work for a few weeks already. What is the reason?

[The Brewdonkey]

I've been experiencing the same thing for the past few days. I'm pretty sure the new Firefox 47.0.1 update broke it.

[Stewart Andreason]

I was running Iceweasel 38.7 when this problem started, and I also still have a seamonkey browser available to run as a secondary check, so I don't think the problem is the newest FIrefox.

[mig]

You should make sure you run libav or ffmpeg that have support for the vp9 video codec.

Try running from a shell avconv -codecs or ffmpeg -codecs

You should get a line like this in the output:

DEV.L. vp9                  Google VP9 (decoders: vp9 libvpx-vp9 ) (encoders: libvpx-vp9 )

Otherwise, for some YouTube videos only streamed using vp9, VDH won't offer the download.

[Stewart Andreason]

That's an angle I had not thought of. But why would an encoder be needed to just save the file as is? and why would the browser (or plugin) ask ffmpeg about it?

My results:
$ ffmpeg -codecs | grep -i VP9
ffmpeg version 2.7.2 Copyright (c) 2000-2015 the FFmpeg developers
  built with gcc 4.9.2 (Debian 4.9.2-10)
  configuration: --enable-gpl --enable-postproc --enable-swscale --enable-libmp3lame --enable-libvorbis --enable-libx264 --enable-pthreads --enable-libopenjpeg --enable-nonfree --disable-ffserver --disable-network --enable-libfreetype --enable-libfontconfig --disable-doc --disable-htmlpages --disable-podpages --enable-shared
  libavutil      54. 27.100 / 54. 27.100
  libavcodec     56. 41.100 / 56. 41.100
  libavformat    56. 36.100 / 56. 36.100
  libavdevice    56.  4.100 / 56.  4.100
  libavfilter     5. 16.101 /  5. 16.101
  libswscale      3.  1.101 /  3.  1.101
  libswresample   1.  2.100 /  1.  2.100
  libpostproc    53.  3.100 / 53.  3.100
 D.V.L. vp9                  Google VP9

I see decoder but no encoder. I'll install libvpx-dev and rebuild ffmpeg-3.1 and see if that fixes it.

[mig]

On Friday, July 8, 2016 at 4:51:46 AM UTC+2, Stewart Andreason wrote:

... But why would an encoder be needed to just save the file as is? ...

 Because there is no such file to just download. The final file you want to obtain does not exist on the YouTube servers, it is made up locally from several pieces downloaded separately (in this YouTube case, 2 pieces: audio and video). Hence the need for a specialized program (ffmpeg or libav) to generate the output file.

[Stewart Andreason]

Okay, I've upgraded my ffmpeg, using the libvpx provided by debian packages.

$ ffmpeg -codecs | grep -i vp9
ffmpeg version 3.1.1 Copyright (c) 2000-2016 the FFmpeg developers

  built with gcc 4.9.2 (Debian 4.9.2-10)

  configuration: --prefix=/usr/local --enable-gpl --enable-postproc --enable-swscale --enable-libmp3lame --enable-libvorbis --enable-libx264 --enable-pthreads --enable-libopenjpeg --enable-nonfree --disable-ffserver --disable-network --enable-libfreetype --enable-libfontconfig --disable-doc --disable-htmlpages --disable-podpages --enable-shared --enable-libvpx --extra-cflags=-I/usr/include --extra-ldflags=-L/usr/lib/i386-linux-gnu
  libavutil      55. 28.100 / 55. 28.100
  libavcodec     57. 48.101 / 57. 48.101
  libavformat    57. 41.100 / 57. 41.100
  libavdevice    57.  0.101 / 57.  0.101
  libavfilter     6. 47.100 /  6. 47.100
  libswscale      4.  1.100 /  4.  1.100
  libswresample   2.  1.100 /  2.  1.100
  libpostproc    54.  0.100 / 54.  0.100

 DEV.L. vp9                  Google VP9 (decoders: vp9 libvpx-vp9 ) (encoders: libvpx-vp9 )

Retried the 2 examples, still says No media to process. Clicked play, Youtube streaming the video okay, click Analyze page, No media to process.

[mig]

Try to restart Firefox and/or click "Recheck converter" in VDH settings > More > Conversion. This will force VDH to reverify whether vp9 is supported or not.

[Stewart Andreason]

Also to be complete, I turned off ad blocking and all local proxy possible interference, tried again, watched the ad, and still no /other/ media to process. It saw the ad, and offered it in ADP format only.Downloaded it, it plays correctly, so I believe that provides a positive verification that vdh worked correctly calling ffmpeg. Took a screenshot in case that is helpful and uploaded it to my server.
Received your reply, maybe the screen shot will not be needed.

Okay found that settings page. It reads "VP9 codec not supported"
Clicked as directed, no change.

Does it output debug text anywhere I can read?

[mig]

Is there any chance you have libav (avconv) installed ? If so, VDH might be using this instead of the ffmpeg you installed. Go to VDH settings > More > Conversion, and check Advanced. You will be displayed the path of the converter binary in a text input.

[Stewart Andreason]

no chance. avconv is not installed.
It _is_ set to /usr/local/bin/ffmpeg
and I tried replacing that with a .sh file that calls the ffmpeg binary, so I can look in /proc/ for anything helpful. I can see it calls it with -codecs just fine, but I can't capture any output for some reason I have not discovered yet.

ffprobe that_ad_ADP.mp4 shows I encoded it, h264 and aac streams.
http://www.seahorsecorral.org/images/tests/firefox-vdh-only-sees-ad-that-played.png

[mig]

Could you try reinstalling VDH from a new Firefox user profile ? Do you still get "VP9 codec not supported" ?

[Stewart Andreason]

Good idea! I created a new profile, installed only 1 plugin, VDH 5.6.1 , changed the storage directory,
Tested 9 youtube videos, 6 still show: "no media to process". 3 videos were resolved, they were available as FLV and ADP formats that I had disabled. Oh no strike 1 FLV back to status "unable to get", all I downloaded was the ad.
http://www.seahorsecorral.org/images/tests/firefox-vdh-screenshot-2.png

Going to conversion and checking that settings tab, it still shows VP9 is not supported.

[Stewart Andreason]

mlg,
After experimenting with encoding to vp9 myself, I am not sure that is the correct goal. For a single core cpu, encoding may take hours. Perhaps VDH only re-encodes the streams and that restriction may not apply.

Also after doing some searching for others with similar problems with youtube in past years, I found an alternative to getting the download working. Using 9xbuddy.com shows all those videos are available in mp4 and webm formats, and not only in vp9 as suggested.

[mig]

MP4 and WEBM are container formats, VP9 is video codec format. VP9 is very likely to be the codec used in WEBM files, as it is in my case with your sample video URLs.

[Wild Willy]

On Saturday, July 9, 2016 at 1:25:56 PM UTC-4, Stewart Andreason wrote:

For a single core cpu, encoding may take hours.

I've seen a few posts about problems with VDH reading YouTube videos so today, I finally tried myself getting 3 different videos from YouTube.  The first 2 were about 3.5 and 4.5 minute-long videos.  It took about 2 hours EACH for the videos to aggregate.  But I did successfully end up with 2 .webm files on my hard drive that do contain the videos, one about 12.4Meg & the other a little over 9Meg.  The third video is a little over 5 minutes long.  But I also notice that the screen resolution offered in VDH is higher for this last one.  It's been about 6 hours aggregating this last video and it appears to be about half way done, according to the progress bar in the VDH dropdown.  Now granted, I'm running Windows 7 and you are running Linux, so there may be a difference there.  However we are both running Firefox.  Still, I have an Intel Core i7 quad CPU clocking at 4GHz in a desktop system (not a laptop, tablet, or other downsize system).  I always thought that was a pretty hefty bit of hardware, but maybe for the purposes of this use, it might be a lightweight.  I am no expert.  It seems to me that a video that will actually play in the space of 5 minutes couldn't possibly have anything about it that could take hours and hours to process.  I mean, if my browser can play the video through in 5 minutes, why does it take what looks like it's going to be TWELVE HOURS to aggregate?  Seems to me like there's an algorithm in there somewhere that might be a candidate for some kind of optimization.

[mig]

The timings you experience are way bigger than what other users get when aggregating. For a 5 minutes video, you should expect a 10-15 secs aggregation.

You have an "Intel Core i7 quad CPU clocking at 4GHz", so the problem is definitely not your CPU.

My best guess is something related to your storage device. Is there any chance your TEMP or final storage directory are mounted over a network connection ? Or maybe some antivirus-like utility analyzing every single packet written or read from the disk ?

[Wild Willy]

No network connection.  This is a self-contained desktop system.  Antivirus could be a factor.  Never thought of that.  Let me see if I can temporarily disable AVG . . .  That doesn't seem to have sped things up appreciably.  I'll leave it off until this aggregation completes.  If I download any more videos, I'll keep this idea in mind.

[mig]

Try check your hard disk too. If it is corrupted (like bad sectors), it could badly impact the performances.

[Stewart Andreason]

Right I am familiar with container vs. video codec. All of the #43 webm container files I have, contain video streams encoded with codec vp8.
    encoder         : Google
    Stream #0:0: Video: vp8, yuv420p, 640x360, SAR 1:1 DAR 16:9, 29.97 fps, 29.97 tbr, 1k tbn, 1k tbc (default)
    Stream #0:1: Audio: vorbis, 44100 Hz, stereo, fltp (default)

Going via 9xbuddy I downloaded the webm version fast, and ffprobe also shows it as vp8.

Revisiting my first 2 examples, they are showing up in VDH as protected now in my normal profile. In fact every youtube video I try this morning is now protected. Is youtube blocking me because I'm signed in?
After over a dozen attempts, I load another firefox with the clean profile, and now I see no media to process, for the same videos. Back in this browser I disable the plugin I added yesterday, but still showing everything protected. I'll try a more complete reset after sending this.

As a test trying to find the vp9 you see, I try downloading from another site with enbedded youtube video, with VDH the ADP variant for a mkv file, and my connection speed shows a 8% trickle, 70K per sec. process list does not show ffmpeg, so this is wierd. h264+vorbis is not vp9. I just don't see any vp9 files.

[mig]

Does this video https://www.youtube.com/watch?v=usAWvTKplIs appears as protected to you ?

When VDH shows a MKV entry to download, this is because the current page is playing video vp8/9 and aac audio, or video h264 and vorbis/opus audio. MP4 and WEBM formats forbid those combinations, only MKV is suitable.

Download speed is irrelevant regarding what VDH does or does not. If it is slow, this might be because of the server, your internet connection, other apps using bandwidth, ... but not VDH as it performs direct TCP/HTTP download using Firefox services.

[Stewart Andreason]

That video works fine on both the test profile, and my normal one.

Your other two paragraphs/explanations sound right.

I tried disabling plugins to find the reason for 80% of my youtube page loads to show protected, but was unable to determine a cause.  Trying the same video(s) in the clean test profile, show No media to process. Would you like a few more examples?

https://www.youtube.com/watch?v=KvaA4-ZJLxc
https://www.youtube.com/watch?v=DAHxbnQpSss

One thing these 2 have in common, is playing an ad first, which VDH does catch and offer in the list as ADP. But I can also show other youtube pages starting with an ad, that after the ad plays, do show the video in the list and non-protected. Then I try it in my normal profile, and it doesn't play the ad, and does show the video in the list just fine and non-protected. so.... resubmitting theory. Try those two.

[Stewart Andreason]

Upon re-reading my post, I feel I should clarify, although these 2 have ads in common, I also have examples that did not play ads, that show protected in my normal profile, and no media in the clean test profile. So although the directed theory is about ads, I don't think it is the complete answer.
I'm probably stopping the ads with Flashstopper in my normal profile. And adding it to the test profile did not change any VDH behavior.

This example exhibits the problem without playing ads first.
https://www.youtube.com/watch?v=DdKrG7hm9-0

[Wild Willy]

On Sunday, July 10, 2016 at 7:23:59 AM UTC-4, mig wrote:

Try check your hard disk too. If it is corrupted (like bad sectors), it could badly impact the performances.

Now that got me thinking.  I'd almost forgotten about this one issue I've always had with Windows 7.  Seems to me I might have had it with Windows XP & Windows 2000 as well.  I don't remember now, it's been so long since I used those other systems I just can't remember.  In any case, here's the problem.  The only time I ever notice the problem is when I open a command window to do a DIR command on my boot partition, which is of course my C-drive.  When I do such a dir command, I get inundated with error messages like this:

The directory name C:\Documents and Settings\All Users\Application Data\Application Data\Application Data\Application Data\Application Data\Application Data\Application Data\Application Data\Application Data\Application Data\Application Data\Microsoft\User Account Pictures\Default Pictures is too long.

I did a DIR command just now so I could capture this sample error message.  There were nearly 900 of these error messages before the dir command told me that the file pattern I was searching for did not exist.

If I do a CHKDSK on my boot partition, these error messages go away.  But this is my boot partition.  The only way to CHKDSK this partition is to schedule a CHKDSK which will not be executed until I next reboot my system.  And my boot partition is 100G so it takes at least an hour to run the CHKDSK.  CHKDSK on every other partition can be run on demand and takes only a few minutes.  But the boot partition is an exception.  It is also the only partition that has this problem with the ridiculous directory names, as in the example error message I've quoted here.  But it gets worse.  Even though I've spent the hour or so letting CHKDSK fix up my boot partition, and the problem is indeed gone right after I reboot, about 10 minutes later the problem comes back.  I don't know what I've done to make it happen.  I'm reasonably well convinced this is a bug in Windows.  I have tried to fix this problem multiple times but it keeps coming back.  There are no symptoms other than this error message snowing me under whenever I do a DIR command for anything on the boot partition.  The system appears to be running flawlessly otherwise.

So your idea prompted me to create a new DWHelper directory on another partition.  Last night, I downloaded this video https://www.youtube.com/watch?v=3VmoZrxXbmg.  In VDH I selected the . . . Wait a minute.  The one I downloaded isn't being shown now.  It was at resolution something like 1920x1080 and the file size was something on the order of 80Meg.  But I'm seeing 3 choices right now, all of them at lower resolutions and smaller file sizes.  That's weird.  Maybe VDH knows I've already downloaded this video once with those particular characteristics & isn't showing me the same one again.  Hmmmm.......  I don't see anything in the VDH preferences that would obviously relate to this.  Maybe if you look at the video, you'll see the choice for the larger file.

In any case, I downloaded this video to a directory NOT on my boot partition.  That took on the order of 5 minutes.  But the aggregation took about 7 hours.  So I'm thinking there is something in VDH that uses another directory on my boot partition.  How do I make sure VDH uses no directories on the boot partition?  I think my Firefox disk cache is on my boot partition but nothing I looked at in about:config appears to allow me to place the disk cache elsewhere.  Seems to me at one time Firefox had a simple settings dialog in which you could specify a cache directory name directly.  But they've improved (ha ha) Firefox so much in recent years that there's less you can control now.

So I am resigned to having to download videos (not something I do all that often) when I can leave my system idling overnight doing the aggregation.

[Wild Willy]

On Wednesday, July 13, 2016 at 1:30:29 AM UTC-4, Wild Willy wrote:

I downloaded this video https://www.youtube.com/watch?v=3VmoZrxXbmg.  In VDH I selected the . . . Wait a minute.  The one I downloaded isn't being shown now.  It was at resolution something like 1920x1080 and the file size was something on the order of 80Meg.  But I'm seeing 3 choices right now, all of them at lower resolutions and smaller file sizes.  That's weird.

Further to that anomaly, I've posted some comments on this https://groups.google.com/forum/#!topic/video-downloadhelper-q-and-a/6Y0GFUaCwdc thread that I believe are connected to this issue.

[mig]

About aggregation performances, try this: open VDH settings > More > Conversion, check Advanced, and try to uncheck parameter Force H264 tuning in aggregation.