Embedded JW Player Videos

[Klein C]

I realized VDH does not detech embedded videos that is using JW Player.
How do i go about download the embedded video with VDH?

[mig]

VDH does not consider the player itself but the network traffic being generated. If the video is streamed over the RTMPx protocol, it won't be seen by VDH. Can you share the URL ?

[dnslo...@gmail.com]

On Monday, June 22, 2015 at 3:56:40 AM UTC-4, mig wrote:

VDH does not consider the player itself but the network traffic being generated. If the video is streamed over the RTMPx protocol, it won't be seen by VDH. Can you share the URL ?

Hi, mig. I greatly appreciate your work to create this extension. I'm writing to give more detail than the OP on this particular problem.

I have encountered the problem sporadically over the past few months, from various sites. Sadly I didn't note the URLs of problem videos
until I had one such "in hand" today and decided to investigate. So I do have one example, but I've encountered many.

First, only a few videos created with JWPlayer (5.8.2011) have the problem. As far as I can remember, they all involve servers of dailymotion.com,
but, again, not all dailymotion.com videos from these servers have the problem. I have no idea how the uploader of the problem videos I have encountered
is creating this problem.

However, here is what I've found.

If you go to http://www.show48.com/akb48-janken-taikai-kashiwagi-yuki-no-real/ you'll find an example. (Ignore the low-res video: it's the
English subtitles that this site supplies that are interesting! :-))

Using Firefox Inspector, choosing the tab Network, you will find the URL:

   http://proxy-60.dailymotion.com/sec(0543d207b5f29a45b5474cb8ac50582c)/video/157/842/43248751_mp4_h264_aac_ld.m3u8

followed by a stream of .ts fragments.

Using curl to get that file (the Inspector gives you a nice command line for getting it but you have to redirect stdout to a file) you will find that
it is indeed a "playlist" looking like this (I just opened it in vi after checking with "file" that it was an ascii file):

   #EXTM3U
   #EXT-X-VERSION:3
   #EXT-X-PLAYLIST-TYPE:VOD
   #EXT-X-TARGETDURATION:20
   #EXTINF:15.849,
   /sec(0543d207b5f29a45b5474cb8ac50582c)/frag(1)/video/157/842/43248751_mp4_h264_aac_ld.ts
   #EXTINF:10.010,
   /sec(0543d207b5f29a45b5474cb8ac50582c)/frag(2)/video/157/842/43248751_mp4_h264_aac_ld.ts
   [...]
   #EXT-X-ENDLIST

Feeding that URL for the playlist to either vlc or mpv works perfectly.
Feeding it to youtube-dl downloads the file perfectly (youtube-dl uses ffmpeg to combine fragments), though
the resulting MPEG4 file is named 43248751_mp4_h264_aac_ld.m3u8 and that extension is wrong.

The MPEG4 file plays just fine with mpv, but vlc can't handle it until you change the file extension from .m3u8 to .mp4 .

Other videos *on this same site* still use JWPlayer 5.8.2011, but don't use dailymotion for the source and/or don't use a
playlist-plus-fragments. Yet note that the fault isn't dailymotion because lots of dailymotion videos are handled
just perfectly by Video Downloadhandler.

Hope this helps...

[mig]

Thanks for those details. This was very helpful.

I investigated on this. The video at http://www.show48.com/akb48-janken-taikai-kashiwagi-yuki-no-real/ is using a MP2T chunked streaming. This was the first time i saw this streaming from dailymotion servers.

Since version 5.3.1, VDH supports MP2T streaming, however no video was detected in the case of your video. This is because a filter being used in VDH excludes the URL before it is recognized as MP2T.

The workaround is simple: open VDH settings > "Behavior", "Advanced" checked. Remove the beginning of parameter "Network filter out" to read "[&\?]range=[0-9]+-[0-9]+|/silverlight/". Reload the page and the video will now be detected.

I'll make sure the next version of VDH takes care of this so you won't have to hack the configuration for that type of video to be captured.

Thanks again.

[dnslo...@gmail.com]

Thanks for replying so quickly, and with a fix too!

Changing the "Network filter out" regular expression as you described does indeed have the effect of
getting DWH to recognize the video. However, so far (I'm still trying!) I have been unable to get any
file created.

I have followed the instruction on the website for using the streaming feature, and it *seems* to work
as far as the dialog boxes that I get up on the screen. But when I click on Assemble-and-Convert,
nothing happens.

I do see some files created (and left) in /tmp:

shrdlu dwh-ovflow $ ls -ltr /tmp/ch*
-rw-r--r--. 1 dns dns   48 Jul  8 15:57 /tmp/chunked-media.mpeg
-rw-r--r--. 1 dns dns 8799 Jul  8 16:12 /tmp/chunked-media-1.mpeg
-rw-r--r--. 1 dns dns   48 Jul  8 16:23 /tmp/chunked-media-2.mpeg
-rw-r--r--. 1 dns dns 8799 Jul  8 16:27 /tmp/chunked-media-3.mpeg
-rw-r--r--. 1 dns dns 8799 Jul  8 16:28 /tmp/chunked-media-4.mpeg
-rw-r--r--. 1 dns dns 8799 Jul  8 16:42 /tmp/chunked-media-5.mpeg
-rw-r--r--. 1 dns dns   48 Jul  8 16:54 /tmp/chunked-media-6.mpeg

/tmp/chunks:
total 14272
-rw-rw-r--. 1 dns dns 489928 Jul  8 16:05 c.ts
-rw-rw-r--. 1 dns dns 456276 Jul  8 16:05 c-1.ts
-rw-rw-r--. 1 dns dns 595960 Jul  8 16:05 c-2.ts
-rw-rw-r--. 1 dns dns 412848 Jul  8 16:05 c-3.ts
-rw-rw-r--. 1 dns dns 379196 Jul  8 16:05 c-4.ts
-rw-rw-r--. 1 dns dns 516624 Jul  8 16:05 c-5.ts
-rw-rw-r--. 1 dns dns 416420 Jul  8 16:05 c-6.ts
-rw-rw-r--. 1 dns dns 454584 Jul  8 16:05 c-7.ts
-rw-rw-r--. 1 dns dns 459660 Jul  8 16:05 c-8.ts
-rw-rw-r--. 1 dns dns 567196 Jul  8 16:05 c-9.ts
-rw-rw-r--. 1 dns dns 524332 Jul  8 16:05 c-10.ts
-rw-rw-r--. 1 dns dns 299296 Jul  8 16:05 c-11.ts
-rw-rw-r--. 1 dns dns 339152 Jul  8 16:05 c-12.ts
-rw-rw-r--. 1 dns dns 525084 Jul  8 16:05 c-13.ts
-rw-rw-r--. 1 dns dns 407396 Jul  8 16:05 c-14.ts
-rw-rw-r--. 1 dns dns 411908 Jul  8 16:05 c-15.ts
-rw-rw-r--. 1 dns dns 378068 Jul  8 16:05 c-16.ts
-rw-rw-r--. 1 dns dns 435220 Jul  8 16:05 c-17.ts
-rw-rw-r--. 1 dns dns 319600 Jul  8 16:05 c-18.ts
-rw-rw-r--. 1 dns dns 245152 Jul  8 16:05 c-19.ts
-rw-rw-r--. 1 dns dns 523956 Jul  8 16:06 c-20.ts
-rw-rw-r--. 1 dns dns 253612 Jul  8 16:06 c-21.ts
-rw-rw-r--. 1 dns dns 374120 Jul  8 16:06 c-22.ts
-rw-rw-r--. 1 dns dns 732260 Jul  8 16:06 c-23.ts
-rw-rw-r--. 1 dns dns 377128 Jul  8 16:07 c-24.ts
-rw-rw-r--. 1 dns dns 344792 Jul  8 16:07 c-25.ts
-rw-rw-r--. 1 dns dns 373932 Jul  8 16:07 c-26.ts
-rw-rw-r--. 1 dns dns 576972 Jul  8 16:07 c-27.ts
-rw-rw-r--. 1 dns dns 361900 Jul  8 16:07 c-28.ts
-rw-rw-r--. 1 dns dns 419428 Jul  8 16:07 c-29.ts
-rw-rw-r--. 1 dns dns 708196 Jul  8 16:08 c-30.ts
-rw-rw-r--. 1 dns dns 356260 Jul  8 16:08 c-31.ts
-rw-rw-r--. 1 dns dns 510608 Jul  8 16:08 c-32.ts
shrdlu dwh-ovflow $

That looks promising!! Seems .ts fragments are being captured!

I worry that some other extension (especially noscript) might be interfering, so I disabled noscript entirely and restarted firefox.
This did not solve the problem.

There are a lot of undefined JS symbols reported to my terminal when I run firefox from it. Probably this is unrelated (JS symbols
are always being reported as undefined in firefox these days). The last I see when I try to capture this video are from sandbox.js.

If you want them I'll send them, but as it's a lot of stuff, I should send via private mail I think and not fill up this forum.

Any suggestions?

[dnslo...@gmail.com]

I forgot to mention in my post a few moments ago that I'm running Firefox 40. I'm on the Beta Channel.

And for completeness, my OS is Linux Fedora 21 fully-updated.

And although according to that "playlist" for this video, there are 62 .ts fragments,
there are only 32 .ts files in /tmp/chunks. And no, I didn't run out of space in /tmp:
I have 32 GB of RAM and a very large tmpfs defined for /tmp. I will clear out /tmp
and try to run the whole thing again for you if you want....

[mig]

Ok, so the chunks (.ts files) are correctly saved, which is normal. When you click "Assemble and Convert" or "Assemble only", those chunks should be processed to generate a single file in the regular storage directory.

Could you post the messages from the browser console (CTRL+SHIFT+J) that appear just after you click Assemble ?

[dnslo...@gmail.com]

Absolutely nothing happens after I click on Assemble-and-Convert (except the asynchronous info about the end of the network stream).

Here is that last of the console record (with CSS and JS turned on as well as Logging: I turned off Network because the entries as the
.ts segments come in clutters up the log -- I assure you they really do all come in):

ReferenceError: reference to undefined property (intermediate value).hasDownloadButton angular.min.js line 106 > Function:2:7
ReferenceError: reference to undefined property (intermediate value).hasDownloadConvButton angular.min.js line 106 > Function:2:7
ReferenceError: reference to undefined property (intermediate value).hasConvertButton angular.min.js line 106 > Function:2:7
ReferenceError: reference to undefined property (intermediate value).hasAssembleButton angular.min.js line 106 > Function:2:7
ReferenceError: reference to undefined property (intermediate value).hasAssembleConvertButton angular.min.js line 106 > Function:2:7
ReferenceError: reference to undefined property c.active angular-animate.min.js:20:411
ReferenceError: reference to undefined property n.event.triggered jquery.min.js:3:4791
ReferenceError: reference to undefined property message.smooth commonPanel.js:88:6
ReferenceError: reference to undefined property a[d] angular.min.js:12:464
not well-formed convconf-editor.html:45:29
ReferenceError: reference to undefined property c[e] angular.min.js:64:439
ReferenceError: reference to undefined property c[a] angular.min.js:136:187
ReferenceError: reference to undefined property b.constant angular.min.js:104:138
ReferenceError: reference to undefined property c.immediatePropagationStopped angular.min.js:32:0
ReferenceError: reference to undefined property s.readonly angular.min.js line 106 > Function:4:0
ReferenceError: reference to undefined property s.title angular.min.js line 106 > Function:4:0
ReferenceError: reference to undefined property Nc[a] angular.min.js:31:336
ReferenceError: reference to undefined property g[h] angular.min.js:197:415
ReferenceError: reference to undefined property s.audioonly angular.min.js line 106 > Function:4:0
ReferenceError: reference to undefined property s.twopasses angular.min.js line 106 > Function:4:0
INFO:HLSNetStream:close

I think there is another problem: So far I have been using the bullet to the left of the entry in the list of videos displayed when I
click on the little 3-ball icon of DWH. When I click on it the list disappears, but if I bring up the list again the bullet has turned
to a square as (I think) it should. However, when I do this the first time, after a few .ts segments are already in, I get the
dialog box for conversion almost immediately (I think not absolutely immediately) and if I click on Assemble-and-convert in
that box, nothing happens and the square goes back to bullet (if I bring up that list again). This is long before all the segments
have come in.  Since the bullet has reappeared, I click on it again, and this time the conversion dialog box doesn't come up
until the end. Don't know what that is about.

I note that all the segments appear in .ts files in /tmp/chunks, and that they all disappear after I click on assemble-and-convert
at the end.Still no conversion.

Can I do more to help?

[mig]

Do you get the same thing when picking "Assemble only" instead of "Assemble & Convert" ?

Taking a few days off. I'll be back soon on this issue.

[dnslo...@gmail.com]

Sorry, can't do anything further on this until flash-plugin is updated to fix the 0-day bugs, and firefox accepts that. Should be tomorrow (I hope).
Until then I can't get enough stuff (in Inspector-tab Network) to learn anything. It stops with the embedded video space telling me that I don't
have flash enabled (which is true of course).

I did do some further investigation, learned a bit about the Apple "HLS" streaming-video format which this seems to be.

However, I also found that DailyMotion (the normal, directly-accessed DailyMotion site, not this access through a proxy from
www.Show48.com) has a higher-resolution copy of exactly the same video (with identical English subtitles and all) which
is delivered by Yet Another streaming format!!

This one has a manifest file with extension .mnft (which is an extension that according to google is supposed to be
used for Java manifest files), and video fragments with the extension .flv (even though, if I download the first such
fragment, does not appear to be a Flash video file: "file" just says "data", and "ffprobe" doesn't recognize it).
I checked a number of DailyMotion videos and they all were being delivered this way.

Perhaps this has something to do with MPEG-DASH, but the format of the ".mnft" file is certainly not DASH.

As soon as I can risk having Flash enabled again (tomorrow??) I'll explore further.

I'm conjecturing that DailyMotion is experimenting with its own brand of adaptive streaming format, as the manifest
file I found (the one with extension ".mnft") is notable for the extremely compressed ascii contents. DASH, being
XML, is many, many times longer.

I do note that Downloadhelper does not understand these new .mnft-type DailyMotion videos. It lists all the fragments
(logical since they end in ".flv").

However, youtube-dl *does* handle this new (?) type. I plan to grab the source for youtube-dl from github (it's
"public domain" (i.e. no licence) so you could look at it without any problem) to see how they recognize it.
But I haven't got to that yet. I want to grep for "mnft" and can't via the web interface to github (as far as I can
tell). So I'll pull all the files and grep locally.

So there's my update. I can't answer your last question about "Assemble Only" until Flash is re-enabled.

Oh, well, here's a teaser: the contents of one of the .mnft files:

{"version":"1","duration":761.796,"bitrate":459,"codecs":"mp4a.40.2,avc1.42001e","audio":{"codec":"AAC/LC","bitrate":96,"channels":2,"samplesize":16,"samplerate":44100},"video":{"codec":"H264/AVC Baseline@L3.0","bitrate":359,"width":512,"height":288,"framerate":29.970},"template":"/sec(54aed00888f4eb37ed0fe226f77d0074)/frag($fragment$)/video/157/842/43248751_mp4_h264_aac.flv","fragments":[[1,15.849],[1,10.010],[1,12.846],[2,10.010],[1,14.481],[3,10.010],[1,12.980],[1,10.010],[1,12.412],[1,11.712],[1,13.947],[1,16.183],[3,10.010],[1,11.078],[1,10.410],[1,11.278],[4,10.010],[1,12.145],[2,10.010],[1,11.345],[1,14.748],[2,10.010],[1,19.219],[1,10.010],[1,13.247],[1,10.010],[1,17.584],[1,10.010],[1,17.117],[1,12.513],[1,9.977],[1,16.350],[1,11.378],[1,12.880],[2,10.010],[1,14.314],[1,12.746],[1,10.010],[1,11.778],[1,12.746],[1,15.949],[1,10.644],[4,10.010],[1,19.119],[1,10.010],[1,12.479],[1,10.010],[1,19.386],[1,16.917],[1,3.738]]}

There does seem to be some DASH-like concepts in that, just no XML. (I hate people who don't use newlines for separators in a format when they could!
It's not vi-friendly!)

[dnslo...@gmail.com]

Flash is working again for me, so I continued my investigations:


NOTE: I don't know anything about Firefox programming. I understand
what a window such as the filename-choosing dialog box is: it stays
up until you either dismiss it or click on a Save or Open or whatever
there is for a finishing action.

I do NOT know what you call the type of display you get when I click
on the dancing-atoms icon: the type that goes away when you click
anywhere else in the firefox main-window, or in another window (e.g.
my Terminal window) elsewhere on the screen. In the case of the
example of this that comes up when I click on the dancing-atoms icon,
this is fine. You can get it back by clicking on the dancing-atoms icon
again. However, when it is a "modal" <whatever you call it> such
as the one that contains the "Assemble-only" and "Assemble-and-Convert"
buttons, there is a problem: if you click anywhere else (e.g. in
the Terminal window to check on contents of a directory), YOU CAN'T
GET THAT THING BACK, so you've lost your work.

====================================================================
When I clicked on the one-and-only item (with small bullet
at extreme left end), that widget went away and nothing else happened
until the end. I clicked on the dancing-atoms icon again to check that
the small bullet had changed to a small square, and it had. Then I
clicked elsewhere just to make that list widget go away.
At the end, the widget with Assemble-only and Assemble-and-Convert
came up...
Assemble-only chosen.
Nothing new appeared in Console after choosing Assemble-only.
Nothing new after clicking on the Save of filename-choosing dialog box.

Contents of Console window with CSS, JS, Security, Logging ON:
INFO:HLSNetStream:play(-1) akb48-janken-taikai-kashiwagi-yuki-no-real
INFO:HLSNetStream:seek(-1) akb48-janken-taikai-kashiwagi-yuki-no-real
INFO:Setting audio track to 0 akb48-janken-taikai-kashiwagi-yuki-no-real
INFO:HLSNetStream:close akb48-janken-taikai-kashiwagi-yuki-no-real

All the *.ts fragments appeared as files (mostly about 1/2 MB) in
/tmp/chunks. This directory had gone away when I checked afterwards.

A file qqq.mp4 (my choice of name) was created, which contained only:
 od -c qqq.mp4
 0000000  \0  \0  \0       f   t   y   p   i   s   o   m  \0  \0 002  \0
 0000020   i   s   o   m   i   s   o   2   a   v   c   1   m   p   4   1
 0000040  \0  \0  \0  \b   f   r   e   e  \0  \0  \0  \0   m   d   a   t
 0000060
 
I find that *sometimes* (I can't be any more specific) the Assemble-only/
Assemble-and-Convert widget comes up spontaneously when only a few of
the .ts fragments have come in -- long before the end. If I ignore
that widget and click somewhere else it goes away, and if I then
click on the dancing-atoms icon to see the list again, the square has
gone back to bullet. Nothing else happens even at the end. So there is
nothing saved. If however I click on Assemble-only I *still* do not
get any file saved at the end, except (sometimes??) a 8900-byte file
that ffprobe says has some issue with a "moov atom" -- sorry I forgot
to record it exactly but perhaps it says the "moov atom" is missing.


OBSERVATION: if I grab the URL ending in .m3u8 from the Network-tab of
the Inspector and feed it to youtube-dl, it saves the file successfully by
invoking ffmpeg: you see all the fragments being processed by ffmpeg
in my Terminal window. A useful MP4 file is produced but has a .m3u8
extension on it. mpv displays this video just fine, but vlc will not
until you change the extension to .mp4. (I guess vlc believes the
extension while mpv actually looks into the file.)

OBSERVATION # 2: If I feed the same URL to ffmpeg *directly* thus:

ffmpeg -i http://proxy-69.dailymotion.com/sec(4fdd7b63a0dace5129f336840ad3492e)/video/237/416/63614732_mp4_h264_aac_hd.m3u8 blah.mp4

then ffmpeg assembles the fragments (you can see them all being obtained in
the Terminal window) and a good blah.mp4 file is produced.

Note that one of the advertised features of youtube-dl is that it will find the
highest resolution file (that you allow) for you. I didn't experiment but
I suspect this doesn't happen if you hand it the .m3u8 URL. Obviously ffmpeg
won't do this either.

Note that youtube-dl *cannot* handle the URL for the webpage containing this
embedded video. I.e. http://www.show48.com/nogibingo-s01e02/  It says
"Unsupported URL".

SUGGESTION: perhaps it is unnecessary to treat these "streaming" cases
as streaming and they can be treated (more or less) just like "normal"
cases, as there is a file listing all the fragments. In a sense it is
not streaming because there is a real "end" and a
real "beginning" unlike in a real "streaming" case. It's just using a
streaming protocol for delivering a file of finite end-points.
As the usual user isn't going to know anything about the protocol being
used, it would be more user-friendly to try to treat this as a normal
file from the GUI point of view, and just locate the .m3u8 file and
hand it to ffmpeg behind-the-scenes.

[mig]

That makes sense. I take good note of this for later versions.

[M.H.]

Hi I'm just confirming VDH does not detect anymore JW Player embedded videos (maybe since last JW player update 6.12.4956 Pro Edition, I don't know if the player was updated but it used to work a week ago). Example : www.film-streaming.co (only detects "screen capture" but no mp4 file is detected and suggested anymore in the list to be downloaded), I'm not a specialist at all and I maybe (probably) understood wrong, that it was because of the HTML5 file conversion which was not doing the job to make this mp4 appear in order for it to be detected by VDH ? Anyway... no solution and not clue

[mig]

Could you give a try using version 5.5.0a4 from https://groups.google.com/forum/#!topic/video-downloadhelper-5/9GqxBGEN-lU ?

[M.H.]

Hi, I downloaded and installed this version but icon still remains in grey and does not detect when the video begins to play

[M.H.]

I think the problem comes from the fact that no downloadable file is found in the source page codes (no .mp4 no .flw or so is available and also no http: so no url also) (although I read that a converter had to be installed which did not have to previously when it used to work, so that's what I did but nothing changed