Experiment with frame number

[David Humphrey]

I talk to a lot of web devs who say they want to know not only the
video's current time, but also the current frame number. I realize the
spec doesn't have any notion of this, but I'd like to do an experimental
build that exposes this, and let them try it--see how significant it
actually is in the real world vs. "if only I could..."

My question is this: I assume we keep track of the current frame (like
we do with current time) somewhere in the media decoder, and I'm
wondering if one of you would be kind enough to point me in the right
direction.

Thanks,

Dave

[Paul ADENOT]

Maybe they could experiment using proprietary API already available on
the <video> element [1].

If those does not suit their needs, could we get a precise description
of what is needed?

Paul

[1]:
http://mxr.mozilla.org/mozilla-central/source/dom/interfaces/html/nsIDOMHTMLVideoElement.idl#31

> _______________________________________________
> dev-media mailing list
> dev-...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-media

[Robert O'Callahan]

On Mon, May 6, 2013 at 5:27 AM, David Humphrey <
david.h...@senecacollege.ca> wrote:

> I talk to a lot of web devs who say they want to know not only the video's
> current time, but also the current frame number. I realize the spec
> doesn't have any notion of this, but I'd like to do an experimental build
> that exposes this, and let them try it--see how significant it actually is
> in the real world vs. "if only I could..."
>

This has been requested before on the WHATWG list.

Some video formats allow each frame to have a different duration, so you'd
have to be quite careful about what you mean by "current frame number".

Rob
--
q“qIqfq qyqoquq qlqoqvqeq qtqhqoqsqeq qwqhqoq qlqoqvqeq qyqoquq,q qwqhqaqtq
qcqrqeqdqiqtq qiqsq qtqhqaqtq qtqoq qyqoquq?q qEqvqeqnq qsqiqnqnqeqrqsq
qlqoqvqeq qtqhqoqsqeq qwqhqoq qlqoqvqeq qtqhqeqmq.q qAqnqdq qiqfq qyqoquq
qdqoq qgqoqoqdq qtqoq qtqhqoqsqeq qwqhqoq qaqrqeq qgqoqoqdq qtqoq qyqoquq,q
qwqhqaqtq qcqrqeqdqiqtq qiqsq qtqhqaqtq qtqoq qyqoquq?q qEqvqeqnq
qsqiqnqnqeqrqsq qdqoq qtqhqaqtq.q"

[Ralph Giles]

On 13-05-05 10:27 AM, David Humphrey wrote:
> I talk to a lot of web devs who say they want to know not only the
> video's current time, but also the current frame number. I realize the
> spec doesn't have any notion of this, but I'd like to do an experimental
> build that exposes this, and let them try it--see how significant it
> actually is in the real world vs. "if only I could..."

Why do they want to know this, and why does dividing currentTime by the
framerate not work?

> My question is this: I assume we keep track of the current frame (like
> we do with current time) somewhere in the media decoder, and I'm
> wondering if one of you would be kind enough to point me in the right
> direction.

You could try exposing VideoData::mTimecode, but that's format-specific.
You'd probably want to add an mFrameIndex instead.

http://dxr.mozilla.org/mozilla-central/content/media/MediaDecoderReader.h#l113

HTH,
-r

[Randell Jesup]

On 5/6/2013 3:31 PM, Ralph Giles wrote:
> On 13-05-05 10:27 AM, David Humphrey wrote:
>> I talk to a lot of web devs who say they want to know not only the
>> video's current time, but also the current frame number. I realize the
>> spec doesn't have any notion of this, but I'd like to do an experimental
>> build that exposes this, and let them try it--see how significant it
>> actually is in the real world vs. "if only I could..."
> Why do they want to know this, and why does dividing currentTime by the
> framerate not work?

framerate is not a constant. ;-)

Applications (especially realtime) will want to show the "current"
framerate (for some definition of current - last second, etc). They may
want to take actions based on it as well.

Other apps use frame arrival as a way to know if a connection is still
working (they can also use the currenttime, though).

--
Randell Jesup, Mozilla

[Ralph Giles]

On 13-05-06 1:01 PM, Randell Jesup wrote:

> framerate is not a constant. ;-)

I was imaging they wanted to find a particular frame to cut-and-paste,
e.g. sprites. Or only run an expensive calculation when the frame data
changes.

> Applications (especially realtime) will want to show the "current"
> framerate (for some definition of current - last second, etc). They may
> want to take actions based on it as well.

Aha. It should be possible to try that with the statistics extensions
Paul mentioned. These are already available in Firefox:

video.mozParsedFrames
video.mozDecodedFrames
video.mozPaintedFrames
video.mozPresentedFrames

See
https://developer.mozilla.org/en-US/docs/DOM/HTMLVideoElement#Gecko-specific_properties
for a brief description.

-r

[Randell Jesup]

On 5/6/2013 5:29 PM, Ralph Giles wrote:
> On 13-05-06 1:01 PM, Randell Jesup wrote:
>
>> framerate is not a constant. ;-)
> I was imaging they wanted to find a particular frame to cut-and-paste,
> e.g. sprites. Or only run an expensive calculation when the frame data
> changes.

In theory (just guessing now), they might want to do a 'cut' on a
specific frame, though one would expect that could be converted to a
time. However, it depends on where they got the cut-point from. And not
all video is high-frame-rate; sometime it's effectively stop-motion -
think a security cam that only sends frames when something "interesting"
is happening. Though again you may be able to key off the current time
- if it doesn't advance on audio, which it may. But as I say, I'm guessing.

>
>> Applications (especially realtime) will want to show the "current"
>> framerate (for some definition of current - last second, etc). They may
>> want to take actions based on it as well.
> Aha. It should be possible to try that with the statistics extensions
> Paul mentioned. These are already available in Firefox:
>
> video.mozParsedFrames
> video.mozDecodedFrames
> video.mozPaintedFrames
> video.mozPresentedFrames

Ok, cool. I wonder how many of those apply to mediastream sources.

--
Randell Jesup, Mozilla

[Ralph Giles]

On 13-05-06 3:13 PM, Randell Jesup wrote:

> Ok, cool. I wonder how many of those apply to mediastream sources.

On http://mozilla.github.io/webrtc-landing/pc_test.html only
mozPaintedFrames seems to update. All the others return zero.

mozFrameDelay works, but I hope it's wrong. Its value varies from 2-10
seconds.

-r