synchronizing metadata with video track

[Oren Shir]

Hi,

In theory, the specification allows us have a text track synchronizes with the video track, so I can send in one bundle video and metadata (text, overlays etc.). I couldn't find an API for that in the browser. It doesn't look like tha data channels guarantee synchronization with the video, and there are only video and audio tracks in a video stream. Am I missing something? Is there a way to receive a metadata stream in the browser?

Thanks,

Oren

[Alexandre GOUAILLARD]

I do not remember such API in the webrtc / device Api spec. Can you point me to the one you are referring to?

Note that the HTML <video> has an API for it, when the source is not a MediaStream. A new group is under discussion at W3C for "Timed Text" if I recall correctly, but hasn't started yet.

datachannels are independent are are not sync with MS  or MST.

There are many ways to receive a metadata stream in the browser. There are (AFAIK) no way to received timed text or data that would be sync'ed with a MediaStream through the webRTC 1.0 APIs, today.

Alex.

--

---
You received this message because you are subscribed to the Google Groups "discuss-webrtc" group.
To unsubscribe from this group and stop receiving emails from it, send an email to discuss-webrt...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/5fe27c21-99fd-4298-8ab9-7a1a8138c819%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Alex. Gouaillard, PhD, PhD, MBA

------------------------------------------------------------------------------------

Principal Architect - Citrix, San Francisco

President - CoSMo Software Consulting, Singapore

------------------------------------------------------------------------------------

sg.linkedin.com/agouaillard

• 
  

[Oren Shir]

There isn't such an API at the PeerConnection level, but at the protocol level nothing is preventing us from describing data streams in SDP and transmitting them over RTP/SRTP. I could change the native code to add support for such channels, but there is no way to access them in the browser. I'll have to check the timed text group progress, and until then hack a solution.

[Alexandre GOUAILLARD]

you said "the specification allows us have a text track synchronizes with the video track".

Can you tell me which specification you were referring to in the above sentence?

APIs in the browsers are specified by the W3C, protocols (and content) used on the wire are specified by the IETF. Something might be possible with the protocols (IETF specs), but if there is no corresponding JS API (W3C spec) your web app will not have access to it. Only the browser can use it.

You can modify browser code if you want, but how would you have your users download the modified version and install them? In practice you are limited by what browser vendors provide, that is, if you want to have a web app.

if you use a native app, and only interact with other native app, you are free to do anything you want, and not interoperate.

Alex.

To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/e40a999b-879b-4729-b369-2dde3fe7b1b1%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Oren Shir]

Sorry, I should have been more clear about which specs I'm talking about.

SDP and RTP can be used to bundle metadata with video.

MediaStream spec has a section about extensions that describes how to add more media types.

I could add metadata to the native library and use it in mobile too, but I'm stuck on the browser because in this case no one is going to install my browser. I'll have to find another solution.

Thanks,

Oren

[Alexandre GOUAILLARD]

oh, I see what you mean now.

Correct,

You can decide to add any RTP Extension, or any new codec for that matter (some browsers actually have more audio codecs than the Mandatory to implement).

However beyond what is MTI, you have no guarantee that the other party (e.g. the browser) will have it. That's why there are mandatory to implement codecs (and corresponding payload, extension, ... specs): to assure minimum interoperability.

Some vendors already add some "optimized codecs" in their mobile SDK. Tokbox adds an SVC-enabled version of VP8 in their mobile SDK. If the mobile app written on top of this SDK tries to connect to a web app, the call will not used that optimized version of VP8 and fallback to VP8 or H264, but if you make a call from mobile to mobile, you will benefit from the optimized codec.

alex.

To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/0881dba2-4db9-4c78-9f7f-05ccc88942aa%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.