Meta data in DASH/HLS

[Aleksey Ivanov]

Hello,

I would like to use Shaka Packager to generate content for streaming via DASH/HLS.

However, I need to be able to add meta data to streams.

As far as I know with HLS this could be done using ID3 tags and DASH has concept of events (Inline and Inband).

I have read Shaka documentation but I didn't noticed anything about working with meta data.

Is there any way to append meta data to generated DASH/HLS content?

Thank you.

[KongQun Yang]

Hi Aleksey,

That is not supported directly right now. Can you elaborate a little bit more on what kinds of metadata are you thinking of?

Another option would be to write a script to post-edit the generated DASH manifest / HLS playlists outside of Shaka Packager.

-- KongQun Yang (KQ)

--
You received this message because you are subscribed to the Google Groups "Shaka Packager Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to shaka-packager-users+unsub...@googlegroups.com.
To post to this group, send email to shaka-packager-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/shaka-packager-users/c464dc83-153b-42ea-affa-f2d3e705f8a2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Aleksey Ivanov]

We already have a working solution with HLS and timed meta-data (as ID3 tags).

Those tags represent time-marks to sync video stream with chat.

As far as I understand we insert current video progress (i.e. time since start) in segments as ID3 tags, and messages in chat also have "time since start".

It allows us to show messages in chat only when a specific video moment (where message is related to) is actually occurs, despite of any delays.

Actually, I would like to be able to do something like this but on-the-fly.

Unfortunately, I need this time-tags for live streaming too, so, I can't just edit a manifest.

It looks like an ability to insert "time since start" would be useful for other developers who want to sync a video with chat (or something else), at least it wouldn't hurt anyone :) 

How do you think, does it make sense to implement such functionality within Shaka Packager (maybe with a flag to enable/disable it)?

Or maybe you know any workarounds to sync video with chat?

Thanks.

[KongQun Yang]

Hi Aleksey,

I see, but it seems redundant to me to add an ID3 tag to indicate time-marks since media files already contain timestamps. 

Have you thought about adding a time mark to chat texts, similar to what WebVtt is doing: https://tools.ietf.org/html/draft-pantos-http-live-streaming-23#section-3.5?

   In order to synchronize timestamps between audio/video and subtitles,
   a X-TIMESTAMP-MAP metadata header SHOULD be added to each WebVTT
   header.  This header maps WebVTT cue timestamps to MPEG-2 (PES)
   timestamps in other Renditions of the Variant Stream.  Its format is:

   X-TIMESTAMP-MAP=LOCAL:<cue time>,MPEGTS:<MPEG-2 time>
   e.g. X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000

It seems to be easier to support and more spec compliant. Let us know what you think.

-- KongQun Yang (KQ)

--

You received this message because you are subscribed to the Google Groups "Shaka Packager Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to shaka-packager-users+unsub...@googlegroups.com.
To post to this group, send email to shaka-packager-users@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/shaka-packager-users/7b8b5bec-419c-48ed-97a8-f77278a93fa6%40googlegroups.com.

[Aleksey Ivanov]

Thank you KQ.

Yes, I wonder if video segments have some kind of timestamps.

But I am not sure what kind those timestamps are of?
And can client code have access to them from a player? For example, can one determine exact timestamp that is playing right now?
Especially, can it be done for live streaming (not just for video on demand)?

> Have you thought about adding a time mark to chat texts, ...

If I understand you right we already do it. 

In the previous message I wrote that we use "time since start" for synchronization. But it loos like we use Unix timestamps.
So, server marks all chat messages with a timestamp once it receives it. Also server include timestamps in HLS segments while generating whose segments.

And then chat on client-side can show a message exactly when timestamp from HLS segment and timestamp of the message matches.

[KongQun Yang]

Hi Aleksey,

Yes, you can get timestamp from a player. +Joey from Shaka Player team who can answer player related questions.

-- KongQun Yang (KQ)

On Tue, Dec 5, 2017 at 3:21 AM, Aleksey Ivanov <iale...@gmail.com> wrote:

Thank you KQ.

Yes, I wonder if video segments have some kind of timestamps.

But I am not sure what kind those timestamps are of?
And can client code have access to them from a player? For example, can one determine exact timestamp that is playing right now?

Especially, can it be done for live streaming (not just for video on demand)?

> Have you thought about adding a time mark to chat texts, ...

If I understand you right we already do it. 

In the previous message I wrote that we use "time since start" for synchronization. But it loos like we use Unix timestamps.
So, server marks all chat messages with a timestamp once it receives it. 

Also server include timestamps in HLS segments while generating whose segments.

That shouldn't be necessary as the media already contains timestamps. 

And then chat on client-side can show a message exactly when timestamp from HLS segment and timestamp of the message matches.

--

You received this message because you are subscribed to the Google Groups "Shaka Packager Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to shaka-packager-users+unsubscrib...@googlegroups.com.

To post to this group, send email to shaka-packager-users@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/shaka-packager-users/a669075e-eec8-446e-8aee-725954640d76%40googlegroups.com.

[Aleksey Ivanov]

Those timestamps would be relative to a video starting moment, right?

Hello Joey, could please take a look at the discussion?

[Aleksey Ivanov]

Hello guys,

please let me share my thoughts about this once again.

We have to sync chat with both live and vod video streams.
I guess live case is more difficult than vod one.

KQ, you mentioned WebVtt uses X-TIMESTAMP-MAP tags in order to sync with the video.
It allow us to map cue timestamp to a video timestamps.

But does it work for live stream? Is WebVtt used for live streaming at all?

Consider the following case:
There is a live stream that has been started, say, 10 minutes ago.
A new listener joins to the stream.

A teacher writes a message to chat. Server marks the message with "time since stream start" timestamp (let it be 10:05) and sends it to clients. 
Each client has a video playback delay. So, the message can't be shown right now because a moment it was sent on isn't yet occurred on the client-side.

At some point a player should somehow understand that "this exact moment the message was sent on by the teacher" is playing right now and show the message.

I wonder, can a listener understand how long ago the stream was started on server when it connects to the stream?

The listener's player should understand that it is playing 10:05 moment and not just 00:05 in order to correctly map timestamps.

It seems to me that there is nothing we could know it from.

The stream is just RTP data. Shaka Packager doesn't support RTP, so we wrap it in MPEG-TS (for instance, with FFmpeg).
I guess RTP doesn't have any information about overall stream duration. Maybe this information is present on MPEG-TS container-level?

Please, let me know if my examples are not clear. I will try to reword them.

Thank you.

[KongQun Yang]

-- KongQun Yang (KQ)

On Tue, Dec 12, 2017 at 1:41 PM, Aleksey Ivanov <iale...@gmail.com> wrote:

Hello guys,

please let me share my thoughts about this once again.

We have to sync chat with both live and vod video streams.
I guess live case is more difficult than vod one.

KQ, you mentioned WebVtt uses X-TIMESTAMP-MAP tags in order to sync with the video.
It allow us to map cue timestamp to a video timestamps.

But does it work for live stream? Is WebVtt used for live streaming at all?

I don't know if WebVtt is used  for live streaming, but I don't see why it wouldn't work.

Consider the following case:
There is a live stream that has been started, say, 10 minutes ago.
A new listener joins to the stream.

A teacher writes a message to chat. Server marks the message with "time since stream start" timestamp (let it be 10:05) and sends it to clients. 
Each client has a video playback delay. So, the message can't be shown right now because a moment it was sent on isn't yet occurred on the client-side.

At some point a player should somehow understand that "this exact moment the message was sent on by the teacher" is playing right now and show the message.

I wonder, can a listener understand how long ago the stream was started on server when it connects to the stream?

The listener's player should understand that it is playing 10:05 moment and not just 00:05 in order to correctly map timestamps.

It seems to me that there is nothing we could know it from.

 Yes, I assume the player is able to know the timestamp it is playing. Joey, any comment?

The stream is just RTP data. Shaka Packager doesn't support RTP, so we wrap it in MPEG-TS (for instance, with FFmpeg).
I guess RTP doesn't have any information about overall stream duration. Maybe this information is present on MPEG-TS container-level?

Yes, there is timestamp information in MPEG-TS container. 

Please, let me know if my examples are not clear. I will try to reword them.

Thank you.

--

You received this message because you are subscribed to the Google Groups "Shaka Packager Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to shaka-packager-users+unsubscrib...@googlegroups.com.
To post to this group, send email to shaka-packager-users@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/shaka-packager-users/9ffa9e38-2987-49d7-856f-10b582be914a%40googlegroups.com.