Sync video frame on receive side with external data

[Yuri]

Hi folks, I'm looking for way to sync. video frame in receiver with external data using webrtc c++. Is any way to add frame identification on "sender side" and then identify it on "receiver side"? (example: sync. subtitles for video stream)

I tried using:

- webrtc::VideoFrame::set_timestamp_ms() but it's change on received side

- webrtc::VideoFrame::set_ntp_time_ms() but get "-1" on received side

- webrtc::VideoFrame::set_id() but get "0" on received side

Any idea? 

Thanks

[Yuri]

Any ideas? 

May using "RTP header extension" like: https://webrtc.github.io/webrtc-org/experiments/rtp-hdrext/abs-capture-time/ can helps? 

[bin...@outlook.com]

You may try to use the multiplex codecs

https://chromium.googlesource.com/external/webrtc/+/lkgr/modules/video_coding/codecs/multiplex/multiplex_encoder_adapter.cc

[Yuri]

Thanks for replay,

It's good solution but unfortunately how i see it augmented_video_frame_buffer and it's change video encoder name in SDP from "vp8/vp9/h264" to "multiplex" so i can't show it frame in chrome browser. 

[bin...@outlook.com]

Yes, it will only works in custom native apps, web browsers support is a problem...

[Yuri]

I listened to "abs-capture-time" seems it can helps me.

I thin it was added in m92 release. (https://bugs.chromium.org/p/webrtc/issues/detail?id=10739)

So I interesting if it really can helps me and how to use it

[刘连响]

Hi  Yuri，You can use webrtc insertable stream to add some meta information to each frame.

Yuri <yu...@commonground-ai.com> 于2021年7月27日周二 下午11:34写道：

--

---
You received this message because you are subscribed to the Google Groups "discuss-webrtc" group.
To unsubscribe from this group and stop receiving emails from it, send an email to discuss-webrt...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/82e3c424-33e0-4e0b-8dcc-467ce01ad3ddn%40googlegroups.com.

[Yuri]

Hi leeoxiang, 

"insertable" is still experiment flag but i think it really good solution for me . Thanks you 

[Yuri]

I found that create "insertable streams" from JS is simple (to example: https://levelup.gitconnected.com/adding-metadata-to-media-frames-via-insertable-streams-api-for-webrtc-47f7a740e457) but didn't found examples to C++. 

Folks share please with some example

[V I]

there are some pointers here: https://groups.google.com/g/discuss-webrtc/c/WkY9Ax2Pnn0

To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/82f87326-2aee-42fa-b498-f9d62cc81d73n%40googlegroups.com.

[Yuri]

Thanks for link but I listened to this and https://groups.google.com/a/chromium.org/g/blink-dev/c/XFO4OXrdSRA and https://groups.google.com/g/discuss-webrtc/c/GWH9ce94Eqw/m/M9uNccrxCQAJ thread  but still thinks it's not complete and maybe wrong ways

[Christophe Chaillon]

Hi,

You should override webrtc::FrameTransformerInterface 2 times, one for the encoder, one for the decoder

And call SetEncoderToPacketizerFrameTransformer(MyFrameEncoder) or SetDepacketizerToDecoderFrameTransformer(myFrameDecryptor); on the video track

I have to add my information at the end of the pFrame otherwise I lose data but it works.

Christophe

To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/0ba5a223-373c-4854-9fe6-b627a1c3c7dbn%40googlegroups.com.