Hello,
I am working on an iOS video telephony app that sends its own custom video source via WebRTC.
This video source is implemented as RTCVideoCapturer, which passes each frame via delegate.capturer(self, didCapture: frame).
For. every frame I generate and set the timeStampNs.
The video stream is then sent to the server and from there to all participants in a session.
Each participant can interact with the video and click with the mouse at a point in the video image. The coordinates are then sent back to the iOS device together with the RTP timestamp.
The iOS device must now process the interaction.
Due to transmission delays and movements of the device, the click coordinate never matches the actual position where the other participant clicked in the video image.
For this reason, I need the timeStampNs I originally generated per frame so I know which video frame the user saw when they clicked to compensate the delays.
However, the RTP timestamp sent by WebRTC, which arrives at the server and the participants, no longer correspond to my timeStampNs of the frame that I generated.
The RTP timestamp that arrives at the server counts up, but starts at a random starting point for each new session, that cannot be read anywhere.
Can I set or read the RTP timestamp that actually goes to the server?
I have already tried to derive from RTCDefaultVideoEncoderFactory to set a callback for the generated encoder to get the RTCEncodedImage, but the callback is never called.
Are there any possibilities or suggestions?
Thank you very much,
Christoph