Audio Video Synchronization in RTP streaming

[vasudh...@yahoo.com]

Hi,
I am streaming aac audio and h264 video( a mp4 stream basically) as
RTP packets.Both audio and video packets are timestamped.
Acording to my knowledge,Directshow takes audio timestamps as the
reference and accordingly freezes video or drops the packets.
In case of video, I am taking the Timestamps from the video packet
payload in the FillBuffer() part of my source filter, converting it to
100 ns clock ( as required by the Dshow) , using pSample-
>SetTime(&tStart,&tStop) to throw it on to the pins. And in Transform
filter, I am again retriving the information using GetTime() and
sending it to my decoder.
But in case of audio, I am taking the Timestamps from the video packet
payload in the FillBuffer() part of my source filter, converting it to
100 ns clock ( as required by the Dshow) , using pSample-
>SetTime(&tStart,&tStop) to throw it on to the pins. But in the
AACTransform filter I have not taken any special care to send
Timestamps to the renderer pins.
So my question is will the Dshow get the Timestamps correctly or not??
Because my code is playing the audio correctly but initial video
frames are getting dropped and the audio-video synchronization is not
taking place.
or will I have to take special care (some code snippet ) in my AAC
transform filter because it is not getting the AAC Timestamps
correctly and not able to synch audio and video?

Regards.

[Geraint Davies]

If I remember, the timestamps on RTP streams are offset by a random number,
so you cannot just convert from the stream clock to 100ns and get
comparable timestamps. You need to get the RTCP sender report, and use the
RTP to NTP mapping to get a common base for the two streams.

And yes, your decoder should output timestamps. (All decoders should output
timestamps on all decoded output, except solely in the case where you have
never received any input timestamp. In other cases you should extrapolate
timestamps if they are not present on some input packets.)

And thirdly:

> But in case of audio, I am taking the Timestamps from the video packet
> payload in the FillBuffer() part of my source filter,

Is this a typo? You need to take the timestamps from the audio packet,
convert it to a common baseline and then to 100ns before setting it on the
sample.

G

[pavankeshav]

Hi,

I am facing similar problem. I am sending correct timestamps to both audio
video decoder and then to renderer, but not able to get audio-video sync.
Can someone plz help me on sync.

Regards,
Pavan

[sraddhan]

Hi,

I am wondering.... how can you directly play h264 frames? I am working on a
project where in we've converted h264 frames to YUV(4:2:0) frames and from
that we will convert it to RGB and then play the video.

Is it possible to play h264 directly <- seems impossible but if is it so
then please let me know.

[Chris P.]

Directly, no. Some sort of decoder is always required, the decoder may or
may not make use of hardware acceleration.

DirectShow can make this easier perhaps. Usually you can render YUV
directly, it shouldn't be necessary to convert to RGB.

--
http://www.chrisnet.net/code.htm
[MS MVP for DirectShow / MediaFoundation]

[imran khan]

HI
how do u solve this problem. i am also facing same problem.plz do reply ok

[jaymaniar]

Hi,
I m working on similar project.
I want to know about how to archive the file of the live streaming of audio & video.

imran khan wrote:

Re:Audio Video Synchronization in RTP streaming
26-Sep-07

HI
how do u solve this problem. i am also facing same problem.plz do reply ok

Previous Posts In This Thread:

On Wednesday, March 14, 2007 2:14 PM
vasudha_hegd wrote:

Audio Video Synchronization in RTP streaming

Hi,
I am streaming aac audio and h264 video( a mp4 stream basically) as
RTP packets.Both audio and video packets are timestamped.
Acording to my knowledge,Directshow takes audio timestamps as the
reference and accordingly freezes video or drops the packets.
In case of video, I am taking the Timestamps from the video packet
payload in the FillBuffer() part of my source filter, converting it to
100 ns clock ( as required by the Dshow) , using pSample-

filter, I am again retriving the information using GetTime() and
sending it to my decoder.
But in case of audio, I am taking the Timestamps from the video packet
payload in the FillBuffer() part of my source filter, converting it to
100 ns clock ( as required by the Dshow) , using pSample-

AACTransform filter I have not taken any special care to send
Timestamps to the renderer pins.
So my question is will the Dshow get the Timestamps correctly or not??
Because my code is playing the audio correctly but initial video
frames are getting dropped and the audio-video synchronization is not
taking place.
or will I have to take special care (some code snippet ) in my AAC
transform filter because it is not getting the AAC Timestamps
correctly and not able to synch audio and video?

Regards.

On Thursday, March 15, 2007 5:11 AM
Geraint Davies wrote:

Re: Audio Video Synchronization in RTP streaming

On 14 Mar 2007 12:14:55 -0700, vasudh...@yahoo.com wrote:

If I remember, the timestamps on RTP streams are offset by a random number,
so you cannot just convert from the stream clock to 100ns and get
comparable timestamps. You need to get the RTCP sender report, and use the
RTP to NTP mapping to get a common base for the two streams.

And yes, your decoder should output timestamps. (All decoders should output
timestamps on all decoded output, except solely in the case where you have
never received any input timestamp. In other cases you should extrapolate
timestamps if they are not present on some input packets.)

And thirdly:

Is this a typo? You need to take the timestamps from the audio packet,

convert it to a common baseline and then to 100ns before setting it on the
sample.

G

On Friday, April 20, 2007 1:08 AM
pavankeshav wrote:

Re:Audio Video Synchronization in RTP streaming
Hi,

Regards,
Pavan

On Saturday, June 02, 2007 5:14 AM
sraddhan wrote:

Re:Audio Video Synchronization in RTP streaming
Hi,

I am wondering.... how can you directly play h264 frames? I am working on a
project where in we've converted h264 frames to YUV(4:2:0) frames and from
that we will convert it to RGB and then play the video.

Is it possible to play h264 directly <- seems impossible but if is it so
then please let me know.

On Sunday, June 03, 2007 11:58 AM
Chris P. wrote:

Re: Audio Video Synchronization in RTP streaming

On Wednesday, September 26, 2007 3:52 AM
imran khan wrote:

Re:Audio Video Synchronization in RTP streaming

HI
how do u solve this problem. i am also facing same problem.plz do reply ok

Submitted via EggHeadCafe - Software Developer Portal of Choice
Dr. Dotnetsky's Cool .Net Tips and Tricks No. 26
http://www.eggheadcafe.com/tutorials/aspnet/8e390106-20d1-4b2b-be30-cde5d852348d/dr-dotnetskys-cool-net.aspx

[jaymaniar]

Hi,

I am working on similar project.
I want to know how can I archive of live streaming into a file. Pls tell me.

imran khan wrote:

Re:Audio Video Synchronization in RTP streaming
26-Sep-07

HI

how do u solve this problem. i am also facing same problem.plz do reply ok

Previous Posts In This Thread:

Regards.

And thirdly:

G

Regards,
Pavan

HI
how do u solve this problem. i am also facing same problem.plz do reply ok

On Friday, February 19, 2010 11:50 PM
jay maniar wrote:

File archive of live streaming

Hi,
I m working on similar project.
I want to know about how to archive the file of the live streaming of audio & video.

Submitted via EggHeadCafe - Software Developer Portal of Choice

Join Lists with LINQ - SharePoint 2010
http://www.eggheadcafe.com/tutorials/aspnet/b0c1cd0d-fe82-444e-a16e-7d3fb7d38eca/join-lists-with-linq--sh.aspx