syncing audio and video

[Daniel]

Hi,

I'm well on my way in creating a basic videoplayer using pyffmpeg and
pygame, but I have some trouble getting the audio and video in sync.
I've tried using the set_hurry() function of videotrack or specifying
the 'hurry_up' parameter in the init of the videotrack class. Neither
had any effect. I've been reading about the syncing topic a bit as
well in other sources and found out I have to use the PTS data to sync
both streams, but I have no idea how to do this. I've tried setting
the pts of the video to the pts of the audio, using the get_cur_pts
from the audiotrack and the seek_to_pts of the videotrack, but this
just seems to get the program into an infinite loop.
Is there any example/documentation/source where I can found out how to
tackle this using pts or the 'hurry' parameters, or has anyone already
solved this using pyffmpeg? I tried using http://dranger.com/ffmpeg/tutorial05.html
as a lead, but didn't get any wiser.

Furthermore, when I play a file which has no audiotrack, the video
gets played way to fast. I've tried to solve this with a wait() of
sleep() after rendering a frame, but this slowed the video down too
much again when I used the simple calculation
pygame.time.wait(frameTime - calculationTime)

where frametime =1000/videotrack.get_fps() (1000 ms / fps of the
movie)
and calculationTime is the time lost by processing the frame and
presenting it to the display.

a quick hack was to just shorten the sleeptime by a fixed number, but
this seems a very bad solution. Is there a way to more efficiently get
the display time per frame to use in a sleep() function or another way
to make sure the movie doesn't haste when there is no sound present?

Thanks very much for any help, and I would like to congratulate the
whole pyffmpeg crew for the great work they have done!


For anyone interested; here's what I have written now:
--------------------------------------------------------------------------------------------------------------------------------------------------

import pyffmpeg
import pygame
import pyaudio
import thread
import sys
import traceback


class MediaPlayer():
def __init__(self, vfile="",screensize="default"):
"""
Initialize module. Link to the video can already be specified
but this is optional
file: The path to the file to be played
screensize: An optional tuple with the preferred (w,h) of the
output frame. Otherwise, the video is played in its original size
"""
pygame.init()
pygame.display.init()
self.mp = pyffmpeg.FFMpegReader(0,False)

self.videoTrack = None
self.audioTrack = None
self.file_loaded = False
self.frameTime = 0
self.screensize = screensize
self.paused = False

if vfile != "":
self.load(vfile)

def load(self,vfile):
"""
Load a videofile and make it ready for playback
vfile: The path to the file to be played
"""
if self.screensize == "default":
TS_VIDEO_RGB24={ 'video1':(0, -1,
{'pixel_format':pyffmpeg.PixelFormats.RGB24,'videoframebanksz':1}),
'audio1':(1,-1,{})}
elif type(self.screensize)==tuple and len(self.screensize)==2:
TS_VIDEO_RGB24={ 'video1':(0, -1,
{'pixel_format':pyffmpeg.PixelFormats.RGB24,'videoframebanksz':
1,'dest_width':self.screensize[0], 'dest_height':self.screensize[1]}),
'audio1':(1,-1,{})}
else:
raise ValueError

#Check if audio stream is present, if not only process the
video
try:

self.mp.open(vfile,TS_VIDEO_RGB24) # will
throw IOError if it doesn't find an audio stream
(self.videoTrack,self.audioTrack) = self.mp.get_tracks()
self.samplingFreq = self.audioTrack.get_samplerate()

#pygame.mixer.init(frequency=self.audioTrack.get_samplerate(),
channels=self.audioTrack.get_channels())
self.audioTrack.set_observer(self.handleAudioFrame)
print "Channels: %s" % self.audioTrack.get_channels()
self.audiostream =
pyaudio.PyAudio().open(rate=self.samplingFreq,

channels=self.audioTrack.get_channels(),

format=pyaudio.paInt16,
output=True)
print "as: %s" % self.audiostream

self.hasSound = True
except:
print "Error: %s:\n%s\n%s" % (sys.exc_info()
[0],sys.exc_info()[1],traceback.print_tb(sys.exc_info()[2]))
try:
del TS_VIDEO_RGB24['audio1']
self.mp.open(vfile)
except IOError:
print "File not found"
sys.exit()
else:
self.videoTrack = self.mp.get_tracks()[0]
self.hasSound = False
print "No audio track found"

self.videoTrack.set_observer(self.handleVideoFrame)

self.videoTrack.check_start()
self.audioTrack.check_start()

print "FPS: %s" % self.videoTrack.get_fps()
self.frameTime = 1000/self.videoTrack.get_fps()
print "Frametime: %s" % self.frameTime

# Set pygame window to size of video, check for
self.screensize in future
if type(self.screensize)==tuple and len(self.screensize)==2:
pygame.display.set_mode( self.screensize,
pygame.DOUBLEBUF)
else:
pygame.display.set_mode( self.videoTrack.get_size(),
pygame.DOUBLEBUF)
# Get the pygame surface to draw to
self.screen = pygame.display.get_surface()
self.file_loaded = True

# The video and audio callback methods.
def handleVideoFrame(self, f):
"""
Callback method called by the VideoTrack object for handling a
video frame
"""
start = pygame.time.get_ticks()
pygame.surfarray.use_arraytype("numpy")
f=f.swapaxes(0,1)
pygame.surfarray.blit_array(self.screen,f)
pygame.display.flip()
self.frame_calc_start = pygame.time.get_ticks()


def handleAudioFrame(self, a):
"""
Callback method called by the AudioTrack object for handling
an audio frame
"""
self.audiostream.write(a[0].data)
#print "Audioframe ready!"


def pause(self):
"""
Pauses playback. Continue with unpause()
"""
self.paused = True

def unpause(self):
"""
Continues playback
"""
self.paused = False

def play(self, eventHandler=None):
"""
Starts the playback of the video file. You can specify an
optional callable object to handle events between frames (like
keypresses)
This function needs to return a boolean, because it determines
if playback is continued or stopped. If no callable object is provided
playback will stop when the ESC key is pressed
"""
if self.file_loaded:
self.playing = True
while self.playing:
try:
if self.paused == False:
print "--- Begin step ---"
self.mp.step()
print "Videotrack pts: %s" %
self.videoTrack.get_cur_pts()
print "Audiotrack pts: %s" %
self.audioTrack.get_cur_pts()
print "Offsycn: %s" %
(self.audioTrack.get_cur_pts()-self.videoTrack.get_cur_pts())

#self.videoTrack.set_hurry(self.audioTrack.get_cur_pts()-
self.videoTrack.get_cur_pts())
print "--- End step ---"

except IOError:
self.playing = False
print "Warning: %s: %s" % (sys.exc_info()
[0],sys.exc_info()[1])
else:
if callable(eventHandler):
result = eventHandler()
if type(result) == bool:
self.playing = result
else:
print "Invalid return type of specified
event handler: must be bool"
self.playing = False
else:
e = pygame.event.poll()
if e.type== pygame.KEYDOWN:
if e.key== pygame.K_ESCAPE:
self.playing = False
if e.key== pygame.K_SPACE:
if self.paused == True:
self.paused = False
else:
self.paused = True
#Sleep for remainder of a frame duration that's
left after all the processing time
if not(self.hasSound):
sleeptime = int(self.frameTime -
pygame.time.get_ticks() + self.frame_calc_start)
print "Sleeptime: %s" % sleeptime
pygame.time.wait(sleeptime)

# Clean up on aisle 4!
self.mp.close()
self.audiostream.close()

else:
print "No video loaded"
return False


if __name__ == "__main__":
vfile = sys.argv[1]

player = MediaPlayer(vfile)
player.play()
pygame.quit()

[Daniel]

Ok, solved it quick 'n dirty for now. I just created an internal
audiobuffer which stores a couple of audioframes before playing them:

def handleAudioFrame(self, a):
"""
Callback method called by the AudioTrack object for handling
an audio frame
"""

if len(self.audioBuffer)==3:
self.audiostream.write(self.audioBuffer.pop(0))

self.audioBuffer.append(a[0].data)

Things seem nicely synchronized for now, but I don't like it that this
doesn't use the pts of audio and video streams.
Whenever I tried to seek to a pts on either stream separately, the
other stream seemed to "go with it", as in that seek_to_pts() for
either audio- or videotrack, will cause the other track to jump to its
corresponding position as well. In other words, you can't use
seek_to_pts() for each stream separately?

There also seems to be a built in parameter in ffmpeg that syncs audio
and video, or at least enables you to set the delay for audio by
specified ms.

ffmpeg -i video_source -itsoffet delay -i audio_source -map 0:x -map
1:y .....

but I guess it is not possible to use this from pyffmpeg now? Has
anybody ever solved this sync problem before?

[Bertrand Nouvel]

Dear Daniel,

Thanks for these comments.  In current implementation, the audio is supposed to be synced with each video-frame.

(However, it is possible that a bug created an offset of a few frames. ) . In current implementation. also 

there is no way to seek one the audio and video-streams separately. And thus it indeed assumes

that the packets of the audio-streams and the packets of the video-streams are not too much 

dephased in the transport layer.

Regarding the player implementation, for playing sound and video at the same time,

The sample players provided where tiny samples aimed at demonstrating how to use the API.

I believe a clean design of a audio-video player  would  be multithreaded,

however so far even if some parts of the current design have been thought with 

"multithreading" in mind, the wrapper is only monothread.

Bertrand Nouvel

2011/3/10 Daniel <dsch...@gmail.com>

--
You received this message because you are subscribed to the Google Groups "pyffmpeg" group.
To post to this group, send email to pyff...@googlegroups.com.
To unsubscribe from this group, send email to pyffmpeg+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/pyffmpeg?hl=en.

[Daniel]

Hi Bertrand,

thanks for your reply. I think it has to do with the missing first
frame of the video decoding (I saw this mentioned as a bug somewhere
else).
I did manage to get pts involved in the syncing process, although not
completely succesfully. The latest handleAudioFrame looks like this:

def handleAudioFrame(self, a):
"""
Callback method called by the AudioTrack object for handling
an audio frame

To maintain sync, audio is stored in a buffer, and only sent
to the audio device
when the audiotrack pts is smaller or equal to the videotrack
pts
"""

# If audio pts is almost the same, play the buffered frame
if len(self.audioBuffer) and self.audioBuffer[0][1] <=
(self.videoTrack.get_current_frame_pts()+90000):
audioFrame = self.audioBuffer.pop(0)
self.audiostream.write(audioFrame[0])

# Store current audio frame in the buffer
self.audioBuffer.append( (a[0].data,
self.audioTrack.get_cur_pts()) )

one can play around with the value which is added to the video pts
(now 90000) to fine tune the sync. Ideally
self.audioBuffer[0][1] <= self.videoTrack.get_current_frame_pts()
should have done the trick, but doing it that way the video came ahead
of sound (instead of the other way around), so I still had to fiddle
around with it a bit.

Also I have some movies of which I get the warning:
[mpeg4 @ 02f5adf0] warning: first frame is no keyframe
after which playback is choppy (you can clearly see videoframes are
skipped). Also after videotrack.seek_to_frame() has been used,
playback is choppy, while playing any movie from its beginning gives
perfect playback throughout the movie. Any idea what causes this?