Check you are using the recommended encoding for the M4V files:
http://www.jplayer.org/latest/developer-guide/#jPlayer-media-encodingIt looks like the Flash is not happy with the way your media is encoded.
I take it you are toggling the OGA files on and off the easy way?
By setting {supplied:"m4v"}
As for speed issues... Firefox was using the OGV files and they seemed to download fast enough for me. Way before the video ended.
Buffering though... jPlayer is not too sensible with the buffering. The Flash will play it in short parts as they become available. The HTML5 media seems to do a similar thing, playing it in small parts. Youtube's Flash player tends to buffer it more sensibly and waits for a larger chunk to download and be ready before continuing.