quick shot: you have mp4 files hosted over http and a user must download the
whole video file before it starts playing. The reason is that the metadata is
at the end of the file instead of the beginning (technically "mdat atom is
before the moov atom").
One solution is 'qtfaststart.py' + 'qt-faststart' from ffmpeg.
In general: These client-side issues are best investigated with Firebug or the
Chrome/Safari Web Inspector (the latter is often even a bit better). Next step
could be a network sniffer (like wireshark).
fs