Ankur,
I use that same basic approach to serve thousands of blobs per day in my app, and the average response time in the app engine logs is 15ms. The blobs I'm serving are text data and they vary in size from a few hundred bytes to 1MB and larger. The average size is probably around 200KB.
I wonder if anyone can comment on the possibility that the blobstore service takes longer to serve video content than it does text content. I doubt it, but I suppose it is possible.
Are you sure that the latency you're seeing is actually in app engine and not perhaps in network transit or even in your browser while it loads a viewer for that mp4 content?
Jim