The issue is that the Flash viewer used byte-ranges when streaming is false. (same applies to the other webviewers)
This works fine in IIS for static files, but in this case the file bytes are sent from a controller action.
There is default support for byte-ranges in the controller action, so this was the problem.
After using this library: