Not specifically, but I did notice that in the preferences there are some options for the viewport hardware renderer. I haven't really looked at them but you might.

I missed your reference to H264 and JPG. You say "whatever the codec" but are they the only two methods you've used?
Both are very compressed formats. Try outputting to a TIF sequence or to a Quicktime with Animation compression. Failing that, output uncompressed -- if that gets rid of it you know it's compression that is the issue
(seems likely anyway).
To obtain the fix for this issue, set MAYA_FLOATING_POINT_RT_PLAYBLAST to 1.