PSA: JS-driven benchmarks may have dubious precision

[Alexis Beingessner]

I believe motionmark uses performance.now to determine how long things take.

https://developer.mozilla.org/en-US/docs/Web/API/Performance/now

To offer protection against timing attacks and fingerprinting, the precision of performance.now() might get rounded depending on browser settings.
In Firefox, the privacy.reduceTimerPrecision  preference is enabled by default and defaults to 20us in Firefox 59; in 60 it will be 2ms. 

In Firefox, you can also enable privacy.resistFingerprinting — this changes the precision to 100ms or the value of privacy.resistFingerprinting.reduceTimerPrecision.microseconds, whichever is larger.

on my dev machine and the toronto reference hardware both of these preferences were enabled. These mitigations do legitimately useful things, so I would maybe consider setting up a separate profile for benchmarking that disables these.

I'm still investigating details.

[Alexis Beingessner]

OK so yeah this seems to be a pretty big deal.

if you look here: https://docs.google.com/spreadsheets/d/1Mf3Mq6cQx8MmYuYycdyCNMaqAjk5uIz8EGFBm9cHBgw/edit#gid=387788813

i ran motionmark under 4 configurations today:

* performance.now unrounded, framerate tied to vsync

* performance.now unrounded, ASAP rendering

* performance.now rounded, framerate tied to vsync

* performance.now rounded, ASAP rendering

With vsync, the results of motionmark are high variance and unreliable

With performance.now rounding, webrender and gecko seem indistinguishable.

In the performance.now unrounded + ASAP configuration, which seems to be the "right" one to use, we get the kind of results we expect from using webrender:

* amazing performance wins on "normal" html content

* no difference for canvas tests (unaffected by wr)

* much small difference for svg tests (blobs are slow)

_______________________________________________
dev-tech-gfx mailing list
dev-te...@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-tech-gfx

[Alexis Beingessner]

minor correction: the 100ms anti-fingerprinting setting wasn't on, but the 2ms one was. Even 2ms rounding is evidently enough to mess up motionmark.