How do we measure FPS when the GPU is the bottleneck?

[Philip Rideout]

Our web app feels laggy on low-end machines because it does a lot of GPU-side work.  In extreme cases, we'd like the app to detect poor performance at run time, and automatically drop into a simplified state.

We're having trouble programmatically determining frame rate when the GPU is the bottleneck.  Measuring the interval between requestAnimationFrame isn't quite right, since it appears Chrome drops frames; RAF gets called more frequently than actual backbuffer presentation.

We've considered using failIfMajorPerformanceCaveat to detect these situations, but some middling machines (such as an Acer Chromebook 11) would make it past the caveat while still being too weak for our app's default visual quality settings.

Advice?

Philip

[Theo Armour]

Philip

Great question. I look forward to reading some clever responses. 

Theo

--
You received this message because you are subscribed to the Google Groups "WebGL Dev List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to webgl-dev-lis...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Philip Rideout]

It would be nice if there were an API for this.  It doesn't need to be fancy; we simply need a way to know when frames are being dropped.

Philip

[Brandon Jones]

Hm... I would say that Chrome frequently dropping frames should be considered a bug. I can see scenarios where we might drop a couple of frames here and there to switch from a high-latency pipeline to a low latency one, but that should be a rare event, and only represent a small blip in your FPS. Otherwise we should be reasonably confident that each rAF will actually make it to screen at some point. If the GPU is loaded down we should be issuing rAFs less often.

We recently found a bug on OSX where this definitely wasn't the case, and are taking steps to fix it, but it sounds like you're specifically seeing issues on Chromebooks? I'm not aware of any issues on that platform, so if you have a good repro case please file a bug about it!

--Brandon

[Philip Rideout]

Hi Brandon,

I'm happy to file a crbug, but I just to double check that it is this really is unexpected behavior when the page does stupidly slow things with GPU.  It seems like it must be somewhat normal, given the existence of this working draft from mpb@google and igrigorik@google:

    http://www.w3.org/TR/frame-timing/

Incidentally, I just came across the above draft and it looks pretty exciting!  In fact, I think it answers my question.  Wish I could +1 it somehow.  :)

Philip

--
You received this message because you are subscribed to a topic in the Google Groups "WebGL Dev List" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/webgl-dev-list/KWTOpCWtUpY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to webgl-dev-lis...@googlegroups.com.

[Theo Armour]

Hi Philip

You seem to have found a future answer to your problems but what about the present?

Your original post covers several topics. This response covers your first point:

Our web app feels laggy on low-end machines because it does a lot of GPU-side work.  In extreme cases, we'd like the app to detect poor performance at run time, and automatically drop into a simplified state.

Since I have wondered about this issue myself and there were no direct responses to this aspect, I have looked into discovering lagginess before it happens.

The seminal work - one of the few I know about - is Don Olmstead's presentation at the 2013 HTML 5 conference in San Francisco titled Optimizing WebGL Animations.

I have tried to penetrate the depth of observations that Don offers more than once, but after some hours my tiny brains fails me. If nothing else there is a whole learning curve regarding successful use of the JavaScript console that I have never completed.

In any case, what I am looking for is something that just pops out a simple number telling me that my script is running slowly. This should be part of the script and have no need to access the console. A search of the web offered no direct response, thus I was forced to do some more investigating myself.

In matters of this nature, the first shoulders I look to stand on are, of course: Mr.doob's. His Stats.js is frequently referenced - even by non-Three.js scripts.

Typically the little Stats.js box that displays on a corner of your screen displays the frames per second that your script is displaying. Clicking in the box cause the milliseconds between frames to display. Clicking again display the memory being used. With luck your script is displaying at 60 fps - the maximum allowed - and thua taking about 16 ms per frame. 

These measurements are great but if your frame rate drops below 60 then it is *already too late*. What is needed is an indication that the script is heading in the direction of dropping below 60 fps well before this event occurs.

So I started thinking about writing such a mechanism, but, of course, Mr.doob has taken care of that already. Stats.js has two calls to help: stats.begin() and stats.end() used this way:

var update = function () {

    stats.begin();

    // monitored code goes here

    stats.end();

    requestAnimationFrame( update );

};

requestAnimationFrame( update );

This display tells you how many milliseconds your script takes to process the frame. Actually I seem to like the measurement done this way:

var update = function () {

    stats.end();

    // monitored code goes here

    stats.begin();

    requestAnimationFrame( update );

};

requestAnimationFrame( update );   

This display tells you how many milliseconds is available to other apps and processes. When this number drops down to low single digits, your script - and mine now - better start reducing its use of resources.

Here is a little 2D, no Three.js demo I built that shows both measurements in one script:

Stats.js - Lorenz Atttractor Demo

Note: this revision of the script does not attempt to self correct for over use of resources, but I do hope to cover this featurein a future release.

So, Philip, thank you for posing the issue - and causing me to look for solutions to my issues. Catching lost frames and that sort of stuff is way beyond my pay grade, but I hope these simple observations will help you somewhat to wrangle your errant milliseconds.

Theo

[Philip Rideout]

Thanks Theo, that's helpful stuff in general, especially that Don Olmstead presentation!

In our case, the per-frame CPU workload is small, so Statsjs-style calipers report just a few milliseconds, nowhere near the feared 16.6.  It's the GPU workload that's causing dropped frames, and we don't have a way of knowing when the backbuffer is presented.

For now, our course of action is (a) make shaders faster (e.g., bake proceduralism into textures as much as possible), and (b) give users a prominent "SIMPLE SHADING" setting so that they can downgrade the visual quality on their own.

The harsh reality of being a WebGL developer: your users have crap machines.  In fact, you cannot fathom the immense crapiness of their machines.

Philip

[Jaume Sánchez]

You might find useful a library I did a while ago: rStats https://github.com/spite/rstats . Some intro and examples here http://spite.github.io/rstats/

It aims to be a performance monitor more flexible than stats.js, albeit less "drag and drop". You can measure any part of the code you're interested, and plot it; it offers more sophisticated features like threshold values for warnings, and aggregated percentage graphs. There's also a basic plug-in system to provide more funcionality, with three.js and WebGL monitor plugins.

There's a few ideas for features I've been brewing lately, regarding WebGL performance and WebVR profiling.

I hope it's of some use for someone, let me know if you have any issues or suggestions :)

[Theo Armour]

Hi Jaume

It's good you bring in rStats. I came across your work while I was drafting my message. It all looks interesting and very comprehensive. I look forward to trying out rStats.

Question: can my app access the data. For example, can I use rStats numbers to make my app ask for less processing to be done?

Also can you make rStats into a bookmarklet the way you can with stats.js?

Nice work as always!

Theo

[Mark Callow]

On Nov 22, 2015, at 4:09 PM, Philip Rideout <phi...@mappable.com> wrote:

Thanks Theo, that's helpful stuff in general, especially that Don Olmstead presentation!

In our case, the per-frame CPU workload is small, so Statsjs-style calipers report just a few milliseconds, nowhere near the feared 16.6.  It's the GPU workload that's causing dropped frames, and we don't have a way of knowing when the backbuffer is presented.

The proposed WEBGL_dynamic_texture extension has a mechanism for determining precisely when the frame is presented: the method getLastDrawingBufferPresentTime. This proposal has languished for some time but there has recently been an uptick of interest. Even if the complete extension is not implemented, perhaps we could make a small extension with just  getLastDrawingBufferPresentTime and related functions.

The harsh reality of being a WebGL developer: your users have crap machines.  In fact, you cannot fathom the immense crapiness of their machines.

I’ve long been of the opinion that developers should be given crap machines. Then they would be able to literally see some of the stupid things, e.g. multiple re-drawings for no discernible reason (not talking about WebGL here) that they do and would be less likely to create bloatware.

Regards

    -Mark

[Jaume Sánchez]

On Tue, Nov 24, 2015 at 10:07 AM, Theo Armour <th...@evereverland.net> wrote:

Hi Jaume

It's good you bring in rStats. I came across your work while I was drafting my message. It all looks interesting and very comprehensive. I look forward to trying out rStats.

Question: can my app access the data. For example, can I use rStats numbers to make my app ask for less processing to be done?

Yes, you can do that, if you know what values are being monitored, by specifying rS( monitor_string ).value(). 

For instance: rS( 'fps' ).value(); or rS( 'bindTexture' ).value()

The value retrieving is a very bare bones API in that sense, it might need some work if it's to be used more intensely.

 

Also can you make rStats into a bookmarklet the way you can with stats.js?

Never actually used stats as a bookmarklet. I've thought of turning rStats into a DevTools panel, but the bookmarklet idea is intriguing.

It would have to assume a few things: rStats relies on the developers heavily marking the relevant points in their code. It doesn't even assume that a rAF is the one being used for the render loop or anything. But some common cases could be assumed, and display info about three.js and WebGL which doesn't need specific instrumentation in the render function (think all the glfaces, glelements, drawcalls, bindtextures, etc.) 

Another issue is that some instrumentation has to happen before the page has started to run any code, so it might be tricky to get good monitoring for some values.

I'll take a look!

 

Nice work as always!

Thanks! You're too kind ^_^ 

Cheers,

Jaume

[Theo Armour]

Philip ( and Mark )

Frame Jankiness

While researching my reply to Jaume regarding the Chrome FPS/HUD display, I came [again] across this page on Chrome developer tips:

https://developer.chrome.com/devtools/docs/tips-and-tricks

It appears to present a bunch of timing viz tricks that might be useful to you.

Crappy Machines

The harsh reality of being a WebGL developer: your users have crap machines.  In fact, you cannot fathom the immense crapiness of their machines.

 

I’ve long been of the opinion that developers should be given crap machines. 

Among my set of toys, I have a Chrome Pixel, 2nd ed with the Core i7 processor = $1,300 of toy + tax. The Pixel is one of the very best computers I have ever owned. And the original 128K Mac was like my tenth computer.

The Pixel with its Intel HD 5000 GPU fails, of course, with any Three.js script that casts shadows/has Phong materials.

What amazes me is that WebGL works on as many devices as it currently does.

And, if my may offer some consolation, users tend to self-select according to their needs.

Thus:

If you notice that your app has issues in certain places. 

Don't forget so have your users.

They have ditched IE6 on their XPs. They are eschewing Safari. Whatever. 

Deliver a great app and your users will - one way or another - acquire the hardware to reap the fruits of your endeavors.

Build for the past. Develop for the future.

Theo

.