We are getting close to the alpha release for Fennec and I think it is important that we all are aware of the criteria for reaching alpha. Stuart will post the notes from the meeting at the summit, where we agreed on the criteria, but there is one criteria I would like to bring up for discussion: performance.
AFAIR the criteria we agreed upon was that it shouldn't "feel slow", or "performance would be acceptable" or similar wordings, but I think we need to get to a more objective criteria for performance. For one, because we never now if we ever hit our target, and secondly so we also can verify we are improving and raising the bar for follow-up releases.
I don't have a good solution to what that criteria for performance should be, but I will suggest two:
1) performance at least on par with MicroB on comparable features 2) x fps during animations
Nailing down a specific metric will be tricky, I agree, but I do want to say that I agree with the emphasis here.
As I've said before, I think that having a pleasing, responsive *feel* is the sine qua non for a mobile browser. I think it is not what will differentiate us, necessarily, other phone-based browsers are getting good at visceral, kinetic appeal. I think we'll differentiate ourselves the way we always do - extensibility, openness, overall performance, standards compliance, ui innovation, security, &c. Those are what will make it a mozilla browser, but I think that none of those things will matter very much at all if the core browsing operations aren't responsive and pleasant to use.
This is also why I've been so belligerently resistant to the suggestion that the platform puts a ceiling on our abilities here. I'm sure that's true at some level, but I really don't want us to give ourselves permission there. As you point out, MicroB is more responsive on the same hardware, so that, at least, should be a performance floor, but I'm not sure I'd consider that fast enough either.
Still, it's arguably a good alpha target, and I know that before I went on vacation in August, Stuart was saying that we had things in the pipeline that should get us clear of that mark, at least. I feel like Vlad and Jeff had thoughts about making us gogogo too. What is the state of those things?
Cheers,
Johnathan
On 5-Sep-08, at 9:05 AM, Christian Sejersen wrote:
> We are getting close to the alpha release for Fennec and I think it is > important that we all are aware of the criteria for reaching alpha. > Stuart will post the notes from the meeting at the summit, where we > agreed on the criteria, but there is one criteria I would like to > bring up for discussion: performance.
> AFAIR the criteria we agreed upon was that it shouldn't "feel slow", > or "performance would be acceptable" or similar wordings, but I think > we need to get to a more objective criteria for performance. For one, > because we never now if we ever hit our target, and secondly so we > also can verify we are improving and raising the bar for follow-up > releases.
> I don't have a good solution to what that criteria for performance > should be, but I will suggest two:
> 1) performance at least on par with MicroB on comparable features > 2) x fps during animations
> What do you think?
> Regards,
> Christian Sejersen > Mozilla Corp. > m: +45 2993 7003
From a human centric performance metric (it rhymes!) perspective:
- if I touch a page with my finger and move my finger, the page should move immediately - if I swipe my finger for some UI gesture, the response should come up in under 200ms - zooming should show some change (even if it's a pixely change) within 100ms
Madhava probably has better guidance.
cheers, mike
On 5-Sep-08, at 11:30 AM, Johnathan Nightingale wrote:
> Nailing down a specific metric will be tricky, I agree, but I do want > to say that I agree with the emphasis here.
> As I've said before, I think that having a pleasing, responsive *feel* > is the sine qua non for a mobile browser. I think it is not what will > differentiate us, necessarily, other phone-based browsers are getting > good at visceral, kinetic appeal. I think we'll differentiate > ourselves the way we always do - extensibility, openness, overall > performance, standards compliance, ui innovation, security, &c. Those > are what will make it a mozilla browser, but I think that none of > those things will matter very much at all if the core browsing > operations aren't responsive and pleasant to use.
> This is also why I've been so belligerently resistant to the > suggestion that the platform puts a ceiling on our abilities here. > I'm sure that's true at some level, but I really don't want us to give > ourselves permission there. As you point out, MicroB is more > responsive on the same hardware, so that, at least, should be a > performance floor, but I'm not sure I'd consider that fast enough > either.
> Still, it's arguably a good alpha target, and I know that before I > went on vacation in August, Stuart was saying that we had things in > the pipeline that should get us clear of that mark, at least. I feel > like Vlad and Jeff had thoughts about making us gogogo too. What is > the state of those things?
> Cheers,
> Johnathan
> On 5-Sep-08, at 9:05 AM, Christian Sejersen wrote:
>> We are getting close to the alpha release for Fennec and I think it >> is >> important that we all are aware of the criteria for reaching alpha. >> Stuart will post the notes from the meeting at the summit, where we >> agreed on the criteria, but there is one criteria I would like to >> bring up for discussion: performance.
>> AFAIR the criteria we agreed upon was that it shouldn't "feel slow", >> or "performance would be acceptable" or similar wordings, but I think >> we need to get to a more objective criteria for performance. For one, >> because we never now if we ever hit our target, and secondly so we >> also can verify we are improving and raising the bar for follow-up >> releases.
>> I don't have a good solution to what that criteria for performance >> should be, but I will suggest two:
>> 1) performance at least on par with MicroB on comparable features >> 2) x fps during animations
>> What do you think?
>> Regards,
>> Christian Sejersen >> Mozilla Corp. >> m: +45 2993 7003
What Mike lays out here are the right end goals for responsiveness. These things, like the 100ms and 200ms timeframes for immediacy and sense-of-causality, respectively, are the sort of non-negotiable "way the brain works" determinants of whether the UI will feel responsive and natural (physical) to people.
That said, whether these are *alpha* goals is a question of scheduling vs. how natural we need it to be at this point, compared to abilities of the browser. Do we have a sense of how close we are to being able to do these kind of things?
> From a human centric performance metric (it rhymes!) perspective:
> - if I touch a page with my finger and move my finger, the page > should move immediately > - if I swipe my finger for some UI gesture, the response should come > up in under 200ms > - zooming should show some change (even if it's a pixely change) > within 100ms
> Madhava probably has better guidance.
> cheers, > mike
> On 5-Sep-08, at 11:30 AM, Johnathan Nightingale wrote:
>> Nailing down a specific metric will be tricky, I agree, but I do want >> to say that I agree with the emphasis here.
>> As I've said before, I think that having a pleasing, responsive >> *feel* >> is the sine qua non for a mobile browser. I think it is not what >> will >> differentiate us, necessarily, other phone-based browsers are getting >> good at visceral, kinetic appeal. I think we'll differentiate >> ourselves the way we always do - extensibility, openness, overall >> performance, standards compliance, ui innovation, security, &c. >> Those >> are what will make it a mozilla browser, but I think that none of >> those things will matter very much at all if the core browsing >> operations aren't responsive and pleasant to use.
>> This is also why I've been so belligerently resistant to the >> suggestion that the platform puts a ceiling on our abilities here. >> I'm sure that's true at some level, but I really don't want us to >> give >> ourselves permission there. As you point out, MicroB is more >> responsive on the same hardware, so that, at least, should be a >> performance floor, but I'm not sure I'd consider that fast enough >> either.
>> Still, it's arguably a good alpha target, and I know that before I >> went on vacation in August, Stuart was saying that we had things in >> the pipeline that should get us clear of that mark, at least. I feel >> like Vlad and Jeff had thoughts about making us gogogo too. What is >> the state of those things?
>> Cheers,
>> Johnathan
>> On 5-Sep-08, at 9:05 AM, Christian Sejersen wrote:
>>> We are getting close to the alpha release for Fennec and I think it >>> is >>> important that we all are aware of the criteria for reaching alpha. >>> Stuart will post the notes from the meeting at the summit, where we >>> agreed on the criteria, but there is one criteria I would like to >>> bring up for discussion: performance.
>>> AFAIR the criteria we agreed upon was that it shouldn't "feel slow", >>> or "performance would be acceptable" or similar wordings, but I >>> think >>> we need to get to a more objective criteria for performance. For >>> one, >>> because we never now if we ever hit our target, and secondly so we >>> also can verify we are improving and raising the bar for follow-up >>> releases.
>>> I don't have a good solution to what that criteria for performance >>> should be, but I will suggest two:
>>> 1) performance at least on par with MicroB on comparable features >>> 2) x fps during animations
>>> What do you think?
>>> Regards,
>>> Christian Sejersen >>> Mozilla Corp. >>> m: +45 2993 7003
On Sep 5, 7:05 am, Christian Sejersen <christ...@mozilla.com> wrote:
> We are getting close to the alpha release for Fennec and I think it is > important that we all are aware of the criteria for reaching alpha. > Stuart will post the notes from the meeting at the summit, where we > agreed on the criteria, but there is one criteria I would like to > bring up for discussion: performance.
The list we discussed at the summit included: - stable - usable/"fast enough" - responsive "enough" - our ui looking how we have it speced out - getting our features in and working - downloads - prefs - easily installable
> AFAIR the criteria we agreed upon was that it shouldn't "feel slow", > or "performance would be acceptable" or similar wordings, but I think > we need to get to a more objective criteria for performance. For one, > because we never now if we ever hit our target, and secondly so we > also can verify we are improving and raising the bar for follow-up > releases.
I agree that the wording is vague, however I believe our starting point for measuring should be what we consider good enough. We have several patches from folks that should be landing in the next few days that should put us at that point, however we should all feel comfortable that that is the case.
> I don't have a good solution to what that criteria for performance > should be, but I will suggest two:
> 1) performance at least on par with MicroB on comparable features
I'm not sure it is fair to anyone to say that an alpha's performance needs to be as fast as a shipping product with years of optimization behind it. Perhaps this is a better beta goal. We should set the bar high for ourselves for our final release and probably have something comparing ourselves to other applications out there.
> 2) x fps during animations
As I mentioned above, this might be a fine way to measure, but I think our starting point should be based on what we feel is usable and then continuously set the bar higher as we move forward. I don't think we gain much by trying to pick a random number for alpha that may or may not be usable.
On Sep 5, 10:21 am, Madhava Enros <madh...@mozilla.com> wrote:
> Hey -
> What Mike lays out here are the right end goals for responsiveness. > These things, like the 100ms and 200ms timeframes for immediacy and > sense-of-causality, respectively, are the sort of non-negotiable "way > the brain works" determinants of whether the UI will feel responsive > and natural (physical) to people.
> That said, whether these are *alpha* goals is a question of scheduling > vs. how natural we need it to be at this point, compared to abilities > of the browser. Do we have a sense of how close we are to being able > to do these kind of things?
> Madhava
Hey Madhava, Mike,
Having this kind of insight is great. If we could put together a list of numbers like those tied to a specific tasks we can measure I think that would go a long way towards us shipping a great final product.
For alpha specifically, I don't know that we have to hit those exact numbers. If we're already there, fantastic, but I believe there is a number a bit higher than those where things are still totally usable and people will be able to use it and give feedback without feeling it is "too slow." I'm not sure how to determine those numbers besides us just using it and saying "this feels like something we can have for alpha."
My feeling is that we're pretty close (once we get a few critical performance fixes in) to where we need to be for alpha, but still have a ways to go before we can ship a final release.
> On Sep 5, 7:05 am, Christian Sejersen <christ...@mozilla.com> wrote: >> We are getting close to the alpha release for Fennec and I think it >> is >> important that we all are aware of the criteria for reaching alpha. >> Stuart will post the notes from the meeting at the summit, where we >> agreed on the criteria, but there is one criteria I would like to >> bring up for discussion: performance.
> The list we discussed at the summit included: > - stable > - usable/"fast enough" > - responsive "enough" > - our ui looking how we have it speced out > - getting our features in and working > - downloads > - prefs > - easily installable
>> AFAIR the criteria we agreed upon was that it shouldn't "feel slow", >> or "performance would be acceptable" or similar wordings, but I think >> we need to get to a more objective criteria for performance. For one, >> because we never now if we ever hit our target, and secondly so we >> also can verify we are improving and raising the bar for follow-up >> releases.
> I agree that the wording is vague, however I believe our starting > point for measuring should be what we consider good enough. We have > several patches from folks that should be landing in the next few days > that should put us at that point, however we should all feel > comfortable that that is the case.
I don't mind the starting point being what we consider "good enough", but I think it has to be tied to some objective numbers, e.g. ms as suggested by Beltzner.
>> I don't have a good solution to what that criteria for performance >> should be, but I will suggest two:
>> 1) performance at least on par with MicroB on comparable features
> I'm not sure it is fair to anyone to say that an alpha's performance > needs to be as fast as a shipping product with years of optimization > behind it. Perhaps this is a better beta goal. We should set the bar > high for ourselves for our final release and probably have something > comparing ourselves to other applications out there.
We should set the bar high, and I think one goal should be: "We are at least on par with MicroB on SunSpider and Dromaeo".
>> 2) x fps during animations
> As I mentioned above, this might be a fine way to measure, but I think > our starting point should be based on what we feel is usable and then > continuously set the bar higher as we move forward. I don't think we > gain much by trying to pick a random number for alpha that may or may > not be usable.
Again, I don't think we can have a serious discussion about what you feel is usable compared to what I feel is usable, but I think you also agree with me, since you are talking about set the bar higher as we move forward, that is has to be measurable in some way.
Of course I am not arguing for picking random numbers for the alpha, I am just suggesting that one approach could be to instrument the code (e.g. measuring ms for zoom, panning, sliding in/out) and then set the targets based on whether the feature feels sluggish or not. We can then continuously measure the performance and raise the bar as we move forward.
|
