I took a look at the OS breakdown of Firefox 4 beta feedback and it's interesting to see that there's a clear difference between Windows, Mac, and Linux.
Disclaimer: I did this in an attempt to spot trends in the feedback, but I don't know if this is considered useful to people? Would love to get feedback if you think this is something the SUMO team should continue to do.
Anyway, the results:
Windows: 54% happy, 45% sad Linux: 42% happy, 57% sad Mac OS X: 26% happy, 73% sad
When taking a closer look at the reasons why people are sad, I see the following trends:
-------------- Linux -------------
In summary, the most common complaint seems to be that Linux development isn't prioritized over Windows/Mac. Users feel they're missing out on the fun.
Most common feedback:
* Regular menus / can't hide the menu / No Firefox button / Not the new design / "visually broken" (by far the most common complaint) * Bookmarks button annoying. * Didn't pass ACID3 test.
Quotes:
"The linux interface is still the same one. Yeah, update the Windows one and not the linux one, just wait for ubuntu & others to ditch you :("
"please, remove menu bar by default"
"i don't want main menu in linux, one button can be enough like windows ff."
"The tabs can't go up to the VERY top of the window like they can in Chrome.
"Firefox on Windows looks very good, but on Linux there is still some inconsistency when you set tabs on top; more so with custom GTK themes."
"Why can't we have the "Firefox" button on Linux too? I use Opera for Linux too, and they managed to add it on Linux."
"the awesome UI isn't available for linux yet..."
"There is not the new menu button available on Linux"
"because ridiculously like always, you guys put linux updates at a lower importance... same horrible visual as always... come on..."
"I've got this annoying Bookmarks button between my search box and the feedback button, and I can't get rid of it."
"Bookmarks menu item to the right of the search bar is unecessary."
"The bookmarks button in the upper right corner is not removable."
"Firefox does not pass the acid3 test. It gets a 97/100"
"It doesnot get 100 Marks."
"failed the Acid 3 Test 97/100"
-------------- Mac OS X -------------
In summary, the most common complaint seems to be the second icon showing up in the Dock when Firefox is running plugins.
Most common feedback:
* More than one instance running (by far the most common complaint) * Flash crashing * Video fullscreen doesn't hide menu bar
Quotes:
"Too many instances in my dock on Mac OS 10.6.3"
"Firefox 4.0b1 keeps opening multiple instances for some reason, unknown to me."
"An extra Firefox icon keeps appearing in my dock. When I right-click it and click Quit, Firefox sats the Adobe Flash Player plugin crashed"
"Firefox is sometimes shown twice in my dock when restarted on Mac OS X 10.6.4."
"The new firefox (4.0b1) opens up a second icon on my mac everytime i use it AND doesnt play YouTube videos - sound comes on but video doesnt"
"Adobe Flash plugin always crashes, even in Youtube"
"Firefox crashes often (I think it has to do with the flash plugin) and when I reopen it, Flash does not play anymore until I kill firefox."
"Adobe Flash keeps on crashing."
"firefox does not hide the OSX menu bar when playing flash videos in full screen."
"because when watching videos in fullscreen, my mac dock still stays at the front"
"video full screen is extremely buggy - will not exit when esc is pressed and will not switch programs. Menu bar (osX) will not hide either."
"there are some problems with the flash player in full-screen mode"
How did you get around the 1000-response limitation of the search? Were you using the raw data?
Generally this is useful, but I find the summary percentage statistics to be a little misleading as they're not correcting for spurious input, repetitive input, etc. The wider differences look to be significant for OSX only, and I'd call the differences between Windows and Linux a wash, statistically speaking :)
I think going forward you should co-ordinate with ddash and aakashd, both of whom are also doing some deeper analysis, but reporting here and to the Wednesday meeting is valuable, yes.
> I took a look at the OS breakdown of Firefox 4 beta feedback and it's interesting to see that there's a clear difference between Windows, Mac, and Linux.
> Disclaimer: I did this in an attempt to spot trends in the feedback, but I don't know if this is considered useful to people? Would love to get feedback if you think this is something the SUMO team should continue to do.
> Anyway, the results:
> Windows: 54% happy, 45% sad > Linux: 42% happy, 57% sad > Mac OS X: 26% happy, 73% sad
> When taking a closer look at the reasons why people are sad, I see the following trends:
> -------------- Linux -------------
> In summary, the most common complaint seems to be that Linux development isn't prioritized over Windows/Mac. Users feel they're missing out on the fun.
> Most common feedback:
> * Regular menus / can't hide the menu / No Firefox button / Not the new design / "visually broken" (by far the most common complaint) > * Bookmarks button annoying. > * Didn't pass ACID3 test.
> Quotes:
> "The linux interface is still the same one. Yeah, update the Windows one and not the linux one, just wait for ubuntu & others to ditch you :("
> "please, remove menu bar by default"
> "i don't want main menu in linux, one button can be enough like windows ff."
> "The tabs can't go up to the VERY top of the window like they can in Chrome.
> "Firefox on Windows looks very good, but on Linux there is still some inconsistency when you set tabs on top; more so with custom GTK themes."
> "Why can't we have the "Firefox" button on Linux too? I use Opera for Linux too, and they managed to add it on Linux."
> "the awesome UI isn't available for linux yet..."
> "There is not the new menu button available on Linux"
> "because ridiculously like always, you guys put linux updates at a lower importance... same horrible visual as always... come on..."
> "I've got this annoying Bookmarks button between my search box and the feedback button, and I can't get rid of it."
> "Bookmarks menu item to the right of the search bar is unecessary."
> "The bookmarks button in the upper right corner is not removable."
> "Firefox does not pass the acid3 test. It gets a 97/100"
> "It doesnot get 100 Marks."
> "failed the Acid 3 Test 97/100"
> -------------- Mac OS X -------------
> In summary, the most common complaint seems to be the second icon showing up in the Dock when Firefox is running plugins.
> Most common feedback:
> * More than one instance running (by far the most common complaint) > * Flash crashing > * Video fullscreen doesn't hide menu bar
> Quotes:
> "Too many instances in my dock on Mac OS 10.6.3"
> "Firefox 4.0b1 keeps opening multiple instances for some reason, unknown to me."
> "An extra Firefox icon keeps appearing in my dock. When I right-click it and click Quit, Firefox sats the Adobe Flash Player plugin crashed"
> "Firefox is sometimes shown twice in my dock when restarted on Mac OS X 10.6.4."
> "The new firefox (4.0b1) opens up a second icon on my mac everytime i use it AND doesnt play YouTube videos - sound comes on but video doesnt"
> "Adobe Flash plugin always crashes, even in Youtube"
> "Firefox crashes often (I think it has to do with the flash plugin) and when I reopen it, Flash does not play anymore until I kill firefox."
> "Adobe Flash keeps on crashing."
> "firefox does not hide the OSX menu bar when playing flash videos in full screen."
> "because when watching videos in fullscreen, my mac dock still stays at the front"
> "video full screen is extremely buggy - will not exit when esc is pressed and will not switch programs. Menu bar (osX) will not hide either."
> "there are some problems with the flash player in full-screen mode" > _______________________________________________ > dev-apps-firefox mailing list > dev-apps-fire...@lists.mozilla.org > https://lists.mozilla.org/listinfo/dev-apps-firefox
> How did you get around the 1000-response limitation of the search? Were you using the raw data?
I didn't get around it, but in all honesty I assumed that the limitation was only for the plain text feedback messages. I realize now that it's also limiting the %-age, so taking a closer look at the raw data definitely sounds like a good idea.
> Generally this is useful, but I find the summary percentage statistics to be a little misleading as they're not correcting for spurious input, repetitive input, etc. The wider differences look to be significant for OSX only, and I'd call the differences between Windows and Linux a wash, statistically speaking :)
You could be right about the Linux/Windows differences being insignificant, though after reading 10 pages of feedback, it at least seemed clear what people were most frequently complaining about (they want the new UI, too!).
As can be expected, Linux fluctuates a lot more because of the smaller dataset.
> I think going forward you should co-ordinate with ddash and aakashd, both of whom are also doing some deeper analysis, but reporting here and to the Wednesday meeting is valuable, yes.
Yep, definitely. This was really just an initial and quick analysis to get a better feel for what we might expect on SUMO once we open up the floodgate. We'll coordinate to make sure we incorporate this input channel in our regular Wednesday/newsletter reporting.
> "I've got this annoying Bookmarks button between my search box and the > feedback button, and I can't get rid of it."
> "Bookmarks menu item to the right of the search bar is unecessary."
> "The bookmarks button in the upper right corner is not removable."
The button is movable and removable in beta 2, we were expecting complains, but code for having it movable was not ready for beta1. So it will be interesting to see if this complain disappears or if users will still complain just for its existence. I'm a bit sad that users still don't get that we always try to make changes allowing customization. Where does this lack of faith come from?
When we accidentally painted over the window controls (minimize, maximize, close) if there was a persona on Windows, most of the comments were of the form "I disagree with your decision to remove the window controls." I think users now expect beta to mean nearly final release. This makes sense if you consider there really aren't that many other organizations publicly showing an unfinished application.
-Alex
On Thu, Jul 22, 2010 at 11:20 AM, Marco Bonardo <mak77NONSPA...@supereva.it>wrote:
> "I've got this annoying Bookmarks button between my search box and the >> feedback button, and I can't get rid of it."
>> "Bookmarks menu item to the right of the search bar is unecessary."
>> "The bookmarks button in the upper right corner is not removable."
> The button is movable and removable in beta 2, we were expecting complains, > but code for having it movable was not ready for beta1. > So it will be interesting to see if this complain disappears or if users > will still complain just for its existence. > I'm a bit sad that users still don't get that we always try to make changes > allowing customization. > Where does this lack of faith come from?
On Thu, Jul 22, 2010 at 4:12 PM, Alex Faaborg <faab...@mozilla.com> wrote:
>> Where does this lack of faith come from?
> When we accidentally painted over the window controls (minimize, maximize, > close) if there was a persona on Windows, most of the comments were of the > form "I disagree with your decision to remove the window controls." I think > users now expect beta to mean nearly final release. This makes sense if you > consider there really aren't that many other organizations publicly showing > an unfinished application.
Yeah; also, there are applications that have tens of millions of users for a year and still say "beta".
A good way to derail this thread would be to propose that we look at our milestone naming, so I won't do that.
> A good way to derail this thread would be to propose that we look at > our milestone naming, so I won't do that.
Right, I'm not proposing we change our naming scheme, just providing the premise that the lack of faith stems from users having a different expectation of what "beta" means. -Alex
On Thu, Jul 22, 2010 at 1:14 PM, Mike Shaver <mike.sha...@gmail.com> wrote: > On Thu, Jul 22, 2010 at 4:12 PM, Alex Faaborg <faab...@mozilla.com> wrote:
> >> Where does this lack of faith come from?
> > When we accidentally painted over the window controls (minimize, > maximize, > > close) if there was a persona on Windows, most of the comments were of > the > > form "I disagree with your decision to remove the window controls." I > think > > users now expect beta to mean nearly final release. This makes sense if > you > > consider there really aren't that many other organizations publicly > showing > > an unfinished application.
> Yeah; also, there are applications that have tens of millions of users > for a year and still say "beta".
> A good way to derail this thread would be to propose that we look at > our milestone naming, so I won't do that.
On Thu, Jul 22, 2010 at 4:18 PM, Alex Faaborg <faab...@mozilla.com> wrote: > Right, I'm not proposing we change our naming scheme, just providing the > premise that the lack of faith stems from users having a different > expectation of what "beta" means.
I am not proposing that you are proposing that we change our naming scheme. I am glad that you are not proposing it. I am making my glad face right now!
Alex Faaborg wrote: >> A good way to derail this thread would be to propose that we look at >> our milestone naming, so I won't do that.
> Right, I'm not proposing we change our naming scheme, just providing the > premise that the lack of faith stems from users having a different > expectation of what "beta" means. > -Alex
Over the years, 'beta' seems to have morphed from 'nearly ready for release, please find bugs/omissions', to 'this is where we are, but not where we are going'. I liked it better the other way.
I do think some thought should be given to the difference between 'simplifying' and 'dumbing down', IF there is any.
> When we accidentally painted over the window controls (minimize, maximize, > close) if there was a persona on Windows, most of the comments were of the > form "I disagree with your decision to remove the window controls." I think > users now expect beta to mean nearly final release.
Yes, you're absolutely right, this is the same trend I'm seeing, but was unable to explain it so clearly as you did. People are considering betas as RCs and alpha as betas.
But could even be we give them a wrong message, or we don't push enough on what's a beta for us?
For example on various comments I've seen people pointing to the fact in the release notes we said: "Firefox 4 Beta (version 1) considered to be stable and safe to use for daily web". Then they complained about a crash or that add-ons were not working or UI was missing pieces. They complained that it was hard to realize how far beta1 was from final release because it was declared already stable and safe that is far from "beta".
> or we don't push enough on what's a beta for us?
Actually I think it is in our interest for people testing betas to have a mindset that it's nearly final. If everyone thought "oh, it's still in development, I'm sure someone else will report the problem, and they'll eventually get around to fixing [broken thing]" then we wouldn't get as much valuable feedback during the beta process.
It can however be a little frustrating to hear that Beta 1 mac users "disagree with our decision to draw tabs incorrectly," but at least they are being super critical!
-Alex
On Thu, Jul 22, 2010 at 2:38 PM, Marco Bonardo <mak77NONSPA...@supereva.it>wrote:
>> When we accidentally painted over the window controls (minimize, maximize, >> close) if there was a persona on Windows, most of the comments were of the >> form "I disagree with your decision to remove the window controls." I >> think >> users now expect beta to mean nearly final release.
> Yes, you're absolutely right, this is the same trend I'm seeing, but was > unable to explain it so clearly as you did. People are considering betas as > RCs and alpha as betas.
> But could even be we give them a wrong message, or we don't push enough on > what's a beta for us?
> For example on various comments I've seen people pointing to the fact in > the release notes we said: "Firefox 4 Beta (version 1) considered to be > stable and safe to use for daily web". Then they complained about a crash or > that add-ons were not working or UI was missing pieces. They complained that > it was hard to realize how far beta1 was from final release because it was > declared already stable and safe that is far from "beta".
On Thu, Jul 22, 2010 at 5:56 PM, Alex Faaborg <faab...@mozilla.com> wrote: > It can however be a little frustrating to hear that Beta 1 mac users > "disagree with our decision to draw tabs incorrectly," but at least they are > being super critical!
"We heard your feedback, and decided to remove the graphical corruption from beta3!"
On Thu, 22 Jul 2010 20:20:49 +0200, Marco Bonardo wrote: > I'm a bit sad that users still don't get that we always try to make > changes allowing customization. > Where does this lack of faith come from?
There is this inexplicable belief that Firefox has been getting more bloated and getting worse since, oh, 0.8. Unfortunately this fraction tends to be the most vocal. We get this in SeaMonkey too where we get accused of trying to ape all the worst bling in Firefox.
Phil
-- Philip Chee <phi...@aleytys.pc.my>, <philip.c...@gmail.com> http://flashblock.mozdev.org/http://xsidebar.mozdev.org Guard us from the she-wolf and the wolf, and guard us from the thief, oh Night, and so be good for us to pass. [ ]Walk through doors, don't crawl through Windows. * TagZilla 0.066.6
