Re: Increasing our Aurora and Nightly user populations?

Showing 1-60 of 60 messages
Re: Increasing our Aurora and Nightly user populations? Johnathan Nightingale 4/29/13 12:32 PM
On Apr 29, 2013, at 3:07 PM, Chris Peterson wrote:

> (Sorry if this is not the right forum for this discussion.)

dev-planning feels right-er here - I suggest follow ups take it thattaway.

J

PS - original post, for dev-planning context:

> To increase our testing populations, I'd like to suggest that we add a periodic "channel upsell" message to the about:home page (of Aurora and Beta) and the about box (of Aurora, Beta, and possibly Release).
>
> Beta and Aurora users have already opted in to a testing channel, so we know they are receptive to beta-quality software. If they have set Beta or Aurora as their default browser and/or have been using it consistently for a few weeks, we could try to upsell Beta users to Aurora and Aurora users to Nightly.
>
> We did something similar with Firefox for Android. After a certain number of browser launches, we display a one-time message suggesting they write a Google Play store review if they like Firefox or contact Firefox Support if they have problems.
>
> Similarly, users who check their browser's about box are clearly interested in version numbers or checking for the latest updates. These users might be receptive to a channel upsell message to get an even bigger version number. :)


---
Johnathan Nightingale
VP Firefox Engineering
@johnath

Re: Increasing our Aurora and Nightly user populations? Alex Keybl 4/29/13 12:37 PM
[d.platform->d.planning]

Grace and Laura have worked on similar initiatives in the past (about:home campaigns for pre-release channel uptake). When we do these sorts of campaigns, we like to set a target so that we don't drain too many users from higher channels.

I would not perform any redistribution of Aurora/Nightly users, since we don't want to dip below our current populations in either. If we advertise Nightly/Aurora to Beta users, we have to keep in mind the fact that many of these users have forgotten that they're using Firefox Beta as opposed to Release. I wouldn't recommend advertising Aurora/Nightly to Release users, only Beta, given the far differing quality bar.

As mentioned in my other reply, there is benefit to doing this with regularity for sake of pre-release population diversity and representativeness.

-Alex

On Apr 29, 2013, at 12:07 PM, Chris Peterson <cpet...@mozilla.com> wrote:

> (Sorry if this is not the right forum for this discussion.)
>
> To increase our testing populations, I'd like to suggest that we add a periodic "channel upsell" message to the about:home page (of Aurora and Beta) and the about box (of Aurora, Beta, and possibly Release).
>
> Beta and Aurora users have already opted in to a testing channel, so we know they are receptive to beta-quality software. If they have set Beta or Aurora as their default browser and/or have been using it consistently for a few weeks, we could try to upsell Beta users to Aurora and Aurora users to Nightly.
>
> We did something similar with Firefox for Android. After a certain number of browser launches, we display a one-time message suggesting they write a Google Play store review if they like Firefox or contact Firefox Support if they have problems.
>
> Similarly, users who check their browser's about box are clearly interested in version numbers or checking for the latest updates. These users might be receptive to a channel upsell message to get an even bigger version number. :)
>
>
> chris p.
> _______________________________________________
> dev-platform mailing list
> dev-pl...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform

Re: Increasing our Aurora and Nightly user populations? Gregory Szorc 4/29/13 12:58 PM
>> Beta and Aurora users have already opted in to a testing channel, so we know they are receptive to beta-quality software. If they have set Beta or Aurora as their default browser and/or have been using it consistently for a few weeks, we could try to upsell Beta users to Aurora and Aurora users to Nightly.

Apparently a significant number of our Beta channel users are "regular"
users who are unknowingly on Beta - carryovers from the Firefox 3.6 -> 4
days when many people installed a beta of 4 so they could run a newer
Firefox. I would question whether these people are "receptive to
beta-quality software."

If we are intent on attracting more people to the pre-release channels,
IMO we should apply targeting of specific populations. Firefox Health
Report data is showing some rather remarkable discrepancies between our
channels. For example, 62.5% of Beta users have 2GB or less of memory
compared with 48.6% of Aurora and 25.9% of Nightly. In terms of CPU
cores, 73.5% of Beta users have 2 or fewer contrasted with 39.1% of
Nightly (I believe this hyperthreaded cores count).

IMO we should strive to have uniformity between the channel populations.
Targeting specific populations also has the desirable side effect that
it requires us to answer important questions such as "is the size of
this population adequate."
Re: Increasing our Aurora and Nightly user populations? chris hofmann 4/29/13 1:28 PM
On 4/29/13 12:58 PM, Gregory Szorc wrote:
>>> Beta and Aurora users have already opted in to a testing channel, so
>>> we know they are receptive to beta-quality software. If they have
>>> set Beta or Aurora as their default browser and/or have been using
>>> it consistently for a few weeks, we could try to upsell Beta users
>>> to Aurora and Aurora users to Nightly.
>
> Apparently a significant number of our Beta channel users are
> "regular" users who are unknowingly on Beta - carryovers from the
> Firefox 3.6 -> 4 days when many people installed a beta of 4 so they
> could run a newer Firefox. I would question whether these people are
> "receptive to beta-quality software."

Considering that much of the outreach for Firefox 4 betas was the
technical press, and not mainstream I'm not sure that I'd agree with
your assertion.   These were people that went out of their way to
download and install beta software.  Many might be sharing their
computers  with others that are less technical, and/or they might
question why they are continuing to get beta software beyond their
choice to get a beta of Firefox 4, and that is something that we should
study.

>
> If we are intent on attracting more people to the pre-release
> channels, IMO we should apply targeting of specific populations.
> Firefox Health Report data is showing some rather remarkable
> discrepancies between our channels. For example, 62.5% of Beta users
> have 2GB or less of memory compared with 48.6% of Aurora and 25.9% of
> Nightly. In terms of CPU cores, 73.5% of Beta users have 2 or fewer
> contrasted with 39.1% of Nightly (I believe this hyperthreaded cores
> count).
>
> IMO we should strive to have uniformity between the channel
> populations. Targeting specific populations also has the desirable
> side effect that it requires us to answer important questions such as
> "is the size of this population adequate."

Sure, targeting might have lots of desirable effects, but we have never
been able to target effectively,  hence the kind of numbers that you
see.   We are dealing with volunteers that are anxious to help us in
several different ways and we should try to plug them in to the project
in the way they feel most confortable in contributing.   FHR might
change that, but we still lack the kind of volumes that we mapped out as
critical levels in the planning of train development model.

Lets also have a discussion about how we can recharge and increase the
participation level on all the channels again.  That's been the only
effective way for us to make progress in getting the the kind of testing
and feedback we need over the long history of the project.
> _______________________________________________
> dev-planning mailing list
> dev-pl...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-planning

Re: Increasing our Aurora and Nightly user populations? Anton Kovalyov 4/29/13 1:39 PM
I have a question (sorry if it is a stupid one): will it be possible to
display that about:home message to users who use Developer Tools on a
regular basis? I'm thinking of some kind of a local counter that triggers
an upsell for the about:home page. Developer Tools team would love web
developers to use more recent versions of Firefox because we try to iterate
very and need more feedback. Also I think web developers are usually
receptive to beta-quality software. I don't have any numbers but from my
anecdotal experience lots of web devs use Chrome Canary which is similar to
Firefox Nightly if I'm not mistaken.

Anton
>  ______________________________**_________________
>> dev-planning mailing list
>> dev-pl...@lists.mozilla.org
>> https://lists.mozilla.org/**listinfo/dev-planning<https://lists.mozilla.org/listinfo/dev-planning>
>>
>
> ______________________________**_________________
> dev-planning mailing list
> dev-pl...@lists.mozilla.org
> https://lists.mozilla.org/**listinfo/dev-planning<https://lists.mozilla.org/listinfo/dev-planning>
>
Re: Increasing our Aurora and Nightly user populations? Benjamin Smedberg 4/29/13 1:56 PM
On 4/29/2013 3:37 PM, Alex Keybl wrote:
> Grace and Laura have worked on similar initiatives in the past (about:home campaigns for pre-release channel uptake). When we do these sorts of campaigns, we like to set a target so that we don't drain too many users from higher channels.
>
> I would not perform any redistribution of Aurora/Nightly users, since we don't want to dip below our current populations in either. If we advertise Nightly/Aurora to Beta users, we have to keep in mind the fact that many of these users have forgotten that they're using Firefox Beta as opposed to Release. I wouldn't recommend advertising Aurora/Nightly to Release users, only Beta, given the far differing quality bar.
>
> As mentioned in my other reply, there is benefit to doing this with regularity for sake of pre-release population diversity and representativeness.
I believe that our original target for Aurora was to have a population
about four times larger than the current population, no? Rather than
targeting and redistributing existing prerelease users, is there a way
we can focus on "outside" users? For example:

* web developer population -> Aurora
* users who experience specific crashes -> the channel the crash is fixed on
* New volunteers: Nightly and Aurora (along with more active followups
to help them help us)

I suspect we'd do a lot better promoting these releases to users at the
time it might benefit them or where they have expressed willingness to
help. And installing Aurora is a pretty low bar for basic volunteering!

--BDS

RE: Increasing our Aurora and Nightly user populations? Karen Rudnitski 4/29/13 1:57 PM
Maybe another route to trial this would be to use our product
announcements feature since it can be much more targeted. I would not be
wholly comfortable at this point to use the snippet real estate (if I was
reading the original request), especially for our GA audience.

I also think we should think about the goal here - which I think would be
to expand the volume numbers of each of these groups and not necessarily
move current users to other products (although I do understand the
rationale prompting this suggestion, generally around to lower friction
level to try less stable software).

If we are agreed on the above, perhaps it is also a discussion to get our
marketing friendlies and contributor engagement folks with an objective to
target and capture new users for these channels.

Karen

-----Original Message-----
From: dev-planning-bounces+krudnitski=mozil...@lists.mozilla.org
[mailto:dev-planning-bounces+krudnitski=mozil...@lists.mozilla.org] On
Behalf Of Johnathan Nightingale
Sent: 29 April 2013 15:32
To: Chris Peterson
Cc: mozilla.dev.planning group; dev-pl...@lists.mozilla.org
Subject: Re: Increasing our Aurora and Nightly user populations?

On Apr 29, 2013, at 3:07 PM, Chris Peterson wrote:

> (Sorry if this is not the right forum for this discussion.)

dev-planning feels right-er here - I suggest follow ups take it thattaway.


J

PS - original post, for dev-planning context:

> To increase our testing populations, I'd like to suggest that we add a
periodic "channel upsell" message to the about:home page (of Aurora and
Beta) and the about box (of Aurora, Beta, and possibly Release).
>
> Beta and Aurora users have already opted in to a testing channel, so we
know they are receptive to beta-quality software. If they have set Beta or
Aurora as their default browser and/or have been using it consistently for
a few weeks, we could try to upsell Beta users to Aurora and Aurora users
to Nightly.
>
> We did something similar with Firefox for Android. After a certain
number of browser launches, we display a one-time message suggesting they
write a Google Play store review if they like Firefox or contact Firefox
Support if they have problems.
>
> Similarly, users who check their browser's about box are clearly
interested in version numbers or checking for the latest updates. These
users might be receptive to a channel upsell message to get an even bigger
version number. :)


---
Johnathan Nightingale
VP Firefox Engineering
@johnath

_______________________________________________
dev-planning mailing list
dev-pl...@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-planning
Re: Increasing our Aurora and Nightly user populations? Chris Peterson 4/30/13 10:54 AM
On 4/29/13 12:37 PM, Alex Keybl wrote:

 > Grace and Laura have worked on similar initiatives in the past
 > (about:home  campaigns for pre-release channel uptake). When we do
 > these sorts of campaigns, we like to set a target so that we don't
 > drain too many users from higher channels.

Alex, how often have Grace and Laura run pre-release channel promotions
in the past? How successful were they?

It sounds like there is general consensus that we would like more Beta
and Aurora testers (especially web developers) and more hardware
uniformity across the channels (and more apple pie). In particular,
Aurora seems to be the underdog channel when comparing the actual ratio
of channel users (1:1.2:25:1000) to the original target ratio
(1:10:100:1000).

Community promotion ideas:
* Use Mozilla blogs and community to evangelize Aurora to web developers
and testers. We could even upsell Aurora on the Firefox Beta download
page. :)

In-product promotion ideas:
* Consider showing Aurora promotions on some Beta users's about:home page.
* Limit the Aurora promotions to a targeted subset of Beta users, such
as people who have low-end hardware or use Firefox's Developer Tools.
Avoid promoting to "accidental testers" who don't know they are running
Beta?
* Do not show any promotions to Nightly, Aurora, or Release users.


chris p.
Re: Increasing our Aurora and Nightly user populations? Alex Keybl 4/30/13 11:05 AM
> Alex, how often have Grace and Laura run pre-release channel promotions in the past? How successful were they?

I'll let them speak to the overall success of the campaigns.

> It sounds like there is general consensus that we would like more Beta and Aurora testers (especially web developers) and more hardware uniformity across the channels (and more apple pie)

This is and basically always has been true. It's just where a new initiative falls within our overall outreach/marketing priorities.

-Alex
RE: Increasing our Aurora and Nightly user populations? Karen Rudnitski 4/30/13 11:13 AM
I am adding Chad and Sam from product marketing who would need to be
engaged with any program to try and increase the numbers in both of these
user groups.

Note that any use of snippet real estate must be dependent on dynamic
snippets landing with good granularity in reaching the intended target.

Karen

-----Original Message-----
From: dev-planning-bounces+krudnitski=mozil...@lists.mozilla.org
[mailto:dev-planning-bounces+krudnitski=mozil...@lists.mozilla.org] On
Behalf Of Alex Keybl
Sent: 30 April 2013 14:06
To: Chris Peterson
Cc: dev-pl...@lists.mozilla.org
Subject: Re: Increasing our Aurora and Nightly user populations?

Re: Increasing our Aurora and Nightly user populations? PhillipJones 4/30/13 4:24 PM
What are the advantages of FF with Aurora. I have on my computer and the
only difference I see is just the Name.  I keep it updated. But I see
little difference.

--
Phillip M. Jones, C.E.T.      "If it's Fixed, Don't Break it"
http://www.phillipmjones.net    mailto:pjon...@comcast.net
Re: Increasing our Aurora and Nightly user populations? Monica Chew 4/30/13 4:31 PM
Is there any possibility of switching to a more generic testing infrastructure, e.g. Chrome's --force_fieldtrials flag:

https://code.google.com/p/chromium/codesearch#chromium/src/base/metrics/field_trial.h
http://www.ghacks.net/2013/04/05/field-trials-in-chrome-how-to-randomize-or-force-them/

It would make releasing new features safer by enabling slow roll outs, and we could eliminate bias in population between channels.

Monica

----- Original Message -----
> [d.platform->d.planning]
>
> Grace and Laura have worked on similar initiatives in the past
> (about:home campaigns for pre-release channel uptake). When we do
> these sorts of campaigns, we like to set a target so that we don't
> drain too many users from higher channels.
>
> I would not perform any redistribution of Aurora/Nightly users, since
> we don't want to dip below our current populations in either. If we
> advertise Nightly/Aurora to Beta users, we have to keep in mind the
> fact that many of these users have forgotten that they're using
> Firefox Beta as opposed to Release. I wouldn't recommend advertising
> Aurora/Nightly to Release users, only Beta, given the far differing
> quality bar.
>
> As mentioned in my other reply, there is benefit to doing this with
> regularity for sake of pre-release population diversity and
> representativeness.
>
> -Alex
>
> On Apr 29, 2013, at 12:07 PM, Chris Peterson <cpet...@mozilla.com>
> wrote:
>
> > (Sorry if this is not the right forum for this discussion.)
> >
> > To increase our testing populations, I'd like to suggest that we
> > add a periodic "channel upsell" message to the about:home page (of
> > Aurora and Beta) and the about box (of Aurora, Beta, and possibly
> > Release).
> >
> > Beta and Aurora users have already opted in to a testing channel,
> > so we know they are receptive to beta-quality software. If they
> > have set Beta or Aurora as their default browser and/or have been
> > using it consistently for a few weeks, we could try to upsell Beta
> > users to Aurora and Aurora users to Nightly.
> >
> > We did something similar with Firefox for Android. After a certain
> > number of browser launches, we display a one-time message
> > suggesting they write a Google Play store review if they like
> > Firefox or contact Firefox Support if they have problems.
> >
> > Similarly, users who check their browser's about box are clearly
> > interested in version numbers or checking for the latest updates.
> > These users might be receptive to a channel upsell message to get
> > an even bigger version number. :)
> >
> >
> > chris p.
> > _______________________________________________
> > dev-platform mailing list
> > dev-pl...@lists.mozilla.org
> > https://lists.mozilla.org/listinfo/dev-platform
>
Re: Increasing our Aurora and Nightly user populations? Laura Forrest 4/30/13 8:30 PM
Love this discussion. I can help by being your marketing contact since Firefox Desktop is my realm.

There are many ways to accomplish what you're suggesting, but first I'd like to get a better sense of what you folks need. For example, is the goal to make sure the pre-release population matches a certain diversity level, or is it to increase Aurora ADIs by 20% in order to make crash reports more significantly significant? It would be great if a engineering point-person/s can help clarify and we can go from there.  

Laura
Re: Increasing our Aurora and Nightly user populations? Jonas Sicking 5/1/13 12:30 AM
I think getting more web developers to use Aurora should be an
explicit goal. For three reasons:

* They are probably best equipped to find regressions in Firefox that
are causing their websites to break.
* We need their input on new features that we are building, many of
which are only enabled on Aurora (especially going forward). In fact,
we are enabling these features only on Aurora specifically to get
their input, so if they are not using Aurora that's a wasted effort.
* This means that we will reach them quicker with improvements of our
web development tools. I.e. we'll more quickly make them happier to
develop directly in/for Firefox.

/ Jonas
Re: Increasing our Aurora and Nightly user populations? Benjamin Smedberg 5/1/13 7:01 AM
On 4/30/2013 11:30 PM, Laura Forrest wrote:
> Love this discussion. I can help by being your marketing contact since Firefox Desktop is my realm.
>
> There are many ways to accomplish what you're suggesting, but first I'd like to get a better sense of what you folks need. For example, is the goal to make sure the pre-release population matches a certain diversity level, or is it to increase Aurora ADIs by 20% in order to make crash reports more significantly significant? It would be great if a engineering point-person/s can help clarify and we can go from there.
I'll volunteer, although akeybl will need to be in the discussion as
well ;-)

I think we have a couple complementary goals here:

* Get more "knowledgable passive testers" on the Aurora channel. Ideally
I'd like to increase the current population by about 100%
** Web developers
** Volunteers who are capable of spending a few minutes to file a good bug

* Get a better representative sample of users on earlier channels,
especially Aurora.
** Users in non-English locales
** Users with "odd" plugins, antivirus software, IMEs,
accessibility/screen readers, etc
** Users with older/slower computers
** Users with very new consumer-grade computers (windows 8 especially)

However, we need to understand that in order to successfully run Aurora,
you have be willing for your extensions to sometimes break, your browser
to perhaps be a bit less stable. So the Aurora channel is not for "most
people".

I wonder sometimes whether the fact that Aurora has nightly updates is
harming our user retention there. Using nightly and aurora can still be
a bit annoying because you'll get toast notifications for updates, and I
still see the dialog update prompt pretty regularly. I wonder if we
should consider some of the following changes to the Aurora channel:

* Turn down the update settings so that you don't get any update toasts
or dialogs more than once a week?
* If user chooses not to update immediately, continue downloading the
latest daily updates instead of waiting for restart.
* Move to twice-weekly updates instead of nightlies?

I also think we should consider having a "channel newsletter" of some
sort, which we could use to provide low-touch information to channel
users such as:

* Tips on filing better bugs.
* Notices about important new features

Of course, I may be saying two opposite things here: "annoy users less"
and "annoy users more". I'd love help reconciling those ;-). I think
we've traditionally limited this kind of information to things like
start-page snippets; but I doubt that most Aurora users see the
start-page snippets on a regular basis. For example, we've been
promoting the Flash beta channel to Aurora and beta users via snippets
for months now, and I still get people saying "I just saw a note about
using the Flash beta".

--BDS

Re: Increasing our Aurora and Nightly user populations? Jeff Griffiths 5/1/13 7:39 AM
FWIW devtools recently started a 'new in Aurora' series of posts on
hacks we intend to push out on or around merge dates highlighting what
has landed in the previous 6 weeks. As we work with new hires on Mark
Coggins' team of FxOS evangelists and ramp up that developer program we
will also message out about Aurora as the preferred version for developers.

Jeff
Re: Increasing our Aurora and Nightly user populations? Monica Chew 5/1/13 8:43 AM
> I think we have a couple complementary goals here:
>
> * Get more "knowledgable passive testers" on the Aurora channel.
> Ideally
> I'd like to increase the current population by about 100%

I don't know if this chart is accurate (from March 2013 stats on http://en.wikipedia.org/wiki/Template:Firefox_usage_share), but if it is, then even increasing Aurora usage by an order of magnitude only gets to 1% of Firefox users. In my experience, 1% of users is often not enough to uncover bugs, especially in corner cases, and especially if the population is not representative.

Firefox 19        72.40%        15.11%
Firefox 20        2.20%        0.46%
Firefox 21        0.14%        0.03%
Firefox 22        0.10%        0.02%

Should we be aiming for a more generic slow rollout mechanism here?

Monica
Re: Increasing our Aurora and Nightly user populations? Axel Hecht 5/1/13 9:23 AM
On 5/1/13 4:01 PM, Benjamin Smedberg wrote:
> * Get a better representative sample of users on earlier channels,
> especially Aurora.
> ** Users in non-English locales
> ** Users with "odd" plugins, antivirus software, IMEs,
> accessibility/screen readers, etc
> ** Users with older/slower computers
> ** Users with very new consumer-grade computers (windows 8 especially)
>
> However, we need to understand that in order to successfully run Aurora,
> you have be willing for your extensions to sometimes break, your browser
> to perhaps be a bit less stable. So the Aurora channel is not for "most
> people".
>
> I wonder sometimes whether the fact that Aurora has nightly updates is
> harming our user retention there. Using nightly and aurora can still be
> a bit annoying because you'll get toast notifications for updates, and I
> still see the dialog update prompt pretty regularly. I wonder if we
> should consider some of the following changes to the Aurora channel:
>
> * Turn down the update settings so that you don't get any update toasts
> or dialogs more than once a week?
> * If user chooses not to update immediately, continue downloading the
> latest daily updates instead of waiting for restart.
> * Move to twice-weekly updates instead of nightlies?

I'd obviously love to get more users on localized builds :-)

I'd also like to keep promptly updates to localized builds for changes
in the localization, though. If we let two or three days pass between a
localizer working on aurora, and being able to test the work, we'll make
it yet less likely that they'll actually test.

Can I get a quick fact check on the following?

IIRC, and if we're still doing it this way, we're building an aurora
nightly if:

releases/mozilla-aurora changed or
any of releases/l10n/mozilla-aurora changed

and then we build for en-US and repack all locales, on all platforms,
with updates for all of that.

Do we know how many days we're actually doing aurora builds in the end,
and what triggered them?

There's a bunch of knobs to turn here, though back when we talked about
this last, much depended on what the updater was capable to do. No idea
what our options are today.

Is there data in FHR that can guide us here? Anecdotal evidence, I often
let my aurora build up for a few days, and then apply the incremental
update just to download a full update right away next, and then let the
build sit there for a few more days in active use.

Axel
Re: Increasing our Aurora and Nightly user populations? Matt Brubeck 5/1/13 9:56 AM
On 4/30/2013 4:24 PM, PhillipJones wrote:
> What are the advantages of FF with Aurora. I have on my computer and the
> only difference I see is just the Name.  I keep it updated. But I see
> little difference.

When you use Aurora, you begin testing new features and fixes 12 weeks
before they are released to the ~500 million users of our stable
channel.  Performance data, bug reports, and crash reports from Aurora
testers like you helps us find and fix problems before they affect the
rest of our users.

And as a bonus, you get to enjoy new features and performance
improvements three months before others.  For example, the current
Aurora version contains these improvements that are not yet available on
the release channel:
http://www.mozilla.org/en-US/firefox/22.0a2/auroranotes/
Re: Increasing our Aurora and Nightly user populations? Alex Keybl 5/1/13 10:46 AM
I'm hearing a couple of goals from people:

* Get more web developers on earlier channels (Nightly/Aurora)
* Increase the overall pre-release population size (especially earlier channels), in hopes that we'll get increased critical feedback earlier
* Increase the diversity of the population (through targeting or overall growth) to have a more representative population, for instance l10n-wise

I think getting web developers on earlier channels is a newish goal, while the others have been ongoing goals. I'd suggest that before we consider the necessity for growth initiatives (outside of targeting web devs), we let FHR data come in across all channels and wait to hear back from the metrics team on our holes in coverage.

-Alex

On Apr 30, 2013, at 8:30 PM, Laura Forrest <lfor...@mozilla.com> wrote:

> Love this discussion. I can help by being your marketing contact since Firefox Desktop is my realm.
>
> There are many ways to accomplish what you're suggesting, but first I'd like to get a better sense of what you folks need. For example, is the goal to make sure the pre-release population matches a certain diversity level, or is it to increase Aurora ADIs by 20% in order to make crash reports more significantly significant? It would be great if a engineering point-person/s can help clarify and we can go from there.  
>
Re: Increasing our Aurora and Nightly user populations? Chris Hofmann 5/1/13 11:04 AM
On 5/1/13 10:46 AM, Alex Keybl wrote:
> I'm hearing a couple of goals from people:
>
> * Get more web developers on earlier channels (Nightly/Aurora)
> * Increase the overall pre-release population size (especially earlier channels), in hopes that we'll get increased critical feedback earlier
> * Increase the diversity of the population (through targeting or overall growth) to have a more representative population, for instance l10n-wise
>
> I think getting web developers on earlier channels is a newish goal, while the others have been ongoing goals.

It's always been a goal, or at least should have been.  Finding and
fixing web compatibility bugs inside the 6 weeks of beta is just too
short a window and adds risk to every release.   It's web developers
that can help us to find most of these compatibility bugs when they are
running aurora against the sites they develop; so lets get them on
Aurora to spot problems early.

>   I'd suggest that before we consider the necessity for growth initiatives (outside of targeting web devs), we let FHR data come in across all channels and wait to hear back from the metrics team on our holes in coverage.

Why wait?  We will always have holes to fill and more data from FHR will
always be nicer, but we should press on with adding to our aurora and
beta populations now that we have some momentum and interest.

Press On!

-chofmann
Re: Increasing our Aurora and Nightly user populations? Mounir Lamouri 5/1/13 11:16 AM
On 01/05/13 18:46, Alex Keybl wrote:
> I'm hearing a couple of goals from people:
>
> * Get more web developers on earlier channels (Nightly/Aurora)

I'm sceptical regarding getting web developers on Nightly. We should
definitely target Aurora for them otherwise once they will hit a highly
discomforting bug they might switch back to whatever they were using.

> I think getting web developers on earlier channels is a newish goal,

What I have been doing such as other people is to ask web developers to
always try Aurora on their website so they can tell us if they have a
problem and we can fix it before users actually get the bug shipped to
them. Any web developer who had his/her website broken by a new browser
release should understand that argument.
They will anyway need a Firefox release installed too and we should make
clear that development should be tested on this version because Aurora
might have experimental features enabled that the release channel does
not have.

--
Mounir
Re: Increasing our Aurora and Nightly user populations? Steve Fink 5/1/13 1:34 PM
On Wed 01 May 2013 12:41:34 PM PDT, Cheng Wang wrote:
> As much as increasing population size is an easy thing to point to, I
> don't think it's what we actually want. We really need to make better
> use of our existing population, give them a reason to stay on
> pre-release builds and help them help us.

This feels more right to me than the other approaches that have been
discussed. I keep thinking in the back of my head that if we did some
big push to increase the Aurora audience, that we'd end up doing a
bunch of work only to see the numbers gradually dwindle back down as
other priorities came to the fore. But if we could make Aurora more
meaningful and valuable *to the user*, then it'd be more sticky
long-term and we'd get an increased population as a side benefit.
Concerning the individual points:

> * Add-ons don't work reliably.

There must be some way to improve this, but I don't know where to best
apply the effort. What are the typical causes of unreliable/unavailable
addons on Aurora? Obviously, some addons need to be updated for Aurora
and won't be before the relevant change is in Aurora, but is this the
bulk of the problems? Are we now too cavalier about addon-incompatible
changes?

> * They feel like they should be ahead of the game but it looks and
> feels JUST LIKE release.
> * We're not giving aurora users anything special. They don't get
> priority on getting their feature requests fixed, they don't get
> special access to developers to talk about things of concern, they
> don't get consulted before we roll out a new feature so there's no
> driver to be using (what's perceived as) less reliable software.

IMHO, it would be worth addressing this directly. Give it branding
that's different enough that a shoulder-surfer will notice and comment
on it. The geek cred ought to be worth something, at least. Expose
release notes prominently, perhaps in the Help menu. Put an
Aurora-specific about:config option in the Help menu that highlights
new options, even if they haven't been changed from their default
values, or options that are relevant to Aurora-only changes. Add a menu
option that goes directly to bugzilla, preferably to an Aurora-specific
landing page that thanks them for testing and suggests helpful
procedures for making a better bug report (eg, try to reproduce on
release and nightly.) Encourage the user to mention in the bug that
they found the problem using Aurora. Tell them how to subscribe to
bugmail for their bugs of interest. None of this is new or unavailable
elsewhere, but clustering it all around Aurora helps point out that
they're doing something special to help.

Perhaps make an Aurora-specific community (#aurora irc channel?) to
help users help each other. Heck, I run into mysterious problems
frequently that aren't clear enough to file a bug on, and that I'd like
to just mention to a community of like-minded users to see if anyone
else is seeing the same thing. I could, and do, use #developers for
this, but only for things that are pretty extreme.

I think the benefits of Aurora specialization far outweigh the benefits
of looking as much like the Beta-to-be as possible.

> * Stability and performance is not worse than release but it's not
> better either.

I have no interest in making Aurora stability *better* than release!
But perf is another matter. I think exposing experimental config
settings more liberally in Aurora might help here. Though I'd kind of
like to set these sorts of preferences in an Aurora-specific overlay,
so I can fall back to a release build and revert those settings without
switching profiles or manually toggling them.

> * Updates are more frequent (still not sure how they know) which
> breaks up the user experience

I think other people were talking about this.

> * Nevermind that we have lots of QA, every bug they hit feels like
> we're releasing shoddy work and making them pick up our slack.

Which is ironic, since if you're using Aurora then finding a bug should
make you happy. You're helping! If people are feeling as Cheng
describes above, it seems like we need to improve messaging related to
the appropriate mindset for an Aurora user.

> * Features come and go with no warning, education or commentary.

An in-browser pointer to a raw changelog would make sense to me, even
if it's in a raw form. You'd only read it when you really care about a
particular feature, and maybe not even then -- just knowing the
information is available would perhaps appease many users.

> Right now, we don't have the staff to handle the feedback we do get, I
> have no idea how to handle 10 times more. Everything we do is reading
> tea leaves and educated guesswork. We read 1000 pieces of unstructured
> feedback and 2 people mention something about tabs lagging a little:
> this could be a blow-up issue on release or it could be nothing. There

That's an example where an Aurora-specific forum might help, if it
resulted in the community doing more of the triage. (Easier said than
done, I know.)

Re: Increasing our Aurora and Nightly user populations? Steve Wendt 5/1/13 3:04 PM
On 5/1/2013 1:34 PM, Steve Fink wrote:

> Expose release notes prominently, perhaps in the Help menu.

SeaMonkey still has Help->Release Notes; I never understood why Firefox
stripped that out (same could be said for lots of other stuff, I suppose).

> Aurora-specific about:config option in the Help menu that highlights
> new options, even if they haven't been changed from their default
> values, or options that are relevant to Aurora-only changes.

Actually detailing them in the release notes is better (can be updated,
can contain links to technical articles, etc.).  The release notes have
been severely over-simplified to a list of executive summary bullet
points.  Compare with SeaMonkey, which links to Bugzilla bugs:
http://www.seamonkey-project.org/releases/seamonkey2.17/changes

Maybe keep the simple version, with a "more details" list?

Re: Increasing our Aurora and Nightly user populations? Cheng Wang 5/1/13 3:29 PM


Steve Fink wrote:
<snip>
>
>> * Add-ons don't work reliably.
>
> There must be some way to improve this, but I don't know where to best
> apply the effort. What are the typical causes of unreliable/unavailable
> addons on Aurora? Obviously, some addons need to be updated for Aurora
> and won't be before the relevant change is in Aurora, but is this the
> bulk of the problems? Are we now too cavalier about addon-incompatible
> changes?

I don't think any binary add-ons work, for example. But there's also the
lag. Let's say it takes an add-on developer a week (out of six) to
update an add-on and once every few releases breaks something and fixes
it next day. That's pretty fast response. Except for a user the
experience is the add-on works, works, works, doesn't work, works,
doesn't work, doesn't work, works, works, everything is broken, works
again. It just feels random which is more annoying than just being broken.
I think we're taking our goodwill for granted. Even people on aurora
aren't super core community. They don't want to take extra time to do
all this stuff just because they love open source. They probably don't
want extra email or extra communication just because they happen to use
Aurora.

In my dream world, we give Aurora users a phone number they can call, we
work with them if they want to help or we just take their report if they
don't. In either case, we fix their problem within the next two weeks.
Unicorn ponies and dancing fairies.

Honestly, we should answer every feedback request (see lack of staffing
below) with something. Either, we're trying to fix it but we need more
info, or it's a result of X. Or their problem isn't Aurora/Firefox
specific, here's help.

Realistically, we can just tell Aurora users what we're changing and
give them easy ways to tell us more. We should shoulder the heavy
lifting: testing debug builds, building simplified testcases, getting
the full STRs etc.

>
> I think the benefits of Aurora specialization far outweigh the benefits
> of looking as much like the Beta-to-be as possible.

This has the result of biasing our feedback and meaning that someone has
to chase down/test/try to reproduce all n=1 pieces of feedback in case
it turns out to be a problem that disproportionately affects release
over this specialized group.
I'd love to get community involved in feedback gathering in general.
We've tried, it sometimes works but it's not reliable. We can't
guarantee that someone has read ALL of the feedback to make sure the one
report that we need isn't overlooked. We've also had it turn into
someone only advocating for their pet bug and only finding examples that
support it.

Cheng.

Re: Increasing our Aurora and Nightly user populations? Cheng Wang 5/1/13 3:59 PM
I feel like we're treating our Aurora testers like full time Mozilla
community who are dedicated to making Firefox better. Sadly, that's not
the case. Let's flip this around.

What if Honda came by every day and "tweaked" your car. Ostensibly for
the better, but sometimes they forget to connect the brakes. You, having
opted into their testing program, are not assured of the full
performance and safety of a "release" car but generally speaking, this
car drives well, possibly even on average better. (Of course, while it
may get you to work on average 30 seconds faster, there's a non-zero
chance it doesn't start, so there's that.)

You're supposed to let them know if you have problems, but they give you
bugzilla or a feedback box that you have to hunt for. There's nothing in
the car that reminds you of feedback. You file bugs and leave feedback
but often hear nothing. When you hear something, it's often a request
for more information. Sometimes, they say, try removing all of your
things from your car to see if it fixes things. Sometimes they say:
here's a debug car, it's really slow but drive it for a week and let us
know. Sometimes they say that your problem doesn't show up in most other
test cars so you're stuck with it. If you don't like it, the car is open
source, here's a wrench.

Oh and by the way, if you don't mind, take extra trips in the car
because we want to know how it does in the desert or when driven with
the emergency brake on.

You're told that your testing is invaluable to making Hondas great for
millions of people and you're getting cutting edge technology weeks
before everyone else but you just get an unknown quality car when you're
trying to get to work each morning.

What would motivate you to stick with this program? After all, you don't
work for the car company.

 From their side, would it really help to have ten times more people in
a program like this?

====

Thinking along these lines, here's what I would want:

* A note from Honda to say what they changed each night
* A personal mechanic who will look at any problem I have right away and
get me on the road as quickly as possible with a more permanent fix
coming as soon as possible.
* Some way to record how the car is doing without me having to do
anything. If the car is stalling at every third stoplight, I shouldn't
have to report it, it should report itself.
* Being able to schedule the updates. Preferably not when I'm in the
middle of the highway trying to make a deadline. Ideally, once a week on
weekends.
* Reminders to leave feedback to someone I know will listen.
* Some VIP treatment/recognition.

Re: Increasing our Aurora and Nightly user populations? Steve Fink 5/1/13 4:12 PM
On Wed 01 May 2013 03:29:42 PM PDT, Cheng Wang wrote:
>
>
> Steve Fink wrote:
> <snip>
>>
>>> * Add-ons don't work reliably.
>>
>> There must be some way to improve this, but I don't know where to best
>> apply the effort. What are the typical causes of unreliable/unavailable
>> addons on Aurora? Obviously, some addons need to be updated for Aurora
>> and won't be before the relevant change is in Aurora, but is this the
>> bulk of the problems? Are we now too cavalier about addon-incompatible
>> changes?
>
> I don't think any binary add-ons work, for example. But there's also
> the lag. Let's say it takes an add-on developer a week (out of six) to
> update an add-on and once every few releases breaks something and
> fixes it next day. That's pretty fast response. Except for a user the
> experience is the add-on works, works, works, doesn't work, works,
> doesn't work, doesn't work, works, works, everything is broken, works
> again. It just feels random which is more annoying than just being
> broken.

Make add-on breakage block updates?

>>> * They feel like they should be ahead of the game but it looks and
>>> feels JUST LIKE release.
>>> * We're not giving aurora users anything special. They don't get
>>> priority on getting their feature requests fixed, they don't get
>>> special access to developers to talk about things of concern, they
>>> don't get consulted before we roll out a new feature so there's no
>>> driver to be using (what's perceived as) less reliable software.
>>
>> IMHO, it would be worth addressing this directly. Give it branding
>> that's different enough that a shoulder-surfer will notice and comment
>> on it. The geek cred ought to be worth something, at least. Expose
>> release notes prominently, perhaps in the Help menu. Put an
>> Aurora-specific about:config option in the Help menu that highlights
>> new options, even if they haven't been changed from their default
>> values, or options that are relevant to Aurora-only changes. Add a menu
>> option that goes directly to bugzilla, preferably to an Aurora-specific
>> landing page that thanks them for testing and suggests helpful
>> procedures for making a better bug report (eg, try to reproduce on
>> release and nightly.) Encourage the user to mention in the bug that
>> they found the problem using Aurora. Tell them how to subscribe to
>> bugmail for their bugs of interest. None of this is new or unavailable
>> elsewhere, but clustering it all around Aurora helps point out that
>> they're doing something special to help.
>>
>> Perhaps make an Aurora-specific community (#aurora irc channel?) to
>> help users help each other. Heck, I run into mysterious problems
>> frequently that aren't clear enough to file a bug on, and that I'd like
>> to just mention to a community of like-minded users to see if anyone
>> else is seeing the same thing. I could, and do, use #developers for
>> this, but only for things that are pretty extreme.
>
> I think we're taking our goodwill for granted. Even people on aurora
> aren't super core community. They don't want to take extra time to do
> all this stuff just because they love open source. They probably don't
> want extra email or extra communication just because they happen to
> use Aurora.

Who comprises the Aurora community? I'd guess some early adopters who
want to see what's coming or just feel on the edge, some people who hit
a bug in release and switched to Aurora because it was fixed there,
some web developers who want to use the new tools without suffering the
instability of Nightly, and some people who want to help. I'm guessing
some of those piles are far larger than others.

Maybe break them down into active testers, passive testers, and
unwilling/unknowing testers. Making the Aurora-ness more obvious helps
the size of the active pool and hurts the size of the unknowing pool.
The effect on the passive pool depends on how cool/fun/scary/... the
differences are, I guess. Having extra communication channels available
doesn't seem like it would hurt, as long as you're not trying to shove
them down anyone's throat. Whether they'd get critical mass to help is
an open question.

We shouldn't assume that someone is super core community just because
they use Aurora, but the onramp to the community ought to be clearly
visible and a gentle slope. Right now, switching to Aurora is something
you can do once and then forget about until it causes issues. The only
path leads out. Then again, more frequent reminders will make more of
the unwilling leave, which sucks for the passive pool, but is perhaps
worth the risk if the active/semi-active users are higher value. (And I
don't know that they are. For problems that are just numbers games, it
seems like having a huge passive pool is way more helpful.)

> Honestly, we should answer every feedback request (see lack of
> staffing below) with something. Either, we're trying to fix it but we
> need more info, or it's a result of X. Or their problem isn't
> Aurora/Firefox specific, here's help.

But we can't, so isn't the next best thing to clearly document that and
provide a default path? Something like "No response after 3 days?
Sorry. Here's why, here's what you would do to make your report
actionable if you chose to do so, here's how to install multiple
versions so you can run the release build when you need to. And here's
a forum you can go to to complain to other people who may not be able
to do anything more than commiserate." It sucks that we can't help
everyone or even give them a personalized response, but many users
understand that. They still deserve to have *some* way to proceed, even
if it requires enough effort that they will choose not to.

>> I think the benefits of Aurora specialization far outweigh the benefits
>> of looking as much like the Beta-to-be as possible.
>
> This has the result of biasing our feedback and meaning that someone
> has to chase down/test/try to reproduce all n=1 pieces of feedback in
> case it turns out to be a problem that disproportionately affects
> release over this specialized group.

Huh? I'm just talking about some additional menu entries and help text.
Well, and access so they can more easily tweak their own installations
knowingly.

>>> Right now, we don't have the staff to handle the feedback we do get, I
>>> have no idea how to handle 10 times more. Everything we do is reading
>>> tea leaves and educated guesswork. We read 1000 pieces of unstructured
>>> feedback and 2 people mention something about tabs lagging a little:
>>> this could be a blow-up issue on release or it could be nothing. There
>>
>> That's an example where an Aurora-specific forum might help, if it
>> resulted in the community doing more of the triage. (Easier said than
>> done, I know.)
>>
>
> I'd love to get community involved in feedback gathering in general.
> We've tried, it sometimes works but it's not reliable. We can't
> guarantee that someone has read ALL of the feedback to make sure the
> one report that we need isn't overlooked. We've also had it turn into
> someone only advocating for their pet bug and only finding examples
> that support it.

I was handwaving about something less structured. Right now, there's a
certain chance that any given bit of feedback is important. If someone
with a real problem not only submits feedback but also goes to the
forums, finds other people who can confirm/reproduce it, and one of
them files a new bug for it, then the chance that any given feedback is
important *by itself* goes down, so the cost of ignoring (not being
able to keep up with) feedback is less. People scanning through
feedback can adjust their filters to ignore more questionable cases and
Mozilla still gets the same value out of the incoming flood.

But that's totally theoretical; I have no personal experience with
wading through the feedback storm, so I don't really know what it's
like. I am grateful for those of you brave enough to do it.

Re: Increasing our Aurora and Nightly user populations? Chris Hofmann 5/1/13 4:46 PM

Do you see!
Do you see!
Do you see what happens when we even hint at adding more aurora and beta
testers!

A whole host of wonderful ideas that help to make our software better,
make it easier to keep it in tip top shipping condition, and ideas that
make it easier for more people to contribute.

Imagine a world where we actually added those people to the aurora ad
beta channels.  We would be forced to respond to all these good ideas.

I love this just as a discussion, but lets not let us hold us back from
actually moving forward in encouraging more people to get on those
channels and push us towards implementing all these things that will
make Firefox better.

Yes, we need better systems and more people to help us process the
additional feedback that we might get.  Always have, and always will.  
That's not an excuse to leverage the systems that we have in place now,
including SUMO analyst like Cheng and others who do an amazing job, and
the more simple systems that have been around for ages like web compat
bugs that just get filed by web developers under the Firefox component
and need to wind their way though the bug system triages to find a good
home with DOM, JS, and other gecko engineers taking several weeks to do
so in some cases.

Yes it sounds like we need yet another round of better systems to avoid
creating unintended addon compat problems, warn addon developers of
empending intended changes, and have addons adjusted to be compatible.

Yes we need better product planning and feature tracking, and to make
sure features land in on central and in aurora to get full cycles of
testing.   Yes we need to make aurora users feel privileged in that they
can have impact in making the features better.

Yes we need better systems to inform our Aurora and Beta helpers about
what's landed and what's coming to encourage and entice them to keep
helping us.  Some of that is just blogging and echoing thought the press
and blogishere, and maybe in the future actual features in the product
that facilitate the communication.

The world won't be perfect for the new users that we attract to aurora
and beta, but hopefully they will put up with some shortcomings in the
short term, and push us to respond to their needs in the longer term.

Facebook tells us that we have over 15 million "friends" (
https://www.facebook.com/Firefox )  that believe in us, and want us to
succeed, and want to help.   Let's challenge them.   Lets find ways that
even just a small pct. of these can really plug in and help us to build
great software, and help them feel satisfied in doing so.

-chofmann
> _______________________________________________
> dev-planning mailing list
> dev-pl...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-planning

Re: Increasing our Aurora and Nightly user populations? WaltS 5/2/13 5:13 AM
On 05/01/2013 07:46 PM, Chris Hofmann wrote:
> Yes we need better systems to inform our Aurora and Beta helpers about
> what's landed and what's coming to encourage and entice them to keep
> helping us.

As a tester one page I used to follow was the Features/Release Tracking
wiki, but even though it is up to date with versions, it is very devoid
of informative content.

<https://wiki.mozilla.org/Features/Release_Tracking>

--
openSUSE 12.3 (64-bit) KDE 4.10.2
Thunderbird Release
Re: Increasing our Aurora and Nightly user populations? jorgev 5/2/13 7:25 AM
On 5/1/13 5:12 PM, Steve Fink wrote:
> On Wed 01 May 2013 03:29:42 PM PDT, Cheng Wang wrote:
>>
>>
>> Steve Fink wrote:
>> <snip>
>>>
>>>> * Add-ons don't work reliably.
>>>
>>> There must be some way to improve this, but I don't know where to best
>>> apply the effort. What are the typical causes of unreliable/unavailable
>>> addons on Aurora? Obviously, some addons need to be updated for Aurora
>>> and won't be before the relevant change is in Aurora, but is this the
>>> bulk of the problems? Are we now too cavalier about addon-incompatible
>>> changes?
>>
>> I don't think any binary add-ons work, for example. But there's also
>> the lag. Let's say it takes an add-on developer a week (out of six) to
>> update an add-on and once every few releases breaks something and
>> fixes it next day. That's pretty fast response. Except for a user the
>> experience is the add-on works, works, works, doesn't work, works,
>> doesn't work, doesn't work, works, works, everything is broken, works
>> again. It just feels random which is more annoying than just being
>> broken.
>
> Make add-on breakage block updates?

I don't think that's reasonable. Most add-ons developers check for
compatibility changes either on beta or just after release. While we try
to communicate compatibility changes in advance, we're very
short-staffed and often end up communicating them only a couple of weeks
before release.

One advantage of having more users on Aurora is that add-on developers
would get more feedback from users in that channel, and possibly fix
things earlier.

There's very little we can do to address this situation for binary
add-ons, since those are generally only built for release versions.
Pretty much all antivirus vendors ship add-ons with binary components,
so any user that thinks those are useful will not use anything other
than release.

Jorge

Re: Increasing our Aurora and Nightly user populations? jorgev 5/2/13 7:25 AM
On 5/1/13 5:12 PM, Steve Fink wrote:
> On Wed 01 May 2013 03:29:42 PM PDT, Cheng Wang wrote:
>>
>>
>> Steve Fink wrote:
>> <snip>
>>>
>>>> * Add-ons don't work reliably.
>>>
>>> There must be some way to improve this, but I don't know where to best
>>> apply the effort. What are the typical causes of unreliable/unavailable
>>> addons on Aurora? Obviously, some addons need to be updated for Aurora
>>> and won't be before the relevant change is in Aurora, but is this the
>>> bulk of the problems? Are we now too cavalier about addon-incompatible
>>> changes?
>>
>> I don't think any binary add-ons work, for example. But there's also
>> the lag. Let's say it takes an add-on developer a week (out of six) to
>> update an add-on and once every few releases breaks something and
>> fixes it next day. That's pretty fast response. Except for a user the
>> experience is the add-on works, works, works, doesn't work, works,
>> doesn't work, doesn't work, works, works, everything is broken, works
>> again. It just feels random which is more annoying than just being
>> broken.
>
> Make add-on breakage block updates?

Re: Increasing our Aurora and Nightly user populations? Zack Weinberg 5/2/13 9:12 AM
On 2013-04-29 4:56 PM, Benjamin Smedberg wrote:
> I believe that our original target for Aurora was to have a population
> about four times larger than the current population, no? Rather than
> targeting and redistributing existing prerelease users, is there a way
> we can focus on "outside" users? For example:
>
> * web developer population -> Aurora

This would be a lot more appealing to web developers if side-by-side
installation of multiple channels was easier.

zw
Re: Increasing our Aurora and Nightly user populations? Robert Strong 5/2/13 9:38 AM
Robert Sayre and I discussed this a couple of times way back when and
the idea that we came up with was a xulrunner app that came with
multiple versions. This was entirely brainstorming, no bug was ever
filed, and no one was ever tasked with implementing it.

Cheers,
Robert
Re: Increasing our Aurora and Nightly user populations? Millwood 5/2/13 9:42 AM
In response to this, I went back to Aurora to be a good citizen.  I'm
able to deal with just about any firefox disaster - good backups.  But I
now remember that Aurora updates every day.  Which will make many
otherwise understanding testers shy away.  I understand why you build
every night, and I understand you'd like quick feedback.  I see no easy
solution except a choice about update rate, since the update requires a
restart and I, for one, normally go weeks without restarting firefox.
Re: Increasing our Aurora and Nightly user populations? Steve Fink 5/2/13 9:56 AM
Yes. And you're far from alone. I'm on Nightly, and I've gotten pretty
good at ignoring update notices. I probably update about once a week,
when the urge strikes me. So there's *some* amount of user control
available -- or rather, this is totally user-controllable, but only by
annoying users and training them to ignore what we say to them. :(

At a higher level, a high frequency of updates allows seeing effects of
changes faster, but shrinks the Aurora pool. Is that the tradeoff we
want?

Re: Increasing our Aurora and Nightly user populations? Robert Strong 5/2/13 9:59 AM
What specifically is it about the update rate? I assume it is about the
notification dialog asking you to restart that is shown every 24 hours?

Thanks,
Robert
Re: Increasing our Aurora and Nightly user populations? Millwood 5/2/13 10:08 AM
Robert Strong wrote:
> What specifically is it about the update rate? I assume it is about the
> notification dialog asking you to restart that is shown every 24 hours?
>
> Thanks,
> Robert
>
Exactly.
Re: Increasing our Aurora and Nightly user populations? Zack Weinberg 5/2/13 10:32 AM
That seems ... overcomplicated.  I'm pretty sure it could be done
entirely in the installer: stick each channel in its own C:\Program
Files\ subdir, give each its own profile directory (bonus points for
auto-configuring Sync) and its own desktop icon / start menu entry, make
sure the release channel remains in charge of double-click on .html
files, and make sure channel A does not consider a running instance of
channel B as mutually exclusive with itself.

If only we had anyone to work on the installer. :)

But seriously, "only if I can install it side-by-side with the version
all my users will have" is *the* thing I always hear from webdevs if I
try to get them interested in beta or aurora.

zw
Re: Increasing our Aurora and Nightly user populations? Robert Strong 5/2/13 10:50 AM
On 5/2/2013 10:32 AM, Zack Weinberg wrote:
> On 2013-05-02 12:38 PM, Robert Strong wrote:
>> On 5/2/2013 9:12 AM, Zack Weinberg wrote:
>>> On 2013-04-29 4:56 PM, Benjamin Smedberg wrote:
>>>> I believe that our original target for Aurora was to have a population
>>>> about four times larger than the current population, no? Rather than
>>>> targeting and redistributing existing prerelease users, is there a way
>>>> we can focus on "outside" users? For example:
>>>>
>>>> * web developer population -> Aurora
>>>
>>> This would be a lot more appealing to web developers if side-by-side
>>> installation of multiple channels was easier.
> >
>> Robert Sayre and I discussed this a couple of times way back when and
>> the idea that we came up with was a xulrunner app that came with
>> multiple versions. This was entirely brainstorming, no bug was ever
>> filed, and no one was ever tasked with implementing it.
>
> That seems ... overcomplicated.  I'm pretty sure it could be done
> entirely in the installer: stick each channel in its own C:\Program
> Files\ subdir, give each its own profile directory (bonus points for
> auto-configuring Sync) and its own desktop icon / start menu entry,
> make sure the release channel remains in charge of double-click on
> .html files, and make sure channel A does not consider a running
> instance of channel B as mutually exclusive with itself.
The Windows installer already installs into separate directories for
everything but Beta and the reason Beta doesn't anymore (it used to) is
due to release drivers wanting Beta to be the same as Release. As for
profile directories, none of our installers understand Mozilla profiles
and it would be extremely overly complicated to make any of them
understand Mozilla profiles along with this would not be something we
would want to distribute to Release users.

There are also plenty of Web Developers that use Mac. This was one of
the reasons why sayre and myself decided on xulrunner to do this. Plus,
it can handle the multiple profiles for all platforms and the issues
with multiple installations on Mac. We really did think this through vs.
just shooting from the hip.

>
> If only we had anyone to work on the installer. :)
Along with a bunch of other things they also work on.

Cheers,
Robert

>
> But seriously, "only if I can install it side-by-side with the version
> all my users will have" is *the* thing I always hear from webdevs if I
> try to get them interested in beta or aurora.
>
> zw
Re: Increasing our Aurora and Nightly user populations? Robert Strong 5/2/13 10:51 AM
BTW: we also discussed basing this on the profile manager xulrunner app
since it already has most of the profile management code needed.
Re: Increasing our Aurora and Nightly user populations? Steve Wendt 5/2/13 11:04 AM
On 5/2/2013 7:25 AM, Jorge Villalobos wrote:

> Pretty much all antivirus vendors ship add-ons with binary components,
> so any user that thinks those are useful will not use anything other
> than release.

Users who think those are useful are probably not the users you want
running Aurora and nightly.  ;-)

Re: Increasing our Aurora and Nightly user populations? Mike Connor 5/2/13 12:56 PM
On 2013-05-01 12:23 PM, Axel Hecht wrote:
> Is there data in FHR that can guide us here? Anecdotal evidence, I
> often let my aurora build up for a few days, and then apply the
> incremental update just to download a full update right away next, and
> then let the build sit there for a few more days in active use.

We don't have full updater tracking, though I filed a bug last week on
that. [1]

We do have versions by day, and current build IDs, so we can at least
figure out how many people are lagging per channel.

-- Mike

[1] https://bugzilla.mozilla.org/show_bug.cgi?id=864993
Re: Increasing our Aurora and Nightly user populations? Michael Hoye 5/2/13 1:49 PM


----- Original Message -----
> From: "Benjamin Smedberg" <benj...@smedbergs.us>
>
> I think we have a couple complementary goals here:
>
> * Get more "knowledgable passive testers" on the Aurora channel.
> Ideally I'd like to increase the current population by about 100%

Though it hasn't been a comprehensive search, I've done a
little bit of digging, and I haven't seen a lot of places that
a civilian, for lack of a better term, can find out what betas,
auroras and nightlies are about, much less be able to make an
informed decision about participation.

We've got a few (very few) onramps to our prerelease channels,
and the signage is basically terrible.

I've filed two bugs:

https://bugzilla.mozilla.org/show_bug.cgi?id=868044

That one is an addition to Bugzilla, and my hope is that by knowing
what channel bug-filers are in, we can later add in a way to upsell
them on the next level up of prerelease (Filed a bug, and on a beta?
Why not try aurora! On aurora, and want to live on the edge?
Here's nightly!).

Alternatively, this will give people using prerelease software
inadvertently an escape route, which may also make their lives better.

Likewise, I've filed this bug:

https://bugzilla.mozilla.org/show_bug.cgi?id=868149

with the ultimate goal of selling the advantages of being a part of the
prerelease channels. You get this bug fixed! And you get this bug fixed!
And you get this bug fixed! EVERYONE ON AURORA GETS THEIR BUGS FIXED except
for the one or two we added in, sorry about that, please report them over here,
and if that's a deal-breaker for you here's the regular old beta...

I also think that a dashboardy page in that spot will let us be more transparent
with our users about the costs of participation. Want to join Aurora? Here's the
good bits! Here's the bad bits (plugin breakage, weekly(?) updates...) and help
people make better-informed choices.

The nice thing about this approach is that uptake is to some extent tunable,
and we can offer people options that direct them to where we'd like them to
be, even as we offer them information we think they should have.

Thanks,

- mhoye
Re: Increasing our Aurora and Nightly user populations? Mounir Lamouri 5/3/13 2:03 AM
On 02/05/13 18:50, Robert Strong wrote:
> The Windows installer already installs into separate directories for
> everything but Beta and the reason Beta doesn't anymore (it used to) is
> due to release drivers wanting Beta to be the same as Release. As for
> profile directories, none of our installers understand Mozilla profiles
> and it would be extremely overly complicated to make any of them
> understand Mozilla profiles along with this would not be something we
> would want to distribute to Release users.

What is Firefox Mobile doing?

--
Mounir
Re: Increasing our Aurora and Nightly user populations? Johnathan Nightingale 5/3/13 6:37 AM
Each app's data on android is sandboxed vs. user-global on desktop machines. The same problem doesn't apply (though sharing profiles is now impossible, which causes its own excitement).

J

---
Johnathan Nightingale
VP Firefox Engineering
@johnath

Re: Increasing our Aurora and Nightly user populations? Zack Weinberg 5/3/13 6:42 AM
On 2013-05-02 1:50 PM, Robert Strong wrote:
> On 5/2/2013 10:32 AM, Zack Weinberg wrote:
>> On 2013-05-02 12:38 PM, Robert Strong wrote:
...
>>> Robert Sayre and I discussed this a couple of times way back when and
>>> the idea that we came up with was a xulrunner app that came with
>>> multiple versions. This was entirely brainstorming, no bug was ever
>>> filed, and no one was ever tasked with implementing it.
>>
>> That seems ... overcomplicated.  I'm pretty sure it could be done
>> entirely in the installer: stick each channel in its own C:\Program
>> Files\ subdir, give each its own profile directory (bonus points for
>> auto-configuring Sync) and its own desktop icon / start menu entry,
>> make sure the release channel remains in charge of double-click on
>> .html files, and make sure channel A does not consider a running
>> instance of channel B as mutually exclusive with itself.
> The Windows installer already installs into separate directories for
> everything but Beta and the reason Beta doesn't anymore (it used to) is
> due to release drivers wanting Beta to be the same as Release. As for
> profile directories, none of our installers understand Mozilla profiles
> and it would be extremely overly complicated to make any of them
> understand Mozilla profiles along with this would not be something we
> would want to distribute to Release users.

I'm sorry, I didn't mean to imply you hadn't thought about it at all.

I don't see why you say "this would not be something we would want to
distribute to Release users".  The target audience here is current users
of Release who don't want to *replace* it with Beta or Aurora but would
be happy to try out one of the newer channels *as well*, so if making
that go smoothly involves adjustments to Release then that seems
worthwhile to me.

zw
Re: Increasing our Aurora and Nightly user populations? Robert Strong 5/3/13 12:29 PM
Because separate profiles means that the history, bookmarks, add-ons,
etc. will not be the same between the profiles and at the very least
providing that as an option in the installer would be a huge foot gun
for some Release users though it is also important to note that there
are other reasons besides that as already pointed out.

Robert
Re: Increasing our Aurora and Nightly user populations? Robert Strong 5/3/13 12:35 PM
For context, this thread is in regards to "web developers" (I just
noticed that the previous references to "web developers" has been
snipped from the thread).

Robert
Re: Increasing our Aurora and Nightly user populations? Ben Hearsum 5/6/13 12:08 PM
On 05/01/13 12:23 PM, Axel Hecht wrote:
> Can I get a quick fact check on the following?
>
> IIRC, and if we're still doing it this way, we're building an aurora
> nightly if:
>
> releases/mozilla-aurora changed or
> any of releases/l10n/mozilla-aurora changed
>
> and then we build for en-US and repack all locales, on all platforms,
> with updates for all of that.

Yup.
Re: Increasing our Aurora and Nightly user populations? Robert Kaiser 5/7/13 10:00 AM
Alex Keybl schrieb:
> I would not perform any redistribution of Aurora/Nightly users, since we don't want to dip below our current populations in either. If we advertise Nightly/Aurora to Beta users, we have to keep in mind the fact that many of these users have forgotten that they're using Firefox Beta as opposed to Release. I wouldn't recommend advertising Aurora/Nightly to Release users, only Beta, given the far differing quality bar.

I think we should aim to get the cross-channel distribution that the
channel model originally aimed at, which is roughly 1:10:100:1000.

As I recently mentioned in a different post, that would help stability
stats and testing quite a bit, and we are far away from that with
Aurora, and sizably away from that with Beta right now.

I agree that we should only advertise Beta to Release users, but we can
advertize Aurora to Beta users. We should not advertise Nightly to
anyone on other channel, IMHO, it's only for the daring. Also, the
Nightly:Release ratio is roughly within the original target, for both
desktop and Android.

Robert Kaiser

Re: Increasing our Aurora and Nightly user populations? Robert Kaiser 5/7/13 11:58 AM
Monica Chew schrieb:
>> I think we have a couple complementary goals here:
>>
>> * Get more "knowledgable passive testers" on the Aurora channel.
>> Ideally
>> I'd like to increase the current population by about 100%
>
> I don't know if this chart is accurate (from March 2013 stats on http://en.wikipedia.org/wiki/Template:Firefox_usage_share), but if it is, then even increasing Aurora usage by an order of magnitude only gets to 1% of Firefox users. In my experience, 1% of users is often not enough to uncover bugs, especially in corner cases, and especially if the population is not representative.

Actually, 1% is enough to uncover a lot more than what we uncover right
now with the roughly .1% coverage on Aurora, e.g. we'd get a better
signal-to-noise ratio in crash stats, a lot more feedback on new
features, etc.

 From a stability POV, there some classes of crashes we easily see with
the 100k-ish audience we have on Nightly, but to get a good
signal-to-noise ratio on the "next round" of issues, good data on URLs
to reproduce, a range of crash reports that can give us good insight and
some usable data on trends, we'd probably need some 700k-1M daily
installations on Aurora, while atm we have ~120k on 22.
That means that many of those things only get spotted on Beta (with ~2M
installations on the latest update) when they could be worked on Aurora
already.

Also, right now, we need to ship release to ~10M active installations to
get good data on yet another class of crashes, so right now we turn off
automated updates after we have those, check where we are and if
everything is alright, and only then open up to everyone. Not too
mention that recently we seem to spot a number of other edge-case issues
in that lot, e.g. proxy or remote profile bugs. If we had that kind of
audience on Beta, we'd have more time to fix those issues and feel more
confident about shipping to everyone and do it faster.

So that's the reasons from my stability-focused POV to want larger
Aurora and Beta audiences - and why the 1:10:100:1000+ ratios
(Nightly:Aurora:Beta:Release) we aimed for initially with the channel
model still sounds like a good deal to me.

Robert Kaiser
Re: Increasing our Aurora and Nightly user populations? Robert Kaiser 5/7/13 12:00 PM
Michael Hoye schrieb:
> We've got a few (very few) onramps to our prerelease channels,
> and the signage is basically terrible.

I actually have indeed a hard time finding Aurora or Beta downloads when
I need them, while nightly.mozilla.org is convenient to find. Thanks for
mentioning onramps here!

Robert Kaiser
Re: Increasing our Aurora and Nightly user populations? Steve Wendt 5/7/13 12:13 PM
On 5/7/2013 12:00 PM, Robert Kaiser wrote:

>> We've got a few (very few) onramps to our prerelease channels,
>> and the signage is basically terrible.
>
> I actually have indeed a hard time finding Aurora or Beta downloads when
> I need them, while nightly.mozilla.org is convenient to find. Thanks for
> mentioning onramps here!

Yes, I complained in the past that this URL hides too much:
http://releases.mozilla.org/pub/mozilla.org/firefox/releases/

And you have to go the FTP route to find betas:
ftp://ftp.mozilla.org/pub/mozilla.org/firefox/releases/

Re: Increasing our Aurora and Nightly user populations? Michael Hoye 5/7/13 12:48 PM
In addition to the two previous bugs I've mentioned, I've just filed this one as well:

https://bugzilla.mozilla.org/show_bug.cgi?id=869616

I think we should consider elevating that to a general principle, in fact: anyone who goes out of their way to tell us about one of the sharp edges of our product should get a thank-you note acknowledging their effort, and another one later when that effort helped produce a result.




- mhoye
Re: Increasing our Aurora and Nightly user populations? Kevin Brosnan 5/7/13 1:11 PM
> _______________________________________________
> dev-planning mailing list
> dev-pl...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-planning

There are two simple aliases.

https://aurora.mozilla.org
https://beta.mozilla.org

Kevin

Re: Increasing our Aurora and Nightly user populations? Robert Kaiser 5/7/13 1:19 PM
Kevin Brosnan schrieb:
> There are two simple aliases.
>
> https://aurora.mozilla.org
> https://beta.mozilla.org

Which both throw certificate errors instead of taking me anywhere (I
know how I can get around them, but people we want to download either
probably get scared - rightfully).

Robert Kaiser
Re: Increasing our Aurora and Nightly user populations? Kevin Brosnan 5/7/13 2:22 PM
Sorry the non https versions redirect correctly. I typed them from
memory. I assumed that since mozilla.org is https that those links
would be too.

http://aurora.mozilla.org
http://beta.mozilla.org

Kevin
Re: Increasing our Aurora and Nightly user populations? Axel Hecht 5/7/13 2:43 PM
Seems those links do a variety of things, at least for localized users.

On a German firefox, aurora.m.o goes to an en-US page offering a German
download, beta.m.o goes to the german channel page.

Not sure if either are great.

And of course the https versions should just work.

Axel
Re: Increasing our Aurora and Nightly user populations? Steve Wendt 5/7/13 2:50 PM
On 5/7/2013 2:43 PM, Axel Hecht wrote:

>>>> https://aurora.mozilla.org
>>>> https://beta.mozilla.org
>
> And of course the https versions should just work.

Not only do they have cert errors (a wildcard SSL cert would do nicely
here), they also prompt for an LDAP login.

Re: Increasing our Aurora and Nightly user populations? PhillipJones 5/7/13 3:41 PM
I test Aurora from time to time.

--
Phillip M. Jones, C.E.T.      "If it's Fixed, Don't Break it"
http://www.phillipmjones.net    mailto:pjon...@comcast.net
More topics »