Defining Shumway release criteria

[Chris Peterson]

In yesterday's meeting, we discussed what would be appropriate,
quantifiable release criteria for each Shumway milestone.

Example release criteria for the M2 milestone ("TypeScript Shumway
extension lands in Nightly, but is disabled by default"):

* Shumway extension is installable.
* All Travis tests pass.
* All AVM1 shell tests pass.
* An agreed upon set of Flash websites (TBD) work as expected, e.g.
Homestar Runner, Candy Crush, and Zombotron?

Example release criteria for the M3 milestone ("Shumway is enabled by
default on Nightly for some Flash ads"):

* All (enabled) automated tests pass.
* All bugs blocking the milestone's meta bug [1] have been closed.
* A set of ads (the oldest 100?) from the SWF Archive [2] work as expected.
* SWFs tagged "WORKS" in the SWF Archive [3] still work as expected.

An appropriate quote from Reid Hoffman, Linked In CEO and Mozilla board
member:

"If you are not embarrassed by the first version of your product,
you’ve launched too late."

:)


chris

[1] https://wiki.mozilla.org/Shumway/Roadmap
[2] http://swf.codeazur.com.br/?all-ads=1
[3] http://swf.codeazur.com.br/?tag=works

[Jet Villegas]

Thanks for sending out this late night e-mail, Chris. This subject keeps me up late too :)

Given that we're being measured against a known quantity (Flash Player 14.0), it's important to manage the user's expectations carefully. Rendering Flash ads as a Shumway 1.0 target is very doable, but the key bit is identifying what we can render faithfully at runtime by the first 60ms of SWF parsing. We need to continue building out that feature, and I'd make that a 1.0 requirement.

--Jet

_______________________________________________
dev-shumway mailing list
dev-s...@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-shumway

[Till Schneidereit]

chiming in quickly ...

On Jul 25, 2014 9:56 AM, "Jet Villegas" <j...@mozilla.com> wrote:
>
> Thanks for sending out this late night e-mail, Chris. This subject keeps
me up late too :)
>
> Given that we're being measured against a known quantity (Flash Player
14.0), it's important to manage the user's expectations carefully.
Rendering Flash ads as a Shumway 1.0 target is very doable, but the key bit
is identifying what we can render faithfully at runtime by the first 60ms
of SWF parsing. We need to continue building out that feature, and I'd make
that a 1.0 requirement.

i agree. that requires bug 1035170 to be implemented. i don't think that's
a ridiculous amount of work. the question is whether we should take it on
before or after m3. to me, that largely depends on whether we expect press
coverage and reviews for m3. my vague guess is "yes", so maybe we should do
it before.

another important point is yank: given that we don't yet have e10s for
content, we should absolutely have it for shumway before m3. i wouldn't
want to read about us making the main thread even more yanky or something
like that, when in fact our final solution should be 100% on-par with flash
in this regard.

other than these two, admittedly large, points, i agree with these
milestones.

till

[Chris Peterson]

On 7/25/14 6:40 AM, Till Schneidereit wrote:
>
> chiming in quickly ...
> On Jul 25, 2014 9:56 AM, "Jet Villegas" <j...@mozilla.com

> <mailto:j...@mozilla.com>> wrote:
> >
> > Thanks for sending out this late night e-mail, Chris. This subject
> keeps me up late too :)
> >
> > Given that we're being measured against a known quantity (Flash
> Player 14.0), it's important to manage the user's expectations
> carefully. Rendering Flash ads as a Shumway 1.0 target is very doable,
> but the key bit is identifying what we can render faithfully at
> runtime by the first 60ms of SWF parsing. We need to continue building
> out that feature, and I'd make that a 1.0 requirement.
>

Jet: where does the "60 ms" number come from?

Besides the ads in the SWF Archive, are there two or three Flash demos
that we'd like to showcase as official regression tests before we can
declare a milestone completed? Some candidates: Candy Crush, Zombotron,
Mining Truck 2, Homestar Runner. We use those Flash games for casual
testing, but we It would be nice if the examples could cover AVM1 and
AVM2 content and sound.

> i agree. that requires bug 1035170 to be implemented. i don't think
> that's a ridiculous amount of work. the question is whether we should
> take it on before or after m3. to me, that largely depends on whether
> we expect press coverage and reviews for m3. my vague guess is "yes",
> so maybe we should do it before.
>
> another important point is yank: given that we don't yet have e10s for
> content, we should absolutely have it for shumway before m3. i
> wouldn't want to read about us making the main thread even more yanky
> or something like that, when in fact our final solution should be 100%
> on-par with flash in this regard.
>

I'm concerned that the requirements for M3 are growing. Instead of being
"Shumway enabled for a small list of ads", we might need some larger
features like e10s support and lazy SWF parsing. I had been assuming
that performance work be fixed in M4 and beyond, but I agree we'll
probably get some press coverage when we enable Shumway by default for
any content. My hunch, though, is that enabling Shumway for just a small
list of ads won't get a lot of press coverage. Banner ads pay for the
web, but they do not make for exciting news headlines. :)

I shuffled a bunch of Shumway bugs between the M3, M4, and 1.0 milestone
meta-bugs based on your feedback here. There are still many GitHub
issues not accounted for in Bugzilla.

https://wiki.mozilla.org/Shumway/Roadmap


chris

> > dev-s...@lists.mozilla.org <mailto:dev-s...@lists.mozilla.org>

> > https://lists.mozilla.org/listinfo/dev-shumway
> > _______________________________________________
> > dev-shumway mailing list

> > dev-s...@lists.mozilla.org <mailto:dev-s...@lists.mozilla.org>
> > https://lists.mozilla.org/listinfo/dev-shumway
>

[Jet Villegas]

60ms is an estimate. I'll take 30ms :)

My point is that we currently take up more than 60ms to start up Shumway, but we need to determine if we can play a SWF well before the time it currently takes to fully boot up.

[Benjamin Smedberg]

On 7/25/2014 3:36 AM, Chris Peterson wrote:
>
> Example release criteria for the M3 milestone ("Shumway is enabled by
> default on Nightly for some Flash ads"):

The quality bar for enabling things by default in nightly is fairly low:
it's ok to expect some breakage as long as users can fix it somehow.

I'm primarily concerned about the definition of "some", and figuring out
what the goals are of turning this on by default.

If the goal is to turn Shumway on for some advertising, do we need to
pre-scan or parse the .swf to determine whether we think we support it?
How does that affect performance? Or are we going to turn shumway on for
all .swf that is certain common advertising sizes? If so, will there be
UI for falling back to Flash in case there's a problem? If so the
milestone ought to include designing that UI.

What is the *goal* of M3? It could be:

* measure memory usage compared to Flash in the wild
* measure performance compared to Flash in the wild (what parts of
performance? Total CPU? Frame rate? Instance startup/teardown time?)
* Get bug reports about broken things from more users.

We should focus the milestone markers around making sure that we can
measure the things we actually want to accomplish with each milestone.

--BDS

[Chris Peterson]

On 8/4/14 11:08 AM, Benjamin Smedberg wrote:
>> Example release criteria for the M3 milestone ("Shumway is enabled by
>> default on Nightly for some Flash ads"):
>
> The quality bar for enabling things by default in nightly is fairly low:
> it's ok to expect some breakage as long as users can fix it somehow.

Till is worried that enabling Shumway, even in Nightly for only a small
set of content, will attract (negative) press coverage if Shumway
performs worse than Flash on content we claim to support. But I'm try to
convince him it won't be so bad. :)

> If the goal is to turn Shumway on for some advertising, do we need to
> pre-scan or parse the .swf to determine whether we think we support it?
> How does that affect performance? Or are we going to turn shumway on for
> all .swf that is certain common advertising sizes? If so, will there be
> UI for falling back to Flash in case there's a problem? If so the
> milestone ought to include designing that UI.

Ad detection (bug 1038444) will probably be based on the ad industry's
standard sizes; URL regex (to match known ad servers or whether the
URL's query string includes a "clicktag" parameter); and SWF version.

I don't know if we can support automatically fallback to Flash if
Shumway can't play the SWF. We have a manual fallback button today, but
it will need to be redesigned for end users. I filed bug 1048565 to
track the automatic fallback UX.

> What is the *goal* of M3? It could be:
>
> * measure memory usage compared to Flash in the wild
> * measure performance compared to Flash in the wild (what parts of
> performance? Total CPU? Frame rate? Instance startup/teardown time?)
> * Get bug reports about broken things from more users.
>
> We should focus the milestone markers around making sure that we can
> measure the things we actually want to accomplish with each milestone.

That's a fair question. We would like to get bug reports (especially
around jank) and collect telemetry on startup time, but we really should
better document our goals how we'll measure them. I've added a note to
our next meeting's agenda.

Thanks for your feedback!

cp