Serving the community better - testing their sites

[Lloyd Hilaiel]

I propose a blog post and open request.

"Do you use and love persona? We love you. Stand up a testing environment and we'll make sure we never break your site."

We ask that a website that uses persona stand up a new environment that mirrors their production site with the only delta of staging urls.

We offer in return to test their site for them before we push changes.

This does not scale to hundreds, but it could scale to 20-30 sites which hopefully is a representative subset that gives us high probability of catching any issues that arise due to odd integrations or public api / semantics that we do not realize are actually part of our contract.

Fabulous QA team, my understanding is you already basically do this during pushes, and the delta would be simply more meaningful sites to test stage against - not an incredible amount of work. Is this right? Can we afford to make this request and promise publicly, with a disclaimer that it's first come first serve till we get 20-30ish takers?

lloyd

[Karl Thiessen]

I'd want to hear from jrgm on this, as he still does the lion's share of
the RP testing.

My gut says two things about this proposal:

1. I'd want a "limited time only" disclaimer -- as you say, this will
not scale out beyond about 40-50 sites.

2. I'd want some real automation emphasis -- a push-button test that
would run with some daily-to-weekly frequency and then again before the
pre-flight manual checklist would help us testing humans focus
effectively, I think.

--KT.

> _______________________________________________
> dev-identity mailing list
> dev-id...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-identity
>

[Lloyd Hilaiel]

On May 17, 2013, at 3:13 PM, Karl Thiessen <kthi...@mozilla.com> wrote:

> 2. I'd want some real automation emphasis -- a push-button test that
> would run with some daily-to-weekly frequency and then again before the
> pre-flight manual checklist would help us testing humans focus
> effectively, I think.

This is a great idea. My first reaction though is because this automation would be rather high cost to maintain given any time one of these sites change their DOM, they might break our tests.

So I have two questions:
1. would the fixed testing cost of manually running through these sites once per train exceed the cost of building and maintaining automation?
2. If we determined that automation would actually be *more* expensive to maintain, would QA still be willing to do this? (if the answer to this is yes, then I think the answer to the higher level question of "should we offer this" may also be yes).

Thinking about #1, I'm not convinced one way or another. We've got a pretty sexy and simple automation setup. I can envision us creating a directory and dropping in about 20 lines of code per site… (and all dialog interactions could re-use what we already have).

Anyone interested in writing those 20 lines to test against an external site to see if this is doable?

lloyd

[Karl Thiessen]

On 2013-05-17 14:21, Lloyd Hilaiel wrote:
> On May 17, 2013, at 3:13 PM, Karl Thiessen <kthi...@mozilla.com

> <mailto:kthi...@mozilla.com>> wrote:
>
>> 2. I'd want some real automation emphasis -- a push-button test that
>> would run with some daily-to-weekly frequency and then again before the
>> pre-flight manual checklist would help us testing humans focus
>> effectively, I think.
>
> This is a great idea. My first reaction though is because this
> automation would be rather high cost to maintain given any time one of
> these sites change their DOM, they might break our tests.

That kind of early breakage is _exactly_ what we want in the early
phases, in my opinion. It serves as an early warning of "Hey! Something
changed over here -- someone should check it out to make sure
everything's ok." That's the kind of tight loop we're going to need
with early adopters to make a program like the one you're proposing
successful.

> So I have two questions:
> 1. would the fixed testing cost of manually running through these sites
> once per train exceed the cost of building and maintaining automation?
> 2. If we determined that automation would actually be *more* expensive
> to maintain, would QA still be willing to do this? (if the answer to
> this is yes, then I think the answer to the higher level question of
> "should we offer this" may also be yes).
>
> Thinking about #1, I'm not convinced one way or another. We've got a
> pretty sexy and simple automation setup. I can envision us creating a
> directory and dropping in about 20 lines of code per site… (and all
> dialog interactions could re-use what we already have).
>
> Anyone interested in writing those 20 lines to test against an external
> site to see if this is doable?

I would love to do this in about two weeks -- but with me going offline
starting next Thursday, I have a ton of stuff to get finished between
now and then. But this would make a lovely Friday project for me, if I
can shake a Friday loose.

In the meantime, if anyone else is interested, please feel free.

--KT.

[Lloyd Hilaiel]

On May 17, 2013, at 3:50 PM, Karl Thiessen <kthi...@mozilla.com> wrote:

>> Anyone interested in writing those 20 lines to test against an external
>> site to see if this is doable?
>
> I would love to do this in about two weeks -- but with me going offline
> starting next Thursday, I have a ton of stuff to get finished between
> now and then. But this would make a lovely Friday project for me, if I
> can shake a Friday loose.

Awesome! Obviously, there's no real urgency. I'm really curious to see what you come up with!

lloyd

[Jonathan Brown]

A demo of the latest version of the Drupal module is always at
http://see.drupalpersona.me/

[John Morrison]

So, during a code push to production, we currently do a rough check of 13
Relying Parties: that we can be signed in before the push, bleed off a
datacenter, change the code in that datacenter, and confirm that signin and
signout work with that datacenter (using local DNS to point a browser at the
datacenter). This, of course, does not include the interaction of the RP
with
our verifier service when that datacenter goes live.

It occurred to me though that as we move to running on aws, this becomes
easier to do before release, since we no longer have the constraint of two
physical datacenters and needing to get back to a redundant state in a short
window of time.

Some number of RP's that choose to use our stage verifier with their stage
implementation would be good to have (like bugzilla has now), but it
might get
complicated to keep RP's in sync until we get settled in aws land.

On automation, I too have pondered the tradeoffs. And that you want to
something a little timing dependent about what point this goes live with
existing previous state. (On the other hand, it's not like I haven't missed
things during the short window (even if it seems long to Gene ;-) ). But
maybe
the tradeoffs go away when we get on aws for everything.

For now, I don't want to commit to more than 20.

John

On 5/17/13 11:21 AM, Lloyd Hilaiel wrote:
> I propose a blog post and open request.
>
> "Do you use and love persona? We love you. Stand up a testing environment and we'll make sure we never break your site."
>
> We ask that a website that uses persona stand up a new environment that mirrors their production site with the only delta of staging urls.
>
> We offer in return to test their site for them before we push changes.
>
> This does not scale to hundreds, but it could scale to 20-30 sites which hopefully is a representative subset that gives us high probability of catching any issues that arise due to odd integrations or public api / semantics that we do not realize are actually part of our contract.
>
> Fabulous QA team, my understanding is you already basically do this during pushes, and the delta would be simply more meaningful sites to test stage against - not an incredible amount of work. Is this right? Can we afford to make this request and promise publicly, with a disclaimer that it's first come first serve till we get 20-30ish takers?
>

[Shane Tomlinson]

On 21/05/2013 03:56, John Morrison wrote:

> On automation, I too have pondered the tradeoffs. And that you want to
> something a little timing dependent about what point this goes live with
> existing previous state. (On the other hand, it's not like I haven't
> missed
> things during the short window (even if it seems long to Gene ;-) ).
> But maybe
> the tradeoffs go away when we get on aws for everything.
>
> For now, I don't want to commit to more than 20.
>
> John

Could you share your list of RPs? Are there Mozilla properties that
would be willing to help us, perhaps not with code, but with guidance -
sites that are relatively stable and we could count on to notify us when
they make breaking changes (like we are *supposed* to do for others -
sorry Stephen D and Zac)?

Shane