Basing Firebug User testing on Test Pilot?

[johnjbarton]

I am investigating tools for measuring feature usage and new feature
adoption in Firebug. I recall reading somewhere that one of the design
goals for Test Pilot was to support other applications. Is there
somewhere I can read more about this option?

jjb

[jinghua zhang]

Hi Johnjb, 

Measuring feature usage and new feature adoption in Firebug is very important to all of us, and we are very interested to dig into it.  So far Test Pilot doesn't instrument Firebug yet, but I'd like to bring this topic up to the Firebug team. So maybe you, Test Pilot team and the Firebug team can figure out how to do it together.  We will love to run the study in the near future. 

Do you have any suggestions/ideas so far how you plan to do it? While I am connecting with Firebug team, we could plan the next steps together. 

Great initiative, thanks John!

Jinghua 

[John J. Barton]

Hi.
I've tried twice to post to the newsgroup without effect.
jjb

[johnjbarton]

On Nov 6, 3:25 pm, jinghua zhang <jinghuaz...@gmail.com> wrote:
> Hi Johnjb,
>
> Measuring feature usage and new feature adoption in Firebug is very
> important to all of us, and we are very interested to dig into it.  So far
> Test Pilot doesn't instrument Firebug yet, but I'd like to bring this topic
> up to the Firebug team. So maybe you, Test Pilot team and the Firebug team
> can figure out how to do it together.  We will love to run the study in the
> near future.
>
> Do you have any suggestions/ideas so far how you plan to do it? While I am
> connecting with Firebug team, we could plan the next steps together.

Broadly I was planning to write a Firebug extension that records
events from Firebug use and posts them some place. The recording would
be per Firebug session, not per user or Firefox session. A Firebug
session would start when you open Firebug on a page and end when you
turn Firebug off for the page, close the tab, or exit Firefox.

The events would be results of user operations, "inspect", "set a
breakpoint", "continue after a breakpoint" and so on. The analysis
would tell us three things:
1) For a group of sessions how does feature use cluster? eg 36% use
only inspect, 34% use inspect and CSS edit, 10% only use console, etc.
2) For a single session, are there common patterns "inspect, set a
breakpoint" or whatever.
3) For a single feature, how does usage change over time.

On top of this I'd like to layer user education work. How do users
learn about new features? How can we help them learn? When they try a
feature and it does not work for them how can we get feedback? Can we
make the feature better?

When I get to the "posts them some place" part of the story, I am
stuck.

jjb

>
> Great initiative, thanks John!
> Jinghua
>

[johnjbarton]

On Nov 6, 3:25 pm, jinghua zhang <jinghuaz...@gmail.com> wrote:

> Hi Johnjb,
>
> Measuring feature usage and new feature adoption in Firebug is very
> important to all of us, and we are very interested to dig into it.  So far
> Test Pilot doesn't instrument Firebug yet, but I'd like to bring this topic
> up to the Firebug team. So maybe you, Test Pilot team and the Firebug team
> can figure out how to do it together.  We will love to run the study in the
> near future.
>
> Do you have any suggestions/ideas so far how you plan to do it? While I am
> connecting with Firebug team, we could plan the next steps together.

(I replied once but it does not show up?)

I plan to write a Firebug extension to measure events in Firebug and
post them to some place. The events will be from the time a user
starts working on a web site until they finish. So they open Firebug
on a site or visit a site previously enabled for Firebug, they use
Firebug, then turn it off or close the tab or exit Firefox. That
session is uploaded.

We can measure things like "inspect used" or "set breakpoint" etc. My
first interest is to record uses of the new breakpoint features we
added in 1.5.

Based on the measurements we can:
1) Classify sesssions: 36% only use inspect; 32% use CSS editor, 10%
only use Console etc.
2) Look for patterns within sessions: inspect HTML Style CSS Style
Edit Inspect ...
3) Look for changes over time: more use of break on XHR.
We would like to couple this with user education efforts: does
blogging. update notifications, etc help.

But I am stuck on any plan because of the "post them to some place"
part. So I am looking into Test Pilot.

jjb

>
> Great initiative, thanks John!
> Jinghua
>

[johnjbarton]

Hi...any feedback on this idea?
jjb

[jinghua zhang]

Hi jjb, 

Sorry for the late reply. 

Yes, you are very welcome to post your work on Test Pilot forum (this one), or even in our blog or website, which we will open to public submission in a couple of months. If you prefer to start on your personal blog,  we can also help you to spread the word around through our Test Pilot channels. 

For your ideas, I think the metrics and questions you proposed are definitely interesting to dig into. We can also provide help to review these metrics together with Firebug team when you are close to execute. From items you listed in the email, it seems a lot of work. It would be great if you can do one small successful case first, then we can work on a bigger one later.  :) 

I am not sure if you need to do any preparations before you instrument Firebug, e.g. data policy. If you think there is something that Test Pilot Data policy can help you, we can talk about it.

Let me know if you have more questions or any other things i can help you with. 

Jinghua  

--

You received this message because you are subscribed to the Google Groups "mozilla-labs-testpilot" group.
To post to this group, send email to mozilla-lab...@googlegroups.com.
To unsubscribe from this group, send email to mozilla-labs-test...@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/mozilla-labs-testpilot?hl=.

[johnjbarton]

On Nov 17, 4:25 pm, jinghua zhang <jinghuaz...@gmail.com> wrote:
> Hi jjb,
>
> Sorry for the late reply.
>
> Yes, you are very welcome to post your work on Test Pilot forum (this one),
> or even in our blog or website, which we will open to public submission in a
> couple of months. If you prefer to start on your personal blog,  we can also
> help you to spread the word around through our Test Pilot channels.

I guess I don't know the correct terms. The question I have is: once I
have code that stores firebug usage data I need to send it somewhere
to be aggregated and analyzed. Can you help with that part of the
problem?

>
> For your ideas, I think the metrics and questions you proposed are
> definitely interesting to dig into. We can also provide help to review these
> metrics together with Firebug team when you are close to execute. From items
> you listed in the email, it seems a lot of work. It would be great if you
> can do one small successful case first, then we can work on a bigger one
> later.  :)

Yes, sure that make sense. But really collecting the data is the easy
part.

>
> I am not sure if you need to do any preparations before you instrument
> Firebug, e.g. data policy. If you think there is something that Test Pilot
> Data policy can help you, we can talk about it.

Well I had been hoping to reuse that bit as well, there is a lot of
good stuff in the Test Pilot Data Policy, but also some problems from
my point of view.

The biggest problem is the user experience. (Eclipse has a UI feedback
tool with similar problems). The policy requires lots of opt-in, which
means the user is faced with multiple long dialogs. I'm sure it makes
sense for Firefox, but for Firebug I want to go with a much simpler
model: "it's a community feature". That is, we have an open source
tool that uploads anonymized data on your use of that tool. You can
opt-out by uninstalling the tool at any time, but otherwise your right
to the tool cares a responsibility to contribute. So the user
experience is a one time decision: "Do I trust the Firebug
developers?" If you do, we include your data on the use of the tool.
If you don't, then you don't want to install the tool anyway.

A smaller problem is the last bit about the control of the raw data.
We obviously don't fit the categories allowed to access it, but also I
would want the raw data to be available to anyone in the Firebug
community. There could be restrictions on use of the data to be
consistent with the privacy goals.

What can we do here? Can I crib from your Data Policy, create our own,
and post it for comments?

>
> Let me know if you have more questions or any other things i can help you
> with.
>
> Jinghua
>
> On Tue, Nov 17, 2009 at 10:39 AM, johnjbarton

> <johnjbar...@johnjbarton.com>wrote:

> > mozilla-labs-test...@googlegroups.com<mozilla-labs-testpilot%2Bunsu...@googlegroups.com>
> > .