Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss
One Metric
2 views
Skip to first unread message
Justin Crawford
Jan 27, 2015, 11:46:51 AM1/27/15
to mdn-drivers
As discussed in today's all-MDN meeting[0], the leads team has been working
to identify a single metric that all contributors to MDN can measure our
success against.
This metric will be an internal measure; it's not a metric we'll include in
the 2015 plans submitted to the Steering Committee.
In 2015, we'd like to measure *quality* (h/t groovecoder). Right now we
don't have any instrumentation that surfaces a number about quality; but we
do have lots of metrics that could inform an aggregate quality measure. For
example...
* inbound links
* visitors to the site
* accounts
* uptime
* response speed
* etc.
Please update this etherpad with your ideas for the metric components of an
all-MDN quality measurement.
https://devengage.etherpad.mozilla.org/quality-measures-201501
The best and most likely ideas will not require engineering time to
instrument.
[0] Today's meeting:
https://devengage.etherpad.mozilla.org/devengage-2015-01-27
Justin Crawford
Product Manager, MDN
hoos...@mozilla.com
to identify a single metric that all contributors to MDN can measure our
success against.
This metric will be an internal measure; it's not a metric we'll include in
the 2015 plans submitted to the Steering Committee.
In 2015, we'd like to measure *quality* (h/t groovecoder). Right now we
don't have any instrumentation that surfaces a number about quality; but we
do have lots of metrics that could inform an aggregate quality measure. For
example...
* inbound links
* visitors to the site
* accounts
* uptime
* response speed
* etc.
Please update this etherpad with your ideas for the metric components of an
all-MDN quality measurement.
https://devengage.etherpad.mozilla.org/quality-measures-201501
The best and most likely ideas will not require engineering time to
instrument.
[0] Today's meeting:
https://devengage.etherpad.mozilla.org/devengage-2015-01-27
Justin Crawford
Product Manager, MDN
hoos...@mozilla.com
Janet Swisher
Jan 27, 2015, 5:22:40 PM1/27/15
to mdn-d...@lists.mozilla.org
I'd like to take a step back and make sure we're clear on what we mean
when we say "quality". That can be an elusive concept [1], and
especially so if we're trying to aggregate a definition across content,
community, technology, and programs.
If this has already been discussed and settled, please point me to where
that was captured, and accept my apologies for being behind the discussion.
We have mission and vision statements on our wiki page [2], but those
are difficult to translate directly into metrics. In 2014, we had KPI
targets related to the number of developers using web technologies, and
found that (a) these KPIs weren't moving in a desirable direction, and
(b) measuring our impact on those metrics was impossible.
My intuitive take on what "success" looks like for MDN is something like
this:
Developers are using MDN (in the comprehensive sense that covers the
website and other programs) to improve their mastery of web
technologies, in order to accomplish their goals.
Then the question becomes: How can we tell if this is happening? That's
where we can start to get into things like inbound links, visitors, or
social media mentions.
One can analyze quality as inherent properties of things, which can be a
useful, if incomplete, picture. For the content side of things, one
highly-regarded book on documentation quality [3] breaks it down as:
* Easy to use: user-centered, accurate, complete
* Easy to understand: clear, concrete, well-styled (in the editing sense)
* Easy to find: well-organized, retrievable, visually effective
Since MDN is a website, we might add:
* Easy to access: highly available, fast loading, usable, accessible to
a wide variety of users and devices
Similar inherent aspects of "quality" can undoubtedly be identified for
other programs such as the planned developer event.
However, all those elements are moot if nobody is using our resources or
if users are not getting value from them. Which leads me back to the
success statement above, which puts it in terms of value to users:
Whatever their level of knowledge and skill about web technologies, MDN
is helping users level-up, and empowering them to more effectively
achieve their (development-related) goals.
Does that seem like a reasonable foundation to build on?
[1] http://en.wikipedia.org/wiki/Pirsig%27s_metaphysics_of_Quality
[2] https://wiki.mozilla.org/MDN/
[3]
http://www.ibmpressbooks.com/store/developing-quality-technical-information-a-handbook-9780131477490
> _______________________________________________
> Mdn-drivers mailing list
> Mdn-d...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/mdn-drivers
--
Janet Swisher <mailto:jREMOVE...@mozilla.com>
Mozilla Developer Network <https://developer.mozilla.org>
MDN Community Manager
when we say "quality". That can be an elusive concept [1], and
especially so if we're trying to aggregate a definition across content,
community, technology, and programs.
If this has already been discussed and settled, please point me to where
that was captured, and accept my apologies for being behind the discussion.
We have mission and vision statements on our wiki page [2], but those
are difficult to translate directly into metrics. In 2014, we had KPI
targets related to the number of developers using web technologies, and
found that (a) these KPIs weren't moving in a desirable direction, and
(b) measuring our impact on those metrics was impossible.
My intuitive take on what "success" looks like for MDN is something like
this:
Developers are using MDN (in the comprehensive sense that covers the
website and other programs) to improve their mastery of web
technologies, in order to accomplish their goals.
Then the question becomes: How can we tell if this is happening? That's
where we can start to get into things like inbound links, visitors, or
social media mentions.
One can analyze quality as inherent properties of things, which can be a
useful, if incomplete, picture. For the content side of things, one
highly-regarded book on documentation quality [3] breaks it down as:
* Easy to use: user-centered, accurate, complete
* Easy to understand: clear, concrete, well-styled (in the editing sense)
* Easy to find: well-organized, retrievable, visually effective
Since MDN is a website, we might add:
* Easy to access: highly available, fast loading, usable, accessible to
a wide variety of users and devices
Similar inherent aspects of "quality" can undoubtedly be identified for
other programs such as the planned developer event.
However, all those elements are moot if nobody is using our resources or
if users are not getting value from them. Which leads me back to the
success statement above, which puts it in terms of value to users:
Whatever their level of knowledge and skill about web technologies, MDN
is helping users level-up, and empowering them to more effectively
achieve their (development-related) goals.
Does that seem like a reasonable foundation to build on?
[1] http://en.wikipedia.org/wiki/Pirsig%27s_metaphysics_of_Quality
[2] https://wiki.mozilla.org/MDN/
[3]
http://www.ibmpressbooks.com/store/developing-quality-technical-information-a-handbook-9780131477490
> Mdn-drivers mailing list
> Mdn-d...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/mdn-drivers
--
Janet Swisher <mailto:jREMOVE...@mozilla.com>
Mozilla Developer Network <https://developer.mozilla.org>
MDN Community Manager
Luke Crouch
Jan 27, 2015, 6:48:24 PM1/27/15
to Janet Swisher, mdn-d...@lists.mozilla.org
+1 to this, though can we start a little above the metaphysical level? ;)
Everything you mention is an attribute of "quality" for MDN.
* Accuracy
* Completeness
* Clarity
* Concreteness
* Style
* Organization
* Retrieveability (can I call it Accessibility?)
We should come up with metrics for these. I propose that most (or all) of
the engineering/operational metrics about response time, error rates,
uptime, etc. are actually part of the "Accessibility" attribute.
I further agree the quality of MDN serves a purpose - i.e., empowering web
developers to make more web. And I would love to have (invent as necessary)
metrics for that as well. Based on my experience that web developers *like*
to *share* knowledge they find empowering, I propose:
* Voters, votes, likes, dislikes, helpfulness, etc.
* Social shares & invites (especially *accepted* invites)
* Back-links
As attributes of an "are we fulfilling our purpose" metric.
-L
Everything you mention is an attribute of "quality" for MDN.
* Accuracy
* Completeness
* Clarity
* Concreteness
* Style
* Organization
* Retrieveability (can I call it Accessibility?)
We should come up with metrics for these. I propose that most (or all) of
the engineering/operational metrics about response time, error rates,
uptime, etc. are actually part of the "Accessibility" attribute.
I further agree the quality of MDN serves a purpose - i.e., empowering web
developers to make more web. And I would love to have (invent as necessary)
metrics for that as well. Based on my experience that web developers *like*
to *share* knowledge they find empowering, I propose:
* Voters, votes, likes, dislikes, helpfulness, etc.
* Social shares & invites (especially *accepted* invites)
* Back-links
As attributes of an "are we fulfilling our purpose" metric.
-L
Ali Spivak
Jan 28, 2015, 3:18:11 AM1/28/15
to Luke Crouch, mdn-drivers, Janet Swisher
I love all of these, however they would take some "development" as we don't
have the ability to track them at the moment.
ali spivak
Manager, MDN Community & Content
asp...@mozilla.com
have the ability to track them at the moment.
Manager, MDN Community & Content
asp...@mozilla.com
Jean-Yves Perrier
Jan 28, 2015, 3:30:56 AM1/28/15
to mdn-d...@lists.mozilla.org
Individually, all these things are valuable and welcome. Combined in one
single number, they becomes useless.
So yes to measure these things, but as Ali said, it will be a
significant amount of dev work and I thought we were on shortage here
because of the MDN Services projects.
Also the MDN Content Team is very busy in Q1 and likely in Q2, so don't
expect us to be much more active than 1 mail/week on these things.
I think the first thing to do is to have a system to display metrics and
to start with the easy one: availability of our services. I guess
nothing more can be done in Q1.
--
Jean-Yves
Jean-Yves Perrier
Senior Technical Writer / Mozilla Developer Network
single number, they becomes useless.
So yes to measure these things, but as Ali said, it will be a
significant amount of dev work and I thought we were on shortage here
because of the MDN Services projects.
Also the MDN Content Team is very busy in Q1 and likely in Q2, so don't
expect us to be much more active than 1 mail/week on these things.
I think the first thing to do is to have a system to display metrics and
to start with the easy one: availability of our services. I guess
nothing more can be done in Q1.
--
Jean-Yves
Senior Technical Writer / Mozilla Developer Network
Janet Swisher
Jan 29, 2015, 4:14:32 PM1/29/15
to Luke Crouch, mdn-d...@lists.mozilla.org
I don't think you can start above the metaphysical level, by definition :-)
A further extension of your point is that "quality" and "success" are
two different things (metaphysically speaking). "Quality" inherently
depends on the type of thing you're discussing the quality of. This is
what makes it difficult to abstract "quality" across many things and
activities.
Success is independent of quality; achieving quality in what we do is
probably necessary, but also probably not sufficient, for achieving
success.
If we're looking for One Metric to rule them all, it should probably be
a success metric. The metrics we are able to track will likely be
approximations of the Platonic ideal (sorry) of a success metric for MDN.
We can also define quality metrics, but implementing ways to track them
may be a longer-term project requiring development effort, which would
need to be evaluated against other priorities. Even something like a
"helpfulness" rating is really about success, not quality, and I'd
rather have that than a way to rate articles on accuracy, completeness, etc.
> hoos...@mozilla.com <mailto:hoos...@mozilla.com>
A further extension of your point is that "quality" and "success" are
two different things (metaphysically speaking). "Quality" inherently
depends on the type of thing you're discussing the quality of. This is
what makes it difficult to abstract "quality" across many things and
activities.
Success is independent of quality; achieving quality in what we do is
probably necessary, but also probably not sufficient, for achieving
success.
If we're looking for One Metric to rule them all, it should probably be
a success metric. The metrics we are able to track will likely be
approximations of the Platonic ideal (sorry) of a success metric for MDN.
We can also define quality metrics, but implementing ways to track them
may be a longer-term project requiring development effort, which would
need to be evaluated against other priorities. Even something like a
"helpfulness" rating is really about success, not quality, and I'd
rather have that than a way to rate articles on accuracy, completeness, etc.
> _______________________________________________
> Mdn-drivers mailing list
> Mdn-d...@lists.mozilla.org
> <mailto:Mdn-d...@lists.mozilla.org>
> Mdn-drivers mailing list
> Mdn-d...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/mdn-drivers
>
>
> --
> Janet Swisher <mailto:jREMOVE...@mozilla.com
> <mailto:jREMOVE...@mozilla.com>>
> Mozilla Developer Network <https://developer.mozilla.org>
> MDN Community Manager
>
> _______________________________________________
> Mdn-drivers mailing list
> Mdn-d...@lists.mozilla.org <mailto:Mdn-d...@lists.mozilla.org>
>
>
> --
> Janet Swisher <mailto:jREMOVE...@mozilla.com
> <mailto:jREMOVE...@mozilla.com>>
> Mozilla Developer Network <https://developer.mozilla.org>
> MDN Community Manager
>
> _______________________________________________
> Mdn-drivers mailing list
0 new messages