Success Metrics for Online Tools for Govt. Transparency
Jennifer Bell
An increasingly popular requirement of granting foundations is that
they must be able to measure the impact of the dollars they give out.
I've been thinking about how the success of online tools for
government transparency could be measured for impact, in a generic
way.
Some ideas for transparency website metrics might include:
- basic website tracking: unique visitors, return visits, visit
length, site referrals
- subscribers to RSS feeds
- if site has embeddable widgets, like a chart that could be embedded
in a blog, the number of times that widget was embedded / viewed
- number of mainstream media news stories produced using results of
tools (if this were trackable in some way)
For sites that provide data to other sites through an API, things are
a little harder to quantify. One approach would be to report the sum
of the impact of the downstream websites, though this might be
misleading.
I would be interested in hearing other ideas or approaches.
Jennifer
Rob Collins
goals. Visits, eyeballs, etc. were discredited in the 90s. If you're going
to measure effectiveness it must be related to the goals - how many people
have actively participated, how much has government changed, how many times
is your effort mentioned in the press, etc. The fact that people visit the
site doesn't indicate any value. The fact that behaviour changed because
they visited the site is what matters.
Rob
Joe Murray
actions, etc. are good. For education oriented campaigns, visit length and
page reads on particular content pages can be useful metrics. For campaigns
that are attempting to effect a policy change by a government or
corporation, the ultimate metric should be whether that goal was achieved.
But the trouble is determining what impact the campaign may or may not have
had on a decision - often that's not easy to determine. So the intermediate,
internal campaign activity metrics become a proxy for measuring the pressure
the campaign has applied. Usually it's a good idea to have metrics for each
step on the ladder of engagement, recognizing that a campaign usually needs
a wide funnel - the long tail - even if most campaign stuff is done by a
small core.
Joe Murray, PhD
President, JMA Consulting
@ The Centre for Social Innovation
215 Spadina Ave, Suite 400, Toronto, Ontario, Canada M5T 2C7
416.466.1281, (416) 644-0116 (f)
joe.m...@jmaconsulting.biz
Skype: josephpmurray
Twitter: JoeMurray
Joe
> -----Original Message-----
> From: visiblegover...@googlegroups.com
> [mailto:visiblegover...@googlegroups.com] On Behalf Of Rob
> Collins
> Sent: January 29, 2009 8:33 AM
> To: visiblegover...@googlegroups.com
> Subject: [VG-Discuss] Re: Success Metrics for Online Tools for Govt.
> Transparency
>
>
Majd Al-shihabi
I'm new here - i though i should contribute :)
here's an interesting blog post by Andrew Chin titled "What matters more? Engagement or pageviews?"
http://andrewchenblog.com/2007/02/20/what-matters-more-engagement-or-pageviews/
Andrew Chin has a lot of content about measuring and evaluating the value of a website/business... look through his blog.
Cheers,
Majd
> Subject: [VG-Discuss] Re: Success Metrics for Online Tools for Govt.
> Transparency
>
>
--
Majd Al-Shihabi
2A Systems Design
VP Internal - Engineers Without Borders, Stream B
email: mals...@engmail.uwaterloo.ca
web: http://twitter.com/majdal
Caitlin Bentley
Could you tell us a bit about who your primary funders are?
Yes I agree that evaluation follows the goals of the project, but acceptable types of measures in the NGO world depend more heavily on the norms of who's giving you the money unfortunately.
C
Jennifer Bell
Thanks Caitlin,
Actually, no grants have been recieved yet, so my question is not
really funder driven. I guess I'm trying to imagine what ought to
exist, intrinsically.
There's actually a spectacular knowledge gap around measuring the
effectiveness of technology in the non-profit sector, as I've seen it
so far. This fall, I was on the phone with a non-profit that received
a very large grant to build a website, and I asked the person managing
the website how many people were following their RSS feeds. The
person on the phone said 'Oh, we don't know that.' The contractor who
had built the website did not think to put in a feedburner or other
subscriber counter, and there was no one in the office who had any
idea how to do that.
True story! Neither the people giving the money, nor the people
spending the money, had the sense to put in measurement metrics to
track how many people were using what they had built. I was appalled.
*ANYWAY* Rob, I hate to disagree with you, but I think pageviews are
still a useful measure. Pageviews can say how effective your
marketing is, as the very start of the funnel. (Are people getting to
the site in the first place?)
I liked what Joe said about metrics at each level of progression
during a campaign. Interesting!
Jennifer
On Jan 29, 12:09 pm, Caitlin Bentley <caitlin.bent...@gmail.com>
wrote:
>
> Could you tell us a bit about who your primary funders are?
>
> Yes I agree that evaluation follows the goals of the project, but acceptable
> types of measures in the NGO world depend more heavily on the norms of who's
> giving you the money unfortunately.
>
> C
>
> > Hello,
>
> > I'm new here - i though i should contribute :)
>
> > here's an interesting blog post by Andrew Chin titled "What matters more?
> > Engagement or pageviews?"
>
> > Andrew Chin has a lot of content about measuring and evaluating the value
> > of a website/business... look through his blog.
>
> > Cheers,
>
> > Majd
>
>
> >> It's important to measure campaign effectiveness...contributions, online
> >> actions, etc. are good. For education oriented campaigns, visit length and
> >> page reads on particular content pages can be useful metrics. For
> >> campaigns
> >> that are attempting to effect a policy change by a government or
> >> corporation, the ultimate metric should be whether that goal was achieved.
> >> But the trouble is determining what impact the campaign may or may not
> >> have
> >> had on a decision - often that's not easy to determine. So the
> >> intermediate,
> >> internal campaign activity metrics become a proxy for measuring the
> >> pressure
> >> the campaign has applied. Usually it's a good idea to have metrics for
> >> each
> >> step on the ladder of engagement, recognizing that a campaign usually
> >> needs
> >> a wide funnel - the long tail - even if most campaign stuff is done by a
> >> small core.
>
> >> Joe Murray, PhD
> >> President, JMA Consulting
> >> @ The Centre for Social Innovation
> >> 215 Spadina Ave, Suite 400, Toronto, Ontario, Canada M5T 2C7
> >> 416.466.1281, (416) 644-0116 (f)
> > web: http://twitter.com/majdal
robco...@cyberus.ca
If you want to find somebody who does this well, try Care Canada.
They are very good at measuring the effectiveness of what they do and
have to do so to keep dollars flowing. They aren't focused on
websites though they do use them. (Sorry, the folks I knew there
aren't there any more.)
Your point of measuring the top of the funnel is valid but only if you
measure the whole funnel. If nothing comes out the bottom, who cares
what went in the top? You must measure an end outcome or you will
never know if you've achieved anything. A million hits and nobody
pushing for more open government cannot be considered success.
Rob
Caitlin Bentley
step on the ladder of engagement" that Joe brought up. I think that's spot on. CIDA's public engagement strategy has this diagram in it which is nice.. It shows different levels of engagement quite clearly, it might be a good resource for you. It's the 'continuum' slide in this doc: http://www.acdi-cida.gc.ca/INET/IMAGES.NSF/vLUImages/Public_Engagement/$file/EngagementEng.pdf
I work in the development industry, and we use results-based management which is unfortunate in some (if not most) cases, but it's the standard in our field, so you might want to check it out. For example, it would be helpful to look more closely at each of your projects individually and examine the expected outcomes respectively. Set up some constructs that describe different categories of participation by different stakeholders and the impact that the project has both on an individual level and on a larger impact level. Once you do that, it really becomes clear what kind of data might be able to answer these questions for you (whether it's going to be possible or not to collect). If you need more info or some examples of RBM let me know.
Last but not least, if you are looking for funders and you don't feel like quantitative measures adequately demonstrate or measure the impact that your tools are having, write up a couple of short case studies, or if you're not there yet, write a story, and use some of your stats and perhaps some primary literature to infer that probabilistically, some portion of your visitors are reacting in such and such ways.
C
Christine Prefontaine
Caitlin Bentley
RBM... scary.... sends shivers down my spine.
LOL, yes I certainly agree..
Claire
I could not agree more about some kind of behaviour change being the
best indicator.
I wanted to stress this rule also applies to media hits. In PR we now
are disregarding the number of times per say some organization is
mentioned in the media as a measure of success. I won't go into
details, let's just say it's a waste of resources if it does not serve
a purpose and one can have a lot of media hits while having a very
limited impact on public opinion or behavior.
For instance, right now we want to increase our newsletter
subscriptions in order to cultivate our relationship with the
subscriber and turn him/her into a donor. We rather do fewer media
apperances but ones that do generate subscribers, because all media
hits are not equal in that regard. For instance, radio seems to work
better for us. Unfortunately a lot of board members do use raw media
hit as a measure of success, and that is a big mistake.
If my board was to give me a media hit objective as an executive
director, I would then go for media coverage when it is easier to get,
in the summer and during Christmas time. The thing is much fewer
people are listening and watching at these times of year. So media hit
numbers can be easily inflated but would have had a very limited
impact. And it's not because you were in the paper commenting a news
story that people do agree with you, or that you made them change
their mind. A good chunk of the readers would not even read the piece
if they're not interested in the subject matter, or stop reading after
the first paragraph. So what's worth being in the newspaper then?
Also, I routinely meet people who tell me they saw me on TV but can't
remember what I was talking about when I ask them.
Sorry for going on and on, I just want to stress this because it's a
very big problem in non profits (board members insisting on using raw
media hits to measure performance). Donors are impressed with these
numbers to, so I know organizations end up being in the media for the
sake of being in the media because it makes donors and board members
happy. I know organizations who do that at the expense of their
mission, they have a lot of media hits but they actually have a very
limited impact in the real word.
Same things with page views. It's easy to generate meaningless
traffic, whether unintentionally or not. For example, we found out 25
% of our web site traffic comes from people typing "municipal taxes"
in search engines. But the page they visit does not contain the very
precise information they are looking for and we know they leave
immediately without visiting any other page. We tried to retain this
accidental traffic but our efforts have failed. So all of this traffic
is dead wood. Yet again, it's not uncommon for board members to want
to use simple web site traffic volume as an indicator of success. To
some extent I would say time spent is tricky too.
Laura
I think results-based management *does* provide an excellent framework
for developing success indicators for your web site that reflect your
long term 'real world' goals.
Jennifer, if you want to develop an evaluation framework for visible
government (the site) it has to be in terms of the long term desired
impact of visible government (the organization). However, that's too
long term to know if your site is contributing to your real world
goals. A logic model (a tool used in RBM) helps you define those
short term (online) and long term (offline) goals so that you can
measure both.
Why measure short term goals? Because you need to make decisions
about how to spend finite resources (time & money) so you only want to
do what is actually contributes to those goals.
I've written about this on my blog with you in mind, because I've seen
the same debate take place many times.
http://usability4government.wordpress.com/2009/01/31/is-results-based-management-practical-for-developing-an-evaluation-framework
(comments welcomed).
Also, in response to the guy that wrote about attribution of your
efforts to the real world results, I'd suggest googling John Mayne and
attribution vs. contribution, or having a look at this article:
http://www.evaluationcanada.ca/distribution/20080514_mayne_john.pdf
Hope this is helpful.
Laura Wesley
lmwe...@hotmail.com
Jennifer Bell
thread is wonderful. You people are amazing.
Jennifer
On Jan 31, 2:03 pm, Laura <lmwes...@hotmail.com> wrote:
> I have to agree with Claire, the same is true in government. However,
> I think results-based management *does* provide an excellent framework
> for developing success indicators for your web site that reflect your
> long term 'real world' goals.
>
> Jennifer, if you want to develop an evaluation framework for visible
> government (the site) it has to be in terms of the long term desired
> impact of visible government (the organization). However, that's too
> long term to know if your site is contributing to your real world
> goals. A logic model (a tool used in RBM) helps you define those
> short term (online) and long term (offline) goals so that you can
> measure both.
>
> Why measure short term goals? Because you need to make decisions
> about how to spend finite resources (time & money) so you only want to
> do what is actually contributes to those goals.
>
> I've written about this on my blog with you in mind, because I've seen
>
> Also, in response to the guy that wrote about attribution of your
> efforts to the real world results, I'd suggest googling John Mayne and
> attribution vs. contribution, or having a look at this article:http://www.evaluationcanada.ca/distribution/20080514_mayne_john.pdf
>
> Hope this is helpful.
> Laura Wesley
Marc-Antoine Parent
Sorry to chime in so late. I realized I never sent you that article,
which I believe to be relevant to your discussion:
http://www.mande.co.uk/docs/MSCGuide.htm
You know I am partial to stories (aka qualitative data!) as a tool for
understanding; I am personally a bit suspicious of methods that rely
on quantifying impact such as RBM, as they are limited to situations
where it is possible to trace a single line of causality... I
understand why donors want such a solid evaluation, but I believe they
have to be taught to look at weaker signals for social change.
Cheers,
Marc-Antoine
Laura
people begin to focus their attention on program impact".
Have to agree with you on the value of qualitative information Marc-
Antoine. RBM encourages a mix of quantitative and qualitative data.
It's just a framework to describe the link between the results of your
ongoing activities to your 'most significant change'. It's too bad
that it's gotten such a bad rap...I blame it on government! ;)
I'll leave you with this oft-used phrase: "What gets measured tends to
get improved." It's worth spending the time deciding what that should
be.
Marc-Antoine Parent
>
> Great article! I'm in favour of anything that gets "whole teams of
> people begin to focus their attention on program impact".
>
> Have to agree with you on the value of qualitative information Marc-
> Antoine. RBM encourages a mix of quantitative and qualitative data.
> It's just a framework to describe the link between the results of your
> ongoing activities to your 'most significant change'. It's too bad
> that it's gotten such a bad rap...I blame it on government! ;)
>
> I'll leave you with this oft-used phrase: "What gets measured tends to
> get improved." It's worth spending the time deciding what that should
> be.
Hello, Laura!
Heh, a long discussion... I am not even sure we disagree, since the
devil's in the details, though we seem to come at this grey area from
either end. And all that follows with the caveat that I am not a
measurement specialist, unlike you! But I am trying to approach this
from a systems perspective.
I will certainly agree with you that measurement is most useful to
improve processes; I was trying to say that real positive impacts of
processes cannot always be measured, beyond a point. I may be
splitting hair, here; I am contrasting "measured" and "detected"...
Most goals are influenced by so many factors, that it is almost
impossible (or absurdly costly) to determine the exact contribution of
each factor. However, as a donor, one would legitimately want to know
that a given factor that you sponsored was at least part of the
equation, though I could not know how much! Hence the value of
detection of weaker signals, as in the form of stories... with the
added uncertainty they entail.
I confess not having first-hand experience with RBM; but my wife had
to use it to justify activities of a small NGO to CIDA, and I
certainly got the strong impression, looking at RBM manuals and the
reports she was producing, that she was expected to quantify
contributions of inputs to outputs, not only "describe" the links, and
this I believe to be pie-in-the sky for many applications. (Though of
course it is appropriate in some cases!)
And it was indeed consuming a very high percentage of her time on a
project where she was otherwise the main producer of value. As I said
above, measurement turned out to be very costly, overall. Scale was a
huge factor, here. RBM requires each team to also be a measurement
researcher; for very small teams (approaching one person!), the
overhead is overwhelming. That is also true for qualitative work,
incidentally; but it is worse with quantitative requirements, which
are always felt to require more evidence. That said, some of this may
be, as you said, specific uses of RBM vs its theory, which did seem to
have more leeway. But another thing I believe in is to always look at
methods and tools the way they tend to get used, not the way they are
meant to be used by their designers, or specialists. (Tearing my hair
at my own tool design right now!) The tools impose a surprising bias,
that few practitioners overcome, and it is often sobering to measure
their real-life cost.
That said, I have only one data point; in this case, it could be
practical and useful to measure quantitatively the total cost of
adopting RBM to our economy. (Has this been done?) But, of course,
that needs to be contrasted with the (much more difficult to quantify)
benefits that this measurement did (undoubtedly) yield on processes.
Nothing is ever simple, is it?
So to summarize: Yes, I believe in evaluation. Some kind of control
should be applied, even though it has a cost as argued above. The
alternative is blindness. But evidence (esp. quantitative) can be
tenuous and/or costly to collect; anything that can lighten
measurement requirements allows more people to act. My extremely
limited experience of RBM, and what I heard about it from other
practitioners, is that it has, in practice, encouraged quantitative
requirements. This is useful, even required, above a certain cost/
scale; but I doubt it scales down easily.
That said... nice discussion, but in the precise case of web sites
such as visible government, a lot of quantitative data (clicks) can be
obtained easily, and should. More than that, citizen intervention can
be tracked; Jennifer, I hope that you give users of your site a way to
indicate if they wrote to their MP about what they saw; even better,
allow them to send the letter through the site. Though... there might
also be political reasons not to do so! If not, a Bcc address would be
a good way to be in the loop. But I think you should go for it.
In the long run, changes in expense patterns could also be tracked and
quantified. But how much of those hoped-for changes could be
attributed to the work of visiblegovernment? Here, quantitative
evaluation will draw a blank, and it is important to acknowledge that
evidence, if any, would be anecdotal. Having a means to capture
anecdotes is thus vital, because it is all you could have.
Cheers,
Marc-Antoine