Fascinating stuff about the real world of ratings

[Chris Messina]

YouTube suggests that its 5-star ratings are useless:

http://youtube-global.blogspot.com/2009/09/five-stars-dominate-ratings.html

http://www.techcrunch.com/2009/09/22/youtube-comes-to-a-5-star-realization-its-ratings-are-useless/

I wonder how this might inform (if at all) our thinking on the rating element in activity streams/hreview?

Chris

--
Chris Messina
Open Web Advocate

Personal: http://factoryjoe.com
Follow me on Twitter: http://twitter.com/chrismessina

Citizen Agency: http://citizenagency.com
Diso Project: http://diso-project.org
OpenID Foundation: http://openid.net

This email is:   [ ] shareable    [X] ask first   [ ] private

[Rob Dolin]

I wonder if this applies on other services.  I’ve anecdotally found Yelp ratings to have a broader distribution.  FWIW—

--Rob

[Kaliya *]

Chris Allen has several posts dating 4 years back about ratings and rankings that can be helpful for this conversation....

2006-08: Using 5-Star Rating Systems

• 2005-12: Systems for Collective Choice
• 2005-12: Collective Choice: Rating Systems
• 2007-01: Experimenting with Ratings
• 2006-01: Collective Choice: Competitive Ranking Systems

[Martin Atkins]

Chris Messina wrote:
> YouTube suggests that its 5-star ratings are useless:
>
> http://youtube-global.blogspot.com/2009/09/five-stars-dominate-ratings.html
> http://www.techcrunch.com/2009/09/22/youtube-comes-to-a-5-star-realization-its-ratings-are-useless/
>
> I wonder how this might inform (if at all) our thinking on the rating
> element in activity streams/hreview?
>

Unfortunately, as long as services want to expose "gave a five star
rating" as an activity I don't think we can ignore it even if it is
"useless".

[Peter Reiser]

ratings are important as an indicator of the the value of a content
object - so there should be a rating element in activity streams.

But it should not be the only indicator of the value of a content
object. Other activities (view, comment, updates etc.) should also be
part of the value evaluation.
Furthermore there should be an incentive for the user which provides
rating ..

As an example in the Community Equity we honor ratings in two ways

1. the information object (and the author) gets the rating value
associated with the 1-5 star rating (Information Equity) and the
person providing the rating gets Participation equity.
(see http://www.flickr.com/photos/peterreiser/3953062406/ as an
example on how to configure the value assignment )


Peter

On Sep 23, 1:00 am, Chris Messina <chris.mess...@gmail.com> wrote:
> YouTube suggests that its 5-star ratings are useless:
>

> http://youtube-global.blogspot.com/2009/09/five-stars-dominate-rating...http://www.techcrunch.com/2009/09/22/youtube-comes-to-a-5-star-realiz...

[Melvin Carvalho]

On Wed, Sep 23, 2009 at 1:00 AM, Chris Messina <chris....@gmail.com> wrote:
> YouTube suggests that its 5-star ratings are useless:
> http://youtube-global.blogspot.com/2009/09/five-stars-dominate-ratings.html
> http://www.techcrunch.com/2009/09/22/youtube-comes-to-a-5-star-realization-its-ratings-are-useless/

Not the greatest signal, but perhaps "useless" is an overstatement.

Ratings systems are still very much in their infancy, and I suspect
will evolve over the course of at least a decade, so it's good to
include popular metrics, while trying to build in a bit of
flexibility.

[mary hodder]

Wow.. I've been waiting for this for 5 years.

Since then I've explained to product ppl, journalists, business partners, VCs, etc, that rating like this collect crap data

that is unreliable and meaningless in its form. And no one believed me. 

I guess you have to wait a long time for ppl to see it.

Kind of a relief really.

mary

[Alexandre Loureiro Solleiro]

Indeed, that's interesting - ratings can be really "likes" or favorites, which sort of "streamifies"  more important Youtube activity and extends the question to other sites. But likes and favorites aren't reviews, are they?

 

Alexandre Loureiro Solleiro
----
Partner at Quodis
M +351 933 655 676
Av. da Liberdade 229, 2º
1250-142 Lisboa
www.quodis.com

labs.quodis.com

[Phil Wolff]

On Tue, Sep 22, 2009 at 4:27 PM, Martin Atkins <ma...@degeneration.co.uk> wrote:

Unfortunately, as long as services want to expose "gave a five star rating" as an activity I don't think we can ignore it even if it is "useless".

 

Martin brings up an interesting point. This feels like a net neutrality issue.

The point of activity streams is to make transmission and consumption of activity work. While the baseline namespace sets direction, its biggest message should be that its designers value carriage more than judgment as to what's carried.

In this case, that means carrying ratings, in whatever forms publisher choose to capture them. Ratings systems are likely to change under evolutionary pressure, just as Google's "page rank" does, blending changing sources of data and their tuning.

We've seen big changes in things like the attributes facebook maps to relationships, with common definitions of gender (from m/f to m/f/t/declinetostate to more), with twitter changing from "friend" to "follow". Locking in the 2009 definitions will work for a while, but the world will continue to drift. That's why I'm so interested in activitystream extensibility; pliability is a protocols survival trait.

- Phil

[Alex de Carvalho]

It's not really a useless or meaningless activity from a social
perspective. It's just a different, lighter and more fleeting activity
than favoriting something; in most implementations, just rating
something will not place the object in a list, like bookmarking would.
Rating becomes another phatic expression available to the user: a type
of lightweight social gesture that the user may choose to convey to
friends and followers across social networks:
http://skitch.com/alexdc/nnnr5/youtube-sharing

The misconception is that you can collect such social gestures in a
scientific way to create some meaningful data, whereas people will
appropriate your features in ways that please them and in ways that
feel more like play than work. Why should viewing fun videos become
work? If you want meaningful measurable shared intelligence, first
understand the motivations at work and then design and test it
properly. For example, create a leaderboard or stake a user's
reputation score based on the quality of ratings (and here you may run
into cultural, socio-economic or gender-based challenges).

-Alex


2009/9/22 Phil Wolff <pwo...@gmail.com>: