Making Ratings Better

[Davey Shafik]

A common discussion I have with other speakers about Joind.in is about the ratings system.

Most people have  a system for how they rate, but obviously no two peoples systems are the same — by it's very nature,

it's a very subjective system. Numbers kinda suck for this purpose — giving some words to accompany them would be good, e.g.

1 - Content was poor

2 - Content was OK

3 - Content was Good, Speaker was Good

4 - Content was Great, Speaker was Good

5 - Content was Great, Speaker was Great

I purposely don't rate the speaker at the lower values, and you can change Speaker to Delivery, to take it away from being Personal.

**But this isn't what this email is about!**

What I do want to talk about is this: https://joind.in/talk/view/8703

I gave this talk at Lonestar last month and got great comments back, with great ratings (4 5 stars, and 1 4 star) — 

but when the talk comes up in a listing, it has zero rating: because they were all anonymous.

I understand why we don't count anonymous ratings, because it's too easy to post an anonymous negative 

rating and impact the score of someones talk, but I'd argue that in this case it has the same effect.

In this case, the audience for the talk was completely beginner — it was likely their first conference,

and they may not have twitter or facebook accounts to do social auth, and decided that since anonymous 

was allowed, why bother?

I'd like to propose making two changes:

1) Count ratings with a comment more than N words (5? 7? 10?) *regardless* of the rating — if someone rates me 1, and

bothers to put a comment also, that's fine to count it negatively. It's only the anonymous, no-comment ones that are BS.

2) *Inform* the user that their ratings have no effect on the score [and assuming the above: if they make no comment] when

they are posting anonymously.

There are also a couple of alternatives:

3) Count anonymous ratings where there are no non-anonymous ones. Obviously indicate this is the case.

4) Somehow extend the ratings widget to include anonymous ratings separately and be clear about what's what. I've no idea how this could be done.

I'm happy to put in the code to do #1 [and possibly #2].

Any thoughts?

- Davey

[Joshua Thijssen]

Davey Shafik wrote:
> A common discussion I have with other speakers about Joind.in is about
> the ratings system.
>
> Most people have a system for how they rate, but obviously no two

> peoples systems are the same � by it's very nature,
> it's a very subjective system. Numbers kinda suck for this purpose �

> giving some words to accompany them would be good, e.g.
>
> 1 - Content was poor
> 2 - Content was OK
> 3 - Content was Good, Speaker was Good
> 4 - Content was Great, Speaker was Good
> 5 - Content was Great, Speaker was Great
>
> I purposely don't rate the speaker at the lower values, and you can
> change Speaker to Delivery, to take it away from being Personal.
> **But this isn't what this email is about!**
>
> What I do want to talk about is this: https://joind.in/talk/view/8703
>
> I gave this talk at Lonestar last month and got great comments back,

> with great ratings (4 5 stars, and 1 4 star) �

> but when the talk comes up in a listing, it has zero rating: because
> they were all anonymous.

This is why zero rated talks don't show any stars. There isn't a "zero
rating" talk possible that way just for this reason. I have many
workshops / talks at non-joindin conferences where people don't comment,
and this way they don't look "bad" (even though we can't be sure that
they were/werent). Anonymous comments are there for a reason: to give
you (constructive) feedback, especially when you know the speaker
personally. It saves relationships :)

> 1) Count ratings with a comment more than N words (5? 7? 10?)

> *regardless* of the rating � if someone rates me 1, and

> bothers to put a comment also, that's fine to count it negatively. It's
> only the anonymous, no-comment ones that are BS.

I've seen lots of 5-star ratings with the comments "Excellent", "best
talk ever" etc.. They should count as well.. Otherwise near-perfect
talks gets a 3 star rating just because somebody selected 3 stars and
decided to type more than 10 words. I guess it's hard to find a good
balance in this. And personally, i like just not-counting the anonymous
talks..

> 2) *Inform* the user that their ratings have no effect on the score [and
> assuming the above: if they make no comment] when
> they are posting anonymously.

I think this is already the case, but i'm not 100% sure.


Personally, i don't like the star-rating. Some countries give 3-4 star
ratings on good talks, while the same talk gets scored 5-star in other
countries. It's very hard to tell how good a talk is just by looking at
the scores. Either have a 10-star rating, or a textual like you
proposed. Maybe even score the two things separately: talk and
presenter. A presenter can have a bad day (social-party hangover), while
the talk itself is perfect (or vice versa).. I think this might be
worth exploring?



regards,
Josh

[Casey Wilson]

Old email? Sure is. I'm that slow.

I'd propose the simplest approach to solving this is by only allowing people to rate if they are signed in :)

If it's worth their time, than they'd sign in. I'd rather be rated by people who found me worth signing in for than those who don't.

Senks :)

-C

On Wed, Jul 17, 2013 at 3:35 AM, Joshua Thijssen <joshua....@gmail.com> wrote:

Davey Shafik wrote:
> A common discussion I have with other speakers about Joind.in is about
> the ratings system.
>
> Most people have  a system for how they rate, but obviously no two

> peoples systems are the same — by it's very nature,
> it's a very subjective system. Numbers kinda suck for this purpose —

> giving some words to accompany them would be good, e.g.
>
> 1 - Content was poor
> 2 - Content was OK
> 3 - Content was Good, Speaker was Good
> 4 - Content was Great, Speaker was Good
> 5 - Content was Great, Speaker was Great
>
> I purposely don't rate the speaker at the lower values, and you can
> change Speaker to Delivery, to take it away from being Personal.
> **But this isn't what this email is about!**
>
> What I do want to talk about is this: https://joind.in/talk/view/8703
>
> I gave this talk at Lonestar last month and got great comments back,

> with great ratings (4 5 stars, and 1 4 star) —

> but when the talk comes up in a listing, it has zero rating: because
> they were all anonymous.

This is why zero rated talks don't show any stars. There isn't a "zero
rating" talk possible that way just for this reason. I have many
workshops / talks at non-joindin conferences where people don't comment,
and this way they don't look "bad" (even though we can't be sure that
they were/werent). Anonymous comments are there for a reason: to give
you (constructive) feedback, especially when you know the speaker
personally. It saves relationships :)

> 1) Count ratings with a comment more than N words (5? 7? 10?)

> *regardless* of the rating — if someone rates me 1, and

> bothers to put a comment also, that's fine to count it negatively. It's
> only the anonymous, no-comment ones that are BS.

I've seen lots of 5-star ratings with the comments "Excellent", "best
talk ever" etc.. They should count as well.. Otherwise near-perfect
talks gets a 3 star rating just because somebody selected 3 stars and
decided to type more than 10 words. I guess it's hard to find a good
balance in this. And personally, i like just not-counting the anonymous
talks..

> 2) *Inform* the user that their ratings have no effect on the score [and
> assuming the above: if they make no comment] when
> they are posting anonymously.

I think this is already the case, but i'm not 100% sure.


Personally, i don't like the star-rating. Some countries give 3-4 star
ratings on good talks, while the same talk gets scored 5-star in other
countries. It's very hard to tell how good a talk is just by looking at
the scores. Either have a 10-star rating, or a textual like you
proposed. Maybe even score the two things separately:  talk and
presenter. A presenter can have a bad day (social-party hangover), while
the talk itself is perfect (or vice versa)..  I think this might be
worth exploring?



regards,
Josh

--
You received this message because you are subscribed to the Google Groups "joindin-developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to joindin-develop...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Casey Wilson
Freelance Programmer
http://www.ap0s7le.com/

[lornajane]

This is such an interesting topic :)

I'm reluctant to ban anonymous ratings for the reasons that Josh cited - sometimes something needs to be said, but in a way that doesn't either damage the speaker's credibility, or lead to them hunting you down.  If you're not going to put your name to your rating, then your vote doesn't count.  That said, the v2 API does not currently support anonymous commenting, so neither does/will the new mobile apps or website until we make a decision on that.

Davey touched on something that I'm really keen on, but which has got kinda stuck while we make progress on getting the basics in place on the new site - and that is to *inform* users.  If you aren't signed in, your vote won't count is good information.  But so is "this talk was rated 4.2 at an event where the average talk rating was 2.9".  There's a definite cultural difference where US audiences will give out 5s pretty much every session, whereas the European audiences absolutely don't feel that's expected.  It's fine, apart from when you're submitting to a conference with speakers who you know will have a long series of perfect 5s on their joind.in page!!  Maybe we should normalise the scores? :)

There will always be people who for some reason come up with an ethical objection to being signed in to our system.  The anonymous commenting is there in order that we don't lose their feedback completely, but I think with a change in user flow on the newer versions of joind.in, we'll see less of this anyway.

Lorna

To unsubscribe from this group and stop receiving emails from it, send an email to joindin-developers+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

[Stefan Koopmanschap]

I was talking to conference organizers  this week that are considering adopting joind.in, and it wasn't until I mentioned the anonymous function that they felt it was a serious option. Perhaps we should add an optional nickname field for anonymous comments? So people who just don't want to sign in to another service can still be identified if they want?

Stefan

Sent from my iPhone

To unsubscribe from this group and stop receiving emails from it, send an email to joindin-develop...@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
Casey Wilson
Freelance Programmer
http://www.ap0s7le.com/

--

You received this message because you are subscribed to the Google Groups "joindin-developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to joindin-develop...@googlegroups.com.