Hybrid Voting System -- Discuss!
2 views
Skip to first unread message
Nick Plante
Jun 8, 2009, 6:09:13 PM6/8/09
to Rails Rumble
To follow up on the previous voting thread, we've been discussing
ideas and playing around with a number of different strategies. Some
have panned out, some have not. This is what I think we've arrived at,
and before making an official post on the subject, we thought that
we'd share it here to get feedback and commentary from the community.
Because you guys are smart, and stuff.
The new strategy, which we're dubbing a hybrid voting model, leverages
both votes from an expert panel as well as public voting. Here's how
it would work:
- The Rails Rumble organizers will select a panel of 'experts' --
people who have experience in startups, community building, design,
and development, perhaps including one or more previous competition
winners. We're hoping to target between 9 and 12 of these folks and
already have a number of interested parties who would be willing to
give us their time.
- After the build portion of the competition wraps, the experts will
be randomly assigned a selection of completed entries to vote on. Each
entry will receive voting marks and comments from only three judges,
in order to keep any one person from having to review too many
entries. Comments will be required, and should note at least one thing
the reviewer really liked or didn't like or one way you could improve
your property, etc.
- Once all entries have been reviewed, organizers will select the top-
rated XX (25? 30?) properties as "best of" entries. These entries will
go on to public voting, whereas the others will be removed from active
competition. The reasoning here is that the general public is
intimated by a huge list of applications; if we can give them a
smaller more concise list of the highest quality apps (as determined
by our 'experts') they're more likely to visit them and actively
participate in ranking and evaluating them. This was suggested by a
couple of the media folks last year. Think American Idol, but without
the music, and less of an emphasis on the guyliner.
- The public, as before, will have a limited amount of time to record
their votes once public voting starts. They may vote on any entry they
wish. Instead of a long, drawn-out voting period though, we'll limit
the voting to just a handful of days (2-3?).
- Final scores will be some combination of expert panel scores and
public scores, with emphasis on public scores. The same anti-cheat
numerical methods that were applied last year will be applied to
combat ballot stuffing attempts, even though we think this revised
system will severely lessen their impact.
If you like the concept, or loathe it, feel free to express yourself
here. I know that we won't be able to come up with one system that
will please everyone, but the goal is to level the playing field as
much as possible, limit ballot stuffing attempts, engage with the
public (your potential userbase), and emphasize feedback and value to
the teams whose properties are being rated.
I think, personally, that this strategy accomplishes that. But we
definitely want to hear what you think!
Thanks --
..nap
ideas and playing around with a number of different strategies. Some
have panned out, some have not. This is what I think we've arrived at,
and before making an official post on the subject, we thought that
we'd share it here to get feedback and commentary from the community.
Because you guys are smart, and stuff.
The new strategy, which we're dubbing a hybrid voting model, leverages
both votes from an expert panel as well as public voting. Here's how
it would work:
- The Rails Rumble organizers will select a panel of 'experts' --
people who have experience in startups, community building, design,
and development, perhaps including one or more previous competition
winners. We're hoping to target between 9 and 12 of these folks and
already have a number of interested parties who would be willing to
give us their time.
- After the build portion of the competition wraps, the experts will
be randomly assigned a selection of completed entries to vote on. Each
entry will receive voting marks and comments from only three judges,
in order to keep any one person from having to review too many
entries. Comments will be required, and should note at least one thing
the reviewer really liked or didn't like or one way you could improve
your property, etc.
- Once all entries have been reviewed, organizers will select the top-
rated XX (25? 30?) properties as "best of" entries. These entries will
go on to public voting, whereas the others will be removed from active
competition. The reasoning here is that the general public is
intimated by a huge list of applications; if we can give them a
smaller more concise list of the highest quality apps (as determined
by our 'experts') they're more likely to visit them and actively
participate in ranking and evaluating them. This was suggested by a
couple of the media folks last year. Think American Idol, but without
the music, and less of an emphasis on the guyliner.
- The public, as before, will have a limited amount of time to record
their votes once public voting starts. They may vote on any entry they
wish. Instead of a long, drawn-out voting period though, we'll limit
the voting to just a handful of days (2-3?).
- Final scores will be some combination of expert panel scores and
public scores, with emphasis on public scores. The same anti-cheat
numerical methods that were applied last year will be applied to
combat ballot stuffing attempts, even though we think this revised
system will severely lessen their impact.
If you like the concept, or loathe it, feel free to express yourself
here. I know that we won't be able to come up with one system that
will please everyone, but the goal is to level the playing field as
much as possible, limit ballot stuffing attempts, engage with the
public (your potential userbase), and emphasize feedback and value to
the teams whose properties are being rated.
I think, personally, that this strategy accomplishes that. But we
definitely want to hear what you think!
Thanks --
..nap
smeade
Jun 8, 2009, 6:38:24 PM6/8/09
to Rails Rumble
I like it! From all angles:
Participant: As a participant it will be great to get constructive
feedback from three industry experts.
Public voter: As a Voter, I'll love that I don't have to wade through
incomplete and junk apps to give constructive ratings.
Spectator: As a spectator, having a 'championship round' type of event
will be fun to watch.
Thanks,
_ Scott
Participant: As a participant it will be great to get constructive
feedback from three industry experts.
Public voter: As a Voter, I'll love that I don't have to wade through
incomplete and junk apps to give constructive ratings.
Spectator: As a spectator, having a 'championship round' type of event
will be fun to watch.
Thanks,
_ Scott
Roland Swingler
Jun 9, 2009, 7:09:33 AM6/9/09
to rails-...@googlegroups.com
Hi
I like the idea of getting "experts" involved - especially if it means
you're guaranteed some comments. I also like the idea of shortening
the voting period - it seemed to drag on for ages last year. I'd be
inclined to have the number of apps available to vote on higher than
25/30 though - maybe the top 50? Single-person teams tend to score
lower, and there were only 3 such teams in the top 25 last year - if
you're still planning on giving a prize for best single person team
then it would be good to have more than 3 in the "final running".
How would you deal with the apps that didn't make the final cut? If
you've spent a lot of effort building something it would be a shame if
it didn't get at least some exposure. It would be good if they still
had a ranking (presumably of the expert scores) - I know I'm looking
to get a higher position this year! Maybe you could have some form of
simple "I like this" button, so the public could still give feedback
on these apps, even if it doesn't count towards winning anything.
Thanks,
Roland
I like the idea of getting "experts" involved - especially if it means
you're guaranteed some comments. I also like the idea of shortening
the voting period - it seemed to drag on for ages last year. I'd be
inclined to have the number of apps available to vote on higher than
25/30 though - maybe the top 50? Single-person teams tend to score
lower, and there were only 3 such teams in the top 25 last year - if
you're still planning on giving a prize for best single person team
then it would be good to have more than 3 in the "final running".
How would you deal with the apps that didn't make the final cut? If
you've spent a lot of effort building something it would be a shame if
it didn't get at least some exposure. It would be good if they still
had a ranking (presumably of the expert scores) - I know I'm looking
to get a higher position this year! Maybe you could have some form of
simple "I like this" button, so the public could still give feedback
on these apps, even if it doesn't count towards winning anything.
Thanks,
Roland
Justin Blake
Jun 9, 2009, 10:02:47 AM6/9/09
to Rails Rumble
I like it too. Everything Nick said sounds like a great way to go.
I also like what Roland said about somehow still giving exposure and
feedback to the apps that don't make the final cut. Not sure the best
way to do that though.
-Justin
On Jun 9, 7:09 am, Roland Swingler <roland.swing...@gmail.com> wrote:
> Hi
>
> I like the idea of getting "experts" involved - especially if it means
> you're guaranteed some comments. I also like the idea of shortening
> the voting period - it seemed to drag on for ages last year. I'd be
> inclined to have the number of apps available to vote on higher than
> 25/30 though - maybe the top 50? Single-person teams tend to score
> lower, and there were only 3 such teams in the top 25 last year - if
> you're still planning on giving a prize for best single person team
> then it would be good to have more than 3 in the "final running".
>
> How would you deal with the apps that didn't make the final cut? If
> you've spent a lot of effort building something it would be a shame if
> it didn't get at least some exposure. It would be good if they still
> had a ranking (presumably of the expert scores) - I know I'm looking
> to get a higher position this year! Maybe you could have some form of
> simple "I like this" button, so the public could still give feedback
> on these apps, even if it doesn't count towards winning anything.
>
> Thanks,
> Roland
>
I also like what Roland said about somehow still giving exposure and
feedback to the apps that don't make the final cut. Not sure the best
way to do that though.
-Justin
On Jun 9, 7:09 am, Roland Swingler <roland.swing...@gmail.com> wrote:
> Hi
>
> I like the idea of getting "experts" involved - especially if it means
> you're guaranteed some comments. I also like the idea of shortening
> the voting period - it seemed to drag on for ages last year. I'd be
> inclined to have the number of apps available to vote on higher than
> 25/30 though - maybe the top 50? Single-person teams tend to score
> lower, and there were only 3 such teams in the top 25 last year - if
> you're still planning on giving a prize for best single person team
> then it would be good to have more than 3 in the "final running".
>
> How would you deal with the apps that didn't make the final cut? If
> you've spent a lot of effort building something it would be a shame if
> it didn't get at least some exposure. It would be good if they still
> had a ranking (presumably of the expert scores) - I know I'm looking
> to get a higher position this year! Maybe you could have some form of
> simple "I like this" button, so the public could still give feedback
> on these apps, even if it doesn't count towards winning anything.
>
> Thanks,
> Roland
>
Nick Plante
Jun 9, 2009, 11:05:01 AM6/9/09
to Rails Rumble
We'd definitely keep the lower tier available for viewing during the
public vote, and encourage visitors to browse through them (perhaps
there would be a link to "see more" or the full list). They just
wouldn't be rankable or show up on the main leaderboard. I would hope
that folks would still visit them and comment on them, and that teams
would visit and comment on each others' work too.
..nap
public vote, and encourage visitors to browse through them (perhaps
there would be a link to "see more" or the full list). They just
wouldn't be rankable or show up on the main leaderboard. I would hope
that folks would still visit them and comment on them, and that teams
would visit and comment on each others' work too.
..nap
> Hi
>
> I like the idea of getting "experts" involved - especially if it means
> you're guaranteed some comments. I also like the idea of shortening
> the voting period - it seemed to drag on for ages last year. I'd be
> inclined to have the number of apps available to vote on higher than
> 25/30 though - maybe the top 50? Single-person teams tend to score
> lower, and there were only 3 such teams in the top 25 last year - if
> you're still planning on giving a prize for best single person team
> then it would be good to have more than 3 in the "final running".
>
> How would you deal with the apps that didn't make the final cut? If
> you've spent a lot of effort building something it would be a shame if
> it didn't get at least some exposure. It would be good if they still
> had a ranking (presumably of the expert scores) - I know I'm looking
> to get a higher position this year! Maybe you could have some form of
> simple "I like this" button, so the public could still give feedback
> on these apps, even if it doesn't count towards winning anything.
>
> Thanks,
> Roland
>
>
> I like the idea of getting "experts" involved - especially if it means
> you're guaranteed some comments. I also like the idea of shortening
> the voting period - it seemed to drag on for ages last year. I'd be
> inclined to have the number of apps available to vote on higher than
> 25/30 though - maybe the top 50? Single-person teams tend to score
> lower, and there were only 3 such teams in the top 25 last year - if
> you're still planning on giving a prize for best single person team
> then it would be good to have more than 3 in the "final running".
>
> How would you deal with the apps that didn't make the final cut? If
> you've spent a lot of effort building something it would be a shame if
> it didn't get at least some exposure. It would be good if they still
> had a ranking (presumably of the expert scores) - I know I'm looking
> to get a higher position this year! Maybe you could have some form of
> simple "I like this" button, so the public could still give feedback
> on these apps, even if it doesn't count towards winning anything.
>
> Thanks,
> Roland
>
Dan Gebhardt
Jun 9, 2009, 12:37:32 PM6/9/09
to Rails Rumble
Nice work! I like everything about this proposal, which is a big
improvement over last year's rather discouraging marathon voting
session. In order to fairly represent single person teams, pick a
minimum number that you'd like to see in the final round (perhaps
five). If that number is not represented in the final 25 or 30, then
add the next highest ranked single person teams until you reach that
number. Personally, I'd rather see 26, 27 or even 32 teams in the
final round than 50.
- Dan
improvement over last year's rather discouraging marathon voting
session. In order to fairly represent single person teams, pick a
minimum number that you'd like to see in the final round (perhaps
five). If that number is not represented in the final 25 or 30, then
add the next highest ranked single person teams until you reach that
number. Personally, I'd rather see 26, 27 or even 32 teams in the
final round than 50.
- Dan
> Hi
>
> I like the idea of getting "experts" involved - especially if it means
> you're guaranteed some comments. I also like the idea of shortening
> the voting period - it seemed to drag on for ages last year. I'd be
> inclined to have the number of apps available to vote on higher than
> 25/30 though - maybe the top 50? Single-person teams tend to score
> lower, and there were only 3 such teams in the top 25 last year - if
> you're still planning on giving a prize for best single person team
> then it would be good to have more than 3 in the "final running".
>
> How would you deal with the apps that didn't make the final cut? If
> you've spent a lot of effort building something it would be a shame if
> it didn't get at least some exposure. It would be good if they still
> had a ranking (presumably of the expert scores) - I know I'm looking
> to get a higher position this year! Maybe you could have some form of
> simple "I like this" button, so the public could still give feedback
> on these apps, even if it doesn't count towards winning anything.
>
> Thanks,
> Roland
>
>
> I like the idea of getting "experts" involved - especially if it means
> you're guaranteed some comments. I also like the idea of shortening
> the voting period - it seemed to drag on for ages last year. I'd be
> inclined to have the number of apps available to vote on higher than
> 25/30 though - maybe the top 50? Single-person teams tend to score
> lower, and there were only 3 such teams in the top 25 last year - if
> you're still planning on giving a prize for best single person team
> then it would be good to have more than 3 in the "final running".
>
> How would you deal with the apps that didn't make the final cut? If
> you've spent a lot of effort building something it would be a shame if
> it didn't get at least some exposure. It would be good if they still
> had a ranking (presumably of the expert scores) - I know I'm looking
> to get a higher position this year! Maybe you could have some form of
> simple "I like this" button, so the public could still give feedback
> on these apps, even if it doesn't count towards winning anything.
>
> Thanks,
> Roland
>
danahern
Jun 9, 2009, 2:48:18 PM6/9/09
to Rails Rumble
This is a huge improvement. I'm really happy with the system set up.
The ability to at least get some comments on what someone thinks you
did right and wrong would be a huge improvement. The shorter voting
cycle would also be beneficial.
What about the toolbar with the top area for comments and voting? I
think that would also help but like you said in the other thread one
of the major OpenID providers goes out of it's way to break iframes.
I really liked the fact that if you setup your site to use OpenID that
it basically meant everyone could signup without a problem but if it
breaks a toolbar that would make it easier to vote I'm not sure which
would be more useful.
The ability to at least get some comments on what someone thinks you
did right and wrong would be a huge improvement. The shorter voting
cycle would also be beneficial.
What about the toolbar with the top area for comments and voting? I
think that would also help but like you said in the other thread one
of the major OpenID providers goes out of it's way to break iframes.
I really liked the fact that if you setup your site to use OpenID that
it basically meant everyone could signup without a problem but if it
breaks a toolbar that would make it easier to vote I'm not sure which
would be more useful.
Nick Plante
Jun 9, 2009, 4:06:42 PM6/9/09
to rails-...@googlegroups.com
Yeah, as mentioned, the frame-busted used by one of the major OpenID
providers makes the voting frame that we used in 2007 impossible if
OpenID is to be used. I do agree though, that the voting frame
facilitates an easier voting workflow if present. If anyone has other
suggestions, we're happy to discuss / explore them.
I'd hate to move away from OpenID just for this; seems silly given the
obvious upside for a competition of this nature.
..nap
providers makes the voting frame that we used in 2007 impossible if
OpenID is to be used. I do agree though, that the voting frame
facilitates an easier voting workflow if present. If anyone has other
suggestions, we're happy to discuss / explore them.
I'd hate to move away from OpenID just for this; seems silly given the
obvious upside for a competition of this nature.
..nap
Dana Jones
Jun 10, 2009, 12:32:30 PM6/10/09
to rails-...@googlegroups.com
Just wanted to chime in and say I love this idea. I haven't
participated in a Rumble before, but hope to this year and am relieved
I wouldn't feel I have to "pimp" myself so much just to get a lot of
votes.
A suggestion about the single vs teams issue - are they separate
categories? If not, could they be? Maybe the first round results in
the top 25 team sites, and the top 25 individual sites, and those 50
are presented for popular vote in separate categories?
Dana
participated in a Rumble before, but hope to this year and am relieved
I wouldn't feel I have to "pimp" myself so much just to get a lot of
votes.
A suggestion about the single vs teams issue - are they separate
categories? If not, could they be? Maybe the first round results in
the top 25 team sites, and the top 25 individual sites, and those 50
are presented for popular vote in separate categories?
Dana
Nick Plante
Jun 11, 2009, 10:04:37 AM6/11/09
to Rails Rumble
Everyone is competing for the same prizes / places. There are four
main categories that users vote on, but these are combined to
determine overall winners; group size is not one of em. In the past
we've just had a 'best solo' special category to recognize the efforts
of those people who were competing alone. Statistically we don't get
enough single person teams to make running a parallel solo competition
meaningful, as the vast majority of entries are teams; I think it'd
also be a distraction to voters who just want to look at cool new web
properties -- they don't necessarily care about team size. Just my two
cents though!
I do believe that we should continue to award the prize; my guess is
that one or more single person teams will sneak into the top 25-30
anyway, but if they don't make the top tier cut, we can simply choose
the next closest one that took top honors from the expert panel. Does
that sound fair?
..nap
main categories that users vote on, but these are combined to
determine overall winners; group size is not one of em. In the past
we've just had a 'best solo' special category to recognize the efforts
of those people who were competing alone. Statistically we don't get
enough single person teams to make running a parallel solo competition
meaningful, as the vast majority of entries are teams; I think it'd
also be a distraction to voters who just want to look at cool new web
properties -- they don't necessarily care about team size. Just my two
cents though!
I do believe that we should continue to award the prize; my guess is
that one or more single person teams will sneak into the top 25-30
anyway, but if they don't make the top tier cut, we can simply choose
the next closest one that took top honors from the expert panel. Does
that sound fair?
..nap
Dana Jones
Jun 11, 2009, 12:56:54 PM6/11/09
to rails-...@googlegroups.com
Ahhh I wasn't aware of what the ratio of teams to individuals was, and
hadn't thought about the logistics of voting on one vs the other. I
appreciate the clarification!
Dana
hadn't thought about the logistics of voting on one vs the other. I
appreciate the clarification!
Dana
danahern
Jun 19, 2009, 10:34:35 AM6/19/09
to Rails Rumble
Nick I totally agree that you really need some sort of single sign in
thing. I actually have been really enjoying using Facebook Connect
recently. With the Facebooker gem and just a little bit of code it is
fairly easy to implement not to mention that it is probably just as
common if not more so that users have a Facebook account as OpenID.
Since there aren't multiple entities providing the service and
Facebook just pops up a little box to login you could keep the
toolbar. If you are using any sort of social application users could
also invite friends to use it if they like the idea of the site and
want to test out social features Facebook Connect makes it easy to
invite your Facebook friends to come use the app. The only concern
might be the amount of wall posts that could theoretically be
generated if people setup Facebook notifications however, it is really
easy from the Facebook site to control how chatty each app is allowed
to be based on individual user preferences.
I dunno, there might be problems I'm not thinking of or reasons why it
might not be a good idea, but it's a suggestion.
thing. I actually have been really enjoying using Facebook Connect
recently. With the Facebooker gem and just a little bit of code it is
fairly easy to implement not to mention that it is probably just as
common if not more so that users have a Facebook account as OpenID.
Since there aren't multiple entities providing the service and
Facebook just pops up a little box to login you could keep the
toolbar. If you are using any sort of social application users could
also invite friends to use it if they like the idea of the site and
want to test out social features Facebook Connect makes it easy to
invite your Facebook friends to come use the app. The only concern
might be the amount of wall posts that could theoretically be
generated if people setup Facebook notifications however, it is really
easy from the Facebook site to control how chatty each app is allowed
to be based on individual user preferences.
I dunno, there might be problems I'm not thinking of or reasons why it
might not be a good idea, but it's a suggestion.
Leah Culver
Jun 23, 2009, 4:33:25 PM6/23/09
to Rails Rumble
From a judging perspective I think the fewer finalists, the better.
I'd rather have the time to be able to look closely at a few
spectacular apps, than have to wade through a list of 50 pretty good
apps.
How about around five or six finalists?
I know this isn't ideal for the participants since it will eliminate a
lot of people, so I agree with the suggestion to post the judges
scores of all (most?) apps. That way, people can see how well they
did, without having to burden the public with more judging.
Leah
I'd rather have the time to be able to look closely at a few
spectacular apps, than have to wade through a list of 50 pretty good
apps.
How about around five or six finalists?
I know this isn't ideal for the participants since it will eliminate a
lot of people, so I agree with the suggestion to post the judges
scores of all (most?) apps. That way, people can see how well they
did, without having to burden the public with more judging.
Leah
Nick Plante
Jun 23, 2009, 6:25:09 PM6/23/09
to rails-...@googlegroups.com
Personally my preference is to have somewhere between one to two dozen
applications open to public judging. I think this is a sweet spot for
me, anyway, where as a public judge, I feel like there are a variety
of apps to look at (possibly spanning a wide range of potential uses)
but not an overwhelming number. I think it's also a good spot for
competitors, who want a solid chance at getting their app to the point
where it's viewable by the public.
That's just my opinion though -- other organizers (Darcy? Erin? Jeff?)
may have other ideas about it and we'll definitely discuss before we
settle on a final number. We're also really interested to hear what
the public has to say; it sounds like there are people on both sides
of the issue right now. If the general consensus is that it should be
less or more, we're happy to weigh that in and adjust accordingly.
Like you said it's important that each team (assuming they complete
and submit an app) get *real* feedback from the expert panel too, and
we'll be working with judges to make sure that happens. That way, no
matter what, they get some valuable advice that they can use in future
iterations.
ttys!
..nap
applications open to public judging. I think this is a sweet spot for
me, anyway, where as a public judge, I feel like there are a variety
of apps to look at (possibly spanning a wide range of potential uses)
but not an overwhelming number. I think it's also a good spot for
competitors, who want a solid chance at getting their app to the point
where it's viewable by the public.
That's just my opinion though -- other organizers (Darcy? Erin? Jeff?)
may have other ideas about it and we'll definitely discuss before we
settle on a final number. We're also really interested to hear what
the public has to say; it sounds like there are people on both sides
of the issue right now. If the general consensus is that it should be
less or more, we're happy to weigh that in and adjust accordingly.
Like you said it's important that each team (assuming they complete
and submit an app) get *real* feedback from the expert panel too, and
we'll be working with judges to make sure that happens. That way, no
matter what, they get some valuable advice that they can use in future
iterations.
ttys!
..nap
Reply all
Reply to author
Forward
0 new messages