Update on providing Challenge scores and rankings

[Justin (Google Employee)]

Many people have been asking about receiving either their individual
scores or a complete list of scores. We've discussed this extensively
and have decided to release each submission's quartile rank through
individual emails within the next few weeks.

Why quartile rankings? Well, we feel they provide more meaningful
information about relative application rankings than just raw scores.
We designed the judging process to identify 50 applications with the
highest scores out of a large pool of excellent submissions. Many
applications received average scores that are within 0.1% of each
other because of the number of submissions. This means that providing
an individual score to each participant does not provide meaningful
information on the relative ranking of that submission.

Another request has been for textual feedback from judges. However,
judges only provided scores for each application; they did not provide
notes or any other comments.

That being said, the number of great applications far exceeds the top
50! We had many judges actually ask for contact information for
developers from the entire pool of applications—and we're forwarding
that information on to the judges for developers who opted-in.

We're excited by the massive response to the challenge and we can't
wait to see more great applications for Android.

Regards,
Justin & the Android Developer Challenge Team @ Google

[Hielko]

Nice, thanks for the update.

[Finn Kennedy]

Any change you could add whether or not an application made it into the second phase of round 1?  (I.e. was one of the ~100 apps in the later stage).

Thanks,

Finn
http://www.teedroid.com

[javqui]

Thank you for this important message.
I hope receive this quartile rank soon.

Javier Quintana

On May 14, 7:16 pm, "Finn Kennedy" <finn.kenn...@gmail.com> wrote:
> Any change you could add whether or not an application made it into the
> second phase of round 1?  (I.e. was one of the ~100 apps in the later
> stage).
>
> Thanks,
>

> Finnhttp://www.teedroid.com

> > Justin & the Android Developer Challenge Team @ Google- Hide quoted text -
>
> - Show quoted text -

[PowerGUI]

All what I want to say is:
Thanks Very Very Much,Dear Googlers.

I am really deeply moved,though I can never expect such result,
but Google do it now.

Regards
PowerGUI

[Free Tibet]

Damn Google!!

An hour ago I hit an Amazon's button to buy the book, "Cocoa(R)
Programming for Mac OS X" and got on my way to iPhone world.

Let me see if it is still in time to cancel the purchase.

Thank you. Thank Google. Thank Bodhisattva.

Free Tibet

On May 14, 4:08 pm, "Justin (Google Employee)" <j...@google.com>
wrote:

[Michael Johnston]

"Many applications received average scores that are within 0.1% of
each other because of the number of submissions. This means that
providing an individual score to each participant does not provide
meaningful information on the relative ranking of that submission."

This is interesting, and I agree, but it kind of casts doubt on the
meaningfulness of the scoring system in general. Nonetheless, the
update and forthcoming release of quartile rankings are much
appreciated. Thanks.

On May 14, 4:08 pm, "Justin (Google Employee)" <j...@google.com>
wrote:

> Jstin & the Android Developer Challenge Team @ Google

[javqui]

Dear Justin

As humbleness suggestion, will be useful get a quartile rank for each
judging criteria. Some participants get more rank in one criteria and
less in other one. This suggestion will help us to optimized resources
distribution for ADCII.

Thank you

Javier Quintana

> > Jstin & the Android Developer Challenge Team @ Google- Hide quoted text -

[GodsMoon]

Thank you Google. I was starting to wonder if you had turned your back
on the developer community.
This sort of transparency is exactly what is needed if Android is to
be successful in the developer community.

Thanks for bring back the Google we all know and love.

David Shellabarger
www.goldfishview.com

[A&&Whole]

THEY ARE JUST TRYING TO BE HUMBLE
THEY KNOW THEY DID A BAD JOB
SO PROVIDING RANKING ETC IS JUST TO BRING US BACK HERE

THEY ARE CRAP CRAP and CRAP

[hitsu_g]

I am grateful that Google has listened to the voice of the developers
and decided to release this information. It is an encouragement to me
and I intend to make the best possible use of this info to help guide
my future efforts developing for Android and iPhone.

[Yassine]

Thank you... i'm very happy to ear that...

[Ram]

Thanks Justin, any feedback will be appreciated and an overall
quartile ranking should help provide some closure to many
participants.
This is close to my secondary suggestion on
http://groups.google.com/group/android-challenge/browse_thread/thread/42e5453bf21f8369/d7a00fd4237f6dd0#d7a00fd4237f6dd0

I think I wrote 85% of my app in the last 8 days of the challenge
timeframe and I'm myself interested in knowing what the judge-scores
for my app were.
I'm sure that app-rankings will be even more appreciated by others
who've spent several months on their app-submissions

> and have decided to release each submission's quartile rank through
> individual emails within the next few weeks

Will it be possible to expedite the emails and send all quartile
rankings within the next week ?

Since you already have the scores for all ~1700 apps, I'm assuming
that it wouldn't take a lot of effort to assign a quartile raking and
send out the emails.
Immediate feedback is likely to be more helpful to most people and it
will also help folks move on more quickly.

Thanks Ram

On May 14, 4:08 pm, "Justin (Google Employee)" <j...@google.com>
wrote:

[Nixarn]

I still hope for some decent constructive feedback. Would be great if
every entry got at least two judges comments. That wouldn't require
alot of time/work from the judges. Getting feedback from all 100 is
too much.

Regards,
Niklas

On May 15, 2:08 am, "Justin (Google Employee)" <j...@google.com>
wrote:

[vetch]

> Why quartile rankings? Well, we feel they provide more meaningful
> information about relative application rankings than just raw scores.

why WE can't choose what is better for us ?

But I do not need answer. I only think loud.

greets,
peter

[george_c]

I agree with Justin's request,  a quartile rank for 'each' judging criteria is what's needed. A simple average doesn't help developers know where to focus their efforts to deliver needed applications as industry experts(judges) have depicted.

There will soon be a very competitive landscape, hence delivering apps that make money across the value chain is critical to know at this point or to educate the developers.

George

[plusminus]

For not native speakers, what does "quartile rank" mean?
Is it that all apps are divided into 4 rank-groups, like:
1. Best
2. Very Good
3. Pretty Good
4. Good

?

On May 15, 11:27 am, george_c <chrisg...@gmail.com> wrote:
> I agree with Justin's request, a quartile rank for 'each' judging criteria
> is what's needed. A simple average doesn't help developers know where to
> focus their efforts to deliver needed applications as industry
> experts(judges) have depicted.
>
> There will soon be a very competitive landscape, hence delivering apps that
> make money across the value chain is critical to know at this point or to
> educate the developers.
>
> George
>

[ryank]

Hi Plusminus,
That's correct. It should tell you whether your app fell between the
top 0-25%, 26-50%, 51-75%, or bottom 76-100%.

plusminus, thanks again for your all your help on AndDev!

[gtandon]

Really appreciate Google's efforts to send the quartile rank to
individuals. Beyond this, if possible to send the scores in each
category, that will help too. When sent to individuals pivately, they
are no longer scores relative to other competitors, but 4 numbers
relative to each other.

So say someone gets 3 on originality, 1 on polish and appeal, 0.5 on
use of Android features and 2 on indispensibility, it helps people
understand the external perception of their application in these
categories. Helps them to see that they could have done better at
using the LBS or accelerometer like the guidelines say, or take
another look at the useability aspects of the app etc. Others in a
50%ile with (7,5,4,6) might take away the same message.

I agree with George that the score (these 4 numbers) will allow
developers to focus on the weakest area, or reinforce their belief in
the strong area.

-Gauri

> > Javier Quintana- Hide quoted text -

[Cow Bay]

no.

it is like

1. top50
2. top100
3. tested
4. untested

[Cow Bay]

i swear with virginity of Cleopatra that there r no more scores; they might
have been destroyed!!!

----- Original Message -----
From: "gtandon" <Gauri....@gmail.com>
To: "Android Challenge" <android-...@googlegroups.com>
Sent: Thursday, May 15, 2008 6:30 AM
Subject: [android-challenge] Re: Update on providing Challenge scores and
rankings

[ConAim]

Its ok Cow, Google already made their commitment and nothing that they
can really do to recall their poor judge process. We can just stay out
and watch to see how they are going to continue trigging the developer
community to delivering some more cooking, weather, mission control
apps for them.

More voice and they will start calling us are whiners....

> > > > Javier Quintana- Hide quoted text -

[Cow Bay]

whenever i think of the four equally-weighted criteria, my heart hurts.

adc breaks many hearts here, i believe, not because their apps are not
better but because a bunch of mobile experts were brain-damaged.

cnn!!

[ajd]

Why just the quartile ranking? What is more meaningful in quartile
ranking than the raw scores?

Okey, raw scores might not have meaning to Google, but those scores
are more useful to the entrants. It will say a lot how the judges
looked at the application and therefore where the app needs to
improve..

Quartile? It just says something on how Google wants us to see the
apps. How does it help that an entrant finds out that his/her app is
in the second quarter rank? I will still be guessing how my app lose.

> > - Show quoted text -- Hide quoted text -

[Cow Bay]

a quartile random number generator uses much less cpu power than a decimal
generator.

dont forget to be green anytime anywhere just as judges want us to
appreciate the carbon meter.

no wonder you are out.

[ajd]

On May 15, 8:39 am, ajd <doming...@yahoo.com> wrote:
> Why just the quartile ranking? What is more meaningful in quartile
> ranking than the raw scores?
>
> Okey, raw scores might not have meaning to Google, but those scores
> are more useful to the entrants. It will say a lot how the judges
> looked at the application and therefore where the app needs to
> improve..
>
> Quartile? It just says something on how Google wants us to see the
> apps. How does it help that an entrant finds out that his/her app is
> in the second quarter rank? I will still be guessing how my app lose.
>

Raw scores for example will reveal how high the weather channel app
(and the other weather app) scored in the originality criteria, and
how high the carbon emission calculator or the cooking app scored in
the inidispensability area. But just having the quartile rankings will
effectively hide how those apps won.

[dr123]

I don't understand what is the problem with the raw scores. Quartiles
of course are, but the other numbers are too.
Let me think what I suspect. If they didn't have some apps evaluated
the cannot send out some rubbish numbers because a creator of a nice
gui with a not-so-nice idea could get the opposite ie big mark on the
idea and zero for the gui. He will then understand that this was
random or never evaluated.

[Rui Martins]

Wow !

Just found out about this !

Great news, but you should really, and I mean REALLY, send everyone
their own 4 scores in the 4 evaluation categories.

I understand the need of google to not fire up the discussion on
ranking and scoring, but this info is really useful to each of us
independently.

And I would like to suggest, next time, judges should be REQUIRED to
justify their scoring, by reasoning something about it in writen form.
This is will make the next contests look&feel professional and mostly
OPEN !

But then again, this info will be invaluable to the next contest
participants.

Thank you for any extra info.

P.S.
I just can't understand why it took so long for google team to
realize this.
It's a given in any public contest, that scoring and final listings
are required.

[jazz]

The judges might have provided a single score (1-10) for each app with
no text comments.

-J

[Rui Martins]

Just a side note, stating quartile doesn't really place you anywhere
in terms of posistion, it only works in terms of score, and let me
explain.

As far as I understood, quartile, means you where in the group which
scored in that quartile (0%-100%),
hence this means that we can have 1500 in first quartile (best score),
200 in second, 50 in third and 38 in the last.
or
50, 1700, 30, 8
or
500, 500, 500, 208
or
10,40, 38,1700
etc...

What I mean is, with this output, it's possible inclusively to
imagine, that not all of the first 50 where in the first quartile.

But this surelly looks like throwing sand, while we have or yes
openned.
The statistically more knowledjable than me, please correct me if I'm
wrong ! doubt it !

Ho well, will have what you will be able to give us :(

[ajd]

Read Justin's original post: it says "quartile ranking" - relative
positions of the apps.

There are three kinds of lies: Lies, damned lies, and statistics.
Keep it simple: send the raw scores.

[Colin Cochran]

Thanks for the update Justin.

In addition to the quartile rankings, is there something holding Google back from giving us the 1-10 raw scores from each judge? We'd like to know how indispensable or original judges felt our applications were. For me at least, this information will help me modify my application.

Thanks
-C

On Wed, May 14, 2008 at 5:08 PM, Justin (Google Employee) <j...@google.com> wrote:

Many people have been asking about receiving either their individual
scores or a complete list of scores. We've discussed this extensively
and have decided to release each submission's quartile rank through
individual emails within the next few weeks.

Why quartile rankings? Well, we feel they provide more meaningful
information about relative application rankings than just raw scores.

[Colin Cochran]

Well put.

I second.

[blues]

So why not give both quartile ranking and raw score?

On May 15, 3:10 pm, "Colin Cochran" <ccochran...@gmail.com> wrote:
> Thanks for the update Justin.
>
> In addition to the quartile rankings, is there something holding Google back

> from giving us the *1-10 raw scores from each judge*? We'd like to know how

> > Justin & the Android Developer Challenge Team @ Google- Hide quoted text -

[hitsu_g]

Hmm. I foolishly assumed that "quartile rank" DID mean that they would
be sending out the rank for each of the four categories that our apps
were judged on - each category comprising one quarter of our score...
So I was foolishly excited. Seeing the detailed score like that helps
me to know WHAT AREA I NEED TO IMPROVE IN, knowing whether I was among
the best, worst or average entries seems only good for bragging rights
(if among the best) or drowning in self-pity rights (if among the
worst)...

[eswar]

Oh this is grt news. Thanks for that. Updates more often would keep
the community alive.


On May 15, 4:08 am, "Justin (Google Employee)" <j...@google.com>

[vetch]

I'm tired of your games with us, Google.

There was a challenge, right ? So, there are challenge results,
right ?

So, why Google behaves like old chinese comunist and hides, or trying
to censure all key information ?

We should know not only all scores, but all scores should be open and
publicated with full public ranking, even stats, available for
EVERYONE, as in every professional contest. This one, in my opinion,
was very unprofessional.

TRUTH, whatever it is, is always good, and surely better than cheating
and hiding facts.


greets,
peter

[efon...@gmail.com]

Hello Justin and Team,

In the interest of taunting and the real benefits it will bring to the
android customer base (both carriers and end customers) please
consider publishing the gallery in rank order :-)

This Michael Porter artical "The Competitive Advantage of Nations"
http://harvardbusinessonline.hbsp.harvard.edu/b01/en/common/item_detail.jhtml?id=90211&referral=7855

explains that if you want to win in the global marketplace, there must
be personal rivalry. Yes, every place he found world domination, there
it was, emperically. (read the pdf version - it has better graphics).

All the ambiguity of middle rank may not be something one should hide
to promote social order and the disinterest that goes with it.

Rather, put stuff in the gallery in rank order (at least as an option)
as a service to the viewers and as a taunting vehicle.

Yes, people would need to know their numbers for taunting to be
easily available.

Just a thought.

Ed




On May 14, 6:08 pm, "Justin (Google Employee)" <j...@google.com>

[vinay]

I agree with George and Gauri.

scores for the four judging criteria is what I am looking for to
improve my app.
and I hope everyone is..winners or loosers.

For your consideration now.

-Vinay