Since we are coming up on the week of May 5th, when is Phase II
starting?
Will the entire panel of judges review the whole set of 100
applications, or is the set of 100 split into groups and distributed
randomly again?
Are there any differences between Phase I and Phase II?
On an slightly related note, is there anything planned for the top 50
planned at Google IO?
Also, for Google IO: If you are traveling from Austin, TX, there are
direct flights from Austin to San Jose International. You can then
take CalTrain (http://www.caltrain.com) from close to the airport to
the San Francisco stop. It is on 4th, same street as the Moscone
Center.
Of course I am not a travel agent/planner, so please double check
everything yourself.
I will be at I/O and I live in Silicon Valley. Here are some travel
info:
Caltrain station is about 4 blocks from the Moscone Center.
Keep in mind that the train from San Jose to San Francisco takes 1
hour for the express "bullet" train which only runs during rush
hours. Also, you need shuttle connection between SJ airport and
Caltrain.
If you fly to SF International, you can take BART right at the airport
to San Francisco downtown. Get off at Powell Station and walk to
Moscone Center (3 blocks or so). It takes about 20-30 minutes.
Also consider Oakland airport which has a shuttle to BART (Coliseum
Station). Take the San Francisco train. It takes about 25 minutes.
In general, SF airport and Oakland airports are therefore a much
better choice than San Jose airport.
Hope this helps. Also check out bart.gov for schedule and fare.
On May 2, 6:58 pm, finnk <finn.kenn...@gmail.com> wrote:
> Since we are coming up on the week of May 5th, when is Phase II
> starting?
> Will the entire panel of judges review the whole set of 100
> applications, or is the set of 100 split into groups and distributed
> randomly again?
> Are there any differences between Phase I and Phase II?
> On an slightly related note, is there anything planned for the top 50
> planned at Google IO?
> Also, for Google IO: If you are traveling from Austin, TX, there are
> direct flights from Austin to San Jose International. You can then
> take CalTrain (http://www.caltrain.com) from close to the airport to
> the San Francisco stop. It is on 4th, same street as the Moscone
> Center.
> Of course I am not a travel agent/planner, so please double check
> everything yourself.
I was just suggesting San Jose as there are no direct flights to the area from where I am to the other airports (at least at the same price levels). It cut my travel time for the flight in more than half. Full disclosure, I am not very familiar with the area as I have only traveled there occasionally on business.
On Fri, May 2, 2008 at 9:14 PM, j <jac...@gmail.com> wrote:
> I will be at I/O and I live in Silicon Valley. Here are some travel > info:
> Caltrain station is about 4 blocks from the Moscone Center.
> Keep in mind that the train from San Jose to San Francisco takes 1 > hour for the express "bullet" train which only runs during rush > hours. Also, you need shuttle connection between SJ airport and > Caltrain.
> If you fly to SF International, you can take BART right at the airport > to San Francisco downtown. Get off at Powell Station and walk to > Moscone Center (3 blocks or so). It takes about 20-30 minutes.
> Also consider Oakland airport which has a shuttle to BART (Coliseum > Station). Take the San Francisco train. It takes about 25 minutes.
> In general, SF airport and Oakland airports are therefore a much > better choice than San Jose airport.
> Hope this helps. Also check out bart.gov for schedule and fare.
> On May 2, 6:58 pm, finnk <finn.kenn...@gmail.com> wrote: > > Having reread the Judging Process (http://code.google.com/android/ > > adc_judging.html), I have a couple of questions.
> > Since we are coming up on the week of May 5th, when is Phase II > > starting?
> > Will the entire panel of judges review the whole set of 100 > > applications, or is the set of 100 split into groups and distributed > > randomly again?
> > Are there any differences between Phase I and Phase II?
> > On an slightly related note, is there anything planned for the top 50 > > planned at Google IO?
> > Also, for Google IO: If you are traveling from Austin, TX, there are > > direct flights from Austin to San Jose International. You can then > > take CalTrain (http://www.caltrain.com) from close to the airport to > > the San Francisco stop. It is on 4th, same street as the Moscone > > Center.
> > Of course I am not a travel agent/planner, so please double check > > everything yourself.
Well, for all intents and purposes, Phase II begins as soon as we announce the 50 Phase I winners. It's not like we could stop the winners from starting right away, anyway. :) It looks like we are still on track to announce those winners next week. A different set of judges is reviewing the 100 applications. The top 100 applications are "reset" and rejudged from scratch by a different group of judges, who have no knowledge of the previous judges' scores.
We're thinking about ways to work with the 50 Phase I winners, but that might not necessarily include anything formal at Google I/O. (We don't want to require anyone to attend, and we don't want to give any of them an unfair advantage.)
> Since we are coming up on the week of May 5th, when is Phase II > starting?
> Will the entire panel of judges review the whole set of 100 > applications, or is the set of 100 split into groups and distributed > randomly again?
> Are there any differences between Phase I and Phase II?
> On an slightly related note, is there anything planned for the top 50 > planned at Google IO?
> Also, for Google IO: If you are traveling from Austin, TX, there are > direct flights from Austin to San Jose International. You can then > take CalTrain (http://www.caltrain.com) from close to the airport to > the San Francisco stop. It is on 4th, same street as the Moscone > Center.
> Of course I am not a travel agent/planner, so please double check > everything yourself.
> Well, for all intents and purposes, Phase II begins as soon as we announce
> the 50 Phase I winners. It's not like we could stop the winners from
> starting right away, anyway. :) It looks like we are still on track to
> announce those winners next week.
> A different set of judges is reviewing the 100 applications. The top 100
> applications are "reset" and rejudged from scratch by a different group of
> judges, who have no knowledge of the previous judges' scores.
> We're thinking about ways to work with the 50 Phase I winners, but that
> might not necessarily include anything formal at Google I/O. (We don't want
> to require anyone to attend, and we don't want to give any of them an unfair
> advantage.)
> - Dan
> On Fri, May 2, 2008 at 6:58 PM, finnk <finn.kenn...@gmail.com> wrote:
> > Since we are coming up on the week of May 5th, when is Phase II
> > starting?
> > Will the entire panel of judges review the whole set of 100
> > applications, or is the set of 100 split into groups and distributed
> > randomly again?
> > Are there any differences between Phase I and Phase II?
> > On an slightly related note, is there anything planned for the top 50
> > planned at Google IO?
> > Also, for Google IO: If you are traveling from Austin, TX, there are
> > direct flights from Austin to San Jose International. You can then
> > take CalTrain (http://www.caltrain.com) from close to the airport to
> > the San Francisco stop. It is on 4th, same street as the Moscone
> > Center.
> > Of course I am not a travel agent/planner, so please double check
> > everything yourself.
thank you for the responses. A couple of follow ups.
With Phase 2 I meant the 100 being winnowed down to 50. From the ADC Judging Process page:
"In Phase 2, the 100 highest-scoring submissions will be all be sent to a new panel of judges (which may or may not include one or more of the judges who participated in Phase I judging). ... The 50 entries with the highest scores in Phase 2 judging will move on to Round 2 of the Challenge..."
Just for clarification, are there again groups of judges assigned to look at a subset of the top 100 entries? Or are the entire set of entries judged by all the judges in Phase 2?
Is the "outlier" procedure still used for Phase 2? By outlier I mean the review of scores not matching the rest of the scores for the application (mentioned on the board).
I totally understand the want to have a fair playing field. It would not be fair to extend advantages to winners that can make the trip to Google I/O.
On Sat, May 3, 2008 at 3:27 PM, Dan Morrill <morri...@google.com> wrote: > Well, for all intents and purposes, Phase II begins as soon as we announce > the 50 Phase I winners. It's not like we could stop the winners from > starting right away, anyway. :) It looks like we are still on track to > announce those winners next week. > A different set of judges is reviewing the 100 applications. The top 100 > applications are "reset" and rejudged from scratch by a different group of > judges, who have no knowledge of the previous judges' scores.
> We're thinking about ways to work with the 50 Phase I winners, but that > might not necessarily include anything formal at Google I/O. (We don't want > to require anyone to attend, and we don't want to give any of them an unfair > advantage.)
> - Dan
> On Fri, May 2, 2008 at 6:58 PM, finnk <finn.kenn...@gmail.com> wrote:
> > Since we are coming up on the week of May 5th, when is Phase II > > starting?
> > Will the entire panel of judges review the whole set of 100 > > applications, or is the set of 100 split into groups and distributed > > randomly again?
> > Are there any differences between Phase I and Phase II?
> > On an slightly related note, is there anything planned for the top 50 > > planned at Google IO?
> > Also, for Google IO: If you are traveling from Austin, TX, there are > > direct flights from Austin to San Jose International. You can then > > take CalTrain (http://www.caltrain.com) from close to the airport to > > the San Francisco stop. It is on 4th, same street as the Moscone > > Center.
> > Of course I am not a travel agent/planner, so please double check > > everything yourself.
We are in the process of discovering how famous Eugene Poddany is and
how expensive it is to have an eagle shake an olive branch to his
music!
Our team just wants to see the product exist and mature in a form that
serves customers in a way that is excellent enough to get network
effects. We really think there is a Nobel Peace Prize in it for every
person who uses it 6 years down the road, yes, really. We would
gladly volunteer to forgo participation in phase II to have great
product exist with the help of all that Google 20% discretionary
time.
Oh, the offer stands even if we were disqualified, or not in the first
50, or 100, or...
> We're thinking about ways to work with the 50 Phase I winners, but that
> might not necessarily include anything formal at Google I/O. (We don't want
> to require anyone to attend, and we don't want to give any of them an unfair
> advantage.)
Ahh -- we've not been rigorous in consistently naming these various rounds and phases. Let me try and adopt that terminology for this thread, and explain again. ADC 1 == this $5,000,000 prize event going on now. ADC 2 == the second $5,000,000 prize event that will begin later this year. ADC 1 Round 1 == open participation with the deadline of 14 April, with 50 winners ADC 1 Round 2 == participation limited to the winners of ADC 1 Round 1, with 20 "final" winners ADC 1 Round 1 Phase 1 == reducing the original set of 1,788 submissions to 100 finalists ADC 1 Round 1 Phase 2 == picking the 50 ADC 1 Round 1 winners from the 100 finalists
Okay, phew. :) With those definitions, here is where we are:
- We sent out the submissions to judging a few days after the submission deadline of 14 April, and judging began. - Our 100 or so judges received the judging guidelines we provided, reviewed their assigned submissions, and reported data back to us. - Late last week, we applied our outlier mitigation techniques, identified the top 100 results, and sent them on to the final, separate panel of 15 or so judges to score and produce the final 50 ADC 1 Round 1 award recipients.
So in other words, we are currently in ADC 1 Round 1 Phase 2 as defined above. Once data from the judges comes in, we will notify the 50 award recipients and ADC 1 Round 2 will begin.
It has not escaped my notice even on vacation that there have been a number of discussions on server hits and so on. Obviously we don't have access to everyone's server logs, and we can't monitor what the judges have actually been doing (nor would we snoop if we could, since that seems really sketchy.) We've tried to automate everything we possibly can about the judging process, but the one thing we can't automate is the actual act of assigning scores, since that requires a human's brain.
The judges were given fairly detailed guidance on how to calibrate their scores, and what to review. For instance, they are aware that they are supposed to read documentation and do their best to test all the features. In the end, though, each judge is going to test to his or her own satisfaction. I'm not sure how reliable it is to correlate judge reviews with observed server hits. Some apps might have sporadic bugs that prevent network accesses. Some judges may have decided they didn't need to see a particular feature. And before you cry foul, know that some people who have inquired about "missing" server hits have actually done quite well. Judges are just as likely to say "this is cool, I don't need to see any more" as they are to say "this is so uncool, I don't need to see any more." On the whole, our judges have been excited to participate, and I expect that they are being as conscientious as they can be.
The one thing I can tell you with certainty is that I have answered quite a few private inquiries, and in all but one case the judges responded with legitimate scores, rather than scores that say something went wrong or the review was incomplete. Our only data points are what the judges give us, because that's the only factor we can't automate. Since the judges are telling us that they reviewed to their satisfaction, we can only take their word for it.
We've tried really hard to make sure that the only thing that affects scoring is what you put in front of the judges. But the entire goal of the ADC is to leverage plain old human judgment.
- Dan P.S. - watch for gory details on the nuts & bolts of all this in the near future.
On Sat, May 3, 2008 at 1:58 PM, Finn Kennedy <finn.kenn...@gmail.com> wrote: > Dan,
> thank you for the responses. A couple of follow ups.
> With Phase 2 I meant the 100 being winnowed down to 50. From the ADC > Judging Process page:
> "In Phase 2, the 100 highest-scoring submissions will be all be sent to a > new panel of judges (which may or may not include one or more of the judges > who participated in Phase I judging). > ... > The 50 entries with the highest scores in Phase 2 judging will move on to > Round 2 of the Challenge..."
> Just for clarification, are there again groups of judges assigned to look > at a subset of the top 100 entries? Or are the entire set of entries judged > by all the judges in Phase 2?
> Is the "outlier" procedure still used for Phase 2? By outlier I mean the > review of scores not matching the rest of the scores for the application > (mentioned on the board).
> I totally understand the want to have a fair playing field. It would not > be fair to extend advantages to winners that can make the trip to Google > I/O.
> Finn
> On Sat, May 3, 2008 at 3:27 PM, Dan Morrill <morri...@google.com> wrote:
>> Well, for all intents and purposes, Phase II begins as soon as we announce >> the 50 Phase I winners. It's not like we could stop the winners from >> starting right away, anyway. :) It looks like we are still on track to >> announce those winners next week. >> A different set of judges is reviewing the 100 applications. The top 100 >> applications are "reset" and rejudged from scratch by a different group of >> judges, who have no knowledge of the previous judges' scores.
>> We're thinking about ways to work with the 50 Phase I winners, but that >> might not necessarily include anything formal at Google I/O. (We don't want >> to require anyone to attend, and we don't want to give any of them an unfair >> advantage.)
>> - Dan
>> On Fri, May 2, 2008 at 6:58 PM, finnk <finn.kenn...@gmail.com> wrote:
>>> Since we are coming up on the week of May 5th, when is Phase II >>> starting?
>>> Will the entire panel of judges review the whole set of 100 >>> applications, or is the set of 100 split into groups and distributed >>> randomly again?
>>> Are there any differences between Phase I and Phase II?
>>> On an slightly related note, is there anything planned for the top 50 >>> planned at Google IO?
>>> Also, for Google IO: If you are traveling from Austin, TX, there are >>> direct flights from Austin to San Jose International. You can then >>> take CalTrain (http://www.caltrain.com) from close to the airport to >>> the San Francisco stop. It is on 4th, same street as the Moscone >>> Center.
>>> Of course I am not a travel agent/planner, so please double check >>> everything yourself.
> Ahh -- we've not been rigorous in consistently naming these various rounds
> and phases. Let me try and adopt that terminology for this thread, and
> explain again.
> ADC 1 == this $5,000,000 prize event going on now.
> ADC 2 == the second $5,000,000 prize event that will begin later this year.
> ADC 1 Round 1 == open participation with the deadline of 14 April, with 50
> winners
> ADC 1 Round 2 == participation limited to the winners of ADC 1 Round 1, with
> 20 "final" winners
> ADC 1 Round 1 Phase 1 == reducing the original set of 1,788 submissions to
> 100 finalists
> ADC 1 Round 1 Phase 2 == picking the 50 ADC 1 Round 1 winners from the 100
> finalists
> Okay, phew. :) With those definitions, here is where we are:
> - We sent out the submissions to judging a few days after the submission
> deadline of 14 April, and judging began.
> - Our 100 or so judges received the judging guidelines we provided,
> reviewed their assigned submissions, and reported data back to us.
> - Late last week, we applied our outlier mitigation techniques,
> identified the top 100 results, and sent them on to the final, separate
> panel of 15 or so judges to score and produce the final 50 ADC 1 Round 1
> award recipients.
> So in other words, we are currently in ADC 1 Round 1 Phase 2 as defined
> above. Once data from the judges comes in, we will notify the 50 award
> recipients and ADC 1 Round 2 will begin.
> It has not escaped my notice even on vacation that there have been a number
> of discussions on server hits and so on. Obviously we don't have access to
> everyone's server logs, and we can't monitor what the judges have actually
> been doing (nor would we snoop if we could, since that seems really
> sketchy.) We've tried to automate everything we possibly can about the
> judging process, but the one thing we can't automate is the actual act of
> assigning scores, since that requires a human's brain.
> The judges were given fairly detailed guidance on how to calibrate their
> scores, and what to review. For instance, they are aware that they are
> supposed to read documentation and do their best to test all the features.
> In the end, though, each judge is going to test to his or her own
> satisfaction. I'm not sure how reliable it is to correlate judge reviews
> with observed server hits. Some apps might have sporadic bugs that prevent
> network accesses. Some judges may have decided they didn't need to see a
> particular feature. And before you cry foul, know that some people who have
> inquired about "missing" server hits have actually done quite well. Judges
> are just as likely to say "this is cool, I don't need to see any more" as
> they are to say "this is so uncool, I don't need to see any more." On the
> whole, our judges have been excited to participate, and I expect that they
> are being as conscientious as they can be.
> The one thing I can tell you with certainty is that I have answered quite a
> few private inquiries, and in all but one case the judges responded with
> legitimate scores, rather than scores that say something went wrong or the
> review was incomplete. Our only data points are what the judges give us,
> because that's the only factor we can't automate. Since the judges are
> telling us that they reviewed to their satisfaction, we can only take their
> word for it.
> We've tried really hard to make sure that the only thing that affects
> scoring is what you put in front of the judges. But the entire goal of the
> ADC is to leverage plain old human judgment.
> - Dan
> P.S. - watch for gory details on the nuts & bolts of all this in the near
> future.
> On Sat, May 3, 2008 at 1:58 PM, Finn Kennedy <finn.kenn...@gmail.com> wrote:
> > Dan,
> > thank you for the responses. A couple of follow ups.
> > With Phase 2 I meant the 100 being winnowed down to 50. From the ADC
> > Judging Process page:
> > "In Phase 2, the 100 highest-scoring submissions will be all be sent to a
> > new panel of judges (which may or may not include one or more of the judges
> > who participated in Phase I judging).
> > ...
> > The 50 entries with the highest scores in Phase 2 judging will move on to
> > Round 2 of the Challenge..."
> > Just for clarification, are there again groups of judges assigned to look
> > at a subset of the top 100 entries? Or are the entire set of entries judged
> > by all the judges in Phase 2?
> > Is the "outlier" procedure still used for Phase 2? By outlier I mean the
> > review of scores not matching the rest of the scores for the application
> > (mentioned on the board).
> > I totally understand the want to have a fair playing field. It would not
> > be fair to extend advantages to winners that can make the trip to Google
> > I/O.
> > Finn
> > On Sat, May 3, 2008 at 3:27 PM, Dan Morrill <morri...@google.com> wrote:
> >> Well, for all intents and purposes, Phase II begins as soon as we announce
> >> the 50 Phase I winners. It's not like we could stop the winners from
> >> starting right away, anyway. :) It looks like we are still on track to
> >> announce those winners next week.
> >> A different set of judges is reviewing the 100 applications. The top 100
> >> applications are "reset" and rejudged from scratch by a different group of
> >> judges, who have no knowledge of the previous judges' scores.
> >> We're thinking about ways to work with the 50 Phase I winners, but that
> >> might not necessarily include anything formal at Google I/O. (We don't want
> >> to require anyone to attend, and we don't want to give any of them an unfair
> >> advantage.)
> >> - Dan
> >> On Fri, May 2, 2008 at 6:58 PM, finnk <finn.kenn...@gmail.com> wrote:
> >>> Since we are coming up on the week of May 5th, when is Phase II
> >>> starting?
> >>> Will the entire panel of judges review the whole set of 100
> >>> applications, or is the set of 100 split into groups and distributed
> >>> randomly again?
> >>> Are there any differences between Phase I and Phase II?
> >>> On an slightly related note, is there anything planned for the top 50
> >>> planned at Google IO?
> >>> Also, for Google IO: If you are traveling from Austin, TX, there are
> >>> direct flights from Austin to San Jose International. You can then
> >>> take CalTrain (http://www.caltrain.com) from close to the airport to
> >>> the San Francisco stop. It is on 4th, same street as the Moscone
> >>> Center.
> >>> Of course I am not a travel agent/planner, so please double check
> >>> everything yourself.
Wow! That is more information than I hoped for before going to sleep.
Thanks a lot!
Muthu,
I cannot believe you were right. I'm really starting to believe that
you have friends at Google. Once again, if you win, I'll be highly
suspicious! :)
Good luck and congratulations to those people that complained a lot
about not seeing server hits. According to Dan some (not all) have
done quite good. It appears that this Monday we may actually know who
the top 50 most promising developers from around the world are. Wow, I
can hardly wait for this Monday to come!
On May 4, 12:22 am, "Dan Morrill" <morri...@google.com> wrote:
> Ahh -- we've not been rigorous in consistently naming these various rounds
> and phases. Let me try and adopt that terminology for this thread, and
> explain again.
> ADC 1 == this $5,000,000 prize event going on now.
> ADC 2 == the second $5,000,000 prize event that will begin later this year.
> ADC 1 Round 1 == open participation with the deadline of 14 April, with 50
> winners
> ADC 1 Round 2 == participation limited to the winners of ADC 1 Round 1, with
> 20 "final" winners
> ADC 1 Round 1 Phase 1 == reducing the original set of 1,788 submissions to
> 100 finalists
> ADC 1 Round 1 Phase 2 == picking the 50 ADC 1 Round 1 winners from the 100
> finalists
> Okay, phew. :) With those definitions, here is where we are:
> - We sent out the submissions to judging a few days after the submission
> deadline of 14 April, and judging began.
> - Our 100 or so judges received the judging guidelines we provided,
> reviewed their assigned submissions, and reported data back to us.
> - Late last week, we applied our outlier mitigation techniques,
> identified the top 100 results, and sent them on to the final, separate
> panel of 15 or so judges to score and produce the final 50 ADC 1 Round 1
> award recipients.
> So in other words, we are currently in ADC 1 Round 1 Phase 2 as defined
> above. Once data from the judges comes in, we will notify the 50 award
> recipients and ADC 1 Round 2 will begin.
> It has not escaped my notice even on vacation that there have been a number
> of discussions on server hits and so on. Obviously we don't have access to
> everyone's server logs, and we can't monitor what the judges have actually
> been doing (nor would we snoop if we could, since that seems really
> sketchy.) We've tried to automate everything we possibly can about the
> judging process, but the one thing we can't automate is the actual act of
> assigning scores, since that requires a human's brain.
> The judges were given fairly detailed guidance on how to calibrate their
> scores, and what to review. For instance, they are aware that they are
> supposed to read documentation and do their best to test all the features.
> In the end, though, each judge is going to test to his or her own
> satisfaction. I'm not sure how reliable it is to correlate judge reviews
> with observed server hits. Some apps might have sporadic bugs that prevent
> network accesses. Some judges may have decided they didn't need to see a
> particular feature. And before you cry foul, know that some people who have
> inquired about "missing" server hits have actually done quite well. Judges
> are just as likely to say "this is cool, I don't need to see any more" as
> they are to say "this is so uncool, I don't need to see any more." On the
> whole, our judges have been excited to participate, and I expect that they
> are being as conscientious as they can be.
> The one thing I can tell you with certainty is that I have answered quite a
> few private inquiries, and in all but one case the judges responded with
> legitimate scores, rather than scores that say something went wrong or the
> review was incomplete. Our only data points are what the judges give us,
> because that's the only factor we can't automate. Since the judges are
> telling us that they reviewed to their satisfaction, we can only take their
> word for it.
> We've tried really hard to make sure that the only thing that affects
> scoring is what you put in front of the judges. But the entire goal of the
> ADC is to leverage plain old human judgment.
> - Dan
> P.S. - watch for gory details on the nuts & bolts of all this in the near
> future.
> On Sat, May 3, 2008 at 1:58 PM, Finn Kennedy <finn.kenn...@gmail.com> wrote:
> > Dan,
> > thank you for the responses. A couple of follow ups.
> > With Phase 2 I meant the 100 being winnowed down to 50. From the ADC
> > Judging Process page:
> > "In Phase 2, the 100 highest-scoring submissions will be all be sent to a
> > new panel of judges (which may or may not include one or more of the judges
> > who participated in Phase I judging).
> > ...
> > The 50 entries with the highest scores in Phase 2 judging will move on to
> > Round 2 of the Challenge..."
> > Just for clarification, are there again groups of judges assigned to look
> > at a subset of the top 100 entries? Or are the entire set of entries judged
> > by all the judges in Phase 2?
> > Is the "outlier" procedure still used for Phase 2? By outlier I mean the
> > review of scores not matching the rest of the scores for the application
> > (mentioned on the board).
> > I totally understand the want to have a fair playing field. It would not
> > be fair to extend advantages to winners that can make the trip to Google
> > I/O.
> > Finn
> > On Sat, May 3, 2008 at 3:27 PM, Dan Morrill <morri...@google.com> wrote:
> >> Well, for all intents and purposes, Phase II begins as soon as we announce
> >> the 50 Phase I winners. It's not like we could stop the winners from
> >> starting right away, anyway. :) It looks like we are still on track to
> >> announce those winners next week.
> >> A different set of judges is reviewing the 100 applications. The top 100
> >> applications are "reset" and rejudged from scratch by a different group of
> >> judges, who have no knowledge of the previous judges' scores.
> >> We're thinking about ways to work with the 50 Phase I winners, but that
> >> might not necessarily include anything formal at Google I/O. (We don't want
> >> to require anyone to attend, and we don't want to give any of them an unfair
> >> advantage.)
> >> - Dan
> >> On Fri, May 2, 2008 at 6:58 PM, finnk <finn.kenn...@gmail.com> wrote:
> >>> Since we are coming up on the week of May 5th, when is Phase II
> >>> starting?
> >>> Will the entire panel of judges review the whole set of 100
> >>> applications, or is the set of 100 split into groups and distributed
> >>> randomly again?
> >>> Are there any differences between Phase I and Phase II?
> >>> On an slightly related note, is there anything planned for the top 50
> >>> planned at Google IO?
> >>> Also, for Google IO: If you are traveling from Austin, TX, there are
> >>> direct flights from Austin to San Jose International. You can then
> >>> take CalTrain (http://www.caltrain.com) from close to the airport to
> >>> the San Francisco stop. It is on 4th, same street as the Moscone
> >>> Center.
> >>> Of course I am not a travel agent/planner, so please double check
> >>> everything yourself.
> Since the judges are telling us that they reviewed to their satisfaction, we can only take their
> word for it.
And herein lies the problem. You seem to be having a blind sport
there. From what I can see, and it seems to coincide with many other's
experiences: who ever ran the app seemed not even bothered with
finding the Menu button on the emulator.
> Wow! That is more information than I hoped for before going to sleep.
> Thanks a lot!
> Muthu,
> I cannot believe you were right. I'm really starting to believe that
> you have friends at Google. Once again, if you win, I'll be highly
> suspicious! :)
> Good luck and congratulations to those people that complained a lot
> about not seeing server hits. According to Dan some (not all) have
> done quite good. It appears that this Monday we may actually know who
> the top 50 most promising developers from around the world are. Wow, I
> can hardly wait for this Monday to come!
> On May 4, 12:22 am, "Dan Morrill" <morri...@google.com> wrote:
> > Ahh -- we've not been rigorous in consistently naming these various rounds
> > and phases. Let me try and adopt that terminology for this thread, and
> > explain again.
> > ADC 1 == this $5,000,000 prize event going on now.
> > ADC 2 == the second $5,000,000 prize event that will begin later this year.
> > ADC 1 Round 1 == open participation with the deadline of 14 April, with 50
> > winners
> > ADC 1 Round 2 == participation limited to the winners of ADC 1 Round 1, with
> > 20 "final" winners
> > ADC 1 Round 1 Phase 1 == reducing the original set of 1,788 submissions to
> > 100 finalists
> > ADC 1 Round 1 Phase 2 == picking the 50 ADC 1 Round 1 winners from the 100
> > finalists
> > Okay, phew. :) With those definitions, here is where we are:
> > - We sent out the submissions to judging a few days after the submission
> > deadline of 14 April, and judging began.
> > - Our 100 or so judges received the judging guidelines we provided,
> > reviewed their assigned submissions, and reported data back to us.
> > - Late last week, we applied our outlier mitigation techniques,
> > identified the top 100 results, and sent them on to the final, separate
> > panel of 15 or so judges to score and produce the final 50 ADC 1 Round 1
> > award recipients.
> > So in other words, we are currently in ADC 1 Round 1 Phase 2 as defined
> > above. Once data from the judges comes in, we will notify the 50 award
> > recipients and ADC 1 Round 2 will begin.
> > It has not escaped my notice even on vacation that there have been a number
> > of discussions on server hits and so on. Obviously we don't have access to
> > everyone's server logs, and we can't monitor what the judges have actually
> > been doing (nor would we snoop if we could, since that seems really
> > sketchy.) We've tried to automate everything we possibly can about the
> > judging process, but the one thing we can't automate is the actual act of
> > assigning scores, since that requires a human's brain.
> > The judges were given fairly detailed guidance on how to calibrate their
> > scores, and what to review. For instance, they are aware that they are
> > supposed to read documentation and do their best to test all the features.
> > In the end, though, each judge is going to test to his or her own
> > satisfaction. I'm not sure how reliable it is to correlate judge reviews
> > with observed server hits. Some apps might have sporadic bugs that prevent
> > network accesses. Some judges may have decided they didn't need to see a
> > particular feature. And before you cry foul, know that some people who have
> > inquired about "missing" server hits have actually done quite well. Judges
> > are just as likely to say "this is cool, I don't need to see any more" as
> > they are to say "this is so uncool, I don't need to see any more." On the
> > whole, our judges have been excited to participate, and I expect that they
> > are being as conscientious as they can be.
> > The one thing I can tell you with certainty is that I have answered quite a
> > few private inquiries, and in all but one case the judges responded with
> > legitimate scores, rather than scores that say something went wrong or the
> > review was incomplete. Our only data points are what the judges give us,
> > because that's the only factor we can't automate. Since the judges are
> > telling us that they reviewed to their satisfaction, we can only take their
> > word for it.
> > We've tried really hard to make sure that the only thing that affects
> > scoring is what you put in front of the judges. But the entire goal of the
> > ADC is to leverage plain old human judgment.
> > - Dan
> > P.S. - watch for gory details on the nuts & bolts of all this in the near
> > future.
> > On Sat, May 3, 2008 at 1:58 PM, Finn Kennedy <finn.kenn...@gmail.com> wrote:
> > > Dan,
> > > thank you for the responses. A couple of follow ups.
> > > With Phase 2 I meant the 100 being winnowed down to 50. From the ADC
> > > Judging Process page:
> > > "In Phase 2, the 100 highest-scoring submissions will be all be sent to a
> > > new panel of judges (which may or may not include one or more of the judges
> > > who participated in Phase I judging).
> > > ...
> > > The 50 entries with the highest scores in Phase 2 judging will move on to
> > > Round 2 of the Challenge..."
> > > Just for clarification, are there again groups of judges assigned to look
> > > at a subset of the top 100 entries? Or are the entire set of entries judged
> > > by all the judges in Phase 2?
> > > Is the "outlier" procedure still used for Phase 2? By outlier I mean the
> > > review of scores not matching the rest of the scores for the application
> > > (mentioned on the board).
> > > I totally understand the want to have a fair playing field. It would not
> > > be fair to extend advantages to winners that can make the trip to Google
> > > I/O.
> > > Finn
> > > On Sat, May 3, 2008 at 3:27 PM, Dan Morrill <morri...@google.com> wrote:
> > >> Well, for all intents and purposes, Phase II begins as soon as we announce
> > >> the 50 Phase I winners. It's not like we could stop the winners from
> > >> starting right away, anyway. :) It looks like we are still on track to
> > >> announce those winners next week.
> > >> A different set of judges is reviewing the 100 applications. The top 100
> > >> applications are "reset" and rejudged from scratch by a different group of
> > >> judges, who have no knowledge of the previous judges' scores.
> > >> We're thinking about ways to work with the 50 Phase I winners, but that
> > >> might not necessarily include anything formal at Google I/O. (We don't want
> > >> to require anyone to attend, and we don't want to give any of them an unfair
> > >> advantage.)
> > >> - Dan
> > >> On Fri, May 2, 2008 at 6:58 PM, finnk <finn.kenn...@gmail.com> wrote:
> > >>> Since we are coming up on the week of May 5th, when is Phase II
> > >>> starting?
> > >>> Will the entire panel of judges review the whole set of 100
> > >>> applications, or is the set of 100 split into groups and distributed
> > >>> randomly again?
> > >>> Are there any differences between Phase I and Phase II?
> > >>> On an slightly related note, is there anything planned for the top 50
> > >>> planned at Google IO?
> > >>> Also, for Google IO: If you are traveling from Austin, TX, there are
> > >>> direct flights from Austin to San Jose International. You can then
> > >>> take CalTrain (http://www.caltrain.com) from close to the airport to
> > >>> the San Francisco stop. It is on 4th, same street as the Moscone
> > >>> Center.
> > >>> Of course I am not a travel agent/planner, so please double check
> > >>> everything yourself.
thanks for enlightening us all,
You seem to be our only contact with google please understand our
anxiety.
For innovative and "crazy" ideas like my submission, I don't care
about server hits (but yes I was whinning). The only thing that we
hoped for, is that 4 judges read at least the documentation because of
the fear that one judge may not grasp the whole meaning.
I must say we wrote the documentation and had in mind that it will be
read loudly in a room of judges like a presentation...(this was false
though we didn't understand correctly the judging proccess).
aris
On 4 Μάϊος, 07:22, "Dan Morrill" <morri...@google.com> wrote:
> Ahh -- we've not been rigorous in consistently naming these various rounds
> and phases. Let me try and adopt that terminology for this thread, and
> explain again.
> ADC 1 == this $5,000,000 prize event going on now.
> ADC 2 == the second $5,000,000 prize event that will begin later this year.
> ADC 1 Round 1 == open participation with the deadline of 14 April, with 50
> winners
> ADC 1 Round 2 == participation limited to the winners of ADC 1 Round 1, with
> 20 "final" winners
> ADC 1 Round 1 Phase 1 == reducing the original set of 1,788 submissions to
> 100 finalists
> ADC 1 Round 1 Phase 2 == picking the 50 ADC 1 Round 1 winners from the 100
> finalists
> Okay, phew. :) With those definitions, here is where we are:
> - We sent out the submissions to judging a few days after the submission
> deadline of 14 April, and judging began.
> - Our 100 or so judges received the judging guidelines we provided,
> reviewed their assigned submissions, and reported data back to us.
> - Late last week, we applied our outlier mitigation techniques,
> identified the top 100 results, and sent them on to the final, separate
> panel of 15 or so judges to score and produce the final 50 ADC 1 Round 1
> award recipients.
> So in other words, we are currently in ADC 1 Round 1 Phase 2 as defined
> above. Once data from the judges comes in, we will notify the 50 award
> recipients and ADC 1 Round 2 will begin.
> It has not escaped my notice even on vacation that there have been a number
> of discussions on server hits and so on. Obviously we don't have access to
> everyone's server logs, and we can't monitor what the judges have actually
> been doing (nor would we snoop if we could, since that seems really
> sketchy.) We've tried to automate everything we possibly can about the
> judging process, but the one thing we can't automate is the actual act of
> assigning scores, since that requires a human's brain.
> The judges were given fairly detailed guidance on how to calibrate their
> scores, and what to review. For instance, they are aware that they are
> supposed to read documentation and do their best to test all the features.
> In the end, though, each judge is going to test to his or her own
> satisfaction. I'm not sure how reliable it is to correlate judge reviews
> with observed server hits. Some apps might have sporadic bugs that prevent
> network accesses. Some judges may have decided they didn't need to see a
> particular feature. And before you cry foul, know that some people who have
> inquired about "missing" server hits have actually done quite well. Judges
> are just as likely to say "this is cool, I don't need to see any more" as
> they are to say "this is so uncool, I don't need to see any more." On the
> whole, our judges have been excited to participate, and I expect that they
> are being as conscientious as they can be.
> The one thing I can tell you with certainty is that I have answered quite a
> few private inquiries, and in all but one case the judges responded with
> legitimate scores, rather than scores that say something went wrong or the
> review was incomplete. Our only data points are what the judges give us,
> because that's the only factor we can't automate. Since the judges are
> telling us that they reviewed to their satisfaction, we can only take their
> word for it.
> We've tried really hard to make sure that the only thing that affects
> scoring is what you put in front of the judges. But the entire goal of the
> ADC is to leverage plain old human judgment.
> - Dan
> P.S. - watch for gory details on the nuts & bolts of all this in the near
> future.
> On Sat, May 3, 2008 at 1:58 PM, Finn Kennedy <finn.kenn...@gmail.com> wrote:
> > Dan,
> > thank you for the responses. A couple of follow ups.
> > With Phase 2 I meant the 100 being winnowed down to 50. From the ADC
> > Judging Process page:
> > "In Phase 2, the 100 highest-scoring submissions will be all be sent to a
> > new panel of judges (which may or may not include one or more of the judges
> > who participated in Phase I judging).
> > ...
> > The 50 entries with the highest scores in Phase 2 judging will move on to
> > Round 2 of the Challenge..."
> > Just for clarification, are there again groups of judges assigned to look
> > at a subset of the top 100 entries? Or are the entire set of entries judged
> > by all the judges in Phase 2?
> > Is the "outlier" procedure still used for Phase 2? By outlier I mean the
> > review of scores not matching the rest of the scores for the application
> > (mentioned on the board).
> > I totally understand the want to have a fair playing field. It would not
> > be fair to extend advantages to winners that can make the trip to Google
> > I/O.
> > Finn
> > On Sat, May 3, 2008 at 3:27 PM, Dan Morrill <morri...@google.com> wrote:
> >> Well, for all intents and purposes, Phase II begins as soon as we announce
> >> the 50 Phase I winners. It's not like we could stop the winners from
> >> starting right away, anyway. :) It looks like we are still on track to
> >> announce those winners next week.
> >> A different set of judges is reviewing the 100 applications. The top 100
> >> applications are "reset" and rejudged from scratch by a different group of
> >> judges, who have no knowledge of the previous judges' scores.
> >> We're thinking about ways to work with the 50 Phase I winners, but that
> >> might not necessarily include anything formal at Google I/O. (We don't want
> >> to require anyone to attend, and we don't want to give any of them an unfair
> >> advantage.)
> >> - Dan
> >> On Fri, May 2, 2008 at 6:58 PM, finnk <finn.kenn...@gmail.com> wrote:
> >>> Since we are coming up on the week of May 5th, when is Phase II
> >>> starting?
> >>> Will the entire panel of judges review the whole set of 100
> >>> applications, or is the set of 100 split into groups and distributed
> >>> randomly again?
> >>> Are there any differences between Phase I and Phase II?
> >>> On an slightly related note, is there anything planned for the top 50
> >>> planned at Google IO?
> >>> Also, for Google IO: If you are traveling from Austin, TX, there are
> >>> direct flights from Austin to San Jose International. You can then
> >>> take CalTrain (http://www.caltrain.com) from close to the airport to
> >>> the San Francisco stop. It is on 4th, same street as the Moscone
> >>> Center.
> >>> Of course I am not a travel agent/planner, so please double check
> >>> everything yourself.
