Judging questions

1 view
Skip to first unread message

finnk

unread,
Apr 24, 2008, 3:15:12 PM4/24/08
to Android Challenge
1. How many judges on each panel?

2. Will there be one or more Google staff on each panel?

3. How much time will a judge spend reviewing each application?

4. Will there be more than one test session? (for the first phase)

5. Do the judges collaborate in scoring?

6. Are judges allowed to contact a contestant?

The reason I ask these questions is my entry is a client server
application and from the server logs it does not look like each judge
reviewed every feature, nor read the documentation. Maybe I went a
little far on the documentation, it is 14 pages, but it does include a
quick start guide. The instructions are hopefully very clear with the
in application icons next to each section. The application also
includes toast popups (can be turned off in preferences) for each
activity giving specific instructions.

However, there is one important instruction (repeated in multiple
places), without which the testing experience would suffer:

Center DPad to Pause/Resume moving along the LBS track. (Hoping maybe
judge will read this :))

I spent a lot of time making sure all the features in the application
were functional. I wrote a complete system for simulating LBS, moving
along subdivided tracks that would makes sense for testing/demoing the
application (using data from Google Earth). This was done after
reading that there might not be "adb push" on the groups, however, the
new system was superior so I do not regret doing it. (If you read this
Dan Morrill, thank you for pushing me in that direction, whether or
not I misunderstood what you meant it turned out better. I blame any
misunderstanding on having too much to do and not enough time to
sleep, it was crazy the last month.)

As it is TeeDroid Caddy is ready for phones, it is feature complete
and field tested. You can check out details on: http://www.teedroid.com.
The site is a work in progress, just something I put together in the
days since the deadline closed to show off the system.

Finn

http://www.teedroid.com

Steven A.

unread,
Apr 24, 2008, 3:27:04 PM4/24/08
to Android Challenge
The only information I have seen reported on this was from a post by
Dan a week ago.

It stated:
"We're providing rather detailed judging guidelines
intended to help judges calibrate their scores. We also plan to
post-process the results looking for outliers (such as cases where one
judge
gave a score that varies significantly from the others') to help make
sure
the results are fair.


Panel size will be at least 3, depending on how many judges we have;
currently the panel size is expected to be 4."

Source: http://groups.google.com/group/android-challenge/browse_thread/thread/9a91f426422fed32/9b31c4cae6f2507a#9b31c4cae6f2507a

ConAim

unread,
Apr 24, 2008, 4:27:20 PM4/24/08
to Android Challenge

I have the same problem.
I look at the server log (which does log every activity came from the
clients). But then all I see is, the judges make the connection and
click on the Ready checkbox and then close it. When I staged clearly
on the documentation that this online poker game need at least 2
players join and ready to initiate the game. I'm not sure if the
Judges does read the documents. :(

Muthu Ramadoss

unread,
Apr 24, 2008, 11:55:57 PM4/24/08
to android-...@googlegroups.com
Any application that would require the judges to read the docs before running them is clearly going to be at a disadvantage.

Why?

The judges are all alone sitting with their laptops, ready to test. Imagine yourself downloading an apk from the net. It comes with a readme, fairenough. But now imagine how many things might go wrong, if the application requires you to read the docs fully and understand it and run the application all by yourself.

I'm not saying this is rocket science, but all am saying is there's a lot of variables that can screw up the experience.

Hope the judges do take some time, completely understand the "readme" and then give their best in fairly testing and judging the applications. I hope the judges are not giving it up after their first try, since there's countless number of quality hours spend by the hard working android developers all over the world. They deserve better.

Good luck on all your submissions.
--
take care,
Muthu Ramadoss.

http://mobeegal.in
find stuff closer.

Chris

unread,
Apr 25, 2008, 12:38:41 AM4/25/08
to Android Challenge
Believe it or not..we had a judge somehow look at the read me before
looking at our app. We have an Android client (obviously), and a web
console to trigger server events. We got hits on the web console
first!

I really hope this doesn't mean they couldn't connect on the
console.....

On Apr 24, 8:55 pm, "Muthu Ramadoss" <muthu.ramad...@gmail.com> wrote:
> Any application that would require the judges to read the docs before
> running them is clearly going to be at a disadvantage.
>
> Why?
>
> The judges are all alone sitting with their laptops, ready to test. Imagine
> yourself downloading an apk from the net. It comes with a readme,
> fairenough. But now imagine how many things might go wrong, if the
> application requires you to read the docs fully and understand it and run
> the application all by yourself.
>
> I'm not saying this is rocket science, but all am saying is there's a lot of
> variables that can screw up the experience.
>
> Hope the judges do take some time, completely understand the "readme" and
> then give their best in fairly testing and judging the applications. I hope
> the judges are not giving it up after their first try, since there's
> countless number of quality hours spend by the hard working android
> developers all over the world. They deserve better.
>
> Good luck on all your submissions.
>

Muthu Ramadoss

unread,
Apr 25, 2008, 1:55:50 AM4/25/08
to android-...@googlegroups.com
The general feedback is that the judges are in fact looking at the readme and the related website if provided.

This is great news.

Incognito

unread,
Apr 25, 2008, 10:03:00 AM4/25/08
to Android Challenge
>>Hope the judges do take some time, completely understand the "readme" and
>>then give their best in fairly testing and judging the applications. I hope
>>the judges are not giving it up after their first try, since there's
>>countless number of quality hours spend by the hard working android
>>developers all over the world. They deserve better.

Agree with you, however, let me play devil's advocate, if your
application takes too long to understand( i.e. a couple of hours as
opposed to just a few minutes) then you will most likely loose a lot
of points, in fact, I'll doubt you will win. Why? Because these are
mobil apps, and mobil apps are supposed to be simple to understand and
use, this is the case even for desktop applications. Even if your
application is doing something really complicated behind scenes it
should still be really easy for a user to get up and running within a
couple of minutes. So if a user cannot get the gist of what your
application does and how to use it after reading just a couple of
paragraphs from your manual or Read Me file then you've probably
already lost.

On Apr 24, 11:55 pm, "Muthu Ramadoss" <muthu.ramad...@gmail.com>
wrote:
> Any application that would require the judges to read the docs before
> running them is clearly going to be at a disadvantage.
>
> Why?
>
> The judges are all alone sitting with their laptops, ready to test. Imagine
> yourself downloading an apk from the net. It comes with a readme,
> fairenough. But now imagine how many things might go wrong, if the
> application requires you to read the docs fully and understand it and run
> the application all by yourself.
>
> I'm not saying this is rocket science, but all am saying is there's a lot of
> variables that can screw up the experience.
>
> Hope the judges do take some time, completely understand the "readme" and
> then give their best in fairly testing and judging the applications. I hope
> the judges are not giving it up after their first try, since there's
> countless number of quality hours spend by the hard working android
> developers all over the world. They deserve better.
>
> Good luck on all your submissions.
>

Muthu Ramadoss

unread,
Apr 25, 2008, 10:05:57 AM4/25/08
to android-...@googlegroups.com
Incognito,

Agreed. You are talking sense for a change ;)

Finn Kennedy

unread,
Apr 25, 2008, 10:27:46 AM4/25/08
to android-...@googlegroups.com
It is not that the application is complicated, it is the testing that might be confusing, as the default mock providers weren't very useful for a golf application.  I provided a nice less than one page quick start, hoping that the judges will take their time to look through it thoroughly.  Plus there are the slightly annoying tutorial popups (can be turned off though).

And since I only started on the website after the deadline had past, and I did not have it up until this week I really hope the judge(s) will make a second visit.

Finn
http://www.teedroid.com

Incognito

unread,
Apr 25, 2008, 10:59:06 AM4/25/08
to Android Challenge
>>Incognito,


>>Agreed. You are talking sense for a change ;)

You are making me blush! :)

On Apr 25, 10:05 am, "Muthu Ramadoss" <muthu.ramad...@gmail.com>
wrote:
> find stuff closer.- Hide quoted text -
>
> - Show quoted text -

Muthu Ramadoss

unread,
Apr 25, 2008, 11:11:23 AM4/25/08
to android-...@googlegroups.com
haha.. that was the whole point :)

Dan Morrill

unread,
Apr 25, 2008, 9:57:05 PM4/25/08
to android-...@googlegroups.com
Great questions!  Here are some answers.

On Thu, Apr 24, 2008 at 12:15 PM, finnk <finn.k...@gmail.com> wrote:
1. How many judges on each panel?

"Panel" can mean multiple things, so the most accurate way to phrase it is that each application will be reviewed by at least 4 judges.
 
2. Will there be one or more Google staff on each panel?

Not necessarily.  Applications are assigned randomly to judges.  Since there are many more representatives from other Alliance companies than there are from Google alone, most applications will not have a Google employee as a judge.
 
3. How much time will a judge spend reviewing each application?

This depends on the judge.  We expect judges to spend as much time reviewing each app as it takes to satisfy themselves that they understand it well enough to fairly score it.
 
4. Will there be more than one test session? (for the first phase)

I am not sure what you mean by a "test session", but each judge evaluates the application once, and scores it based on that experience.
 
5. Do the judges collaborate in scoring?

Definitely not.  Each judge does his or her reviews alone, and we will aggregate their scores once all reviews are in.
 
6. Are judges allowed to contact a contestant?

I don't know if there's a specific prohibition against it, but I expect not.  The judges judge whatever .apk file was submitted, in a clean emulator.  It's reviewed on its merits, meaning that they review it based on whatever appears in front of them when they start the app, good or bad.
 
- Dan

Muthu Ramadoss

unread,
Apr 25, 2008, 11:31:30 PM4/25/08
to android-...@googlegroups.com
Dan,

Thanks for the answers. It helps us keeping sane.

When is the next SDK, btw?

Hielko

unread,
Apr 26, 2008, 6:51:36 AM4/26/08
to Android Challenge
Quote: I don't know if there's a specific prohibition against it, but
I expect not.
The judges judge whatever .apk file was submitted, in a clean
emulator.
It's reviewed on its merits, meaning that they review it based on
whatever
appears in front of them when they start the app, good or bad.

How will applications be judged that require multiple users online to
do something usefull? For example all GTalk based applications, must
social networking apps, multiplayer games etc?


On Apr 26, 5:31 am, "Muthu Ramadoss" <muthu.ramad...@gmail.com> wrote:
> Dan,
>
> Thanks for the answers. It helps us keeping sane.
>
> When is the next SDK, btw?
>
>
>
>
>
> On Sat, Apr 26, 2008 at 7:27 AM, Dan Morrill <morri...@google.com> wrote:
> > Great questions!  Here are some answers.
>

Dan Morrill

unread,
Apr 26, 2008, 4:38:18 PM4/26/08
to android-...@googlegroups.com
The judging system has a way to start multiple emulators so that judges can simulate two users. 

- Dan

Hielko

unread,
Apr 26, 2008, 5:01:35 PM4/26/08
to Android Challenge
I hope these emulators are not started on the same pc, since there is
a bug in the location service that prevents our application from
working properly when two emulators are running on the same pc (the
second emulator will always block when we request the current
location). I have no idea how this is possible, but there seems to be
a issue with the mock location provider when multiple threads and
multiple simulators are used on the same system. Luckily we make most
of our calls to the location provider in a seperate thread so it does
not always block our GUI, but it limits the functionality.

We have tested this on Windows XP/Vista, but the bug probably also
exists in the Linux version.

If you want to reproduce this bug you can simply start two emulators
on one pc, load our application (Snap! - www.snap-project.net) on both
emulators and check the logcat of the second emulator. See also issue
639 on the Issue Tracker.

(this is also documented in the readme of our application, but not in
depth)
> > > - Show quoted text -- Tekst uit oorspronkelijk bericht niet weergeven -
>
> - Tekst uit oorspronkelijk bericht weergeven -

jim.renkel

unread,
Apr 26, 2008, 8:45:35 PM4/26/08
to Android Challenge
My application also requires two communicating emulators, and because
of known and documented bugs in the emulator and SDK, the two
emulators cannot be on the same host.

I was told to point this out in the readme.pdf that accompanied the
submission, which I did, and the application would not be marked down
because of deficiencies in the emulator / SDK.

But I don't know how they will be able to do that because without
running the application in two emulators on different PCs, how will
the judges know what to not mark down from?

I included a very detailed test script with many screen shots. Maybe
they'll trust that that is the way the application works and grade it
based on that? I would like that, but I can see where some would think
that unfair, or something.

No hits on my servers yet, but then there wouldn't be until the
applications in the two emulators tried to talk to each other.

Good luck to all!

Jim Renkel

On Apr 26, 4:01 pm, Hielko <hie...@tweakers.net> wrote:
> I hope these emulators are not started on the same pc, since there is
> a bug in the location service that prevents our application from
> working properly when two emulators are running on the same pc (the
> second emulator will always block when we request the current
> location). I have no idea how this is possible, but there seems to be
> a issue with the mock location provider when multiple threads and
> multiple simulators are used on the same system. Luckily we make most
> of our calls to the location provider in a seperate thread so it does
> not always block our GUI, but it limits the functionality.
>
> We have tested this on Windows XP/Vista, but the bug probably also
> exists in the Linux version.
>
> If you want to reproduce this bug you can simply start two emulators
> on one pc, load our application (Snap! -www.snap-project.net) on both

Kevin

unread,
Apr 26, 2008, 10:44:33 AM4/26/08
to Android Challenge
I've been watching my logs, and I want to contact somebody about it.
Not sure what to do. I have a user signup process that doesn't
require reading documents, but you sort of have to finish it. I have
a bug. The user menu becomes active before the setup is done.
Originally, you couldn't click around the main app until the user
setup was complete. However, I changed the app towards the end to do
the menus in a common subclass, so the user setup procedure starts
making the menu available earlier than it should be.

So. Two judges have logged in. The first one got about a screen into
user setup, then stopped completely. That was frustrating. That guy
just stopped. The second one also didn't finish the setup, but then
clicked around the app for a while. At least that one got to see it.
Neither did any data submission (which is what the app does), and
neither logged into the exiting user I set up for the purpose of
evaluation and put in the docs I sent.

Most of these apps are going to be very early stage. Its also a new
type of platform for most of us (I'd bet 90%+ come from a web
programming background). What that boils down to is interface design
that might not be as clear as possible. Right? Its easy to send a
link to a buddy to take a look. Not so easy to send an apk file and
an emulator, then explain what a command line is (most of the people I
have look at stuff are not computer people, which is very helpful).
In my case, the screen that these guys are dropping out on has 1
button, which actually says "Next". Yet neither clicked it. Thinking
back, I should make it red or something, maybe animate it around, huge
flashing arrow pointing at it. Whatever. But, come on. It says
"Next". Its the only button. Top/right of the screen.

There are 1788 submissions. I don't know how many judges there are.
So that's a lot of work. But if you figured out how much work we all
put into the submissions, a total of a couple minutes is not
sufficient. At least, you know, if "Next" is confusing, have somebody
send me an email. I'll explain it.

Lets imagine my app is out of the contest now, and I know from the
logs that the 2 judges I got were really phoning it in (pun intended.
By that I mean they're barely trying). I'm going to have a real
problem with that. This whole 'good will' exercise would have the
opposite effect. If they gave my app a full look, and then rejected
it, well hey. Don't play if you can't handle losing. But without
really evaluating it?

Anybody else have a similar situation? The tough love answer is, "Hey
man, if your app isn't getting these guys to want to use it, maybe its
not good". It isn't good. Its alpha quality software. Barely.
Right? Now, to be clear, I switched the app from a different one
earlier. I had a lot less time than if I'd started with this one in
the beginning. That was my decision, but still. Its a new platform.
Android is not without its own bugs. I sort of assumed "proof of
concept" was what they were looking for anyway.

You know what they say about assumptions.

Incognito

unread,
Apr 27, 2008, 11:40:39 AM4/27/08
to Android Challenge
>"Hey
>man, if your app isn't getting these guys to want to use it, maybe its
>not good".

I'm glad you said it yourself, I was thinking the same thing but
didn't want to sound cynical. Crying and whining will not help your
case man and will only annoy everybody . There are 1700+ apps and you
can bet that 50 of those will be much better than the rest of the
other apps (including yours and mine). i.e., it will wow the judge on
first impression, it will not have a bug with the menu like yours, it
will work without a glitch, and the concept will be a good one.

Truth is, a user or judge will have made a decision about your app on
the first impression and if your App has a bug in the menu and or
simply does not seem appealing than you are screwed.
> > Judges does read the documents. :(- Hide quoted text -

Kevin Galligan

unread,
Apr 27, 2008, 12:08:54 PM4/27/08
to android-...@googlegroups.com
Yeah. I was frustrated when I wrote that one. I assume its a very
long shot to get into the top 50, but the first hit on the server was
like the person opened it and closed it. Your app would have to be
horrible, or have an error, for that to be the end of the review. You
know what I mean?

I heard back the short ones are spot checks. So far just one actual review.

I've seen in various places that the reviews would be done by May 5.
That's like a week. Guess this week is going to be interesting.

Reply all
Reply to author
Forward
0 new messages