Application Judging

1 view
Skip to first unread message

tomgibara

unread,
May 1, 2008, 4:58:36 AM5/1/08
to Android Challenge
I scanned a log of the #android IRC channel this morning and now I'm
confused (concerned?) by the judging process. Dan Morrill said that
judging had progressed to halving the final 100 applications to 50.

I'm not logging much information on the server side of my Moseycode
application, but I do know how many devices (emulators) have
authenticated with the demo account I provided for testing: 2. But as
far as I know, each application is supposed to be judged by 4 judges.

The server hosting my application has failed twice in the last month
(for about 3hrs each time) which I'm upset about, but there's nothing
I can do (of course it hadn't failed for at least 6 weeks _before_ the
deadline). If a judge had attempted to use the application during one
of these periods they would have received a red message box warning
them of the problem on the application's home screen (this was also
explained in the supplied readme). It seems very unlikely (and
unlucky) that two judges would have tested the application at these
times; even more unfortunate if they are given an explicit warning
that there was a problem connecting to the server but chose to
disregard it.

What troubles me more is that the part of the application that is
usable without a camera feed, barcode publishing, does not appear to
have been tested either, even though it's a very accessible part of
the application. One judge published one barcode. Is that the testing
that an application which took 6 months to write merits?

Since my application requires interactive use of the camera, I was
resigned to judges not actually being able to use the scanning part of
the application without setting-up a camera (and I know from my logs
that the judges certainly did not scan any barcodes) so I made a video
of that, but I did expect judges to fully explore the other elements
of the application.

My Moseycode application is being developed with the goal of fully
realizing a new barcode system, not just as an entry into the ADC.
Ever since explanations about the judging process were forthcoming I
always felt that it probably wouldn't do well in the challenge because
of its reliance on a real camera feed. But my concern is more general
than that: what degree of testing did all the other applications
receive?

Of course this is all conjecture, perhaps my Moseycode application was
too buggy, or perhaps two judges did try to use the application while
the server was down, but I am disappointed by Moseycode's ignominious
evaluation.

living_sword

unread,
May 1, 2008, 5:26:01 AM5/1/08
to Android Challenge
take a look at this thread -> http://groups.google.com/group/android-challenge/t/30b3e76aca29cca4
. that should satisfy ur general concern.

Incognito

unread,
May 1, 2008, 5:34:16 AM5/1/08
to Android Challenge

> The server hosting my application has failed twice in the last month
> (for about 3hrs each time) which I'm upset about, but there's nothing
> I can do (of course it hadn't failed for at least 6 weeks _before_ the
> deadline). If a judge had attempted to use the application during one
> of these periods they would have received a red message box warning
> them of the problem on the application's home screen (this was also
> explained in the supplied readme). It seems very unlikely (and
> unlucky) that two judges would have tested the application at these
> times; even more unfortunate if they are given an explicit warning
> that there was a problem connecting to the server but chose to
> disregard it.

Well, how do you know if wasn't down more than 3 hrs each time?

> What troubles me more is that the part of the application that is
> usable without a camera feed, barcode publishing, does not appear to
> have been tested either, even though it's a very accessible part of
> the application. One judge published one barcode. Is that the testing
> that an application which took 6 months to write merits?


Judges are not testers. i.e. The judge will not go through every
single feature to verify that it works. This is what I think is really
happening. First they read the first pages of your manual, or at least
the introduction (or watch a video demo if you have one) to understand
what your application is about. Based on this they'll know how to
rate it for the inovation criteria. Second, they may open it just to
check out the GUI. If they really like it they may explore more of
your app. However, if all it's doing is just scanning a bar code then
they believe you that it does that and they don't verify it
themselves. Or it could just be that your application failed and it
didn't allow the judge to continue. Only way to know for sure is to
ask the judges.

If you provided a video it could be that they are relying heavily on
that do rate your application and just opening your app for a minute
or two to verify that your app opens. I know I would if I had 76
applications to judge and running out of time.

tomgibara

unread,
May 1, 2008, 6:18:42 AM5/1/08
to Android Challenge
A monitoring script notifies me of any disruption and logs on the
server leave a precise record - 3hrs is actually a rounded up.

Your last comment about the video is pretty much my concern, which as
I stated is not for my application but for others: what about the
entrants who produced capable applications, but no videos? I spent
time doing both.

I'm concerned that entrants who committed all of their time to
producing a good application and not a highly polished readme with
supporting videos etc. might be unfairly disadvantaged. Both should
certainly get credit, but if time constraints mean that judges just
read documentation and don't use the application much, then what is
being judged is the presentation of ideas, whereas my understanding
was that Google's staff were looking for good applications.

Kevin Galligan

unread,
May 1, 2008, 8:33:00 AM5/1/08
to android-...@googlegroups.com
I have the same issue. I've had two people look at my app. One for a
very short period. I do not know of my server ever being down.

From the info I had, from Dan and general info, one of them was almost
certainly a spot check. However, every day it seems less likely that
somebody is going to stop by.

q2dm1

unread,
May 4, 2008, 8:30:32 AM5/4/08
to Android Challenge
I have a backup server, so it is very unlikey they both failed at the
same time. That said, I had 2 spot checks or whatever in total (<2
minutes each). Don't think I'd waste my time on ADC2.

On May 1, 2:33 pm, "Kevin Galligan" <kgalli...@gmail.com> wrote:
> I have the same issue.  I've had two people look at my app.  One for a
> very short period.  I do not know of my server ever being down.
>
> From the info I had, from Dan and general info, one of them was almost
> certainly a spot check.  However, every day it seems less likely that
> somebody is going to stop by.
>

tomgibara

unread,
May 4, 2008, 11:47:22 AM5/4/08
to Android Challenge
I certainly don't think I've wasted my time by entering the ADC, even
though I currently think it's unlikely that I will win a final 50
place. I have always been interested in the potential of the Android
platform and not primarily in the challenge (which I have always
believed Google to have run in good faith).

The concerns I expressed previously arose because I failed to
understand that judging was more evenly weighted between the
application and its readme, than I had previously thought. This has
since been clarified by Dan Morrill in posts to this forum. If I have
any criticisms of the challenge, they are minor, but one would be that
this wasn't clear prior to submission - I certainly remain positive
about the opportunities that Google has already provided for Android
developers via this challenge.

Of course, the extent to which preparing a submission to the ADC (or
ADC II) deviates from ones development roadmap will determine the
amount of 'additional' (and possibly wasted) work that the submission
will require. This will naturally vary between applications, but I for
one certainly won't automatically disregard entering the ADC II.

Tom.

phil

unread,
May 4, 2008, 12:03:13 PM5/4/08
to android-...@googlegroups.com
I am not saying anything about the platform, my main criticism is that
people where deceived about the judging process. If they were going to
judge an application by the readme alone, they should've said so and spare
us the hassle of deadlines and all that stress. On the contrary, it was
very clearly stated that working and feature-complete applications will be
preffered.

I'm not complaining because I'm not gonna win (maybe I will, who knows, my
readme is by no means bad), but because of all the time I used to develop
a working application NOW (instead of when the first handsets come out and
there's a stable and working SDK) when in fact a pdf and a login screen
would have sufficed.

--
Using Opera's revolutionary e-mail client: http://www.opera.com/mail/

Dan Morrill

unread,
May 4, 2008, 12:43:21 PM5/4/08
to android-...@googlegroups.com
Well, that's a pretty extreme overstatement.  Nobody is judging applications "by the readme alone".  However, judges are naturally going to judge everything you put in front of them. For example, if the application doesn't run or crashes on startup, judges explicitly mark it as such, even if the README is a work of art.

On the other hand, if somebody provided a really good README file and a judge tries out a few functions of the app which are also very good, it seems perfectly reasonable to me that they're going to give the rest the benefit of the doubt and move on to the next app.  In both cases, someone operating a server would see an "incomplete" session.

As we've said all along, the ADC is about cultivating great ideas and assessing potential, not quality assurance.  Judges aren't sitting there with a checklist like a driving instructor, taking a point off every time you forgot to signal a lane change.

- Dan

phil

unread,
May 4, 2008, 1:05:21 PM5/4/08
to android-...@googlegroups.com
I know what you mean but I'm not talking about incomplete sessions, I'm
talking about no sessions at all. If your statement that every application
is going to be tested by 4 judges is true, than I'm not the only one whose
application just wasn't started 4 times. In this case I can only deduce
that:

a. many applications, including mine, crash ~50% of the time before
sending a packet to the server
b. ~50% of the time the applications aren't tested at all

Given that my application didn't crash once before showing the login
screen I'm pretty sure the former is not the case. It's true that
unforeseen things may happen but it'd be a real coincidence if the latter
was true, given the numerous reports in this group.

javqui

unread,
May 4, 2008, 1:41:20 PM5/4/08
to Android Challenge
After read the “terms and conditions”, the FAQ and the judging
process, I was searching for a middle point between protection and
innovation release before submit the readme and apk documents. I
sacrifice “polish and appeal” points (with a deliberated limited
android apk demo) and some non relevant “Originality” points.
It was my play.
I not sure if the judges evaluate the full application or not. I’m not
sure if I have any chance to win with android. But I’m sure that with
the limited information submitted, every judge has enough information
to take a decision if my ideas and work has a chance or not. Judges
did not need tons of information and did not need test the full
application for hours. Judges only need experience, common sense and
future vision, and I’m pretty sure that Google and android team select
the judges according.

A favorable consideration in the challenge will trigger important
events for us and for many other participants. From our point of view,
a favorable consideration is a position on the 50 or in the worst
case, a constructive feedback for our concepts if we are not in the
50.

Remember the two worlds from matrix, (the red and the blue pill). This
is the real outside world.

I will take the opportunity to say “thank you” for this experience to
everybody involved on this challenge, including the people of android-
developers discussion group.

Muthu Ramadoss

unread,
May 5, 2008, 6:42:46 AM5/5/08
to Android Challenge
Well said. Hats off to you.
Reply all
Reply to author
Forward
0 new messages