RMRC update + advice for next year's Qualification TMPs
33 views
Skip to first unread message
Raymond Sheh
Nov 3, 2020, 2:45:28 PM11/3/20
to The Open Academic Robot Kit
Hi All!
I hope you, your families, friends, classmates, teachers, and
colleagues are all staying safe in your parts of the world. I hear
some of you are getting back to normal while others are still facing
challenges. All the best to all of you in your studies, teaching, and
activities!
It has been a few months since we released the opening status survey
and we received very few responses so far, certainly not enough to
make decisions on. We're assuming that this is because, for most of
you, things are still too uncertain to meaningfully answer.
We'll keep the survey up so as things change and become more certain,
please do respond. If you've responded already and things have
changed, feel free to write a new response, keeping the same team
name.
This also means that we won't be making a decision on a virtual RMRC
until sometime next year. In the meantime, those of you who are in
locations where lockdowns are lifting and are able to resume work on
your robots, please do feel free to post updates, pictures, and videos
of your robots performing over the standard tests. We can still share
what we've done, and practice remotely scoring and doing the various
parts of a virtual RMRC in an ad-hoc manner, without making it a full
competition just yet.
I have also sent TDM feedback to teams who didn't pre-qualify for next
year. Note that because this year's RoboCup was cancelled prior to
pre-qualification announcements, we set the pre-qualification bar very
high with the goal of keeping as many spots available as possible for
next year, to avoid disadvantaging teams who might have been able to
turn in a good entry but for early by COVID-related disruptions. We
usually pre-qualify around 10-15 teams, depending on anticipated team
size and local capacity. This year we pre-qualified only 7 teams,
based either on being finalists or from their Qualification Team
Description Materials (Qualification TDMs) achieving almost perfect
scores. In normal times, an additional 3-8 teams would have qualified
based on the Qualification TDM.
I also have some more general feedback and will be updating the
guidance on the website (currently
http://oarkit.intelligentrobots.org/home/competitions/robocuprescue-rmrl/world-championship-2018/2018-robocuprescue-robot-league-rapidly-manufactured-robot-challenge-opening-announcement/
) to reflect this. Note that this feedback is general in nature, a lot
of the Qualification TDMs received did not have these issues so they
may not all apply to you.
In reading the Qualification TDMs, I think the biggest issue that
lower scoring teams had was losing sight of the purpose of the
Qualification TDM. The purpose of the Qualification TDM is to convince
us that you know what you're doing and you have a good chance of doing
well in the competition, with a secondary purpose of being a reference
for other teams. The qualification decision is based on a ranking of
all teams, using this criteria.
The list on the website is a list of points that we are focusing on,
and a guide to structuring your report. It seems like some teams are
treating the list as a checklist, whereby as long as they have a
section with that name, and the section has some text in it that is
vaguely related to the name, they check the box. Such an approach is
not going to result in a particularly high scoring Qualification TDM,
compared to teams who used each of those headings to inspire a section
that discussed how they have, or plan to, tackle that aspect of their
entry.
Describing why you did, or didn't, make a particular choice, and how
it fits into your system, and even things you encountered that didn't
work, as some teams did, does show us that you made a choice based on
a proper understanding. Describe what it is about your design that
means that those specifications on the servo are appropriate. Describe
how you're using Python and C++, where you're using one and where
you're using the other, and more importantly, why. Describe how the
Raspberry Pi and Arduino are connected and what each one does and how
they complement each other. When we say "describe", we are giving you
the opportunity to describe your entry, which is targeted at a
particular application and capability, not just the individual
components. Ideally, the resulting document should also be useful to
other teams, although that is more the focus of the Updated TDM, to be
presented closer to the competition.
A lot of the lower scoring Qualification TDMs also spent a lot of
words summarising the goals of the competition, why rescue robotics is
a good thing to get into and why do 3D printing. This is all
interesting information but it doesn't really tell us how well you are
likely to do.
Statements to the effect that the team is highly motivated and that
the mentor is well qualified are also interesting, but we can only
score your demonstration of this through a comprehensive and well
researched description of the different components of your system and
how they come together. The exception is discussion about prior
performance in this competition (or highly related competitions),
which is useful as part of a wider description of how the team has
changed in the meantime.
Mission strategy, experiments and testing, and strengths of the team
were also severely lacking in a lot of lower scoring TDMs. Some of you
may find it easier to describe your strengths further up, in the
system description section, such as "We chose this wheel design to
give us a particular strength in the stepfields, due to ...", we would
accept that as well (although we would also expect to see some
justification, ideally in the form of test results, even if basic).
Even new teams have something that is being focused on so every team
should be able to say something here. Again, we're talking about
strengths in the approach, specific to the competition, rather than
how qualified the team and mentor are.
Many TDMs were missing some of the requested appendices. Again, these
are necessary, particularly where it relates to open source. We want
to see that you recognise where you've built on other peoples' work
and how you might contribute back. We do realise that especially newer
teams may be at too preliminary a stage to contribute back to
community projects and we will account for this.
Finally, please make sure you do basic checks of your document. Run it
through a spelling and grammar checker. Make sure that you don't have
copy and paste errors. If you have hyperlinks, make sure they survived
the conversion to PDF.
As we start to update the rules and website, in preparation for a
*possible* competition next year (online or in person), we are still
soliciting comments and suggestions, both for improvements to the
in-person competition, as well as for how we might run an online
competition. Refer to my earlier emails about how such a competition
might run.
Cheers!
- Raymond
--
Dr. Raymond Sheh
Research Professor
Institute for Soft Matter Synthesis and Metrology
Georgetown University
506 Reiss Science Building
37th and O Streets NW, Washington DC 20057, USA
Guest Researcher
Intelligent Systems Division
National Institute of Standards and Technology
U.S. Department of Commerce
100 Bureau Drive, Stop 8230, Gaithersburg MD 20899-8230, USA
Senior Lecturer (Adjunct)
Department of Computing
Curtin University of Technology
Street: Building 314, Kent Street, Bentley WA 6102, AUSTRALIA
Mail: GPO Box U1987, Perth WA 6845, AUSTRALIA
I hope you, your families, friends, classmates, teachers, and
colleagues are all staying safe in your parts of the world. I hear
some of you are getting back to normal while others are still facing
challenges. All the best to all of you in your studies, teaching, and
activities!
It has been a few months since we released the opening status survey
and we received very few responses so far, certainly not enough to
make decisions on. We're assuming that this is because, for most of
you, things are still too uncertain to meaningfully answer.
We'll keep the survey up so as things change and become more certain,
please do respond. If you've responded already and things have
changed, feel free to write a new response, keeping the same team
name.
This also means that we won't be making a decision on a virtual RMRC
until sometime next year. In the meantime, those of you who are in
locations where lockdowns are lifting and are able to resume work on
your robots, please do feel free to post updates, pictures, and videos
of your robots performing over the standard tests. We can still share
what we've done, and practice remotely scoring and doing the various
parts of a virtual RMRC in an ad-hoc manner, without making it a full
competition just yet.
I have also sent TDM feedback to teams who didn't pre-qualify for next
year. Note that because this year's RoboCup was cancelled prior to
pre-qualification announcements, we set the pre-qualification bar very
high with the goal of keeping as many spots available as possible for
next year, to avoid disadvantaging teams who might have been able to
turn in a good entry but for early by COVID-related disruptions. We
usually pre-qualify around 10-15 teams, depending on anticipated team
size and local capacity. This year we pre-qualified only 7 teams,
based either on being finalists or from their Qualification Team
Description Materials (Qualification TDMs) achieving almost perfect
scores. In normal times, an additional 3-8 teams would have qualified
based on the Qualification TDM.
I also have some more general feedback and will be updating the
guidance on the website (currently
http://oarkit.intelligentrobots.org/home/competitions/robocuprescue-rmrl/world-championship-2018/2018-robocuprescue-robot-league-rapidly-manufactured-robot-challenge-opening-announcement/
) to reflect this. Note that this feedback is general in nature, a lot
of the Qualification TDMs received did not have these issues so they
may not all apply to you.
In reading the Qualification TDMs, I think the biggest issue that
lower scoring teams had was losing sight of the purpose of the
Qualification TDM. The purpose of the Qualification TDM is to convince
us that you know what you're doing and you have a good chance of doing
well in the competition, with a secondary purpose of being a reference
for other teams. The qualification decision is based on a ranking of
all teams, using this criteria.
The list on the website is a list of points that we are focusing on,
and a guide to structuring your report. It seems like some teams are
treating the list as a checklist, whereby as long as they have a
section with that name, and the section has some text in it that is
vaguely related to the name, they check the box. Such an approach is
not going to result in a particularly high scoring Qualification TDM,
compared to teams who used each of those headings to inspire a section
that discussed how they have, or plan to, tackle that aspect of their
entry.
Describing why you did, or didn't, make a particular choice, and how
it fits into your system, and even things you encountered that didn't
work, as some teams did, does show us that you made a choice based on
a proper understanding. Describe what it is about your design that
means that those specifications on the servo are appropriate. Describe
how you're using Python and C++, where you're using one and where
you're using the other, and more importantly, why. Describe how the
Raspberry Pi and Arduino are connected and what each one does and how
they complement each other. When we say "describe", we are giving you
the opportunity to describe your entry, which is targeted at a
particular application and capability, not just the individual
components. Ideally, the resulting document should also be useful to
other teams, although that is more the focus of the Updated TDM, to be
presented closer to the competition.
A lot of the lower scoring Qualification TDMs also spent a lot of
words summarising the goals of the competition, why rescue robotics is
a good thing to get into and why do 3D printing. This is all
interesting information but it doesn't really tell us how well you are
likely to do.
Statements to the effect that the team is highly motivated and that
the mentor is well qualified are also interesting, but we can only
score your demonstration of this through a comprehensive and well
researched description of the different components of your system and
how they come together. The exception is discussion about prior
performance in this competition (or highly related competitions),
which is useful as part of a wider description of how the team has
changed in the meantime.
Mission strategy, experiments and testing, and strengths of the team
were also severely lacking in a lot of lower scoring TDMs. Some of you
may find it easier to describe your strengths further up, in the
system description section, such as "We chose this wheel design to
give us a particular strength in the stepfields, due to ...", we would
accept that as well (although we would also expect to see some
justification, ideally in the form of test results, even if basic).
Even new teams have something that is being focused on so every team
should be able to say something here. Again, we're talking about
strengths in the approach, specific to the competition, rather than
how qualified the team and mentor are.
Many TDMs were missing some of the requested appendices. Again, these
are necessary, particularly where it relates to open source. We want
to see that you recognise where you've built on other peoples' work
and how you might contribute back. We do realise that especially newer
teams may be at too preliminary a stage to contribute back to
community projects and we will account for this.
Finally, please make sure you do basic checks of your document. Run it
through a spelling and grammar checker. Make sure that you don't have
copy and paste errors. If you have hyperlinks, make sure they survived
the conversion to PDF.
As we start to update the rules and website, in preparation for a
*possible* competition next year (online or in person), we are still
soliciting comments and suggestions, both for improvements to the
in-person competition, as well as for how we might run an online
competition. Refer to my earlier emails about how such a competition
might run.
Cheers!
- Raymond
--
Dr. Raymond Sheh
Research Professor
Institute for Soft Matter Synthesis and Metrology
Georgetown University
506 Reiss Science Building
37th and O Streets NW, Washington DC 20057, USA
Guest Researcher
Intelligent Systems Division
National Institute of Standards and Technology
U.S. Department of Commerce
100 Bureau Drive, Stop 8230, Gaithersburg MD 20899-8230, USA
Senior Lecturer (Adjunct)
Department of Computing
Curtin University of Technology
Street: Building 314, Kent Street, Bentley WA 6102, AUSTRALIA
Mail: GPO Box U1987, Perth WA 6845, AUSTRALIA
Raymond Sheh
Nov 3, 2020, 3:55:55 PM11/3/20
to The Open Academic Robot Kit
Oops of course, in my previous email I had meant to say "... advice for
next year's Qualification TDMs" ... I'm sure you knew what I meant!
Cheers!
- Raymond
Intelligent Robots Group
www.intelligentrobots.org
next year's Qualification TDMs" ... I'm sure you knew what I meant!
Cheers!
- Raymond
www.intelligentrobots.org
Reply all
Reply to author
Forward
0 new messages