Re: Request: user-level RobotTestBed

[Pavel Šavara]

Hi Philip,

Great idea, love it.
I didn't had chance to watch the video yet, but I will do soon. So
just quick thoughts now.

1) RobotTestBed is rather integration test, not unit test, so students
should be advised about difference.
2) Because I didn't watched the video, I don't understand how your
code is integrated to robocode. My feeling about proper way how to do
that is to use Control API (from looking at the code seems to be the
case :-)
3a) we could include it directly into robocode.jar as control API
extension. We must not add dependency on junit to robocode.jar
3b) or together with other 'utils' for testing in something like
robocode.testing-x.y.z.jar package.
4) I would like to hear feedback about stability/deterministic
behaviour. Robocode is quite complex.
5) combination of this testing tool with custom battlefields (maze)
would be great fun for students, because you could define battlefield
goals (so it will motivate students)
6) we need to rename it to not confuse it with our test bed. We may
rename our existing one, because this Philip's will become API.
7) actually I'm not sure if inheritance is good pattern for API. Could
you please try to rework it for composition somehow ? I'm not sure it
will be nicer, but let's try.
8) The fact that it will become API means it will stay very stable.
Please hide/drop whatever is not pure gold.
9) we need the contribution under CPL, probably not problem
10) we would appreciate donation of user-land unit tests for few of
our sample robots. Maybe that would form extra deployment package
together with robocode.testing-x.y.z.jar and bundled junit.jar ?

Please join this group to reply to this email, so other core members
could notice the conversation. I cross posted.
http://groups.google.com/group/robocode-developers?hl=en

Thanks a lot
Pavel

> Hi Pavel,
> We have got a request with a user-level RobotTestBed:
> https://sourceforge.net/projects/robocode/forums/forum/116459/topic/3415539
> I really like the idea, and I am impressed with his video and the quality of
> his work.
> Perhaps we should ask him if we could include his work into Robocode?
> - Flemming

[Philip Johnson]

> 1) RobotTestBed is rather integration test, not unit test, so students
> should be advised about difference.

That's true. In the video, I distinguish between "acceptance
tests" (where you just test that your robot can beat another robot),
"behavioral tests" (where you test that your robot implements a given
strategy), and "unit tests" (where you are basically testing a helper
method to correct input-output behavior). Integration test is
probably a combination of my acceptance and behavioral tests.
Robocode is pretty unique as a student-accessible programming
framework in terms of the potential sophistication of the tests. As
you can see from watching the video, I am pretty impressed with it.

> 2) Because I didn't watched the video, I don't understand how your
> code is integrated to robocode. My feeling about proper way how to do
> that is to use Control API (from looking at the code seems to be the
> case :-)

Yes.

> 3a) we could include it directly into robocode.jar as control API

> extension. We must not add dependency on junit to robocode.jar.

Hmm. I see what you're saying, but unfortunately both your
RobotTestBed and my version of it include @Test, @Before, and @After
annotations. So, there is a dependency on JUnit.

> 3b) or together with other 'utils' for testing in something like
> robocode.testing-x.y.z.jar package.

Yes, that would work fine and keep robocode.jar free of JUnit.

> 4) I would like to hear feedback about stability/deterministic
> behaviour. Robocode is quite complex.

So far, so good! Of course, the robots in my class are quite simple.
I was concerned about the students degenerating into "programming via
downloading robots from the internet", so I require them to extend
Robot (not AdvancedRobot). That bugs some of the better students, but
it has the effect of making 99% of the cool robot implementations
nontrivial to use in class.

> 5) combination of this testing tool with custom battlefields (maze)
> would be great fun for students, because you could define battlefield
> goals (so it will motivate students)

Very cool indeed.

> 6) we need to rename it to not confuse it with our test bed. We may
> rename our existing one, because this Philip's will become API.

This is completely up to you. Feel free to rename mine if that would
be appropriate.

> 7) actually I'm not sure if inheritance is good pattern for API. Could
> you please try to rework it for composition somehow ? I'm not sure it
> will be nicer, but let's try.

I will take a look at this. Although I agree that composition is
preferable to inheritance, the current implementation seems pretty
reasonable to me.

> 8) The fact that it will become API means it will stay very stable.
> Please hide/drop whatever is not pure gold.

There are two lines I left in because I didn't know what they did:
System.setProperty("EXPERIMENTAL", "true");
System.setProperty("TESTING", "true");

If these are vestigial, they can be removed.

> 9) we need the contribution under CPL, probably not problem.

No problem.

> 10) we would appreciate donation of user-land unit tests for few of
> our sample robots. Maybe that would form extra deployment package
> together with robocode.testing-x.y.z.jar and bundled junit.jar ?

Here's an almost-ready "acceptance test":

http://code.google.com/p/robocode-pmj-dacruzer/source/browse/trunk/src/pmj/TestDaCruzerVersusSittingDuck.java

All you have to do is replace "pmj.DaCruzer" with, for example,
"sample.Walls".

Come to think of it, here's a "behavioral" test that with slight
modification would work for sample.Corners:

http://code.google.com/p/robocode-pmj-dacruzer/source/browse/trunk/src/pmj/TestDaCruzerMovement.java

I go over both of these in the video.

BTW, if you're interested, here's my course schedule page, which has
links to various videos and assignments for Robocode:

http://groups.google.com/group/ics-software-engineering-fall-2009/web/00-schedule

The class version of the roborumble server is at:

http://csdl.ics.hawaii.edu/~johnson/rumble/

Thanks!
Philip

[Pavel Šavara]

>> 2) Because I didn't watched the video

Watched it.good stuff.

>> 7) actually I'm not sure if inheritance is good pattern for API.

> current implementation seems pretty reasonable to me.

Another thing, if you throw exception from assert in onTunrEnded or
any other event, it is not propagated to junit console.
It's because it's excecuted on battle thread, not on your unit tests thread.
It's rather dumped into error messages and then we count expected
count of errors.
This could be probably done better, by stopping the engine, if
exception is thrown in event handler.
As well see 12),14)

> There are two lines I left in because I didn't know what they did:
> System.setProperty("EXPERIMENTAL", "true");
> System.setProperty("TESTING", "true");

Without going into details, leave them there for now.

on top of your idea:
11) we could create robot training kit
a) learn how to move: avoid the walls - Asses that robot never hit the wall
b) learn your robot how to aim level 1 - Asses that robot could hit
sample.crazy or sample.corner in 90% cases
c) learn how to avoid bullets level 1: Asses moves away from bullet
direction (see snapshot) in 60% cases
d) learn how to keep the distance
e) level 2 goals, more oponents .....
The user or student would pass the in known order and improve his robot.

12) the above scenarios don't exclude each other, so I think we could
make the test-bed extensible with watch-dogs which will care about
it's specific metric, or rule. The test-run itself could be just
composed from those test guards. Guard itself should implement
IBattleListener, but would not be junit aware or battle setup aware.

13) there should be standardized set of opponents to train against,
training sets, kind of benchmark

14) Idea 12) attract me to composition over inheritance again. We
could have something similar to RobocodeEngine, which will be
extensible with guards. The unit test class would be user code only
(so we get rid of junit dependency), it will support pluggable
watchdogs, and it could run whole series of battles (with different
sets of opponents), and the assertion exceptions would be propagated.
Maybe I'm just dreaming ?

15) To really unit test (units of robot) the robot itself should be
build modular way. Module for tracking others, module for movement
decisions, module for aiming. For such modules, if they are done
properly, we could create testing framework. I would imagine sequences
of robot events to form the scenarios. Example: testing detect fire,
two subsequent onScan events for same oponent would have expected
delta in energy. Another test for onScan event with different
oponents, and another for too big or too small energy delta, both of
them should not excite tested module. This is what I imagine as unit
test for robots. For the framework to be reusable by all robot
developers, it should probably stick to Robot API/events. Is this idea
interesting for you ?

16) are you keen to become robocode developer and do the work ? We
could create branch for you. That would get your contribution done
much faster, as we are busy with something else now with Flemming.
There is no pressures on you timewise, we do it for fun, so we take
our time to do it and to do it right.

Pavel

[Mat]

Great stuff, guys. I love what you're doing with Robocode!
-Mat

[Philip Johnson]

Hi Pavel,

Thanks for the reply; you make a lot of excellent points.

A couple additions:

> 11) we could create robot training kit

This is a cool idea. I did something much more basic in my class: I
made them write a bunch of simple robots with simple defined
behaviors. Here's the assignment:

http://groups.google.com/group/ics-software-engineering-fall-2009/web/06-robocode

I am sure there are better/additional "training" assignments that more
experienced robot developers could come up with.

> 16) are you keen to become robocode developer and do the work ?

That's very nice of you to offer. I will definitely not have time in
the near term, but next summer is a possibility. It seems like the
work should be cut up into slices:

(1) First, just get a new robocode-testing.jar into the distribution
that has an "official" RobotTestBed class.

(2) Start work on a more advanced testing framework.

Philip

[Pavel Šavara]

> (1) First, just get a new robocode-testing.jar into the distribution
> that has an "official" RobotTestBed class.

Hi Philip, all,

I did all the changes necessary for phase 1)

Please download, install and test this binary distribution
http://robocode.sourceforge.net/files/robocode.testing-1.7.1.4-setup.jar
It should install itself into robocode structure this way.
junit-4.5.jar and robocode.testing.jar should go into robocode/libs
new directory robocode/testing should contain the rest.

The code could be seen here
https://robocode.svn.sourceforge.net/svnroot/robocode/robocode/trunk/plugins/testing/

Philip,
please review the code, Read.Me file and provide feedback, if I
screwed something.
Or you have any improvement or idea. I think your students could
contribute more sample tests :-)
Did I forgot to do something ?

Flemming,
could you please create unix versions of batch files ?
The deployment binary is created by running mvnassembly.cmd from
plugins\testing directory.
I think that we could release it in next full release.

- - - -
A) During development I hit one problem. As you know we don't version
robocode.jar filname to not break peoples classpath with robocode
upgrades. I did same for robocode.testing.jar. Is it good idea ? Or
people could live with robocode.testing-1.7.1.4.jar ? If we should
strip version, should we do that for junit-4.5.jar as well ?

B) I placed the RobotTestBed into robocode.control.testing and
TestWallBehavior is in sample package. Objections ?

Cheers
Pavel

[Flemming N. Larsen]

Thanks Pavel for implementing this! :-D

>could you please create unix versions of batch files ?

Yes, I will + test in under Ubuntu 9.10. ;-)

>I think that we could release it in next full release.

Version 1.7.2 or 1.7.1.5. I think we should wait for 1.7.2, as the 1.7.1.x is currently is for bugfixes. I am not so religious about it.

>B) I placed the RobotTestBed into robocode.control.testing and
>TestWallBehavior is in sample package. Objections ?

No objections.

>A) During development I hit one problem. As you know we don't version
>robocode.jar filname to not break peoples classpath with robocode
>upgrades. I did same for robocode.testing.jar. Is it good idea ? Or
>people could live with robocode.testing-1.7.1.4.jar ? If we should
>strip version, should we do that for junit-4.5.jar as well ?

Hmm.. I think we should stick to the names with versions where possible. Perhaps we could also do this for Robocode.jar in the future? (Will it brake the classpath, as it will "just" copy a new version into their existing /libs folder. It might be a problem with the startup batch files, which people can modify, and which will (currently) always contain "-cp robocode.jar". But perhaps we could look for it and replace it with the new version, when installing Robocode on an existing folder, and perhaps copy the old batch files to a backup? Just an idea.

-----Oprindelig meddelelse-----
Fra: robocode-...@googlegroups.com [mailto:robocode-...@googlegroups.com] På vegne af Pavel Šavara
Sendt: 10. oktober 2009 21:11
Til: robocode-...@googlegroups.com
Emne: [robocode-developers] Re: Request: user-level RobotTestBed

[Pavel Šavara]

> Version 1.7.2 or 1.7.1.5. I think we should wait for 1.7.2, as the 1.7.1.x is currently is for bugfixes. I am not so religious about it.

Version 1.7.2. I committed into trunk, because I didn't realized that
we have branch for 1.7.2
I think we could merge into branch and ignore that in trunk for now.

[Flemming N. Larsen]

Okay. I'll merge it later. :-)

- Flemming

-----Oprindelig meddelelse-----
Fra: robocode-...@googlegroups.com [mailto:robocode-...@googlegroups.com] På vegne af Pavel Šavara

Sendt: 11. oktober 2009 11:26

Til: robocode-...@googlegroups.com
Emne: [robocode-developers] Re: Request: user-level RobotTestBed