Seperation of the extensions

[Joshua Galecki]

Hello everyone.

I haven't died, even if you haven't heard from me in a month :) I've
sorted out the issues with the Extension API. There is a generic
IExtensionApi interface on the RobocodeAPI. An interface within the
Extension implements this and is passed to RobotPeer through
CustomRules. Users can call getExtensionAPI(), typecast it as a
specific Extension API, then call any functions the extension author
offers. I should have a demo of this up later this week (with a few
simple CTF robots).

The last major change is separating the extension from the Robocode
core (you may recall the extension loading is currently hardcoded into
ModuleFactory.) I've been looking into this, and I would like to let
you know my general plan in case I'm about to do something horribly
wrong.
The repository functionality and access is spread out between several
packages but is concentrated in robocode.repository. The class
"database" has a load() function that gets the paths of the robots. I
would also have it get the paths of any extensions (I picture
extensions being selectable much like robots are). I've created an
ExtensionSpecification class that the extension information will be
feed into. As the battle starts, this will become the Custom Rules for
that battle.
If you guys can think of any simpler / better solutions, I'd value
your input.

Regards,
Josh

[Flemming N. Larsen]

Hi Josh, :-)

Nice to hear from you again. I think most of us have been on summer
vacation, at least I have. So we have not been so productive on Robocode as
we are when we are not on vacation. I know I can only speak for myself. :-)

Your solution/idea with using the Extension API sounds like the right
solution to me. I should love to see the code for it, so perhaps you include
this in your next 0.5 release?

I don't see any problems with using the robocode.repository like you
suggest, but I think Pavel is better at judging this part, as he rewrote big
parts of this recently in order to speed up the robot class loading etc.
(nice speedup Pavel, by the way! ;-)

Do you miss any information from us? I am sorry that I did not write about
how all the scorings are computed for the existing version of Robocode. At
that time, there were several issues/bugs in Robocode, which are solved now,
and which I have merged into your branch.

Currently, I am working on fixing issues for the coming version 1.7.1.4
Beta, and some issues seen with the RoboWiki community with the new robot
movement (updateMovement() and getNextVelocity() in RobotPeer), which needs
to be adjusted one more time. After that, I plan to get back to work with
the obstacle/object detection with the radar etc. for the custom
battlefields, and I want to see if I can implement the R+ algorithm + your
"painters algorithm" like you explained - unless you still want to do it?

Best regards,
- Flemming

-----Oprindelig meddelelse-----
Fra: robocode-...@googlegroups.com
[mailto:robocode-...@googlegroups.com] På vegne af Joshua Galecki
Sendt: 3. august 2009 22:59
Til: robocode-developers
Emne: [robocode-developers] Seperation of the extensions

[Joshua Galecki]

Implementing my original idea (with EngineSpecification) led me to
believe it's a terrible idea :) My current strategy is to register the
necessary classes with PicoContainer and call instances or a singleton
from that (I think this is what Pavel had in mind all along :)
I have a jar file with all the extension classes in it, but I'm having
trouble loading from the jar. I've been trying to add registration
code to the Container class (inside its static {} block). I've tried
creating a new instance of the class loader engineLoader with a path
to the extension jar, but it is not able to find any classes. Any
ideas which direction I should go? I've never worked with class
loaders before, so I'm not sure what proper protocol is.

Thanks,
Josh

On Aug 4, 7:57 am, "Flemming N. Larsen" <flemming.n.lar...@gmail.com>
wrote:

[Flemming N. Larsen]

Hi Josh,

Yes, the classes should be registered with PicoContainer. You should decide
if the individual class must be instantiated from cache (singleton) or if a
new instance must be created each time you need return an instance using the
factory method. Remember that you should use interfaces everywhere possible
instead of a specific class (of course there are exceptions to this rule).
PicoContainer will create a concrete (actually inject) an instanced object
everywhere where you use the interfaces declared on our static body in the
Module.java file (there is one per project/module). But as I read your mail,
I am confident that you are on the right track here. :-)

Regarding the class loading. Again you are on the right track. However, I am
not sure how to add extension jars to the classpath as I have not tried out
this myself yet. Did you use the addURL() (to your class path) on the
EngineClassLoader instance? If not, this might be the reason why you cannot
"see" the classes from the class loader.

Best,
- Flemming

-----Oprindelig meddelelse-----
Fra: robocode-...@googlegroups.com
[mailto:robocode-...@googlegroups.com] På vegne af Joshua Galecki

Sendt: 5. august 2009 22:44
Til: robocode-developers
Emne: [robocode-developers] Re: Seperation of the extensions

[Pavel Šavara]

Hi Josh,

sorry about the delay, I was busy recently.

> I have a jar file with all the extension classes in it, but I'm having
> trouble loading from the jar.

For more info look at
net.sf.robocode.core.Container class
loadFromPath(), loadJars(),
look at system property robocode.class.path

Or even deeper
net.sf.robocode.security.HiddenAccess
getClassLoader()
createClassLoader()

It seems that I load robocode.core+pico+robocode.jar into system classloader.
Then I enumerate rest of jar files in lib directory and create
robocode.class.path from them.
Then I call Container in core, which creates EngineClassLoader, which
uses robocode.class.path.

So, the rule is to have your .jar in correct directory and then it
will be accessible in engine classloader.
But not in class loader of any robot. Which is maybe your problem.
If yes, please let me know. I do have already solution for that in my
.NET branch, so I can merge it.
It's new mechanism of registering exceptions to visibility rules for
classloaders and propagating the class to common system classloader
which is parent of both engine and robot's classloader. Therefore the
class is then visible to both spaces.

But before we get too far with that, it would be extension of Robot
API. Is there more than onObjectScanned() interface now ?
My understanding was that you will extend robocode.jar which is in
robot classloader all the time.

Cheers
Pavel

[Nat Pavasant]

Hi Josh,

Nice to have another version committed. I haven't try it yet, but I
wonder why don't you have CTFBot in project robocode.sample?

Best,
- Nat

2009/8/12 Pavel Šavara <pavel....@gmail.com>:

[Joshua Galecki]

As you may have noticed, I’ve updated once again. I’ll attempt to
explain my changes here.

I’ve managed to separate the extension code from the core Robocode
modules. I’ve added getExtensionDirectory() to FileUtil, this should
work the same way as the getMapDirectory() function Flemming added.
The specific jar file and custom rules to be used (within the jar) can
be specified using new text fields on the NewBattleRulesTab dialog,
through .battle files, or through command line arguments. However,
the .battle files don’t save this new information (they only load it)
and the command line argument is only implemented for one of two of
the specific files. Right now, these strings are hardcoded within
BattleManager for convenience. BattleManager has getCustomRules(),
which looks for the specified jar file and loads the custom rules
class and its dependencies (such as custom statistics).

I’ve decided to merge together BattlefieldSetup and CustomRules. I
originally kept these two separate to follow the Single Responsibility
Principle, but I’ve come across situations where Custom Rules should
know things from BattlefieldSetup. For example, if robots respawn, the
rules might wish the robot to respawn at the starting location
(defined within the BattlefieldSetup). I still think SRP should be
followed, so I will probably recreate BattlefieldSetup but keep it
within CustomRules instead of Battle. I’ve added new extension points
to CustomRules so there are extension points at the start and the
finish of every battle, round, and turn. In addition, the entire
Battle object is being passed in now instead of just robots and
robjects. I’ve added several functions to Battle (such as getters and
setters for contestants) and made a few functions public
(getRobotsAtRandom(), getBulletsAtRandom()) so the extension author
can access these within CustomRules.
I’ve changed CustomRules so that the default functions are the Classic
Robocode functions. I thought this would be easier for the extension
author who, for example, wants to change the way the robjects are
setup but not the way the scoring works.

CaptureTheFlagExtensionApi contains functions a robot would need to
call, such as getEnemyFlagLocation(). This class (or any other
Extension API) needs to access CustomRules. Since robots cannot access
ExtensionApi (since it accesses CustomRules) or CustomRules directly
(since it accesses on Battle), robots access ICaptureTheFlagApi in the
ExtensionApi project. This interface is very much like the IRobotPeer
class and allows information to be passed from the extension to the
robot. This will compile onto a separate jar that robots be dependent
on. (The reason FlagSeeker isn’t in robocode.sample is that I didn’t
want to introduce a dependency on the ExtensionApi project for all of
the sample bots.)

The teams in my extension are slightly different than normal teams.
Instead of having to compile a set of robots as a team, my extension
simply assigns each robot to a team in alternating order. For example,
if the order of robots selected for a battle is Fire, SittingDuck,
Fire, SittingDuck, then the two Fire robots will be on a team against
the SittingDucks. Note that the two Fire robots won’t realize that
they are on the same team because they never call the
CaptureTheFlagApi. Another change to the way teams work is that I’ve
stopped messages from going through any robjects.
I’ve changed ContestantStatistics around a bit so that a Statistic
doesn’t need to represent a robot. I’ve implemented an example of this
with CaptureTheFlagTeamStatistics, which represents a team of robots.
Robots’ individual scores aggregate to this TeamStatistics.

Next I’ll be attempting to move the class loading to a better place.
Right now the extension jar loads lazily and it would be better to
load this along with the other jars. Some questions about the move:
We won’t know what the class names will be. Should we mandate a
standard class name, or is there another way to do this?
Should we load every extension jar at this point?
If I load the extensions into Picocontainer at this point, can I just
call Container.getInstanceOf(Key) (or some similar function call) to
get an instance, or is the process more complicated?

I’ve included a FlagSeeker robot to demonstrate how the Extension API
is used. It works, sometimes. The main problem right now is that when
FlagSeeker is loaded into the repository, Robocode can’t find the
ICaptureTheFlagApi (from the ExtensionApi project) because it isn’t
loaded yet. FlagSeeker then gets kicked out of any battle it tries to
join. I hope to address this when I move the loading to the proper
place. I was somehow able use some black magic to get it to load
before, but I’ve not been able to reproduce this for a few days. I can
verify that the Extension API is properly communicating with the
engine.

Before the new version will work, there’s a little manual labor you
must do. An “extensions” folder must be created in the working
directory. After this, copy and paste the jar file within
“robocode.extenions\target” to this new folder and rename it “ex.jar”.

A few more questions for debate:
Should we encourage creating extensions within the
“robocode.extensions” and “robocode.extensionsApi” projects I’ve
created or should we make sure that extensions can be built without
using Maven?
In the NewBattleDialog, how difficult would it be to create a
selectable list of extensions, similar to the selectable list of
robots?

This will be my last huge update, I promise :)

Regards,
Josh

[Pavel Šavara]

Hey Josh, All

sounds like you are real expert on Robocode engine now :-D
Good work, thanks a lot!

Only comment I have now, without looking at code yet, is that I
expected that we will create one general API and include it into
robocode.jar as part of Robot API. You rather propose that each
extension vendor will distribute his own library with robocode and
that robots will be now compiled with more than one library.

I'm not sure why I don't like it, but have mixed feelings about that.
It sounds to me as path to anarchy, where lot of people will create
lot of extensions and break APIs often. This will kill ground for
competition and stability. Probably they will start implement
cheatable rules and undermine security by bad design of extensions.

Maybe that just me. What all people around think ? Should we retain
strict control over Robot API or we should be more open ?
Guys, please elaborate, this is key question.

Thanks
Pavel

[Flemming N. Larsen]

Hi all,

I have not looked into all the details yet, and I really look forward to review the code in depth. I hope I will be able to do this in the end of the weekend. :-)

At first when I read about the extension, I thought why not, this would be a good idea. But when Pavel's is bringing up the issues about security (making harm to the users system), cheating (through the robots) and also stability, I am really getting worried about this feature. As it is now, we already fight with stability, but here we are in total control of developing on every part of Robocode. If we allow extensions, we should ensure that the security cannot be compromised, and also make sure that cheating is not possible. Regarding stability, we'll never be able to guarantee this when somebody is responsible for his/her extension. Perhaps we could "just" put a disclaimer on this part. But the security might be very challenging, unless we make sure that the extension will be ruled by the RobocodeSecurityEngine, and extend it everywhere where we find holes. I also expect that we need more intelligent class loading that will filter out bad code, if it is detected. Perhaps all this is possible, but we need to do a lot of hacking attempts etc. to test that the security is preserved.

I will return with review comment, when I have had a good look at the source code.

- Flemming

-----Oprindelig meddelelse-----
Fra: robocode-...@googlegroups.com [mailto:robocode-...@googlegroups.com] På vegne af Pavel Šavara
Sendt: 14. august 2009 22:03
Til: robocode-...@googlegroups.com

Emne: [robocode-developers] Re: Seperation of the extensions

[Joshua Galecki]

My general idea was to include objects (robjects) as part of the
Robocode API. All of this code would be shared by all robots and the
extension could manipulate objects any way it wants. An extension
doesn’t need to have its own API, but the extension author may want to
give special information to the robot participants. For example, my
extension API includes functions for finding who is on your team,
where the flags are, etc. This ability to tell robots specific
information opens up a lot of possibilities.
Of course, it opens up a lot of security holes as well :) I had always
assumed that we would run the extension through the security manager
at some point and disallow things such as file access outside
Robocode, network access, and so on.
As far as competition goes, there might be some fracturing of types of
new robots created. They likely would not all be Classic Robocode
robots, as people would write robots for the new extensions. This
might not be a bad thing, however. People have been writing Classic
Robocode robots for nearly a decade now and might like to try
something new. And remember that not every extension robot written
would have been a classic robot – some robot authors who are simply
maintaining and polishing their robots might write new ones for the
extensions.
Thinking about the extension API regarding competition, remember that
a CTF robot won’t fare to well in a Classic Robocode match, and vice
versa – whether or not the extension API is kept.
As far as stability goes, I propose we ourselves (or me myself, if no
one else wants to do it :) develop and maintain two extensions to go
along with Classic Robocode: Deathmatch (classic rules with obstacles)
and Capture the Flag. We can ensure stability on these, at least. If
someone else comes up with a cool extension that gets really popular,
good for them! But stability would be mostly their issue.
Cheating in extensions would be easy to do, seeing as how the
extension author can set victory conditions (if (robot.getName().equals
(“myBot”)) :) This would be difficult to stop. My idea here is to
require open source code for all the extensions. After that, if
everyone enjoys playing a rigged game, who are we to stop them? The
open source idea would also help with security, as the code could be
checked by anyone.

Regards,
Josh

[Flemming N. Larsen]

Hi Josh,

I agree with you regarding how you want the robjects as a part of the
Robocode API etc.

Regarding security, the extensions are running inside Robocode, which is the
one defining the SecurityManager for the whole JavaVM instance/process.
Hence, the extensions will automatically have to obey the policies and
restrictions we have in our RobocodeSecurityManager. However, we need to
make sure to find all (new) security holes that we can possible find. We
should do this when testing the extensionAPI.

Hmm.. so whould we have ClassicRobocode as an extension? Then I understand
your idea in the code with letting ClassicRules inherit from CustomRules.
Sorry that I did not get that. I wanted the ClassicRules as build-in rules
in Robocode, but actually it might be better to provide this in an
independent module/extension, which the users can remove if he/she don't
want it. :-)

I also like your idea of requiring extension to be open-source. At least we
should recommend this, and then let it up to the user if he/she will risk to
use it.

So in general, I agree with you. :-)

Best,
- Flemming

-----Oprindelig meddelelse-----
Fra: robocode-...@googlegroups.com
[mailto:robocode-...@googlegroups.com] På vegne af Joshua Galecki
Sendt: 15. august 2009 18:09
Til: robocode-developers

Emne: [robocode-developers] Re: Seperation of the extensions

[Pavel Šavara]

I'm still not convinced that we should do that. I mean I would not
like to open Robot API for anyone. At least not in first round.

But I can see why you ask for that. For example ICaptureTheFlagApi
contains isOwnFlagAtBase() method.
My proposal is to expose new general way how to query battlefield
state without necessity of very specific API.
It could be simple name-value collection. Something like

String getBattleFieldState(String key)

Which could be used this way

getBattleFieldState(isOwnFlagAtBase).equals("true")

Then in robots land, people could implement libraries to wrap that for
specific rules. I.e. CaptureTheFlagApi wrapper.
This will give us generic API for robocode, which trusted data
transmision, but keep the opportunity to extend.

By trusted transmission I mean the way how exec commands and exec
results are used.

Pavel

[Pavel Šavara]

More explanation.

I introduced ExecCommands and ExecResults as safe communication
between robot and battle. (ver 1.6.2)
ExecCommands are coming from robot to battle. ExecResults contains
state of battle field and all events for robot.
They are immutable as soon as they leave sender. Exchange is done
using interloacked primitives, so it's really fast.

It brings
- security -> no shared object to block(synchronize) on
- better synchronization -> speed, up to 9k tps.
- serlizability -> .NET robots, robots over network

Please do _not_ communicate with robot by any other way, it's really
internal API between battle and robot.
If you need to pass state of flag for example, you make ExecResults
extensible and the use that as one way ship to send it to robot.

After you send something to robot, you should not change or access it
from battle thread.
Easier way how to imagine what I'm saying is this: let's have each
robot on different machine and battle class as well. Then all you
could do is to send messages. In order to do that the messages must
serialize and deserialize. See serialize() and deserialize() methods.

The extension of ExecCommands could be simillar to DebugProperties
which are there already.
Same for extension of ExecResults.

I think I use List because it's better serializable than dictionary,
but not sure now.
Then in robot space, you could expand it into hash for fast access, if
you feel that necessary.

[Joshua Galecki]

Thanks for the explanation Pavel. I'll try to implement this soon.

Regarding the Classic Robocode mode, I believe we should keep this
within the source of Robocode as opposed to an external extension. The
reason why is that Robocode would always have these rules to fall back
on.
Consider a user who isn't interested in extensions. He might delete
the extensions folder, taking the Classic extension with it. Robocode
wouldn't be able to find any rules and wouldn't work. Even if the
Classic extension wasn't within the extensions folder, there is still
a possibility that Robocode will not be able to find any external
extension modules. I think its better to keep Classic rules within
Robocode as a dafault and as emergency rules if no extensions are
found.

Since I'm keeping ClassicRules within robocode, would it be better to
use this as the base class and get rid fo the abstract base class
CustomRules (soon to be BattleRules)?

Regarding security, I've already been able to use basic FIle IO to the
robocode directory within my extension. Is this supposed to be
allowable?

Josh

> > Pavel- Hide quoted text -
>
> - Show quoted text -

[Pavel Šavara]

> Regarding the Classic Robocode mode, I believe we should keep this
> within the source of Robocode as opposed to an external extension. The
> reason why is that Robocode would always have these rules to fall back
> on.

Absolutely.

> Regarding security, I've already been able to use basic FIle IO to the
> robocode directory within my extension. Is this supposed to be
> allowable?

In extension of robocode yes. Because it's considered trusted code.
In robot which is using extension, no.