RL-Library on Windows
W Bradley Knox
I'm putting RLGlue on yet another computer. This is the first one
that's a Windows machine, like the others we'll be using for some
upcoming experiments.
I'm using the Getting Started instructions on your website for
RL-Library, and the run.bash is set up for, well, the bash
environment. I'm not used to working in Windows and am having some
issues converting run.bash to a Windows version. Do you have a run.bat
or something like that which would work on XP? Or do you recommend
using CygWin for RL-Library?
Thanks again.
- Brad
Brian Tanner
We can definitely make it work, because it worked for the competition,
but there are some ugly hacks, even in Cygwin (paths are different
structure and stuff).
If you tell me exactly how you want to run things, I can make some
advice, probably. For example:
1) Are you running everything in Java?
2) Do you need sockets support?
3) Are you running RLVizApp, or console-based experiments?
When I hear back, I'll give you some ideas. Hint: If the answers are
Yes,No,RLVizApp, then it might be trivial(ish).
Brian Tanner
Ph.D Student
University of Alberta
br...@tannerpages.com
W Bradley Knox
On Wed, Oct 7, 2009 at 3:48 PM, Brian Tanner <br...@tannerpages.com> wrote:
>
> Ugh. RL-Library has not been vetted for Windows, even with Cygwin.
> We can definitely make it work, because it worked for the competition,
> but there are some ugly hacks, even in Cygwin (paths are different
> structure and stuff).
>
> If you tell me exactly how you want to run things, I can make some
> advice, probably. For example:
> 1) Are you running everything in Java?
> 2) Do you need sockets support?
> 3) Are you running RLVizApp, or console-based experiments?
Brian Tanner
that sockets are necessary.
Just looking at rl-library-includes.sh right now. Looks like it has
some Cygwin in there, so this might be easier than I thought.
The first step is to try and launch each of the components
independently and see if they work. From there you can use your .bat
skillz to make it work more easily.
I'm really not familiar with running Python and Java stuff in Windows.
It's possible that you can set it up so that you can just
double-click the Python agent and have it connect to RL-Glue. The
same might be true of the RL-Glue executable.
RLVIZAPP (Experiment)
------------------------------
This is less likely for RLVizApp because the main class probably isn't
set right for that. Maybe one day I should make a config panel that
pops up for you to choose where to look for environments and stuff.
In any case, it's not there now. So, you'll have to figure out the
right line to launch RLVizApp from a batch script. I'm looking at
rl-library-includes.sh, and the line you want is something like:
I'm assuming you want to use the fancy tools to dynamically pick an
environment and configure it from a list, but you will just launch the
agent from a terminal window with pre-specified settings.
java -enableassertions -Xmx128M $osExtras -jar $VIZAPP_JAR
agent-environment-jar-path=$ENV_AGENT_JARS_PATH list-environments=true
env-viz=true
- osExtras is Mac only, so you can ditch that
- VIZAPP_JAR should be something like
C:\path\to\rl-library\system\common\libs\rl-viz\RLVizApp.jar
- ENV_AGENT_JARS_PATH should be something like
C:\path\to\rl-library\products <--- or wherever your environment jars
are
You might be wondering why we need the Jar path to the environment
since we are going to be connecting it through sockets anyway. The
reason is that the visualization relies on the experiment having the
visualizer classes available, even if the environment classes are not.
That's it!
EnvironmentShell (Environment Loader)
-------------------------------------------------------
java -enableassertions -Xmx128M -jar $ENVSHELL_JAR
environment-jar-path=$ENV_AGENT_JARS_PATH
- ENVSHELL_JAR should be something like
C:\path\to\rl-library\system\common\libs\rl-viz\EnvironmentShell.jar
- ENV_AGENT_JARS_PATH should be something like
C:\path\to\rl-library\products <--- or wherever your environment jars
are
So, if you launch those two java processes, plus rl_glue.exe and your
agent, it should maybe just all work. Let me know, please, if it does
or where things go south.
On Wed, Oct 7, 2009 at 3:23 PM, W Bradley Knox <brajava
-enableassertions -Xmx128M $osExtras -jar $VIZAPP_JAR
Brian Tanner
Since Python and Java agents can run without Cygwin, you can probably
do this all in vanilla windows.
If you get it all working, let's put a tutorial up on the new
http://library.rl-community.org/
W Bradley Knox
are all working in Cygwin. I'm having some trouble with the java
classpath to get SampleExperimentRLGlue to work. I'll let you know how
it goes.
Brad
u.a.b.u...@gmail.com
Hi Brian Tanner,
I am following your guidance above and running RLVizApp in Windows.
Unfortunately, when running either "AgentShell.jar" or "EnvironmentShell.jar" I got error messages.
SET ProductPath=D:\RL-Library\products
SET RLVizApp=D:\RL-Library\system\common\libs\rl-viz
>>java -enableassertions -Xmx128M -jar %RLVizApp%\EnvironmentShell.jar environment-jar-path=%ProductPath%
Error messages as below :-
Exception in thread "main" java.lang.NoClassDefFoundError: rlVizLib/dynamicLoadi
ng/DynamicLoaderInterface
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:791)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:14
2)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:423)
at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:791)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:14
2)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:423)
at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
at org.rlcommunity.rlviz.environmentshell.EnvironmentShell.refreshList(U
nknown Source)
at org.rlcommunity.rlviz.environmentshell.EnvironmentShell.<init>(Unknow
n Source)
at org.rlcommunity.rlviz.environmentshell.EnvironmentShell.main(Unknown
Source)
Caused by: java.lang.ClassNotFoundException: rlVizLib.dynamicLoading.DynamicLoad
erInterface
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:423)
at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
... 25 more
Cheers.
Regards,
Baker.
Brian Tanner
Can you tell me what is in the D:\RL-Library\system\common\libs\rl-viz folder?
--
Brian Tanner
Operations Manager / Fire Inspector
br...@fireplan.ca
> You received this message because you are subscribed to the Google Groups "rl-library" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to rl-library+...@googlegroups.com.
> To post to this group, send email to rl-li...@googlegroups.com.
> Visit this group at http://groups.google.com/group/rl-library.
> For more options, visit https://groups.google.com/groups/opt_out.
u.a.b.u...@gmail.com
Answer: the files are as below :-
AgentShell.jar
EnvironmentShell.jar
RLVizApp.jar
RLVizLib.jar
RLVizLib-src.jar
Regards
Baker.
Brian Tanner
Obviously it's something to do with the right classes not being found, I am not sure if they are usually embedded in EnvironmentShell.jar or if they are in RLVizLib.jar (probably both). I guess you could unpack/unzip EnvironmentShell.jar and see if rlVizLib/dynamicLoading/DynamicLoaderInterface.class is in there.
u.a.b.u...@gmail.com
Many thanks for your little help, and sorry for taking your busy times. Yes I already posted in the google group.
The "rlVizLib/dynamicLoading/DynamicLoaderInterface.class" is FOUND in RLVizLib.jar but NOT in EnvironmentShell.jar .
However I think this is not the issue.
2) Congratulation that you are running your own business. I could try my luck to drop my CV (smile).
Regards,
Baker
Brian Tanner
Otherwise I don't have a lot of great ideas. But I do want you to succeed.
The other thing you could do is download the 2011 reinforcement learning competition code. That code for sure successfully can fire up RLViz in Windows. So it would at least give you a working prototype to investigate with…
I think it's available from here: http://rl-competition-code.googlecode.com
However, since my frustratingly poor Internet at my hotel is not working, I cannot verify at the moment. But rest assured there is a code distribution out there from the 2011 competition (and the 2010 competition) that should work out of the box on Windows.
Brian Tanner
u.a.b.u...@gmail.com
Thanks for your reply. And thanks for your moral support !.
The manifest looks like below :-
Manifest-Version: 1.0
Ant-Version: Apache Ant 1.7.0
Created-By: 1.5.0_16-133 (Apple Inc.)
RLVizLib-Link-Version: 1.3
Main-Class: org.rlcommunity.rlviz.agentshell.AgentShell
Class-Path: RLVizLib.jar
Name: agentShell/
Specification-Title: AgentShell
Specification-Version: 1.0
Specification-Vendor: RLAI
Implementation-Title: AgentShell
Implementation-Version: Build:405
Implementation-Vendor: Brian Tanner
2) The “EnvironmentShell.jar” from “rl-competition” seems work well than the .jar I had.
However, I got new type of error message as below , I think caused by missing of main () method in the code.
Exception in thread "main" java.lang.NullPointerException
at sun.launcher.LauncherHelper.getMainClassFromJar(LauncherHelper.java:399)
at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:463)
3)>>The other thing you could do is download the 2011 reinforcement learning competition code.
Cheers,
Baker.
Brian Tanner
go into the public directory.
In the "trainers" directory there is consoleTrainerJava folder. This has code to launch experiments from the command line. This should work on Windows in Cygwin.
There is also guiTrainerJava which will launch the RLViz Gui.
There are some special flags and stuff for these for use in the competition, but if you look in the public/system/libraries directory you will see that the setup is similar to yours, and if you trace the includes in the bash scripts that run the experiments you may be able to duplicate things.
Basically, I think you just need to do something with class paths or where things are stored.
I just looked at public/system/rl-competiion-includes.sh
This is the code to run the environment:
startEnvShellInBackGround(){
java -enableassertions -Xmx128M -jar $ENVSHELL_JAR environment-jar-path=$ENV_AGENT_JARS_PATH &
envShellPID=$!
echo "-- Starting up dynamic environment loader - PID=$envShellPID"
}
But first, the function for windows specifically is called:
setCygwinPaths ()
{
if [[ `uname` == CYGWIN* ]]
then
glueExe="$glueExe.exe"
ENV_AGENT_JARS_PATH=`cygpath -wp $ENV_AGENT_JARS_PATH`
VIZAPP_JAR=`cygpath -wp $VIZAPP_JAR`
rlVizLibPath=`cygpath -wp $rlVizLibPath`
ENVSHELL_JAR=`cygpath -wp $ENVSHELL_JAR`
AGENTSHELL_JAR=`cygpath -wp $AGENTSHELL_JAR`
fi
}
Also, as a total aside - are you running all of your components (agent, environment, and experiment) in Java? If so there is a far easier way to do all of this. Let me know if that's true and I'll send you a link.
W Bradley Knox
Good luck!
--------------------------------------------------------
--------------------------------------------------------
If installing in Windows (otherwise, skip to Installing RLGlue and RL-Library):
--------------------------------------------------------
Install notepad++ for all text editing.
Install Cygwin (http://www.cygwin.com/). During installation, you should have a chance to choose packages to install (this is right after choosing a mirror for downloads). If you need to go back and install packages later, execute the setup.exe file that was downloaded for Cygwin's installation. Install the following packages: X windows; Python; in Web, wget; in Editors, Emacs and Vim; in Devel, subversion, make, and gcc:g++. (Brian, if this is wrong at all, feel free to fix it.) You will need to a text editor to follow these instructions. Emacs and Vim are most popular (you can Google to get a list of basic commands for either editor). If you use Emacs, add
set "CYGWIN=tty"
to C:\cygwin\cygwin.bat.
-------------------------------------------
-------------------------------------------
Installing RLGlue and RL-Library:
-------------------------------------------
Make sure that you have Java JDK, Python, and Apache Ant installed (see below for Windows users).
[For Cygwin users]: {
Apache Ant install:
"wget http://www.apache.org/dist/ant/binaries/apache-ant-1.8.0-bin.tar.gz"
"tar zxvf apache-ant-1.8.0-bin.tar.gz"
Python should have been installed as a Cygwin package. For Java and Apache Ant, you'll probably need to add the binary directories to your Windows PATH variable. To do this, right-click on 'My Computer', go to Properties/Advanced/Environment Variables. There, click New in the System variables box to add each of the following System environment variables:
Variable Name: PATH Variable Value: PATH_TO_JDK_BINARIES;PATH_TO_APACHE_ANT_BINARIES (e.g., C:\Program Files\Java\jdk1.6.0_20\bin\;C:\Program Files\apache-ant-1.8.0\bin)
Variable Name: JAVA_HOME Variable Value: PATH_TO_JDK_BINARIES [not sure if you need the last \bin\ on this one]
Variable Name: CLASSPATH Variable Value:
Variable Name: RLLIBRARY Variable Value: PATH_WHERE_YOU_LATER_PUT
Example user variables (different per machine):
JAVA_HOME = C:\Program Files\Java\jdk1.6.0_20
PATH = C:\Program Files\Java\jdk1.6.0_20\bin\;C:\Program Files\apache-ant-1.8.0\bin
RLLIBRARY = C:\rlglue-3.04\rl-library
}
[Almost all of the directions from this point on can be skipped if the installTamer.bash script is used. If this message still doesn't have a link to the script, email me and I'll fix that.]
These directions assume you are working within the bash environment.
Download and install rl-glue 3, following the directions on the following web pages (including their instructions on how to extract tar.gz files; Windows' default extraction can mess things up).
"tar -zxf GLUE-DEV-FILE-BASE.tar.gz"
"cd /path/to/GLUE-DEV-FILE-BASE"
"./configure"
"make"
------------------------------
http://code.google.com/p/rl-glue-ext/wiki/RLGlueCore
download and install codecs:
http://code.google.com/p/rl-glue-ext/wiki/Python (Install the user distribution of the python-codec, not the Windows one.)
http://code.google.com/p/rl-glue-ext/wiki/Java
Download rl-library
-------------------
Follow Getting Started guide, downloading and installing as the example indicates):
http://code.google.com/p/rl-library/wiki/GettingStarted
IMPORTANT: Put the rl-library directory just within the RLGlue directory, as in: rlglue-3.02/rl-library/
[If using Cygwin on Windows (do everything in the brackets)]:
{
In rl-library/SampleExperimentRLGlue-Java-R1229/run.bash, change
java -classpath $GluePath:./classes SampleExperiment
to
java -classpath "$GluePath;./classes" SampleExperiment
replacing any ":" with ";" and adding quotations around the entire classpath. Any other java classpath containing colons will require the same treatment. (This is explained at http://blog.buluschek.com/?p=69.)
}
To test run.bash for either SampleExperimentRLGlue or (later) SampleExperimentRLVizApp, you may need to make the corresponding run.bash file executable via the command "chmod a+x run.bash" given while within the directory where run.bash is located
(Optional) Get the Epsilon-Greedy Agent (I can explain what that means later):
http://code.google.com/p/rl-library/wiki/EpsilonGreedyTileCodingSarsaLambdaJava
Get me (brad...@cs.utexas.edu) to give you an account on my svn repository.
Checkout and compile Tetris. Within the rl-library directory, type :
svn --username yourUsername co http://z.cs.utexas.edu/svn/bradknox/Tetris-Java-R1225 Tetris-Java-R1225
do "ant all" inside Tetris-Java-R1225 directory
Checkout and compile Mountain Car. Within the rl-library directory, type :
svn --username yourUsername co http://z.cs.utexas.edu/svn/bradknox/MountainCar-Java-R1229 MountainCar-Java-R1229
do "ant all" inside MountainCar-Java-R1229 directory
Checkout and compile tamerCommon, which facilitates receiving feedback from the user via the keyboard. Within the rl-library directory, type:
svn --username yourUsername co http://z.cs.utexas.edu/svn/bradknox/tamerCommon tamerCommon
do "ant all" inside tamerCommon directory
Follow these directions to get RLViz downloaded and running:
http://code.google.com/p/rl-library/wiki/GraphicalExperimentsWithRLViz
When you've got the Random Agent working on Mountain Car and Random Agent working on Tetris, move on to the next step.
Get the more recent version of RLViz. Within the rl-library directory, type:
svn checkout http://rl-viz.googlecode.com/svn/trunk/ rl-viz-read-only
Inside rl-viz-read-only directory, do "ant all" and then "cp -r system/dist/* ../system/common/libs/rl-viz/"
Make sure that you can still run on of the aforementioned agents on Mountain Car or Tetris.
-----------------------------------------------------------------------------------
-----------------------------------------------------------------------------------
To set up RLViz to allow Python agents (or agents in any other supported language):
-----------------------------------------------------------------------------------
This disables automatic loading of agents (which can currently only be done for Python agents).
- comment out the line in rl-library/SampleExperimentRLVizApp-Java-R1207/run.bash that starts the AgentShell and the line that waits for it to die (startAgentShellInBackground & waitForAgentShellToDie)
- in rl-library/system/scripts/rl-library-includes.sh, inside startNetGuiTrainerDynamicEnvAgentBothViz(), change from
java -enableassertions -Xmx128M $osExtras -jar $VIZAPP_JAR agent-environment-jar-path=$ENV_AGENT_JARS_PATH list-agents=true list-environments=true env-viz=true agent-viz=true
to
java -enableassertions -Xmx128M $osExtras -jar $VIZAPP_JAR agent-environment-jar-path=$ENV_AGENT_JARS_PATH list-agents=false list-environments=true env-viz=true agent-viz=true
------------------------
------------------------
Set RLLIBRARY environment variable
------------------------
The following command can either be executed every time you start a terminal window (or Cygwin window) or it can be added to a start up script such as .bashrc or .profile to run automatically.
export RLLIBRARY=/wherever/you/put/the/folder/rl-library
------------------------
------------------------
Getting the TAMER agents
------------------------
After I've given you access to the svn repository, check out the agents via svn from within your rl-library directory:
----
Mountain Car agent:
svn --username yourUsername co http://z.cs.utexas.edu/svn/bradknox/mcshapeagent mcshapeagent
----
Tetris agent:
svn --username yourUsername co http://z.cs.utexas.edu/svn/bradknox/tetrisshapeagent tetrisshapeagent
--------------------------
--------------------------
(Optional for now) Install the scipy library
--------------------------
http://www.scipy.org/Installing_SciPy/Linux
-------------------------------------------------------
-------------------------------------------------------
(IGNORE THIS SECTION. No longer necessary, since Mountain Car comes from my own svn repository.) Changing Mountain Car to my specification
-------------------------------------------------------
The only real changes are to make the car randomly start within a region near the middle of the mountains (in the valley) and insert my graphical representation of the car.
in rl-library/MountainCar-Java-R1229/src/org/rlcommunity/environments/mountaincar/MountainCarState.java, add to the class variables:
public double easyLftRegionMax = -0.6;
public double easyRtRegionMin = 0.4;
public double startRegionLftBorder = -0.7;
public double startRegionRtBorder = -0.3;
in rl-library/MountainCar-Java-R1229/src/org/rlcommunity/environments/mountaincar/MountainCar.java, add these class variables to class MountainCar:
private static final long randomSeed = 140823235235345435L;
private Random seededRdmGen = new Random(randomSeed);
private int epCounter = 0;
private static final int epsUntilRepeat = 200;
private static final boolean repeatInitState = true;
in rl-library/MountainCar-Java-R1229/src/org/rlcommunity/environments/mountaincar/MountainCar.java, replace function env_start() with:
public Observation env_start() {
if (theState.randomStarts) {
do {
double randStartPosition;
if (repeatInitState) {
if (epCounter % epsUntilRepeat == 0) {
seededRdmGen = new Random(randomSeed);
}
randStartPosition = (seededRdmGen.nextDouble() * (theState.startRegionRtBorder + Math.abs((theState.startRegionLftBorder)))
- Math.abs(theState.startRegionLftBorder));
}
else {
randStartPosition = (randomGenerator.nextDouble() * (theState.startRegionRtBorder + Math.abs((theState.startRegionLftBorder)))
- Math.abs(theState.startRegionLftBorder));
}
theState.position = randStartPosition;
} while (theState.inGoalRegion());
}
else {
theState.position = theState.defaultInitPosition;
}
//System.out.println("Initial Position: " + theState.position + ". Episode Counter: " + epCounter + ".");
theState.velocity = theState.defaultInitVelocity;
epCounter++; // Therefore the first episode is counted as number 0 (not 1).
return makeObservation();
}
public Observation env_start() {
if (theState.randomStarts) {
double randStartPosition = (randomGenerator.nextDouble() * (theState.startRegionRtBorder + Math.abs((theState.startRegionLftBorder)))
- Math.abs(theState.startRegionLftBorder));
theState.position = randStartPosition;
while (theState.inGoalRegion()) {
randStartPosition = (randomGenerator.nextDouble() * (theState.easyRtRegionMin + Math.abs((theState.easyLftRegionMax)))
- Math.abs(theState.easyLftRegionMax));
theState.position = randStartPosition;
}
} else {
theState.position = theState.defaultInitPosition;
}
theState.velocity = theState.defaultInitVelocity;
return makeObservation();
}
also in rl-library/MountainCar-Java-R1229/src/org/rlcommunity/environments/mountaincar/MountainCar.java, change the line in getDefaultParameters() as shown below:
p.addBooleanParam("randomStartStates", true); // was false
in rl-library/MountainCar-Java-R1229/src/org/rlcommunity/environments/mountaincar/visualizer/CarOnMountainVizComponent.java, add (or uncomment if they are still there):
import java.awt.geom.Rectangle2D;
and change render(Graphics2D g) to:
public void render(Graphics2D g) {
g.setColor(Color.RED);
// AffineTransform saveAT = g.getTransform();
// g.scale(.005, .005);
//to bring things back into the window
double minPosition = mcv.getMinValueForDim(0);
double maxPosition = mcv.getMaxValueForDim(0);
int lastAction = mcv.getLastAction();
double transX = UtilityShop.normalizeValue(this.mcv.getCurrentStateInDimension(0), minPosition, maxPosition);
//need to get he actual height ranges
double transY = UtilityShop.normalizeValue(
this.mcv.getHeight(),
mcv.getMinHeight(),
mcv.getMaxHeight());
transY = (1.0 - transY);
// transY = (1.0 - transY-.05);
// transX*=200.0d;
// transY*=200.0d;
double rectWidth = .05;
double rectHeight = .05;
// double theta=mcv.getSlope()*1.25;
Rectangle2D fillRect = new Rectangle2D.Double(transX - rectWidth / 2.0d, transY - rectHeight / 2.0d, rectWidth, rectHeight);
g.fill(fillRect);
// AffineTransform theTransform= AffineTransform.getTranslateInstance(transX-theCarImage.getWidth(null)/2.0d, transY-theCarImage.getHeight(null)/2.0d);
// theTransform.concatenate(AffineTransform.getRotateInstance(theta,theCarImage.getWidth(null)/2,theCarImage.getHeight(null)/2));
// g.drawImage(theCarImage,theTransform, null);
if (showAction) {
double actionRectWdthOffset = 0;
if (lastAction == 0) {
actionRectWdthOffset = 0;
} else if (lastAction == 1) {
actionRectWdthOffset = 7 * (rectWidth / 16);
} else if (lastAction == 2) {
actionRectWdthOffset = 14 * (rectWidth / 16);
}
g.setColor(Color.CYAN);
Rectangle2D fillActionRect = new Rectangle2D.Double((transX - rectWidth / 2.0d) + actionRectWdthOffset,
transY - rectHeight / 2.0d,
rectWidth / 8.0, rectHeight);
g.fill(fillActionRect);
}
mcv.getForceDrawRefresh();
// g.setTransform(saveAT);
}
in rl-library/MountainCar-Java-R1229/, type:
ant all
If it says "BUILD SUCCESSFUL", then the specification change should be complete. Try running Mountain car in RLViz to see the changes.
-------------------------------------
Getting rid of time step build ups (important for some applications)
-------------------------------------
in rl-library/rl-viz-read-only/projects/rlVizApp/src/org/rlcommunity/rlviz/app/gluestepper/FixedIntervalGlueRunner.java,
add:
import java.util.Date;
and replace:
currentTimer.scheduleAtFixedRate(new TimerTask() {
public void run() {
theGlueLogic.step();
}
}, 0, timeStepDelay);
with:
currentTimer.schedule(new TimerTask() {
public void run() {
theGlueLogic.step();
}
}, new Date(), timeStepDelay);
then in rl-library/rl-viz-read-only, type:
ant all
cp -r system/dist/* ../system/common/libs/rl-viz/
-------------------------------
************************
-------------------------------
************************
-------------------------------
-------------------------------
Running the agents in RLViz
(This will work but they can now be run in the same window. I'll update the directions later.)
-------------------------------
To run Tetris agent and environment in the same window:
enter directory rl-library/tetrisagent/
./startTetris.bash
(for the win PCs in maddoxlab, make sure startTestris.bash is in rl-glue root dir)-- i should probably change this (-Brian)
----------------
(1) Start RLViz (in one window)
----------------
enter directory rl-library/SampleExperimentRLVizApp-Java-R1207/
./run.bash
---------------------
(2) Run the agents (in another window)
---------------------
Mountain Car agent:
Ways to run (while RLViz has been started and the automatic agent loader is disabled in the steps above):
- Sarsa Agent:
cd rl-library/mcshapeagent/
./sarsa.bash
- TAMER Agent:
cd rl-library/mcshapeagent/
./runExperiment.bash
----
Tetris agent:
Ways to run (while RLViz has been started and the automatic agent loader is disabled):
- TAMER Agent:
cd rl-library/tetrisagent/
./run.bash
u.a.b.u...@gmail.com
Apologies for this late reply, I was away.
I have tested your guidance above but unfortunately the same thing has reoccurred. I even added the classpath (RLVizLib.jar). As below :-
SET ProductPath=D:\RL-Library\products
SET RLVizApp=D:\RL-Library\system\common\libs\rl-viz
echo "-- Starting up dynamic agent loader -"
java -classpath %RLVizApp%\RLVizLib.jar -enableassertions -Xmx128M -jar %RLVizApp%\AgentShell.jar agent-jar-path=%ProductPath%
I got the same error message as aforementioned.
2) Yes I am running everything on Java (Environment , Agent, RL-Glue).
I am notfamiliar with Cygwin. I just simply use batch command, notepad and Java in my Windows.
Hope to hear from you.
Thanks.
Baker.
Brian Tanner
W Bradley Knox
u.a.b.u...@gmail.com
Thanks for your reply. But Cygwin gives installation issues.
After the mirror installation.....
u.a.b.u...@gmail.com
Thanks for your reply. Cygwin is just another problem.
>>Install the following packages: X windows; Python; in Web, wget; in Editors, >>Emacs and Vim; in Devel, subversion, make, and gcc:g++.
There were no X Windows and Python found in Web directory. Python has its own directory with many2x sub directories.
This is just another frustration.
u.a.b.u...@gmail.com
I just wish to have RlVizApp runs on my Windows as picture's below :-
W Bradley Knox
I won't be able to help further with this direction. I should note that the picture you linked to was taken on a Mac, judging by the window.
I just wish to have RlVizApp runs on my Windows as picture's below :-
http://xn--p8ja5bwe1i.jp/wiki/?plugin=attach&refer=%E5%BC%B7%E5%8C%96%E5%AD%A6%E7%BF%92%2FMac%E3%81%A7RL-Glue%E3%82%92%E4%BD%BF%E3%81%86&openfile=RLVizApp.png
u.a.b.u...@gmail.com
u.a.b.u...@gmail.com
Dear Gentlemen,
My apologies for keep posting questions during your busy hours. I much appreciate your moral support.
I found and got RL-Applet working well in my windows here (smile).
Got it from the link below:-
https://groups.google.com/forum/#!topic/rl-library/Ptm3Og1MvT8
But with limitation for CartPole environment, wondering if this could also be working with different environment (e.g Puddle world, Mountain Car).
Many thanks.
-Baker
W Bradley Knox
Here are more recent instructions: http://www.cs.utexas.edu/~bradknox/rlapplet/RL-Applet.html.
Just download the jar file for the desired environment from RL-Library the instructions I linked.
u.a.b.u...@gmail.com
Yes !!!! ....everything works like charm in Applet.......
AgentInterface agent = new RandomAgent();
EnvironmentInterface env = new GridWorld();
Brad, I am attempting to implement QLearning (optimal policy) for your GridWorld.
Do you mind to guide me how to replace RandomAgent by implementing my own QLearningAgent?
Best regards,
Baker.
W Bradley Knox
agent_end() is called when an episode ends.
I hope that helps!
u.a.b.u...@gmail.com
Hi Brad,
RLGlue framework works like charm, it is good idea to put "agent_step()" at Agent/QTable side and "environment_step()=reward" at uncharted Environment side.
Wonderful!.
For now I have my own 2 environments, GridWorld and Cartesian Coordinate...really cool !.
I have a question, as for now I am attempting to solve Tower Of Hanoi problem, and usually Tower Of Hanoi involving only 3 rings on 3 towers.
The question is, what could we do if we have 800 Towers with 800 rings? Don't we are facing great scalability issue? :-( It sounds scary !.
Baker.
W Bradley Knox
Baker.
u.a.b.u...@gmail.com
Dear Brad,
Thanks for your suggestion.Anyway, could you be more specific of which rl-list email would be more suitable? Apologies that I am a newbie for this google posting (smile)
Regards,
Baker.