Web Optimizer for JBookTrader

95 views
Skip to first unread message

new_trader

unread,
Aug 18, 2010, 8:53:53 AM8/18/10
to JBookTrader
Under this link
http://groups.google.com/group/jbooktrader/web/JBTWebOptimizer.jar
you will find a web based version of the JBT optimizer.
It is intended to be used in headless environments with access via web
browser.
My personal use case:
I am using it in the Amazon cloud for heavy duty optimization jobs.
On my local machine some optimization jobs are running for ~8 hours.
In the Amazon cloud it is possible to use the Cluster Compute Instance
of type Quadruple Extra Large which costs $1.60 per hour.
There is an Amazon Machine Image (AMI) with the id ami-7ea24a17 with
CentOS which can be run with this instance type. The JBT benchmark
with the Equalizer strategy needs 377 seconds to complete. On Eugene’s
Intel i7-920 machine this benchmark needed 688 seconds – see this
thread for reference:

http://groups.google.com/group/jbooktrader/browse_thread/thread/cb5a20bdcfc47c81/eea0042e413c196f

So JBT in the Cloud is very fast!

How to use the JBT Web Optimizer (JBTWO):
-----------------------------------------------------------------

Start it on the command line as you start JBT. My recommended setup:
“java –server –Xmx12288M JBTWebOptimizer.jar <directory where you
start JBTWO>”
Important is the –server switch: here the Java VM runs in server mode.
In many cases this doubles the execution speed of Java programs, in
the JBT case it doubles the speed.
The Quadruple Extra Large Instance type has 23 GB RAM so 12GB for
JBTWO is OK. Of course for starters you can use –Xmx2014M
The above command line starts JBTWO. It starts an inline Jetty servlet
container. The JBTWO servlet is listening on port 80. It has no
timeout so the optimization session can run endlessly.
When testing on your local machine simply enter http://localhost in
your favorite browser. When running in the Cloud you have to enter the
public IP address of your Amazon AWS instance.
The start screen lists the strategies contained in the JAR file.
Select one strategy and press the Optimize… button. This brings you to
the optimizer screen. You have to enter manually the path of the data
file with your tick data. In contrast to JBT no file selector is
available. You can select the data range. The calendar selector seems
to have a little display bug when it is set to status disabled. I will
have to check this. But it is only cosmetics. In the parameters
section you can edit your parameters as you are used to from JBT. When
pressing the Advanced… button you see the advanced optimization
options. As I have no slider control you have to enter manually the
parameter for the density of the Divide&Conquer optimizer.
When you have setup all your stuff you can press the Optimize button.
The optimizer starts as you know it from Swing JBT. The screen behaves
nearly exactly like his Swing based big brother.
There are some layout issues but this is only cosmetics.
There is an issue when the optimization run has a lot of results. Then
the results table becomes unresponsive, de facto you cannot use it.
This can happen when you set the number of trades to a very low
number.
In the jar file I have included the strategies delivered with JBT
7.06. If you want to test your own strategies simply put them in the /
com/jbooktrader/strategy folder in the JAR file. Technically a Java
JAR file is a ZIP file so you can do this with Winzip. In the JAR file
are many other files since I have packed it as a single Jar file, so
don’t worry.
JBTWO is based on the JBT706 code base with only one little
modifications to com.jbooktrader.platform.startup.JBookTrader.java and
com.jbooktrader.platform.util.ClassFinder.java

I will do some cleanup on my code and then publish it on this web site
in the next few days.

John-Crichton McCutcheon

unread,
Aug 18, 2010, 10:54:44 AM8/18/10
to jbook...@googlegroups.com
This sounds awesome. JBT in a client-server architecture is
interesting.


On 8/18/2010 8:53 AM, new_trader wrote:
> Under this link
> http://groups.google.com/group/jbooktrader/web/JBTWebOptimizer.jar
> you will find a web based version of the JBT optimizer.
> It is intended to be used in headless environments with access via web
> browser.
> My personal use case:
> I am using it in the Amazon cloud for heavy duty optimization jobs.
> On my local machine some optimization jobs are running for ~8 hours.
> In the Amazon cloud it is possible to use the Cluster Compute Instance
> of type Quadruple Extra Large which costs $1.60 per hour.
> There is an Amazon Machine Image (AMI) with the id ami-7ea24a17 with
> CentOS which can be run with this instance type. The JBT benchmark

> with the Equalizer strategy needs 377 seconds to complete. On Eugene�s
> Intel i7-920 machine this benchmark needed 688 seconds � see this


> thread for reference:
>
> http://groups.google.com/group/jbooktrader/browse_thread/thread/cb5a20bdcfc47c81/eea0042e413c196f
>
> So JBT in the Cloud is very fast!
>
> How to use the JBT Web Optimizer (JBTWO):
> -----------------------------------------------------------------
>
> Start it on the command line as you start JBT. My recommended setup:

> �java �server �Xmx12288M JBTWebOptimizer.jar<directory where you
> start JBTWO>�
> Important is the �server switch: here the Java VM runs in server mode.


> In many cases this doubles the execution speed of Java programs, in
> the JBT case it doubles the speed.
> The Quadruple Extra Large Instance type has 23 GB RAM so 12GB for

> JBTWO is OK. Of course for starters you can use �Xmx2014M


> The above command line starts JBTWO. It starts an inline Jetty servlet
> container. The JBTWO servlet is listening on port 80. It has no
> timeout so the optimization session can run endlessly.
> When testing on your local machine simply enter http://localhost in
> your favorite browser. When running in the Cloud you have to enter the
> public IP address of your Amazon AWS instance.
> The start screen lists the strategies contained in the JAR file.

> Select one strategy and press the Optimize� button. This brings you to


> the optimizer screen. You have to enter manually the path of the data
> file with your tick data. In contrast to JBT no file selector is
> available. You can select the data range. The calendar selector seems
> to have a little display bug when it is set to status disabled. I will
> have to check this. But it is only cosmetics. In the parameters
> section you can edit your parameters as you are used to from JBT. When

> pressing the Advanced� button you see the advanced optimization


> options. As I have no slider control you have to enter manually the
> parameter for the density of the Divide&Conquer optimizer.
> When you have setup all your stuff you can press the Optimize button.
> The optimizer starts as you know it from Swing JBT. The screen behaves
> nearly exactly like his Swing based big brother.
> There are some layout issues but this is only cosmetics.
> There is an issue when the optimization run has a lot of results. Then
> the results table becomes unresponsive, de facto you cannot use it.
> This can happen when you set the number of trades to a very low
> number.
> In the jar file I have included the strategies delivered with JBT
> 7.06. If you want to test your own strategies simply put them in the /
> com/jbooktrader/strategy folder in the JAR file. Technically a Java
> JAR file is a ZIP file so you can do this with Winzip. In the JAR file
> are many other files since I have packed it as a single Jar file, so

> don�t worry.

Eugene Kononov

unread,
Aug 18, 2010, 10:18:57 PM8/18/10
to jbook...@googlegroups.com
Thanks. There are no source files in that JAR. I look forward to looking at how you've done it.


--
You received this message because you are subscribed to the Google Groups "JBookTrader" group.
To post to this group, send email to jbook...@googlegroups.com.
To unsubscribe from this group, send email to jbooktrader...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/jbooktrader?hl=en.


Klaus

unread,
Aug 21, 2010, 1:09:41 PM8/21/10
to JBookTrader
This is really cool work.
I could not test it yet. I am only wondering: are heatmaps
(visualization of the results) also supported?

Klaus


On Aug 19, 4:18 am, Eugene Kononov <eugene.kono...@gmail.com> wrote:
> Thanks. There are no source files in that JAR. I look forward to looking at
> how you've done it.
>
>
>
> On Wed, Aug 18, 2010 at 8:53 AM, new_trader <loc...@gmx.de> wrote:
> > Under this link
> >http://groups.google.com/group/jbooktrader/web/JBTWebOptimizer.jar
> > you will find a web based version of the JBT optimizer.
> > It is intended to be used in headless environments with access via web
> > browser.
> > My personal use case:
> > I am using it in the Amazon cloud for heavy duty optimization jobs.
> > On my local machine some optimization jobs are running for ~8 hours.
> > In the Amazon cloud it is possible to use the Cluster Compute Instance
> > of type Quadruple Extra Large which costs $1.60 per hour.
> > There is an Amazon Machine Image (AMI) with the id ami-7ea24a17 with
> > CentOS which can be run with this instance type. The JBT benchmark
> > with the Equalizer strategy needs 377 seconds to complete. On Eugene’s
> > Intel i7-920  machine this benchmark needed 688 seconds – see this
> > thread for reference:
>
> >http://groups.google.com/group/jbooktrader/browse_thread/thread/cb5a2...
> > jbooktrader...@googlegroups.com<jbooktrader%2Bunsubscribe@googlegr oups.com>
> > .

new_trader

unread,
Aug 21, 2010, 5:48:54 PM8/21/10
to JBookTrader
heatmaps of the com.jbooktrader.platform.chart.OptimizationMap class
are not implemented.
my use case does always brute force optimization so I do not need heat
maps.

the org.jfree.chart.JFreeChart class needs a Java2D graphics device.
Will be hard to implement.
I think it can be done when JFreeChart is drawing the chart to a JPEG
stream which is then displayed as an org.wings.SIcon object.

Klaus

unread,
Aug 22, 2010, 5:20:47 AM8/22/10
to JBookTrader
Thanks for clarifying.
I thought it would be difficult, that's why I am asking.
In my use case I examine the heatmaps on top of the optimization.
If the spot with good results is too small, I dump the approach,
because then the
resulting method is probably too brittle in practice.
But your mileage may vary. Good luck!

Klaus

new_trader

unread,
Aug 22, 2010, 1:39:41 PM8/22/10
to JBookTrader
OK, I see your point, a good one!
if Eugene is willing to merge the JBTWO source into the JBT code line
I will integrate heat maps into JBTWO

Javier

unread,
Sep 1, 2010, 7:31:28 AM9/1/10
to JBookTrader
> How to use the JBT Web Optimizer (JBTWO):
> -----------------------------------------------------------------
>
> Start it on the command line as you start JBT. My recommended setup:
> “java –server –Xmx12288M JBTWebOptimizer.jar <directory where you
> start JBTWO>”


I've tried to run JBTWebOptimizer under windows and linux but I'm
having problems when running, see below:

Under windows:

>java -version

java version "1.6.0_19"
Java(TM) SE Runtime Environment (build 1.6.0_19-b04)
Java HotSpot(TM) 64-Bit Server VM (build 16.2-b04, mixed mode)


>java -server -jar JBTWebOptimizer.jar .

2010-09-01 13:18:02.833:INFO::Logging to STDERR via
org.mortbay.log.StdErrLog
2010-09-01 13:18:02.889:INFO::jetty-6.1.x
01-sep-2010 13:18:02 org.wings.session.WingServlet init
INFO: Initializing wingS global servlet with configuration:
01-sep-2010 13:18:02 org.wings.session.WingServlet init
INFO: wings.mainclass =
com.jbooktrader.weboptimizer.wings.MainScreen
01-sep-2010 13:18:02 org.wings.session.WingServlet initLookupName
INFO: use session servlet lookup name
SessionServlet:com.jbooktrader.weboptimizer.wings.MainScreen
2010-09-01 13:18:02.958:WARN::failed SocketC...@0.0.0.0:80:
java.net.Socket Exception: Unrecognized Windows Sockets error: 0:
JVM_Bind
2010-09-01 13:18:02.959:WARN::failed Server@2acdb06e:
java.net.SocketException:
Unrecognized Windows Sockets error: 0: JVM_Bind
java.net.SocketException: Unrecognized Windows Sockets error: 0:
JVM_Bind
at java.net.PlainSocketImpl.socketBind(Native Method)
at java.net.PlainSocketImpl.bind(Unknown Source)
at java.net.ServerSocket.bind(Unknown Source)
at java.net.ServerSocket.<init>(Unknown Source)
at java.net.ServerSocket.<init>(Unknown Source)
at
org.mortbay.jetty.bio.SocketConnector.newServerSocket(SocketConnector.java:
80)
at
org.mortbay.jetty.bio.SocketConnector.open(SocketConnector.java:73)
at
org.mortbay.jetty.AbstractConnector.doStart(AbstractConnector.java:
283)
at
org.mortbay.jetty.bio.SocketConnector.doStart(SocketConnector.java:
147)
at
org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:
50)
at org.mortbay.jetty.Server.doStart(Server.java:235)
at
org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:
50)
at
com.jbooktrader.weboptimizer.wings.JBTWebOptimizerStarter.<init>(JBTWebOptimizerStarter.java:
64)
at
com.jbooktrader.weboptimizer.wings.JBTWebOptimizerStarter.main(JBTWebOptimizerStarter.java:
84)




Under Linux:

# java -version
java version "1.5.0_11"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_11-b03)

# java -server -jar JBTWebOptimizer.jar
Exception in thread "Main Thread"
java.lang.UnsupportedClassVersionError: unsupported classversion 50.0
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:620)
at
java.security.SecureClassLoader.defineClass(SecureClassLoader.java:
124)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:
260)
at java.net.URLClassLoader.access$100(URLClassLoader.java:56)
at java.net.URLClassLoader$1.run(URLClassLoader.java:195)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:
268)
at java.lang.ClassLoader.loadClass(ClassLoader.java:251)

new_trader

unread,
Sep 1, 2010, 8:36:45 AM9/1/10
to JBookTrader
Hi Javier,
first thanks for testing the JBT Web Optimizer.

you have mentioned two problems, one on windows and one on linux.

your windows problems seems is network related.
JBTWO tries to open TCP port 80 and then listens for incoming request.
when your firewall is blocking port 80 this could be a problem.
another cause for this problem can be that another application is
already using port 80. you can check this with the netstat command in
the windows command shell:
netstat -an shows you all opened ports.

your linux related problem is a Java problem. JBT and JBTWO must have
Java version 1.6
On your linux box the default Java system has the version 1.5.0_11
if you upgrade to Java version 1.6 this error should disappear.
> 2010-09-01 13:18:02.958:WARN::failed SocketConnec...@0.0.0.0:80:

John-Crichton McCutcheon

unread,
Sep 2, 2010, 9:12:29 AM9/2/10
to jbook...@googlegroups.com
Perhaps this explains why the order book is not a consistent
predictor of future direction.

http://online.wsj.com/article/SB10001424052748703882304575465990082237642.html

Klaus

unread,
Sep 2, 2010, 9:39:19 AM9/2/10
to JBookTrader
A very interesting article. Thanks for posting.
However, I do not understand what is the purpose of posting
it in this thread, let alone hijacking the thread by modifying the
title!?

Therefore I modified the title back.

Cheers
Klaus


On 2 Sep., 15:12, John-Crichton McCutcheon
<john.crichton.mccutch...@gmail.com> wrote:
>   Perhaps this explains why the order book is not a consistent
> predictor of future direction.
>
> http://online.wsj.com/article/SB1000142405274870388230457546599008223...

John-Crichton McCutcheon

unread,
Sep 2, 2010, 12:01:59 PM9/2/10
to jbook...@googlegroups.com
Sorry, My bad. Was in a hurry.

Klaus

unread,
Sep 2, 2010, 12:22:26 PM9/2/10
to JBookTrader
..and again. Looks like you really do not like the thread topic.
Just for the records: whenever you set a topic explicitly it is
marked as new topic of the thread and becomes the only one shown in
the overview.. :)


On 2 Sep., 18:01, John-Crichton McCutcheon

Javier

unread,
Sep 14, 2010, 1:21:57 PM9/14/10
to JBookTrader
> I am using it in the Amazon cloud for heavy duty optimization jobs.

I'd like to reuse some old desktop computers to create a cluster for
heavy duty optimization jobs, do you know if JBTWebOptimizer could be
used in a cluster using ParallelKnoppix?

More info about PararllelKnoppix:
http://idea.uab.es/mcreel/ParallelKnoppix/

new_trader

unread,
Sep 15, 2010, 4:05:31 AM9/15/10
to JBookTrader
>> I'd like to reuse some old desktop computers to create a cluster for
> heavy duty optimization jobs, do you know  if JBTWebOptimizer could be
> used in a cluster using ParallelKnoppix?

Interesting idea :-)

JBTWebOptimizer in its current state is just another frontend for
JBookTrader.
This means that JBTWO is utilizing the JBT code base including a
slightliy adopted version of the strategy optimizer.
So JBTWO is not capable for clustered computing.

An architecture for using JBT in a clustered environment could follow
the MapReduce approach (http://en.wikipedia.org/wiki/MapReduce).

I think in the Map step the parameter sets for the strategies should
be determined and distributed to the available computing nodes.
On these nodes the data file with the back testing data should reside
because transfering 100MB of data file contents to each node would be
a bottleneck.

I think there are two possibilities to enable JBT for clustered
computing:
- develop an adoption to the Hadoop framework. Then you can run it
directly in the Amazon cloud, it uses the hadoop framework in its
Amazon Elestic MapReduce service
- using GridGain as framework for clustering. By using this approach
you have more manual work to do but also you have more control. For
your scenario this would be an intersting option

Javier

unread,
Sep 15, 2010, 5:22:06 AM9/15/10
to JBookTrader
> your linux related problem is a Java problem. JBT and JBTWO must have
> Java version 1.6
> On your linux box the default Java system has the version 1.5.0_11
> if you upgrade to Java version 1.6 this error should disappear.

Confirmed, you were right, it's working with JVM 1.6. JBTWO is a good
solution to work with amazon cloud!

I've a couple of suggestions for JBTWO:

- login protection for the web interface
- possibility of specify the listening port when starting the .jar
(port 80 is very used by other apps)

new_trader

unread,
Sep 15, 2010, 5:40:54 AM9/15/10
to JBookTrader
> I've a couple of suggestions for JBTWO:
>
> - login protection for the web interface

-> in the next release I will use the user/pwd combination from the
JBT web console

> - possibility of specify the listening port when starting the .jar
> (port 80 is very used by other apps)

-> very good point, will be in the next release, too

Javier

unread,
Sep 15, 2010, 10:04:43 AM9/15/10
to JBookTrader
> -> very good point, will be in the next release, too

I forgot another point:

If I'm running an optimization and after some time I close the browser
or the internet connection is interrupted, when I open again the
browser and point to the previous address I can't see the previous
optimization (still in progress). I guess this could be fixed easily,
too.

Klaus

unread,
Dec 27, 2011, 7:51:26 AM12/27/11
to jbook...@googlegroups.com
Hi, 

coming back to work a bit with AWS, I am wondering what happened to this thread and the associated software?
Is someone still using it?
As far as I can see, there results were not merged with JBT. (a pity)
(If I look at the current AWS pricing, today, a micro instance (spot) is about 0.006 USD, i.e., renting 100 for an hour would be just $0.6 ;-) or more than 150 for a USD/hour..
My experience so far, is the micro instances are not very powerful, but they might have the best cost-benefit for massive parallel problems..
Alternatively: Eight Extra Large 88 EC2 Compute Units (eight-core 2 x Intel Xeon) cost $0,54  (spot price at time of writing)
One or two of them would also form a powerfull possibility..  (at the time of the thread start above less than half the power did cost $1.60 ... and was already twice as fast as the computing reference above..)
For bursts the the micro instances have 2 EC2, meaning for $0.54 this would be about 180 EC2. - However, I am not sure about AWS definition of "short burst".. i.e., wether this is seconds or minutes. Moreover, this would require significant autoscaling and automated load distribution..

Cheers
 Klaus



Judson Wilson

unread,
Dec 27, 2011, 1:19:07 PM12/27/11
to jbook...@googlegroups.com
The one time I used a micro instance, it seemed worse than useless.  It REALLY is bursty - i.e. it is fast for about 10 seconds and then it just staaaallllllllllssssss. 

 Klaus



--
You received this message because you are subscribed to the Google Groups "JBookTrader" group.
To view this discussion on the web visit https://groups.google.com/d/msg/jbooktrader/-/S4aSbnG0agUJ.

To post to this group, send email to jbook...@googlegroups.com.
To unsubscribe from this group, send email to jbooktrader...@googlegroups.com.

Judson Wilson

unread,
Dec 27, 2011, 1:23:19 PM12/27/11
to jbook...@googlegroups.com

coming back to work a bit with AWS, I am wondering what happened to this thread and the associated software?
Is someone still using it?

As far as I can see, there results were not merged with JBT. (a pity)

I just realized... are you talking about grid-gain, etc?  I too am interested in this, but never saw anything come of it.

Klaus

unread,
Dec 27, 2011, 2:21:19 PM12/27/11
to jbook...@googlegroups.com
ok, I will see. - So far, this has certainly not been the case. - It is highly responsive. With OpenNX it is even more reponsive in X-Windows use than another server in Germany where I use VNC. It might be different, if AWS is highly loaded.
(There were rather extreme spot prices in December. - probably due to high christmas shopping load.
This might have given negative effects also to responsiveness.)
Currently I am collecting data on AWS and the server I used for this for many months already.
I will check the resulting data and see.

Regarding your other post: grid gain? I do not know the name, but I am referring to the context of the thread I am posting this in. (See opening message by new_trader)

Reply all
Reply to author
Forward
0 new messages