JBT in the cloud

[Klaus]

Hi, 

already for some time I thought about putting JBT somewhere on a virtual server. 

Due to recent experience: https://groups.google.com/d/msg/jbooktrader/acqYD4iYpSo/u52Rlb4wql0J

it might well pay off. 

Has someone specifically positive experience with a provider (especially for trading ES at GLOBEX). 

I am not looking for the highest end solution, but a price-effective V-server solution, which nevertheless has 

good response time and good connections to IB.

Klaus

[Judson Wilson]

linode.com, 256m of RAM option running ubuntu.

--
You received this message because you are subscribed to the Google Groups "JBookTrader" group.
To view this discussion on the web visit https://groups.google.com/d/msg/jbooktrader/-/If_vaGaQIGEJ.
To post to this group, send email to jbook...@googlegroups.com.
To unsubscribe from this group, send email to jbooktrader...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/jbooktrader?hl=en.

[Eugene Kononov]

On Sun, Jul 31, 2011 at 12:21 AM, Judson Wilson <wilson...@gmail.com> wrote:

linode.com, 256m of RAM option running ubuntu.

Did you compare this to Amazon, Judson? A small instance of Linux on Amazon is $7.32 per month. I am not sure if it's apples-to-apples comparison, though. I did talk to someone who was running JBT on Amazon, and I even looked at his event report. Executions were in the 100ms to 300ms range, which is pretty good.

A small instance on Amazon includes:
1.7 GB memory
1 EC2 Compute Unit (1 virtual core with 1 EC2 Compute Unit)
160 GB instance storage
32-bit platform
I/O Performance: Moderate


http://calculator.s3.amazonaws.com/calc5.html
http://aws.amazon.com/ec2/instance-types/

[Eugene Kononov]

1.7 GB memory
1 EC2 Compute Unit (1 virtual core with 1 EC2 Compute Unit)
160 GB instance storage
32-bit platform
I/O Performance: Moderate

I forgot to mention: the $7.32 per month price fro this service is based on the 10hr/day usage. There is also a 24hr/day usage of a "micro" instance for $5.12 a month. 

[Klaus]

The amazon options are indeed competitive:

the micro instance would be $14,4 / month (30 x 24h) max. 

Or with a fixed booking $54 / year + $5,04 / month

.. and it seems the micro instance should be enough (with Linux) 613MB RAM. It is only unclear what bursts of CPU means and 

what their standard is (i.e., continuous CPU usage ok?)

Amazons network connection should be fine.. :)

[Eugene Kononov]

.. and it seems the micro instance should be enough (with Linux) 613MB RAM. It is only unclear what bursts of CPU means and 

what their standard is (i.e., continuous CPU usage ok?)

Yes, 613MB RAM should be enough. You can confirm it experimentally by running JBT in the trading or forward testing modes with this options: -Xms512M -Xmx512M. In regards to the CPU usage, JBT consumes very little, unless it's running the optimizer. The only possible issue withe "micro" instance is that it lists the I/O performance as "low", compared to the "moderate" for the small instance. If the "I/O" refers to the disk I/O, we don't care, as JBT uses very little of that. However, if this refers to the network throughput, it could mean that the latency may be more significant with the "micro" instance.

Micro Instance 613 MB memory
Up to 2 EC2 Compute Units (for short periodic bursts)
EBS storage only
32-bit or 64-bit platform
I/O Performance: Low
API name: t1.micro

Small Instance

1.7 GB memory
1 EC2 Compute Unit (1 virtual core with 1 EC2 Compute Unit)
160 GB instance storage
32-bit platform
I/O Performance: Moderate

API name: m1.small

I'll probably sign up for the "micro" instance, and see how it goes.

[Judson Wilson]

I've played with micro in different contexts, and I was not impressed.  I would not use it.  It goes from fast to VERY slow over different periods of time.

I did compare small for 24 hours a day, I think, a while ago, and it just didn't make sense.  Maybe it does now.  I am too lazy right now to wake up at like 3AM to turn it on in the morning.

256megs of ram is enough, no problems there, so long as you are running one single instance of JBT from the command line (not from within eclipse).

Back when I did the math, linode was the best choice.

--
You received this message because you are subscribed to the Google Groups "JBookTrader" group.

[Eugene Kononov]

Thanks, Judson. I have a question about the EC2 micro, since you used it. What would you use to run GUI based apps such as TWSGateway and JBT on a Linux micro EC2 instance remotely?

[Ali]

hi guys,

any security concerns over the TWS login info; and the code being
deployed in the hosted services?

cheers.

issy

On Aug 1, 2:11 am, Eugene Kononov <eugene.kono...@gmail.com> wrote:
> Thanks, Judson. I have a question about the EC2 micro, since you used it.
> What would you use to run GUI based apps such as TWSGateway and JBT on a
> Linux micro EC2 instance remotely?
>

> On Sun, Jul 31, 2011 at 1:57 PM, Judson Wilson <wilson.jud...@gmail.com>wrote:
>
> > I've played with micro in different contexts, and I was not impressed.  I
> > would not use it.  It goes from fast to VERY slow over different periods of
> > time.
>
> > I did compare small for 24 hours a day, I think, a while ago, and it just
> > didn't make sense.  Maybe it does now.  I am too lazy right now to wake up
> > at like 3AM to turn it on in the morning.
>
> > 256megs of ram is enough, no problems there, so long as you are running one
> > single instance of JBT from the command line (not from within eclipse).
>
> > Back when I did the math, linode was the best choice.
>

> > On Sun, Jul 31, 2011 at 9:27 AM, Eugene Kononov <eugene.kono...@gmail.com>wrote:
>
> >>> .. and it seems the micro instance should be enough (with Linux) 613MB
> >>> RAM. It is only unclear what bursts of CPU means and
> >>> what their standard is (i.e., continuous CPU usage ok?)
>
> >> Yes, 613MB RAM should be enough. You can confirm it experimentally by
> >> running JBT in the trading or forward testing modes with this options:
> >> -Xms512M -Xmx512M. In regards to the CPU usage, JBT consumes very little,
> >> unless it's running the optimizer. The only possible issue withe "micro"
> >> instance is that it lists the I/O performance as "low", compared to the
> >> "moderate" for the small instance. If the "I/O" refers to the disk I/O, we
> >> don't care, as JBT uses very little of that. However, if this refers to the
> >> network throughput, it could mean that the latency may be more significant
> >> with the "micro" instance.

> >> *
> >> Micro Instance* 613 MB memory

> >> Up to 2 EC2 Compute Units (for short periodic bursts)
> >> EBS storage only
> >> 32-bit or 64-bit platform
> >> I/O Performance: Low
> >> API name: t1.micro

> >> *
> >> Small Instance*

[Judson Wilson]

I only tried gnome on ubuntu because that's all I really know.  I am sure there are other options, but there is definitely plenty of RAM for running it.  You might be able to run something smaller to help keep the computations down, but I am not the person to ask about that.

Also, you can try micro instances out for FREE for a good amount of time on Amazon, at the moment.  Nothing to lose there.

I had to dig around but I found the "OFFICIAL" ubuntu images to run off of.  Note that many (most?) of the images (AMI?) that are out there are people's home brew images, which may contain malicious software in them.

[Judson Wilson]

On Sun, Jul 31, 2011 at 11:35 AM, Ali <alphad...@gmail.com> wrote:

hi guys,

any security concerns over the TWS login info; and the code being
deployed in the hosted services?

cheers.

issy

The same concerns any sys-admin has when holding personal information.  Don't take security lightly.

 

[Judson Wilson]

Oh yes, one last thing, I set up a "Headless VNC Server". 

That should be enough info for you to hunt around on google. :)

[nonlinear]

As it turned out, the micro instance on EC2 is free with certain Ubuntu images, as Judson mentioned. So I signed up and followed this useful guide:
http://blog.restbackup.com/2011/01/how-to-use-amazon-ec2-as-your-desktop.html

My ec2 instance is now up and running. I'm in the middle of installing  TWS, JBT, and Java now.

[Klaus]

Dear Nonlinear: two questions: 

- regarding the free micro instance, I suppose you mean the one year for free option?

- did you already have experiences? Does it work for you?
  (I suppose the highly volatile market in the last two weeks should place extraordinarily high demands. If a micro

    instance is sufficient then, then it is probably always sufficient..)

Klaus

 

[Knm]

I've noticed the IBGateway run param specifies -Xmx512M. Since the
micro instance only has 613mb memory, I don't think micro is
sufficient to run both Jbooktrader and gateway simultaneously....

[Judson Wilson]

512M is a suggesting used as a starting place that assumes you will want to do backtesting and optimizing.  Clearly you would not do those on a micro instance - you would just do trading (which uses MUCH less memory, because you are not loading months of data into RAM).

256M of ram is enough to run Gnome, TWS, and JBT.  I have been doing it for months.

--
You received this message because you are subscribed to the Google Groups "JBookTrader" group.

[Knm]

Oh right I didn't realize Xmx means max heap size unlike Xms. Thanks

[Klaus]

.. I am wondering whether someone used a micro-instance in forwarding trading or even live trading in august.

Was ist able to provide the necessary performance in the volatile markets?

[Klaus]

Just a comment: if you are trying NX as is proposed in the guidance 

http://blog.restbackup.com/2011/01/how-to-use-amazon-ec2-as-your-desktop.html

I can recommend (at least on Mac / Lion) to use OpenNX. This went flawlessly also for session suspend / resume, 

and screen size issues, where I got many problems with the widely proclaimed NoMachine client software.

While using JBT from Germany gave me pig delays of 170-300 ms, using East-Zone AWS gives ping delays of about 25 ms ;-)

This is really nice. - And with FreeNX even using this remote is better than most other remotedesktop-system.

Have a nice day.

[Judson Wilson]

Old response, but I thought I would comment:

On Sun, Sep 4, 2011 at 11:46 AM, Klaus <klaus.s...@googlemail.com> wrote:

.. I am wondering whether someone used a micro-instance in forwarding trading or even live trading in august.

Was ist able to provide the necessary performance in the volatile markets?

My understanding is that there should be no difference with normal trading, from a software performance standpoint.
I believe IB sends 4 updates per second, and your indicators will update once a second, regardless of the volitility.

Now if it takes you 3 seconds to fill, it will always take 3 seconds to fill, and that could cause more slippage if
prices are more volatile - which I can't comment on.  It would be interesting to see people's average time to fill.
 

[Eugene Kononov]

Now if it takes you 3 seconds to fill, it will always take 3 seconds to fill, and that could cause more slippage if
prices are more volatile - which I can't comment on.  It would be interesting to see people's average time to fill.
 

Over hundreds of trades that I placed, the average time to fill was around 250 milliseconds. My network connection speed is about 5 mbps download, 0.7 mbps upload. I am in North Carolina, USA.

[Klaus]

I cannot comment on fill times with the AWS setup, as I only did set it up over the last few days 

and for now only for gathering data, but the previous setup (with ping times of around 300 ms) often required 

about 700ms, however with quite some fluctuation (which are probably due to the connection). 

So, I would expect with this setup to come close to 250 ms as well. - Although there might also be different delays

if the IB servers are heavily loaded (i.e., during real trading time). 

I will experiment with gathering data for the next few weeks from AWS. - And if this works nicely, I will probably move the trading to AWS as well. 

[Klaus]

Hi,

for those who are still interested. I had a JBT instance running in trade mode in parallel now for four weeks on the AWS instance. 

(It traded a non-trading system, so I did not yet make experience with running it in full swing.)

The setup was rather stable. (I.e., it only had problems relating to IB (i.e., connection loss that was the case also in my

regular solution)

With OpenNX it is also very nice to use, when one locks in for maintenance. (Much better than the previous one. Highly responsive.)

I now cross-checked the data on my previous setup with the current one. 

The obvious experience is: the second stamps are not in parallel, thus also exact price and volume may vary. (But I guess

this is the case for any two IB-setups.)

I also wrote a small script to further analyze the data (manually scanning 10 MB is no fun). - you find it below, so you can compare on your

own data. 

What I found with this is that it seems that at around 1:30 (am) several glitches happen, usually (perhaps due to AWS maintenance). 

Further I have quite regular each our one second of (do not know why, perhaps time correction). Apart from this, the other glitches are rather minor. (Especially, if compared to my previous setup, which also included a hosted virtual server.)

I will now switch the main (trading account) to AWS. Let's see. 

 

Overall I am quite happy as the AWS instance is quite cheap. However, this is only possible as the setup does not consume 

a lot of power. (It hardly creates load, micro instances are not for heavy load.)

Below the script (works on Mac, should work on Linux as well, Windows - uhm..

#!/bin/bash

# Support for checking time integrity of data files. 

# Usage for example:  ./checktime file1

# Provides the lines where an unexpected time laps is found. 

# Note that every minute 059->100 a pseudo-lapse occurs. This is taken care of. 

awk -F ',' '{if (!(($2 == A+1) || (($2 % 100 == 0) && ($2 = A+41)))) {print $2, A;}; A=$2; }' $1