intermediate status of downloader

[new_trader]

the backtest data downloader has the following intermediate status:
- when downloading two instruments with BarSize=1sec TWS very often
raises a pacing violation (IB error 162). the cause is that within 10
min only 60 requests are allowed. A pause of 11 seconds between each
request does not cure that. It seems that TWS is generating a pacing
violation much earlier. i will have to do more research on this.
- the RTH (regular trading hours) times for different instruments are
often different. the EUR CASH contract e.g. has other RTH than the EUR/
USD future. I will create a dummy TradingSchedule object from 9:30 to
22:00 which defines the boundaries of what should go into the backtest
file. this has to be done to align the time series.

the rest of the downloader is working so far. instruments with aligned
RTH can already be downloaded.

[nonlinear5]

> the backtest data downloader has the following intermediate status:
> - when downloading two instruments with BarSize=1sec TWS very often
> raises a pacing violation (IB error 162). the cause is that within 10
> min only 60 requests are allowed. A pause of 11 seconds between each
> request does not cure that. It seems that TWS is generating a pacing
> violation much earlier.

There is an additional limitation that only one historical data
request can be done at a time. In JSystemTrader, this is handled in
the following code. Are you doing something similar?

// Only one historical data request can hit the socket at a time, so
// we wait here to be notified that no historical requests are
pending.
synchronized (trader) {
while (trader.getIsPendingHistRequest()) {
trader.wait();
}

trader.setIsPendingHistRequest(true);
long elapsedSinceLastRequest = System.currentTimeMillis()
- lastRequestTime;
long remainingTime = MAX_REQUEST_FREQUENCY_MILLIS -
elapsedSinceLastRequest;
if (remainingTime > 0) {
Thread.sleep(remainingTime);
}
}

[new_trader]

I have used the code from JST, so this issue is handled correctly.

Now it is working fine, I just had to increase the wait time between
two calls to 20 seconds. now it downloads everything properly. it just
takes a while to download 1 day of data, approx. 20-30 minutes.

in the moment I am implementing the time series alignment with the
TradingSchedule class. Works fine, just have to do some extra handling
on the "edges" of the TradingSchedule.
When time series a is larger than time series bthe first and the last
Pricebar have to be cut off.

[new_trader]

have just uploaded a first file from the downloader:
http://groups.google.com/group/jarbitrager/web/DWNLDR_EUR_EUR.txt
-> the time window is from 09:30:01 to 15:59:59 on July 9

instrument 1 is the EUR cash contract, instrument 2 is the EUR future
expiration Sep/2010

when backtesting, it only generates 17 trades.
when backtesting with the recorded file (JAB 102 format) from our test
session form July 9, the backtester generates 188 trades.

-> I think we will have to do some analysis on this

the next days I will clean up the code of the downloader and post it.

[new_trader]

I have just uploaded an intermediate version of the downloader based
on the JAB 1.02 code base:
http://groups.google.com/group/jarbitrager/web/source.zip

It is for those of you wo want to be early adaptors. note that this
is a version still in heavy development.
But the class structure will remain the same.
note that in BackDataDownloader.java there is a debugging code line
which allows me to serialize downloaded data to disk. This saves lots
of time when doing the time series alignment and normalization.
with the boolean class member
private boolean bBarSave = true/false;
you can toggle if the data should be saved to disk or loaded from
disk. when it is set to false no IB download action takes place and
for time series alignment the previously downloaded bars are loaded
from disk.
for this class I am still in heavy testing. but it produces already
some good results.

i have tested each instrument pair combination of the sample
strategies:
GBP_GBP
EUR_EUR
ES_NQ
-> these worked fine. 1 second data was downloaded successfully

NQ_QQQQ
ES_SPY
YM_DIA
-> here TWS delivered erroneous data! duplicate timestamps where
downloaded

please play around and test :-)

[Eugene Kononov]

The link seems to be broken, and when I tried to download directly from the "Files" section, it complains that it's not a valid zip file. Can you upload again, please?

[new_trader]

> The link seems to be broken, and when I tried to download directly from the
> "Files" section, it complains that it's not a valid zip file. Can you upload
> again, please?

it looks like google groups has problems with uploaded ZIP files!
here is a new link:
http://groups.google.com/group/jarbitrager/web/jab_downloader_source.jar