Interactive Brokers Historical Data Limitations

[new_trader]

IB has some cumbersome limitations when downloading historical data.
unter http://www.interactivebrokers.com/php/apiUsersGuide/apiguide.htm
search for "historical data limitations"

this will be a big obstacle for our downloader!

for example a citation from the IB guide:
"
The following conditions can cause a pacing violation:
Making identical historical data requests within 15 seconds;
Making six or more historical data requests for the same Contract,
Exchange and Tick Type within two seconds.

Also, observe the following limitation when requesting historical
data:
· Do not make more than 60 historical data requests in any
ten-minute period.
" citation end.
1 second bars can only be downloaded for the duration window of 1800 S
(=30 minutes).
if you want to have an 8 hour trading day this will be 16 calls to IB.
within the above mentioned limit of 60 calls in a ten-minute period
you can grab 3 days of data in 10 minutes.
to download a month of data will take more than an hour.

I will have to do some research on this...

[nonlinear5]

The data downloaded in JSystemTrader already takes these limitations
into consideration, so you may want to reuse that code.

On Jul 9, 10:53 am, new_trader <loc...@gmx.de> wrote:
> IB has some cumbersome limitations when downloading historical data.

> unterhttp://www.interactivebrokers.com/php/apiUsersGuide/apiguide.htm

[new_trader]

That's great if it is considered!
here you can see a little screenshot of our downloader:
http://groups.google.com/group/jbooktrader/web/JAB_Downloader.png

have you ever experimented with the BID, ASK and BID_ASK parameter in
reqHistoricalData()?

I have played around with them in the IB sample program and got the
fowllowing result:
B/A = results with BID_ASK parameter
B/A: id=2 date = 20100709 16:20:18 open=44.34 high=44.35 low=44.34
close=44.35 volume=-1 count=-1 WAP=-1.0 hasGaps=false
BID: id=2 date = 20100709 16:20:18 open=44.34 high=44.34 low=44.34
close=44.34 volume=-1 count=-1 WAP=-1.0 hasGaps=false
ASK: id=2 date = 20100709 16:20:18 open=44.35 high=44.35 low=44.35
close=44.35 volume=-1 count=-1 WAP=-1.0 hasGaps=false

in the B/A I can not identify ASK or BID fields :-(

[new_trader]

update on BID_ASK Parameter from IB doku 8.61:
http://www.ibinstitutions.com/en/software/apiReleaseNotes/api_86.php?ib_entity=inst
citation:
The “whatToShow” parameter determines the nature of the data
extracted , with allowable values being “TRADES,” “MIDPOINT,” “BID,”
“ASK,” or “BID_ASK.” Bars of the first four types contain the start
time, open, high, low, close, volume, and weighted average price
during the time slice in question. The contents of bars returned in
response to a BID_ASK query differ from those returned by the other
query types, in that the open and close values are actually the time
weighted average bid, and time weighted average offer, respectively.
This makes these bars identical in nature to TWS’s “BID_ASK”
candlestick chart bars.
citation end.
essence: -> the open and close values are actually the time weighted
average bid, and time weighted average offer, respectively.

I will use these values as the bid/ask values

If you want to have it the offer way as you proposed earlier I will
have to download BID separately and ASK separately. But this will make
things a lot more complicated.

please give feedback.

[nonlinear5]

> That's great if it is considered!
> here you can see a little screenshot of our downloader:http://groups.google.com/group/jbooktrader/web/JAB_Downloader.png
>

It's looking good, new_trader, good job.

> have you ever experimented with the BID, ASK and BID_ASK parameter in
> reqHistoricalData()?
>

Nope, I've only dealt with the downloader to the extend that
JSystemTrader needed, and that was a while ago.

[nonlinear5]

> The “whatToShow” parameter determines the nature of the data
> extracted , with allowable values being “TRADES,” “MIDPOINT,” “BID,”
> “ASK,” or “BID_ASK.” Bars of the first four types contain the start
> time, open, high, low, close, volume, and weighted average price
> during the time slice in question. The contents of bars returned in
> response to a BID_ASK query differ from those returned by the other
> query types, in that the open and close values are actually the time
> weighted average bid, and time weighted average offer, respectively.
> This makes these bars identical in nature to TWS’s “BID_ASK”
> candlestick chart bars.
> citation end.
> essence: -> the open and close values are actually the time weighted
> average bid, and time weighted average offer, respectively.
>
> I will use these values as the bid/ask values
>
> If you want to have it the offer way as you proposed earlier I will
> have to download BID separately and ASK separately. But this will make
> things a lot more complicated.
>
> please give feedback.

I think the BID_ASK type is the most appropriate in our situation.
Once your downloaded is done, we'll simply download some recent period
which we've already recorded live, and compare the data sets (either
directly or by running backtests against both sets).

[new_trader]

> I think the BID_ASK type is the most appropriate in our situation.
> Once your downloaded is done, we'll simply download some recent period
> which we've already recorded live, and compare the data sets (either
> directly or by running backtests against both sets).

OK, great, then we will procede this way.
I think we will only be able to compare by backtest, since your new
format is completely different from the downloaded format, e.g.
duplicate lines etc. as you have mentioned before.

I will browse the JST code base if it is has automatic handling of the
most liquid contract for futures. I will have to consult you when I am
at this point.