RAW data format

[dyno]

I have made a special version just for recording raw market data. A
sample data file is listed here. Your inputs on the raw data format
are welcome.
Data converter will be needed to convert the raw data into whatever
data format you want. I will post the special build if there are
enough interests.

-dyno

http://jbooktrader.googlegroups.com/web/EUR.txt?gsc=wE31LgsAAAB86QbR_HOI7QPX0R4iq3uL

-------------------------------------------------------------------------------------------------------------------
# This historical data file is created by JBookTrader
# Each line represents a series of raw market update at a particular
time:
# date, time, number of updates;
# 1. date is in the MMddyy format
# 2. time is in the HHmmss.SSS format, followed by token ';'
# 3. Number of updates in this series, followed by token ';'
# 4. for each update, the format is (position, operation, side, price,
size), followed by token ';'

timeZone=America/New_York

060308,210541.629;24;0,0,1,1.5445,27632405;1,0,1,1.54445,10000000;2,0,1,1.5444,3000000;3,0,1,1.54435,4000000;4,2,1,1.5441,1000000;4,0,1,1.5441,1000000;0,0,0,1.5447,20500000;1,0,0,1.54475,10000000;2,0,0,1.5448,10131093;3,0,0,1.54485,4000000;4,2,0,1.545,1000000;4,0,0,1.545,1000000;0,1,1,1.5445,23132405;1,1,1,1.54445,14500000;0,1,0,1.54465,4500000;1,1,0,1.5447,16000000;2,1,0,1.54475,10000000;3,1,0,1.5448,10131093;4,1,0,1.54485,4000000;0,1,0,1.5446,7000000;1,1,0,1.54465,4500000;2,1,0,1.5447,9000000;3,1,0,1.54475,10000000;4,1,0,1.5448,10131093;
060308,210541.841;5;0,1,0,1.5446,12000000;2,1,0,1.5447,4000000;1,1,0,1.54465,14500000;3,1,0,1.5448,10131093;4,1,0,1.54485,4000000;
060308,210542.075;2;3,1,0,1.54475,4000000;4,1,0,1.5448,10131093;
060308,210546.935;9;0,1,0,1.5446,5000000;2,1,0,1.5447,11000000;1,1,0,1.54465,4500000;3,1,0,1.54475,14000000;0,1,0,1.54465,4500000;1,1,0,1.5447,16000000;2,1,0,1.54475,14000000;3,1,0,1.5448,10131093;4,1,0,1.545,1000000;
060308,210547.143;7;0,1,1,1.5445,27632405;1,1,1,1.54445,10000000;0,1,0,1.5447,20500000;1,1,0,1.54475,14000000;2,1,0,1.5448,10131093;3,1,0,1.545,1000000;4,1,0,1.54565,3125000;
060308,210547.377;7;0,1,1,1.5445,23132405;1,1,1,1.54445,14500000;0,1,0,1.54465,4500000;1,1,0,1.5447,16000000;2,1,0,1.54475,14000000;3,1,0,1.5448,10131093;4,1,0,1.545,1000000;
060308,210547.637;13;0,1,0,1.5446,7000000;1,1,0,1.54465,4500000;2,1,0,1.5447,9000000;3,1,0,1.54475,14000000;4,1,0,1.5448,10131093;0,1,0,1.5446,12000000;2,1,0,1.5447,4000000;1,1,0,1.54465,14500000;3,1,0,1.54475,4000000;0,1,1,1.5445,27632405;1,1,1,1.54445,10000000;1,1,0,1.54465,10000000;2,1,0,1.5447,8500000;
060308,210547.871;2;0,1,0,1.5446,7000000;2,1,0,1.5447,13500000;
060308,210548.131;10;0,1,0,1.54465,10000000;1,1,0,1.5447,20500000;2,1,0,1.54475,4000000;3,1,0,1.5448,10131093;4,1,0,1.545,1000000;0,1,0,1.5447,20500000;1,1,0,1.54475,14000000;2,1,0,1.5448,10131093;3,1,0,1.545,1000000;4,1,0,1.54565,3125000;
060308,210548.911;3;1,1,0,1.54475,10000000;3,1,0,1.54485,4000000;4,1,0,1.545,1000000;
060308,210552.629;7;0,1,1,1.54455,4500000;1,1,1,1.5445,23132405;2,1,1,1.54445,10000000;3,1,1,1.5444,3000000;4,1,1,1.54435,4000000;0,1,0,1.5447,16000000;1,1,0,1.54475,14500000;
060308,210552.863;10;0,1,0,1.5447,15000000;2,1,0,1.5448,12131093;0,1,1,1.5446,5000000;1,1,1,1.54455,4500000;2,1,1,1.5445,18132405;3,1,1,1.54445,10000000;4,1,1,1.5444,3000000;4,1,0,1.5451,1000000;0,1,1,1.5446,12000000;2,1,1,1.5445,11132405;
060308,210553.071;10;2,1,1,1.5445,14132405;4,1,1,1.54435,4000000;0,1,0,1.5447,12000000;2,1,0,1.5448,15131093;0,1,1,1.5446,13000000;2,1,1,1.5445,13132405;2,1,0,1.5448,14131093;1,1,1,1.54455,14500000;3,1,1,1.54435,4000000;4,1,1,1.5442,1000000;
060308,210553.929;1;3,1,1,1.54445,4000000;
060308,210603.025;2;4,1,1,1.5441,1000000;4,1,0,1.545,1000000;
060308,210603.259;21;1,1,1,1.54455,10000000;2,1,1,1.5445,17632405;0,1,0,1.5447,16500000;1,1,0,1.54475,10000000;0,1,1,1.5446,6000000;2,1,1,1.5445,24632405;0,1,1,1.5446,1000000;2,1,1,1.5445,29632405;0,1,1,1.54455,10000000;1,1,1,1.5445,31632405;2,1,1,1.54445,4000000;3,1,1,1.5441,1000000;4,1,1,1.5437,20000;1,1,1,1.5445,30632405;0,1,0,1.5447,17500000;2,1,0,1.5448,13131093;1,1,1,1.5445,27632405;3,1,1,1.5444,3000000;4,1,1,1.5441,1000000;0,1,0,1.5447,20500000;2,1,0,1.5448,10131093;
060308,210603.493;5;0,1,1,1.5445,27632405;1,1,1,1.54445,14000000;2,1,1,1.5444,3000000;3,1,1,1.5441,1000000;4,1,1,1.5437,20000;

[Shane]

Good idea!  Hopefully a few people can volunteer to record the raw data each day, so we will get at least 1 good file per day.  They can be uploaded to the account Eugene set up for this purpose.

[Robert]

Any reason not to use this format :
http://groups.google.com/group/jbooktrader/browse_thread/thread/d11bf92425ed22b6/13ed24a5b8310ea8
?

> Data converter will be needed to convert the raw data into whatever data format you want.

How will you manage to use this "whatever data format you want" as
input for your backtests ?

Anyway it's a good idea and it would be useful to test the raw format
against the 3.03 format, in forward test mode but also in backtest
mode.

I'd be interested in participating in such tests.

[Kelvin]

I find that the new format is more spatial efficient than the previous
one.
But what is the file size every day using this new format?

On Jun 4, 8:48 am, Robert <RobertdeChastel...@gmail.com> wrote:
> Any reason not to use this format :http://groups.google.com/group/jbooktrader/browse_thread/thread/d11bf...

[dyno]

I have recorded 6 days of data in this format so far. The file size
for ES, NQ, ER2 are 140, 106, 86MB respectively. The average daily
size for ES is about 24MB. I am going to write a RAW data converter to
convert the data to 303 format this weekend. In theory, we should be
able to convert the raw data into any previous or future data
formats.

> > I'd be interested in participating in such tests.- Hide quoted text -
>
> - Show quoted text -

[dyno]

Finally I finished the converter for 3.04 data format. I also play
around by creating other data format, such as replacing midBalance
with meanBalance. So far so good.

> > - Show quoted text -- Hide quoted text -

[nonlinear5]

> I also play
> around by creating other data format, such as replacing midBalance
> with meanBalance.
>

I like that idea. Mean balance may be more representative than the mid
balance. In fact, I think that for historical data file, the "High-Low-
Mean" format may be better than the current "Open-High-Low-Close"
format.

Have you been recording the raw ES depth data continuously since June
3, dyno?

[dyno]

Yes I got all the data of regular trading hour for the trio of ES/NQ/
ER2 except for one day due to power outrage.

I agree with you that that it's hard to find the appropriate usage for
open and close data.

[nonlinear5]

> Yes I got all the data of regular trading hour for the trio of ES/NQ/
> ER2 except for one day due to power outrage.
>

That's very nice. When you work on the conversion, can you convert
your raw ES data to 1-second "High-Low-
Mean" format, please? I'd like to run some tests to see if we can make
it a new official JBT format.

[Gab]

It was a really good idea to record raw MarketData.  Thanks for that Dyno!

[Kelvin]

Can you further reduce the size of the raw data format?
For example, do not record the mmddyy in each line, but add it in the
datafile header?

On Jun 25, 6:48 am, Gab <gabrieldanca...@gmail.com> wrote:
> It was a really good idea to record raw MarketData.  Thanks for that Dyno!
>

> On 6/25/08, nonlinear5 <eugene.kono...@gmail.com> wrote:
>
>
>
>
>
> > > Yes I got all the data of regular trading hour for the trio of ES/NQ/
> > > ER2 except for one day due to power outrage.
>
> > That's very nice. When you work on the conversion, can you convert
> > your raw ES data to 1-second "High-Low-
> > Mean" format, please? I'd like to run some tests to see if we can make

> > it a new official JBT format.- Hide quoted text -

[Shane]

That's a problem when you have multiple days in one file.

[nonlinear5]

> For example, do not record the mmddyy in each line, but add it in the
> datafile header?
>

Well, it's a little trickier, because a single data file may cover
multiple days, months, or even years.

[HF Dave]

Since the data file is processed in a linear way, date could be an
optional field. If not present on the current line, last date seen/
parsed should be assumed.

[nonlinear5]

> Since the data file is processed in a linear way, date could be an
> optional field. If not present on the current line, last date seen/
> parsed should be assumed.

Yes, it could be done that way. I'll look into it in the nearest
future.

[Kelvin]

We can append a data header at the begining of each day. In the
backtestfilereader, we change the date if we meet a new data header.

[Kelvin]

Good idea! Even the HHMMSS can be implemented in this way

On Jun 25, 11:25 am, HF Dave <tmi...@gmail.com> wrote:

[Kelvin]

Further more, since at most of the time, the prices do not change.
Even the prices can be implemented in this way.

> > parsed should be assumed.- Hide quoted text -

[dyno]

Sure. I will work on that.

I also have implemented a new way to calculate balance that I would
like to welcome your comments. The new method can eliminate the
effect of price movement on balance due to the limits on size of the
book. Considering the following example right before price move up:

Bid Ask
size price price size
1302.25 100
1302.00 100
1301.75 100
1301.50 100
1301.25 100
1301.00 1
99 1300.75
100 1300.5
100 1300.25
100 1300
100 1299.75
100 1299.5

This is a well balanced book. The book only show best of 5 asks. The
size of best ask (at 1301) is only 1 so this slot is close to empty
right before the price move up. The calculate balance is 11. After an
order of limit buy 2 contracts @1301, the book becomes

Bid Ask
size price price size
1302.25 100
1302.00 100
1301.75 100
1301.50 100
1301.25 100
1 1301.00
99 1300.75
100 1300.50
100 1300.25
100 1300.00
100 1299.75
100 1299.50

The best bid is close to empty, the calculated balance is -11.

The new balance calculation is to compensate this effect by
subtracting

cumulativeBid -= bid4Size*bid0Size/(bid0Size+ask0Size);
cumulativeAsk -= ask4Size*ask0Size/(bid0Size+ask0Size);

Using the new method, the values of balance before and after are both
close to 0.

It makes sense, the effect of placing an order for 2 contracts should
not have significant effect on the balance of the book with 900
contracts.

[dyno]

This remind me of the good old day when I have to handle sparse matrix

> > - Show quoted text -- Hide quoted text -

[nonlinear5]

> The new balance calculation is to compensate this effect by
> subtracting
>
>         cumulativeBid -= bid4Size*bid0Size/(bid0Size+ask0Size);
>         cumulativeAsk -= ask4Size*ask0Size/(bid0Size+ask0Size);
>
> Using the new method, the values of balance before and after are both
> close to 0.
>
> It makes sense, the effect of placing an order for 2 contracts should
> not have significant effect on the balance of the book with 900
> contracts.
>

That's very interesting, and yes, it does make sense. I am anxious to
backtest my strategies using this new way of calculating the balance.
Good work, dyno.

[dyno]

I have uploaded the file. The data is automatic stored between 9:15AM
and 4:15PM. I set the start time to 9:16AM since it needs some time to
get the market depth complete. During some short period the market
depth contains error and those period is excluded. The market depth is
reset if the data gap is greater than 10 minutes. Other than those
little things, the rides are quite normal.

[nonlinear5]

> I have uploaded the file. The data is automatic stored between 9:15AM
> and 4:15PM. I set the start time to 9:16AM since it needs some time to
> get the market depth complete. During some short period the market
> depth contains error and those period is excluded. The market depth is
> reset if the data gap is greater than 10 minutes. Other than those
> little things, the rides are quite normal.
>

Thanks, dyno. I ran a couple of quick tests and found that the results
with the "mean" balance are slightly worse than those with the "mid"
balance. This could have been an isolated result, though, so I need to
experiment more with it.

I am also interested in the effects of compensating the balance for
the top bid and ask size, as in your formulas:

cumulativeBid -= bid4Size*bid0Size/(bid0Size+ask0Size);
cumulativeAsk -= ask4Size*ask0Size/(bid0Size+ask0Size);

Would you please post an ES data file which uses this adjustment?
Thanks.

[nonlinear5]

> I ran a couple of quick tests and found that the results
> with the "mean" balance are slightly worse than those with the "mid"
> balance. This could have been an isolated result, though, so I need to
> experiment more with it.
>

I realized that I didn't run the test quite right, so I'll take back
what I said before. I'll re-run the test and post the update here.

[nonlinear5]

> I realized that I didn't run the test quite right, so I'll take back
> what I said before. I'll re-run the test and post the update here.

Ok, here is what I did. Ran two optimizations. One uses midBalance to
calculate the indicators, and the other one uses meanBalance to
calculate the indicator. Everything else is the same: the data file,
the optimization ranges, and the strategies. The results are very
similar. It actually makes sense, because the difference between the
min and max for each 1-second balance is typically very small, so the
mean and mid point for each one second sample are going to be very
close, resulting in nearly the same strategy results.

[Eugene Kononov]

Attached are the two optimization maps, one for midBalance, one for meanBalance.

[dyno]

I forgot to mention that the data is converted using the new balance
method. I have uploaded the comparison files:

304a.png
http://jbooktrader.googlegroups.com/web/304a.png?gda=atNSxDkAAACxCPCFuIK3A7OuJ-UzQ7on8qGI9o_qJ-KjM8kRKvfSF2G1qiJ7UbTIup-M2XPURDQWBAJ3MCAETANj_NPWy7Qt&gsc=vR6rSgsAAACAks5Ok3q1lS9IvXCqXQe3

and 304b.png
http://jbooktrader.googlegroups.com/web/304b.png?gda=vDoLaTkAAACxCPCFuIK3A7OuJ-UzQ7on8qGI9o_qJ-KjM8kRKvfSF2G1qiJ7UbTIup-M2XPURDQFa75jr7l5RXpZxlLV_jDg&gsc=vR6rSgsAAACAks5Ok3q1lS9IvXCqXQe3

where 304a applies the standard method to calculate balance and 304b
applies the compensated method mentioned above. The range of balance
is tighter in 304b than in 304a, especially during the period of
10:07-10:17AM.

[dyno]

In most of the cases, the difference should be small. meanBalance is a
good candidate for calculating the moving average of the balance. I
also have tried using standard deviation of the balance as an
indicator but found nothing special so far. I will try more parameters
later.

On Jun 27, 3:50 pm, "Eugene Kononov" <eugene.kono...@gmail.com> wrote:
> Attached are the two optimization maps, one for midBalance, one for
> meanBalance.
>

>  midBalance.png
> 28KViewDownload
>
>  meanBalance.png
> 29KViewDownload

[nonlinear5]

> I forgot to mention that the data is converted using the new balance
> method.

So, the balances in es-IB-RAW-May30ToJune26.zip are already
compensated for the top bid and top ask, according to these two
formulas?

cumulativeBid -= bid4Size*bid0Size/(bid0Size+ask0Size);
cumulativeAsk -= ask4Size*ask0Size/(bid0Size+ask0Size);

If so, I should then compare the performance of a strategy using two
different data files. One is yours (compensated), and one is mine
(uncompensated).

[Robert]

even if the results are similar, there are more orange/red colours in
the top-right part of meanBalance.png

On Jun 27, 3:50 pm, "Eugene Kononov" <eugene.kono...@gmail.com> wrote:

> Attached are the two optimization maps, one for midBalance, one for
> meanBalance.
>

>  midBalance.png
> 28KViewDownload
>
>  meanBalance.png
> 29KViewDownload

[nonlinear5]

>
> So, the balances in es-IB-RAW-May30ToJune26.zip are already
> compensated for the top bid and top ask, according to these two
> formulas?
>
> cumulativeBid -= bid4Size*bid0Size/(bid0Size+ask0Size);
> cumulativeAsk -= ask4Size*ask0Size/(bid0Size+ask0Size);
>
> If so, I should then compare the performance of a strategy using two
> different data files. One is yours (compensated), and one is mine
> (uncompensated).

OK, I ran another set of tests to measure the effect of compensation.
Specifically, I optimized WildCat using your (compensated) dataset,
and using my (uncompensated) dataset, covering the same time period.
The compensation makes a lot of sense. However, what I found out is
that it produces significantly inferior strategy results. That is,
using uncompensated balances resulted in steeper and smoother equity
curves compared to those when using compensated balances. I've used
Profit Factor and Performance Index measures to evaluate the results.

Dyno, let us know if you find any evidence to the contrary.

[dyno]

Eugene, I didn't compare the effect of the compensated method on
strategy performance. I still use version 3.04 where the visual
comparison is not possible. I will upgrade to current later.

It is possible that the uncompensated method is better in reflecting
the effect of price movement.

[xjustin]

Hi Dyno,

I am very interested in working on recording and efficiently storing
raw data on my summer vacation. Where to get your special version of
recording raw data?

On Jun 3, 9:22 pm, dyno <dynobr...@gmail.com> wrote:
> I have made a special version just for recording raw market data. A
> sample data file is listed here. Your inputs on the raw data format
> are welcome.
> Data converter will be needed to convert the raw data into whatever
> data format you want. I will post the special build if there are
> enough interests.
>
> -dyno
>
> http://jbooktrader.googlegroups.com/web/EUR.txt?gsc=wE31LgsAAAB86QbR_...
>
> ---------------------------------------------------------------------------­----------------------------------------
> # This historical data file is created by JBookTrader
> # Each line represents a series of raw market update at a particular
> time:
> # date, time, number of updates;
> # 1. date is in the MMddyy format
> # 2. time is in the HHmmss.SSS format, followed by token ';'
> # 3. Number of updates in this series, followed by token ';'
> # 4. for each update, the format is (position, operation, side, price,
> size), followed by token ';'
>
> timeZone=America/New_York
>
> 060308,210541.629;24;0,0,1,1.5445,27632405;1,0,1,1.54445,10000000;2,0,1,1.5­444,3000000;3,0,1,1.54435,4000000;4,2,1,1.5441,1000000;4,0,1,1.5441,1000000­;0,0,0,1.5447,20500000;1,0,0,1.54475,10000000;2,0,0,1.5448,10131093;3,0,0,1­.54485,4000000;4,2,0,1.545,1000000;4,0,0,1.545,1000000;0,1,1,1.5445,2313240­5;1,1,1,1.54445,14500000;0,1,0,1.54465,4500000;1,1,0,1.5447,16000000;2,1,0,­1.54475,10000000;3,1,0,1.5448,10131093;4,1,0,1.54485,4000000;0,1,0,1.5446,7­000000;1,1,0,1.54465,4500000;2,1,0,1.5447,9000000;3,1,0,1.54475,10000000;4,­1,0,1.5448,10131093;
> 060308,210541.841;5;0,1,0,1.5446,12000000;2,1,0,1.5447,4000000;1,1,0,1.5446­5,14500000;3,1,0,1.5448,10131093;4,1,0,1.54485,4000000;
> 060308,210542.075;2;3,1,0,1.54475,4000000;4,1,0,1.5448,10131093;
> 060308,210546.935;9;0,1,0,1.5446,5000000;2,1,0,1.5447,11000000;1,1,0,1.5446­5,4500000;3,1,0,1.54475,14000000;0,1,0,1.54465,4500000;1,1,0,1.5447,1600000­0;2,1,0,1.54475,14000000;3,1,0,1.5448,10131093;4,1,0,1.545,1000000;
> 060308,210547.143;7;0,1,1,1.5445,27632405;1,1,1,1.54445,10000000;0,1,0,1.54­47,20500000;1,1,0,1.54475,14000000;2,1,0,1.5448,10131093;3,1,0,1.545,100000­0;4,1,0,1.54565,3125000;
> 060308,210547.377;7;0,1,1,1.5445,23132405;1,1,1,1.54445,14500000;0,1,0,1.54­465,4500000;1,1,0,1.5447,16000000;2,1,0,1.54475,14000000;3,1,0,1.5448,10131­093;4,1,0,1.545,1000000;
> 060308,210547.637;13;0,1,0,1.5446,7000000;1,1,0,1.54465,4500000;2,1,0,1.544­7,9000000;3,1,0,1.54475,14000000;4,1,0,1.5448,10131093;0,1,0,1.5446,1200000­0;2,1,0,1.5447,4000000;1,1,0,1.54465,14500000;3,1,0,1.54475,4000000;0,1,1,1­.5445,27632405;1,1,1,1.54445,10000000;1,1,0,1.54465,10000000;2,1,0,1.5447,8­500000;
> 060308,210547.871;2;0,1,0,1.5446,7000000;2,1,0,1.5447,13500000;
> 060308,210548.131;10;0,1,0,1.54465,10000000;1,1,0,1.5447,20500000;2,1,0,1.5­4475,4000000;3,1,0,1.5448,10131093;4,1,0,1.545,1000000;0,1,0,1.5447,2050000­0;1,1,0,1.54475,14000000;2,1,0,1.5448,10131093;3,1,0,1.545,1000000;4,1,0,1.­54565,3125000;
> 060308,210548.911;3;1,1,0,1.54475,10000000;3,1,0,1.54485,4000000;4,1,0,1.54­5,1000000;
> 060308,210552.629;7;0,1,1,1.54455,4500000;1,1,1,1.5445,23132405;2,1,1,1.544­45,10000000;3,1,1,1.5444,3000000;4,1,1,1.54435,4000000;0,1,0,1.5447,1600000­0;1,1,0,1.54475,14500000;
> 060308,210552.863;10;0,1,0,1.5447,15000000;2,1,0,1.5448,12131093;0,1,1,1.54­46,5000000;1,1,1,1.54455,4500000;2,1,1,1.5445,18132405;3,1,1,1.54445,100000­00;4,1,1,1.5444,3000000;4,1,0,1.5451,1000000;0,1,1,1.5446,12000000;2,1,1,1.­5445,11132405;
> 060308,210553.071;10;2,1,1,1.5445,14132405;4,1,1,1.54435,4000000;0,1,0,1.54­47,12000000;2,1,0,1.5448,15131093;0,1,1,1.5446,13000000;2,1,1,1.5445,131324­05;2,1,0,1.5448,14131093;1,1,1,1.54455,14500000;3,1,1,1.54435,4000000;4,1,1­,1.5442,1000000;
> 060308,210553.929;1;3,1,1,1.54445,4000000;
> 060308,210603.025;2;4,1,1,1.5441,1000000;4,1,0,1.545,1000000;
> 060308,210603.259;21;1,1,1,1.54455,10000000;2,1,1,1.5445,17632405;0,1,0,1.5­447,16500000;1,1,0,1.54475,10000000;0,1,1,1.5446,6000000;2,1,1,1.5445,24632­405;0,1,1,1.5446,1000000;2,1,1,1.5445,29632405;0,1,1,1.54455,10000000;1,1,1­,1.5445,31632405;2,1,1,1.54445,4000000;3,1,1,1.5441,1000000;4,1,1,1.5437,20­000;1,1,1,1.5445,30632405;0,1,0,1.5447,17500000;2,1,0,1.5448,13131093;1,1,1­,1.5445,27632405;3,1,1,1.5444,3000000;4,1,1,1.5441,1000000;0,1,0,1.5447,205­00000;2,1,0,1.5448,10131093;
> 060308,210603.493;5;0,1,1,1.5445,27632405;1,1,1,1.54445,14000000;2,1,1,1.54­44,3000000;3,1,1,1.5441,1000000;4,1,1,1.5437,20000;

[dyno]

I have uploaded the source files. I have tested the zipped files on my
other machine and it works. Forward testing mode works OK. Let me know
if you have any problem.

http://jbooktrader.googlegroups.com/web/3.01Acom.zip?gda=AuJiwz0AAACxCPCFuIK3A7OuJ-UzQ7onO_sf8TpjO5cdmG99By6vomG1qiJ7UbTIup-M2XPURDSkhAbpRDVZLkGzrWX33L1P&gsc=jzM1hgsAAABZL9cYR4MvzGsjTujOElzB

> > 060308,210541.629;24;0,0,1,1.5445,27632405;1,0,1,1.54445,10000000;2,0,1,1.5­­444,3000000;3,0,1,1.54435,4000000;4,2,1,1.5441,1000000;4,0,1,1.5441,100000­0­;0,0,0,1.5447,20500000;1,0,0,1.54475,10000000;2,0,0,1.5448,10131093;3,0,0­,1­.54485,4000000;4,2,0,1.545,1000000;4,0,0,1.545,1000000;0,1,1,1.5445,2313­240­5;1,1,1,1.54445,14500000;0,1,0,1.54465,4500000;1,1,0,1.5447,16000000;2,­1,0,­1.54475,10000000;3,1,0,1.5448,10131093;4,1,0,1.54485,4000000;0,1,0,1.5­446,7­000000;1,1,0,1.54465,4500000;2,1,0,1.5447,9000000;3,1,0,1.54475,10000­000;4,­1,0,1.5448,10131093;

> > 060308,210541.841;5;0,1,0,1.5446,12000000;2,1,0,1.5447,4000000;1,1,0,1.5446­­5,14500000;3,1,0,1.5448,10131093;4,1,0,1.54485,4000000;
> > 060308,210542.075;2;3,1,0,1.54475,4000000;4,1,0,1.5448,10131093;

> > 060308,210546.935;9;0,1,0,1.5446,5000000;2,1,0,1.5447,11000000;1,1,0,1.5446­­5,4500000;3,1,0,1.54475,14000000;0,1,0,1.54465,4500000;1,1,0,1.5447,160000­0­0;2,1,0,1.54475,14000000;3,1,0,1.5448,10131093;4,1,0,1.545,1000000;
> > 060308,210547.143;7;0,1,1,1.5445,27632405;1,1,1,1.54445,10000000;0,1,0,1.54­­47,20500000;1,1,0,1.54475,14000000;2,1,0,1.5448,10131093;3,1,0,1.545,10000­0­0;4,1,0,1.54565,3125000;
> > 060308,210547.377;7;0,1,1,1.5445,23132405;1,1,1,1.54445,14500000;0,1,0,1.54­­465,4500000;1,1,0,1.5447,16000000;2,1,0,1.54475,14000000;3,1,0,1.5448,1013­1­093;4,1,0,1.545,1000000;
> > 060308,210547.637;13;0,1,0,1.5446,7000000;1,1,0,1.54465,4500000;2,1,0,1.544­­7,9000000;3,1,0,1.54475,14000000;4,1,0,1.5448,10131093;0,1,0,1.5446,120000­0­0;2,1,0,1.5447,4000000;1,1,0,1.54465,14500000;3,1,0,1.54475,4000000;0,1,1­,1­.5445,27632405;1,1,1,1.54445,10000000;1,1,0,1.54465,10000000;2,1,0,1.544­7,8­500000;
> > 060308,210547.871;2;0,1,0,1.5446,7000000;2,1,0,1.5447,13500000;
> > 060308,210548.131;10;0,1,0,1.54465,10000000;1,1,0,1.5447,20500000;2,1,0,1.5­­4475,4000000;3,1,0,1.5448,10131093;4,1,0,1.545,1000000;0,1,0,1.5447,205000­0­0;1,1,0,1.54475,14000000;2,1,0,1.5448,10131093;3,1,0,1.545,1000000;4,1,0,­1.­54565,3125000;

> > 060308,210548.911;3;1,1,0,1.54475,10000000;3,1,0,1.54485,4000000;4,1,0,1.54­­5,1000000;

> > 060308,210552.629;7;0,1,1,1.54455,4500000;1,1,1,1.5445,23132405;2,1,1,1.544­­45,10000000;3,1,1,1.5444,3000000;4,1,1,1.54435,4000000;0,1,0,1.5447,160000­0­0;1,1,0,1.54475,14500000;
> > 060308,210552.863;10;0,1,0,1.5447,15000000;2,1,0,1.5448,12131093;0,1,1,1.54­­46,5000000;1,1,1,1.54455,4500000;2,1,1,1.5445,18132405;3,1,1,1.54445,10000­0­00;4,1,1,1.5444,3000000;4,1,0,1.5451,1000000;0,1,1,1.5446,12000000;2,1,1,­1.­5445,11132405;
> > 060308,210553.071;10;2,1,1,1.5445,14132405;4,1,1,1.54435,4000000;0,1,0,1.54­­47,12000000;2,1,0,1.5448,15131093;0,1,1,1.5446,13000000;2,1,1,1.5445,13132­4­05;2,1,0,1.5448,14131093;1,1,1,1.54455,14500000;3,1,1,1.54435,4000000;4,1­,1­,1.5442,1000000;

> > 060308,210553.929;1;3,1,1,1.54445,4000000;
> > 060308,210603.025;2;4,1,1,1.5441,1000000;4,1,0,1.545,1000000;

> > 060308,210603.259;21;1,1,1,1.54455,10000000;2,1,1,1.5445,17632405;0,1,0,1.5­­447,16500000;1,1,0,1.54475,10000000;0,1,1,1.5446,6000000;2,1,1,1.5445,2463­2­405;0,1,1,1.5446,1000000;2,1,1,1.5445,29632405;0,1,1,1.54455,10000000;1,1­,1­,1.5445,31632405;2,1,1,1.54445,4000000;3,1,1,1.5441,1000000;4,1,1,1.5437­,20­000;1,1,1,1.5445,30632405;0,1,0,1.5447,17500000;2,1,0,1.5448,13131093;1­,1,1­,1.5445,27632405;3,1,1,1.5444,3000000;4,1,1,1.5441,1000000;0,1,0,1.544­7,205­00000;2,1,0,1.5448,10131093;
> > 060308,210603.493;5;0,1,1,1.5445,27632405;1,1,1,1.54445,14000000;2,1,1,1.54­­44,3000000;3,1,1,1.5441,1000000;4,1,1,1.5437,20000;- Hide quoted text -

[Kelvin]

I really think that the raw data is the best data format. But we
should try to compress the size as small as possible.

On Jul 2, 12:44 am, dyno <dynobr...@gmail.com> wrote:
> I have uploaded the source files. I have tested the zipped files on my
> other machine and it works. Forward testing mode works OK. Let me know
> if you have any problem.
>

> http://jbooktrader.googlegroups.com/web/3.01Acom.zip?gda=AuJiwz0AAACx...

> > > 060308,210541.629;24;0,0,1,1.5445,27632405;1,0,1,1.54445,10000000;2,0,1,1.5­­­444,3000000;3,0,1,1.54435,4000000;4,2,1,1.5441,1000000;4,0,1,1.5441,10000­0­0­;0,0,0,1.5447,20500000;1,0,0,1.54475,10000000;2,0,0,1.5448,10131093;3,0­,0­,1­.54485,4000000;4,2,0,1.545,1000000;4,0,0,1.545,1000000;0,1,1,1.5445,2­313­240­5;1,1,1,1.54445,14500000;0,1,0,1.54465,4500000;1,1,0,1.5447,1600000­0;2,­1,0,­1.54475,10000000;3,1,0,1.5448,10131093;4,1,0,1.54485,4000000;0,1,­0,1.5­446,7­000000;1,1,0,1.54465,4500000;2,1,0,1.5447,9000000;3,1,0,1.54475­,10000­000;4,­1,0,1.5448,10131093;

> > > 060308,210541.841;5;0,1,0,1.5446,12000000;2,1,0,1.5447,4000000;1,1,0,1.5446­­­5,14500000;3,1,0,1.5448,10131093;4,1,0,1.54485,4000000;
> > > 060308,210542.075;2;3,1,0,1.54475,4000000;4,1,0,1.5448,10131093;

> > > 060308,210546.935;9;0,1,0,1.5446,5000000;2,1,0,1.5447,11000000;1,1,0,1.5446­­­5,4500000;3,1,0,1.54475,14000000;0,1,0,1.54465,4500000;1,1,0,1.5447,16000­0­0­0;2,1,0,1.54475,14000000;3,1,0,1.5448,10131093;4,1,0,1.545,1000000;
> > > 060308,210547.143;7;0,1,1,1.5445,27632405;1,1,1,1.54445,10000000;0,1,0,1.54­­­47,20500000;1,1,0,1.54475,14000000;2,1,0,1.5448,10131093;3,1,0,1.545,1000­0­0­0;4,1,0,1.54565,3125000;
> > > 060308,210547.377;7;0,1,1,1.5445,23132405;1,1,1,1.54445,14500000;0,1,0,1.54­­­465,4500000;1,1,0,1.5447,16000000;2,1,0,1.54475,14000000;3,1,0,1.5448,101­3­1­093;4,1,0,1.545,1000000;
> > > 060308,210547.637;13;0,1,0,1.5446,7000000;1,1,0,1.54465,4500000;2,1,0,1.544­­­7,9000000;3,1,0,1.54475,14000000;4,1,0,1.5448,10131093;0,1,0,1.5446,12000­0­0­0;2,1,0,1.5447,4000000;1,1,0,1.54465,14500000;3,1,0,1.54475,4000000;0,1­,1­,1­.5445,27632405;1,1,1,1.54445,10000000;1,1,0,1.54465,10000000;2,1,0,1.­544­7,8­500000;
> > > 060308,210547.871;2;0,1,0,1.5446,7000000;2,1,0,1.5447,13500000;
> > > 060308,210548.131;10;0,1,0,1.54465,10000000;1,1,0,1.5447,20500000;2,1,0,1.5­­­4475,4000000;3,1,0,1.5448,10131093;4,1,0,1.545,1000000;0,1,0,1.5447,20500­0­0­0;1,1,0,1.54475,14000000;2,1,0,1.5448,10131093;3,1,0,1.545,1000000;4,1,­0,­1.­54565,3125000;

> > > 060308,210548.911;3;1,1,0,1.54475,10000000;3,1,0,1.54485,4000000;4,1,0,1.54­­­5,1000000;

> > > 060308,210552.629;7;0,1,1,1.54455,4500000;1,1,1,1.5445,23132405;2,1,1,1.544­­­45,10000000;3,1,1,1.5444,3000000;4,1,1,1.54435,4000000;0,1,0,1.5447,16000­0­0­0;1,1,0,1.54475,14500000;
> > > 060308,210552.863;10;0,1,0,1.5447,15000000;2,1,0,1.5448,12131093;0,1,1,1.54­­­46,5000000;1,1,1,1.54455,4500000;2,1,1,1.5445,18132405;3,1,1,1.54445,1000­0­0­00;4,1,1,1.5444,3000000;4,1,0,1.5451,1000000;0,1,1,1.5446,12000000;2,1,­1,­1.­5445,11132405;
> > > 060308,210553.071;10;2,1,1,1.5445,14132405;4,1,1,1.54435,4000000;0,1,0,1.54­­­47,12000000;2,1,0,1.5448,15131093;0,1,1,1.5446,13000000;2,1,1,1.5445,1313­2­4­05;2,1,0,1.5448,14131093;1,1,1,1.54455,14500000;3,1,1,1.54435,4000000;4­,1­,1­,1.5442,1000000;

> > > 060308,210553.929;1;3,1,1,1.54445,4000000;
> > > 060308,210603.025;2;4,1,1,1.5441,1000000;4,1,0,1.545,1000000;

> > > 060308,210603.259;21;1,1,1,1.54455,10000000;2,1,1,1.5445,17632405;0,1,0,1.5­­­447,16500000;1,1,0,1.54475,10000000;0,1,1,1.5446,6000000;2,1,1,1.5445,246­3­2­405;0,1,1,1.5446,1000000;2,1,1,1.5445,29632405;0,1,1,1.54455,10000000;1­,1­,1­,1.5445,31632405;2,1,1,1.54445,4000000;3,1,1,1.5441,1000000;4,1,1,1.5­437­,20­000;1,1,1,1.5445,30632405;0,1,0,1.5447,17500000;2,1,0,1.5448,131310­93;1­,1,1­,1.5445,27632405;3,1,1,1.5444,3000000;4,1,1,1.5441,1000000;0,1,0,­1.544­7,205­00000;2,1,0,1.5448,10131093;

> > > 060308,210603.493;5;0,1,1,1.5445,27632405;1,1,1,1.54445,14000000;2,1,1,1.54­­­44,3000000;3,1,1,1.5441,1000000;4,1,1,1.5437,20000;- Hide quoted text -
>

> > - Show quoted text -- Hide quoted text -

[HF Dave]

On Jul 11, 11:38 pm, Kelvin <baoxing...@gmail.com> wrote:
> I really think that the raw data is the best data format. But we
> should try to compress the size as small as possible.

In my humble opinion, the best way is to record and store raw data.
Compressed to the death and burned onto a DVD it costs nothing. Then,
the raw data could be converted (like the CME data) to any format
anyone (any version of JBT or anything else) would need. I think that
we cannot loss this way.

[nonlinear5]

Dyno,

I hope you are still recording the raw depth data. If so, would you be
kind enough to convert the ES data to this format:

# Each line represents the order book at a particular time and
contains 5 columns:
# date, time, balance, lowPrice, highPrice

# 1. date is in the MMddyy format

# 2. time is in the HHmmss format
# 3. balance is the period's book balance
# 4. lowPrice is the period's lowest price
# 5. highPrice is the period's highest price

where "balance" is based on the cumulative bid size and cumulative ask
size adjusted with your formulas:

cumulativeBid -= bid4Size*bid0Size/(bid0Size+ask0Size)
cumulativeAsk -= ask4Size*ask0Size/(bid0Size+ask0Size)

Thanks.

[nonlinear5]

> In my humble opinion, the best way is to record and store raw data.
> Compressed to the death and burned onto a DVD it costs nothing. Then,
> the raw data could be converted (like the CME data) to any format
> anyone (any version of JBT or anything else) would need. I think that
> we cannot loss this way.

I agree that storage is cheap and it's not an issue. I also agree that
the raw format is the most detailed, descriptive, and universal.
However, the backtesting and optimizing based on raw data would be
simply impractical. At 2 million raw data pieces a day, a one year
data set would have half a billion (!) lines. I don't think there is
any software that could backtest and optimize these gigantic amounts
of data efficiently. It can be done, but it would take hundreds of
times longer than it now takes with JBT. One can of course convert the
raw data into some other format (similar to JBT format) prior to
backtesting and optimization, but this would introduce a lot more
complexity and maintenance to JBT. My take on this is quite pragmatic:
if the current format allows for creation of clearly profitable and
consistent strategies, that would do it for me. From my various
experimentation with the raw data, I don't see any evidence that more
microscopic view of the data (such as bids and asks for all levels
separately) provides any additional "edge" over the existing format
and the strategies. If anyone has such evidence, I'd love to see it.
Until then, I don't see any convincing reasons to support raw depth
data storage and processing, with the (what I believe) ephemeral hope
that it would pay off later.

Having said that, I am glad that Dyno has been scrupulously collecting
and maintaining raw data with his customized version of JBT. His "top
bid"/"top ask" adjustment methodology makes a lot of sense, and I am
leaning toward using it for a new historical data format. However,
this new format will actually be smaller and simpler that the existing
one. I am planning to drop "openBalance", "closeBalance",
"highBalance", and "lowBalance" and replace them with the single "Dyno-
adjusted" balance for each 1-second sample. Even without the "Dyno-
adjustment", the existing JBT format can be simplified. As can be seen
from the sample strategies, none of them actually use openBalance,
closeBalance, highBalance, and lowBalance. All they need is a single
figure, which is currently the mid-point balance.

[dyno]

Would you please tell me the detailed setting:

sampling frequency = 1Hz?
balance = meanBalance or MidBalance?

[dyno]

I agree with you on raw data format. To me, raw data format is used
for recording market data and experimenting new parameters.

I haven't finished comparing the two methods on optimization map yet.
I just updated my customized version to 4.02 this weekend. Hope I can
do some catch up work later this week.

[nonlinear5]

> Would you please tell me the detailed setting:
>
> sampling frequency = 1Hz?
> balance = meanBalance or MidBalance?
>

Sampling frequency should be the same as it's currently, 1000
milliseconds. The balance should be mean balance calculated using
cumulativeBid and cumulativeAsk which are pre-adjusted with your
formulas.

[dyno]

done, check the files section please.

[nonlinear5]

> done, check the files section please.
>

Thanks Dyno, I'll run some tests on this data set.

[HF Dave]

On Jul 14, 4:17 am, nonlinear5 <eugene.kono...@gmail.com> wrote:
> I agree that storage is cheap and it's not an issue. I also agree that
> the raw format is the most detailed, descriptive, and universal.
> However, the backtesting and optimizing based on raw data would be
> simply impractical. At 2 million raw data pieces a day, a one year
> data set would have half a billion (!) lines. I don't think there is
> any software that could backtest and optimize these gigantic amounts
> of data efficiently. It can be done, but it would take hundreds of
> times longer than it now takes with JBT.

Sure, I have never thought of passing the whole set of raw data into

JBT.


> One can of course convert the
> raw data into some other format (similar to JBT format) prior to
> backtesting and optimization, but this would introduce a lot more
> complexity and maintenance to JBT.

I did not know it would be that dramatic. JBT is already capable to
accept raw data from IB, convert it to the proper format and store it
into a file. I hoped it would take the data from a file instead of IB
and convert it without big changes.

> My take on this is quite pragmatic:
> if the current format allows for creation of clearly profitable and
> consistent strategies, that would do it for me. From my various
> experimentation with the raw data, I don't see any evidence that more
> microscopic view of the data (such as bids and asks for all levels
> separately) provides any additional "edge" over the existing format
> and the strategies. If anyone has such evidence, I'd love to see it.
> Until then, I don't see any convincing reasons to support raw depth
> data storage and processing, with the (what I believe) ephemeral hope
> that it would pay off later.

I agree. I personally do not need more precise data. The only point I
see is the possibility that JBT data format will change and we lost
all the stored IB data.

> Having said that, I am glad that Dyno has been scrupulously collecting
> and maintaining raw data with his customized version of JBT. His "top
> bid"/"top ask" adjustment methodology makes a lot of sense, and I am
> leaning toward using it for a new historical data format. However,
> this new format will actually be smaller and simpler that the existing
> one. I am planning to drop "openBalance", "closeBalance",
> "highBalance", and "lowBalance" and replace them with the single "Dyno-
> adjusted" balance for each 1-second sample. Even without the "Dyno-
> adjustment", the existing JBT format can be simplified. As can be seen
> from the sample strategies, none of them actually use openBalance,
> closeBalance, highBalance, and lowBalance. All they need is a single
> figure, which is currently the mid-point balance.

Like this. Do I understand it right? Will this change abandon the
already collected IB data?

[nonlinear5]

> Like this. Do I understand it right? Will this change abandon the
> already collected IB data?

Yes, the currently collected IB dataset will be discarded. However,
luckily for us, Dyno has all the raw data, so we can re-create the IB
data in the new format.

[nonlinear5]

Dyno, my previous tests comparing midBalance with "compensated mean
balance" were not quite "apples-to-apples", since I was using two
different datasets. To make the tests more consistent, I need both
formats be generated from the same dataset. You've already posted the
one containing "compensated mean balance" covering May 30 to July 11.
Can you please generate another dataset which contains uncompensated
midBalance, for the same period?

The sampling frequency should be the same, 1000 milliseconds.
"MidBalance" should be half point between the lowest and the highest
balance during each one second period. The format is:

# Each line represents the order book at a particular time and
contains 5 columns:

# date, time, midBalance, lowPrice, highPrice

# 1. date is in the MMddyy format

# 2. time is in the HHmmss format
# 3. midBalance is the period's book mid balance

# 4. lowPrice is the period's lowest price
# 5. highPrice is the period's highest price

Once I have this second dataset, I would be able to run some true
comparison tests to see if "compensated mean balance" is a better
representation than the "mid-balace", which is what JBT uses
currently. Thanks.

[dyno]

I have uploaded the file. I will do some comparisons tomorrow.

[nonlinear5]

> I have uploaded the file. I will do some comparisons tomorrow.
>

Dyno, the new JBT data format in 4.04 should make it easy to run
comparisons. Since there is a single field for "balance" in the new
format, there is no need to modify the code base to test "midBalance"
and "adjusted mean balance". Simply swap the historical data file. The
code doesn't need to change.

[dyno]

That's good. less work is always better than more work

[Florent Guiliani]

nonlinear5 a écrit :

> Having said that, I am glad that Dyno has been scrupulously collecting
> and maintaining raw data with his customized version of JBT. His "top
> bid"/"top ask" adjustment methodology makes a lot of sense, and I am
> leaning toward using it for a new historical data format. However,
> this new format will actually be smaller and simpler that the existing
> one. I am planning to drop "openBalance", "closeBalance",
> "highBalance", and "lowBalance" and replace them with the single "Dyno-
> adjusted" balance for each 1-second sample. Even without the "Dyno-
> adjustment", the existing JBT format can be simplified. As can be seen
> from the sample strategies, none of them actually use openBalance,
> closeBalance, highBalance, and lowBalance. All they need is a single
> figure, which is currently the mid-point balance.
>

I would like to remember to you that the high/low technique was
introduce to work arround a problem with the data bursts. Every user
receive a different set of book update items. Even with a fixed one
second sampling, all users will get different balance data. The high/low
technique aims to provide the same balance for all users modulo one or
the next burst.

If you drop the high/low technique, I guess we'll return to the starting
point when differents users get differents results on the same trading
session. Or maybe I missed something...

--
Florent,

[Eugene Kononov]

If you drop the high/low technique, I guess we'll return to the starting
point when differents users get differents results on the same trading
session. Or maybe I missed something...

We ran a multi-user test a few days ago using the "midBalance" method for all strategies. All results matched pretty closely.

[Skyisthelimit]

Dyno

Would you mind converting your data to JBT404 and JBT403 format? I
wish to get May 30th to June1st. Thanks.

[nonlinear5]

> I have uploaded the file. I will do some comparisons tomorrow.

Dyno,

How do you comparisons look like? I am running some at the moment.
I'll post a comparison report here when it's done.

[dyno]

It is already in files section

http://jbooktrader.googlegroups.com/web/ES-IB-RAW-May30ToJuly15-V402C.zip?gda=43nFa1IAAAB6KKTNDT2qYzZLIUf2PZN0jshquR-lYP8zhSdrtaEO1GG1qiJ7UbTIup-M2XPURDQiQ9aoBPRIdOaP-8YXQ1ncEAuuzAvy1UWB7dJ0ru8_jkdxBfjKiV3Zvs76YMEWYL0&gsc=G8HYqgsAAADq2Wo3g7Del5rLcEXCtOie

[dyno]

I haven't made any progress. Your comparison looks good

[jeb211b]

Version 4.02 needs data with 8 columns. Is there any data available
for 4.02?

On Jul 20, 10:47 am, dyno <dynobr...@gmail.com> wrote:
> It is already in files section
>

> http://jbooktrader.googlegroups.com/web/ES-IB-RAW-May30ToJuly15-V402C...

>
> On Jul 18, 9:48 am, Skyisthelimit <thulig...@gmail.com> wrote:
>
>
>
> > Dyno
>
> > Would you mind converting your data to JBT404 and JBT403 format? I

> > wish to get May 30th to June1st. Thanks.- Hide quoted text -

[nonlinear5]

Dyno,

Can you please post your code changes to MarketBook.java (and possibly
other classes) which contain "adjusted mean balance" calculations? I
could code it from your formulas, but I'd like to be sure that we have
the same interpretation of these adjustments.

Thanks.

[dyno]

I have uploaded the source code. It doesn't works in forward/trade
mode. It only works for data converting.

I basically leave MarketBook.java unchanged except for removing some
lines due to complain from the compiler.

[dyno]

I don't have data for 4.02. You can try the latest version. It's good
to keeps everybody on the same page.

> > - Show quoted text -- Hide quoted text -

[nonlinear5]

Dyno,

I am almost ready with my changes. At the end of the day, can you
please convert your entire dataset (May20-July22) one more time? The
format should be the same as before: adjusted mean balance, 1000 ms
samples, 5 columns. Starting from the next release, JBT will record
and use adjusted mean balances.

[dyno]

Eugene,

The file has been uploaded. Are you going to make a new version soon?

[nonlinear5]

> The file has been uploaded. Are you going to make a new version soon?
>

Thanks. Yes, the new version is coming up today or tomorrow. I am
converting CME data to the new format.

[jeb211b]

Thanks for the reply. I haven't tried any new ideas since version 2.12
and 4.04 is very different. I finally managed to try out most of my
new ideas using 4.04.