A suggestion on futuretest optimization philosophy

[Kelvin]

Hi folks,

I think our backtest optimization criterion is not correct. Our
current optimization philosophy only finds the best backtest
performance. However, a high backtest performance does not mean what
will happen in the future.

Let us perform the futuretest instead of backtest. The philosophy is
that, we use a sweeping time window of the historical data to optimize
the parameters. Instead of optimize for the best backtest performance.
We evalute the performance of the "future trade". The "future trade"
is based on the historical data in ahead of the sweeping time window.

Another problem is that how to determine the length of the sweeping
time window. In the time window, the historical data should generate
enough trades. It could be 100 trades per parameter.

How do you guys think?

[Gab]

How do you guys think?

I'm sorry Kelvin, but I don't understand your idea.  Could you explain it in other words please?

[Shane]

When I was doing Rapid Miner tutorials, they would do use cross validation.  You feed the data in and it would take a subset of the data to build the model and then test the model on the rest of the the data.  Is this what you are talking about?

From wikipedia

http://en.wikipedia.org/wiki/Cross-validation

Cross-validation, sometimes called rotation estimation, is the statistical practice of partitioning a sample of data into subsets such that the analysis is initially performed on a single subset, while the other subset(s) are retained for subsequent use in confirming and validating the initial analysis.

The initial subset of data is called the training set; the other subset(s) are called validation or testing sets.

[rickty]

In essence you're proposing walk-forward optimization;
which is definitely the way to go. However, it will be
even more time consuming. Nevertheless, I imagine,
in practice, reoptmization need only to be done once
a month.

[HF Dave]

If you think not to optimize on all the data but on a floating time
frame (always several days/weeks/months back) ... I am thinking about
this for very long time. I have mentioned before that I want to test
this using the CLI. In my opinion optimization method could be more
important than the strategy itself. You can make money on simple
stupid MA cross over strategy once you know how to optimize it right.

I have been gone for some days and I have to read all the posts
now. :)

[nonlinear5]

As rickty mentions, it sounds like Kelvin is talking about walk-
forward optimization. Specifically, optimize the strategy on the
period of time from T1 to T2 and evaluate its performance on the
period from T2 to T3. Then one can shift periods forward: optimize on
T1+delta to T2+delta, and evaluate on T2+delta to T3+delta. Is that
what you are proposing, Kelvin?

So, here is the question. Suppose we run a sequence of this
optimizations on a simple MA crossover system, and find this:

In January, the best combination of parameters was 5, 20
In February, the best combination of parameters was 10, 30
In March, the best combination of parameters was 100, 300

Now what?

[Gab]

Ok, I think I understand better what you guys are talking about.  But the simplest way to do is, and correct me if I'm wrong, would be with a database?

[lon]

I think all the optimization is looking for a market attitude, the
more precise the more success will be. From this point, the attitude
of market from last month is hardly to reflect the current or future
movement.

Theoretically, a dynamic or real time optimization maybe interesting,
let’s say while JBT doing forward/trading session, an optimization
process keeps optimizing past daily market data, and update the
trading strategy with latest parameters …, sounds crazy, but may be
interesting for those extreme momentum trading strategies.

[rickty]

Nonlinear,

While you're waiting for Kelvin to reply let me have a go:

"In January, the best combination of parameters was 5, 20
In February, the best combination of parameters was 10, 30
In March, the best combination of parameters was 100, 300


Now what? "

I don't see the problem. Using your example if through
optimization it is detemined that the best parameters
for January are 5,20 then these are used to trade the month
of February and the results become part of the out-of-sample
performance. Next an optimization is done with data through
to the end of February then these (possibly) new parameters
are used to trade March and the performance for March
is appended to the out-of-sample performance data. And so on.

[Gab]

I don't see the problem. Using your example if through

optimization it is detemined that the best parameters
for January are 5,20 then these are used to trade the month
of February and the results become part of the out-of-sample
performance. Next an optimization is done with data through
to the end of February then these (possibly) new parameters
are used to trade March and the performance for March
is appended to the out-of-sample performance data. And so on.

I tried this with Florent this summer.  We splited the data in time frames of 2 weeks and did exactly that.  The results were horrible.  It worths the shot to try it again, but the time frame has to contain enough trades to mather and I don't think one month is enough. 

[Eugene Kononov]

Theoretically, a dynamic or real time optimization maybe interesting,
let's say while JBT doing forward/trading session, an optimization
process keeps optimizing past daily market data, and update the
trading strategy with latest parameters …, sounds crazy, but may be
interesting for those extreme momentum trading strategies.

I thought about dynamic parameters, too. This would become a sort of a "meta-optimization". For example:

1. Optimize on a particular subset of data (say the last 20 trading days)

2. Use the best combination of parameters found in step 1 to back test the next immediate one day

3. Shift one day forward, and optimize on the previous 20 days

4. Use the best combination of parameters found in step 2 to back test the next immediate one day

5. Proceed with steps 1 to 4 until the end of data

6. Repeat the entire sequence from 1 to 5, using different moving window size, to find out which window size yields the best results.

This is entirely possible with JBT, although it would be *very* time consuming. The benefit is that it would effectively create a strategy with dynamic parameters which would change every day (based on the recent market behavior), and potentially capture the ever changing dynamics of the market.

[nonlinear5]

> I don't see the problem. Using your example if through
> optimization it is detemined that the best parameters
> for January are 5,20 then these are used to trade the month
> of February and the results become part of the out-of-sample
> performance. Next an optimization is done with data through
> to the end of February then these (possibly) new parameters
> are used to trade March and the performance for March
> is appended to the out-of-sample performance data. And so on.
>

Right. I was just thinking that Kelvin had something else in mind.

[Gab]

I thought about dynamic parameters, too. This would become a sort of a "meta-optimization". For example:

1. Optimize on a particular subset of data (say the last 20 trading days)

Yes that could work.  Would you add the 20 (days) as a parameter optimizable?

[Eugene Kononov]

Yes that could work.  Would you add the 20 (days) as a parameter optimizable?

Yes, that would be a "meta" optimization parameter. The entire job would constitute the optimization of optimizations. I am just inventing my own terms here. Essentially, it's just another step beyond walk forward optimization.

[rickty]

Gab,

"It worths the
shot to try it again, but the time frame has to contain enough trades
to
mather and I don't think one month is enough. "

We have to be very clear about the fact that there are two different
intervals associated with walk forward optimization. The first is the
reoptimization period. This the interval of time between optimizations
and is the one month period I suggested above. The second interval
is the interval over which the optimization is performed. There are
two
possibilities here:1) use a moving window which is sufficiently long
to get a required number of trades, e.g. 100 trades; 2) use the
so-called anchored walk-forward where all previous data is used and
grows in the walk-forward process.

[HyperD]

[Backtesting or Testing Optimization Results without Fanaticism],

"I suppose this diagram is quite clear. For example, we take the year
2007 as the analyzing period. For the first optimization the period of
optimization will be from 01.01.2007 to 31.03.2007, and the testing
period - from 01.04.2007 to 31.05.2007. After testing and recording
the results, we shift optimization and testing periods by one month
forward: optimization period will be from 01.02.2007 to 30.04.2007,
and testing period - from 01.05.2007 to 30.06.2007. And so on. As a
result we have a table for recording results of each optimization, for
example this one:"

http://articles.mql4.com/622


Regards

[Kelvin]

> As rickty mentions, it sounds like Kelvin is talking about walk-
> forward optimization. Specifically, optimize the strategy on the
> period of time from T1 to T2 and evaluate its performance on the
> period from T2 to T3. Then one can shift periods forward: optimize on
> T1+delta to T2+delta, and evaluate on T2+delta to T3+delta. Is that
> what you are proposing, Kelvin?

Yes, it is what I am proposing. The theory behind is that the current
market has larger correlatoin to the latest history data, and less
correlation to the historical data long time ago. For example, current
market is bear, and the future market will most likely perform like
the last three month. But the historical data of the last year could
be less useful.

>
> So, here is the question. Suppose we run a sequence of this
> optimizations on a simple MA crossover system, and find this:
>
> In January, the best combination of parameters was 5, 20
> In February, the best combination of parameters was 10, 30
> In March, the best combination of parameters was 100, 300
>
> Now what?

Yes, this will definitely happen. But, since the current market is
more related to the latest historical data. We should choose the
latest optimized parameters.

The backtest result of January and Febuary are also very useful. They
should be used for evaluate the consistance of the strategy. The max
down and profit variance can be used to tell the future possible risk
using the latest parameters, which is more realistic.

I also agree that this will increase the optimization complexity a
little bit. But, we have powerful machines.

[Kelvin]

I personally think that the number of trade per parameter is the key.
For example, one strategy has 2 parameters, it could need 200 trades
in the time window. Another strategy has 4 parameters. It could
require 400 trades in the time window.

> > mather and I don't think one month is enough.- Hide quoted text -
>
> - Show quoted text -