Pretesting the Effectiveness of Genetic Programming

[Rick]

I understand that the idea was to show the GP can produce systems with
improved OOS performance for fittest members over random members and
that can serve as a validation of the process. However, there are a
couple of problems with the study IMO which I find significant:

1. Comparing to a random strategy usually means in trading system
development a strategy that is selected randomly but makes sense for
the comparison. Not a totally random construction. This is important.
For example, a common random strategy is to use the same exit rules
but with random entry points. This is because what we are testing is
predictive ability, a.k.a. timing. Given that the majority of
strategies that I have seen generated by AB have random entries,
improvement of exit rule for the fittest member will always outperform
any random member with inferior exit rules in the OOS.

2. The study involved comparing losing strategies other than the one
using daily data. Does it make sense to say that GP is a valid method
because on the average you will lose less in OOS than you would lose
with a random strategy? Obviously, this still implies that the process
is a losing one although due to (1) above the results may look better
in the OOS.

Thank you

[Michael R. Bryant]

-----Original Message-----
Subject: Pretesting the Effectiveness of Genetic Programming

I understand that the idea was to show the GP can produce systems with
improved OOS performance for fittest members over random members and
that can serve as a validation of the process. However, there are a
couple of problems with the study IMO which I find significant:

1. Comparing to a random strategy usually means in trading system
development a strategy that is selected randomly but makes sense for
the comparison. Not a totally random construction. This is important.
For example, a common random strategy is to use the same exit rules
but with random entry points. This is because what we are testing is
predictive ability, a.k.a. timing. Given that the majority of
strategies that I have seen generated by AB have random entries,
improvement of exit rule for the fittest member will always outperform
any random member with inferior exit rules in the OOS.

It all depends on what you're testing. A common test is to use completely
random trades with comparable length and number of wins and losses.
Obviously, that would be an easier comparison. By the way, that's the common
comparison for estimating the sampling bias.

In my case, I wanted to see how the GP process improved upon the initial
population. Certainly, the initial population is not total random because
each strategy has to abide by the semantic rules of strategy construction.
On other hand, if I wanted to test the entry conditions, I could have used a
different comparison, as you suggest.

2. The study involved comparing losing strategies other than the one
using daily data. Does it make sense to say that GP is a valid method
because on the average you will lose less in OOS than you would lose
with a random strategy? Obviously, this still implies that the process
is a losing one although due to (1) above the results may look better
in the OOS.

I discussed this in the article: (1) The point was not to demonstrate that
GP is always a valid method. Quite to the contrary, it was to show that
pretesting the market can be useful in determining how likely it is that
you'll be successful using GP, and (2) I only used a small population with a
small number of generations and made no attempt to improve upon the results,
as would normally be done in practice.

Mike Bryant

[Mark Knecht]

Yeah, I short of have a few problems with that stuff also, but here's
how I'm thinking about it. Shoot me down. It's for my own good.

1) I am NOT questioning the use of Builder to create the strategies.
For this discussion it's a given. Sounds like for you that might not
be true.

2) Given that I'm _GOING_ to use Builder the question then is how to
use it best, most effectively, etc., so that I have a chance at
finding something successful.

3) I choose some data set - daily, 5 minute, 1 minute, etc., because I
have some interest in that data. That decision is totally outside of
Builder. There's no right or wrong. We all have our interests.

4) Having chosen the data I'm now interested in understanding whether
Builder is likely to be successful if I give it some time. Successful
in this case, and as I understand Mike's paper, is more about its
ability to find something that can be exploited in the data set
without over optimizing to the point of curve fitting.

5) To evaluate #4 I choose metrics and give them weightings. I run a
moderate sized population (100) for a few iterations (5) and keep the
complete population. This is to ensure that I'm keeping the bad models
along with the good. (I'd like to investigate that comment with Mike &
the group, but I'll do that in some other thread I think.) If the
metrics and weightings I choose produce more winning strategies than
losing strategies then I have a solution space that might be
interesting.

6) Assuming I get this far then I start doing big runs to see what
Builder produces. Only now do I do populations of 5000 over 20 or 30
generations, etc., and even then because I learned in 5 that maybe
only 70% of the 100 population runs produced and OOS profit I may end
up doing this step multiple time. However, at least I'm not doing it
blind. I have reasons to believe it might be successful instead of
just hoping it will be successful.

I've been playing with this since last week. I can say right now that
for the markets & bar types I'm looking at there are a LOT of metrics
that don't have much chance of succeeding. To me this pretest profit
allows me to discover than more quickly and focus in on the metrics
that have a better chance.

Or so I hope... :-)

Cheers,
Mark