Strategy to perform beyond out-of-sample period.

[Rocky]

I appreciate any comments and advise to my approaches to obtain a
robust strategy.
I have Adaptrade Builder generated many strategies which produce very
good equity curve in-sample and out-of-sample period. But the
strategy does not perform (equity curve goes down) with new data
beyond the out of sample period. I may miss something important or not
on the right track.
I obtained the strategies in 3 different ways:
1. Set out-of-sample net profit target after typical 20 generations.
Rebuild if the target is not met and run until getting the strategy
with the target out-of-sample profit.
2. Rebuild if out-of-sample profit is negative after 5 generations and
then run to 20 generations.
3. If the strategies do not converge after 20 generations, pick the
best out-of-sample profit strategy with good in-sample fitness.
All strategies obtained this way have good in-sample and out-of-sample
performance. But they don’t perform on new data beyond out-of-sample
period.
I used
1. At least one year 30 and 5 minute data of ES, SPY and other index
ETF.
2. Population of 2000 to 5000 and at least 20 generations.
3. In-sample data 80% and forward out-of-sample 20%.
4. Build goals are Net profit, corr. coeff, significance, pct wins,
drawdown and max MAE.
5. Reduced complexity to avoid curve fitting does not improve the
performance on new data.
I obtained many strategies look very robust. They can be applied to
many markets and different time frames and have nice equity curve. But
they all have huge MAE (maximum adverse excursion) without stops and
usually long side only as mentioned by Tom. It is too risky to use it
when get caught in a bear market.
I appreciate your comments, advice and suggestions on how to get
robust strategy to perform on new data.

[Michael R. Bryant]

When you say the strategies don't perform on new data, what kind of testing
are you talking about? How many trades and over what time period? The real
time tracking needs to be for a long enough time period and over enough
trades to be statistically significant. If that's the case and the results
are not more or less in line with the OOS period results, then there are
basically two possibilities. One is that the strategy is overly complex and
overly fit to market features that are not likely to recur. Even if the
strategy is not too complex, it can sometimes happen by chance that the
logic fits the market too well. That's why we do OOS and real time tracking.
Second, it may be that the point at which you stopped testing in Builder and
started tracking in real time unfortunately represents a change in the
market. All trading strategies need to be rebuilt or re-optimized as some
point as the market changes. It could be that your strategy needs to be
rebuilt at the end of the OOS period. To see if that's the case, try moving
your data window to encompass the new data and repeat with new real time
tracking.

Mike Bryant

[Rocky]

Thanks for your response. Here are 5 examples. They all have profit
peak at the end of OOS. I don't know the reasons.

(1) SPY 30m 1000sh from 6-28-2010 with 21 inouts & the cost is $30/
trade
Period Date Net Profit No Trade Note
End in-sample 4/16/2011 24k 221
End out-of-sample 6/28/2011 34k 46 Profit peak
Beyond OOS 8/24/2011 28k 33 No trades from 7/28 to
Current 11/4/2011 14k 64 8/11 as mkt down 20%

(2) SPY 30m 1000sh from 3-29-2010 - symmetry with 5 inputs
Period Date Net Profit No Trade
Note
End in-sample 1/15/2011 40k 260
End out-of-sample 3/29/2011 44k 56 Profit peak
Beyond OOS 7/21/2011 29k 107 EQ down
Current 11/4/2011 -4k 94

(3) TNA 30m 1000sh from 5-6-2009 -- symmetry with 12 inputs
Period Date Net Profit No Trade Note
End in-sample 12/10/2010 100k 120
End out-of-sample 5/10/2011 122k 32 Profit peak
Beyond OOS 7/27/2011 98k 16 EQ down
Current 11/4/2011 104k 20 Profit from mkt down

(4) TNA 30m 1000sh from 4-6-2009 -- 18 inputs
Period Date Net Profit No Trade Note
End in-sample 10/10/2010 107k 120
End out-of-sample 4/5/2011 137k 50 Profit peak
Beyond OOS 7/29/2011 86k 40 EQ down
Beyond OOS 9/8/2011 104k 10 Profit from mkt down
Current 11/4/2011 100k 22

(5) TNA 30m 1000sh from 4-6-2009 --same as (4), re-optimized on
9/8/2011 on TS
Period Date Net Profit No Trade Note
End in-sample 10/10/2010 120k 120
End out-of-sample 4/5/2011 189k 54 Profit peak
Beyond OOS 7/29/2011 165k 45 EQ down
Re-optimized on TS 9/8/2011 182k 20 Profit from mkt down
Current 11/4/2011 150k 22



On Nov 6, 1:37 pm, "Michael R. Bryant" <m...@BreakoutFutures.com>
wrote:

[Michael R. Bryant]

I have several recommendations in the user's guide for improving OOS and
real-time performance. Beyond that, I would probably try a looser use of the
Reset on OOS Performance option. Invoking that too much may bias the results
towards the OOS data, effectively rendering the OOS period in-sample data.
However, my prior comments still apply. It may simply be that the strategy
logic is not suitable for the market because it fits the noise rather than
the signal or because it fits a part of the signal that's changing too
quickly. That's the purpose of testing after all -- to weed out such
strategies.

I also noticed that all your symbols are 30 minutes. Sometimes all it takes
is trying a different bar size. Although I think I already mentioned this,
it probably bears repeating: try including the recent data in the in-sample
period to (1) extend the length of the build (in-sample) period and to (2)
include the more difficult period in the build process. Then perform your
OOS testing in real time. The in-sample periods you're using are not
particularly long either, so it may be a good idea in general to use more
data.

Mike Bryant

[Mark Knecht]

Hi Mike,
If the problem here is really one of the market character changing
and thus negatively effecting the OP's ability to build models that
work well during forward testing then it might be advantageous if
Builder could take clues from the user about when he/she feels the
market character is similar.

One idea I had was a user supplied indicator that would act
specifically as a Trading Filter. Take the excessively simple case of
building a model for trading long with daily bars. If I was interested
in a model that only traded when the 50 day MA was above the 200 day
MA, then I could supply an indicator that was a 1 anytime that 50MA >
200MA and 0 at all other times. If Builder limited its entries to only
those bars when the Trading Filter was a 1 then I would have control
over finding models that operate at times I feel the market character
is similar enough to warrant using the model.

If Builder supported just __ONE__ Trading Filter input then I could
generate my filter data at any level of complexity I felt worked in my
favor and (I hope) the changes to Builder would be fairly minimal,
along the lines of what it already does with the trading time period.
Just add one more term to TimeOK such as:

TimeOK = (Time >= StartTrading) and (Time <= EndTrading) and
(TradingFilter = 1);

Just an idea.

Cheers,
Mark

[Michael R. Bryant]

You may be able to do pretty much what you want by defining a custom indicator that returns a logical (true/false) type. You can't currently guarantee that it will be included with "and" and compared to "true", but those capabilities will be forthcoming.

Mike Bryant

[Mark Knecht]

Yep, I've already tried that. The problem I ran into was that if I
made the trading period fairly selective, say 20% of the bars, then
Builder always ignored it and built models for all bars. If I make it
cover too many bars then I'm not IMO making the model as focused as I
might want it to be.

It's pretty easy to imagine a number of market characteristics that
start to limit the percentage of bars you're looking at. Even a moving
average pair, along with price position above or below, and maybe high
& low volatility, starts to get you into some fairly small groups,
like less than 20% of the bars.

I don't expect Builder today to understand that I can trade 5
different models, each covering 20% of the bars. It just doesn't see
that sort of thing. (Yet!)

Thanks for keeping the idea on the radar.

Cheers,
Mark

[Jim Ching Shir]

Hi! Mike,

I am unable to post this reply on the Google Group. I send it by e-mail.

Thanks for your advice. I like to implement your advice after I understand
some of your points.
You mentioned two main reasons for the strategies don't perform on new data
are that the strategy is over complex and overly fit as well as the market
changes.
1. The strategy is over complex.
How to measure the complexity of a strategy? Is number of inputs counted for
the complexity?
How many inputs are considered too complex? Among my five examples, the
input number ranges from 5 to 21. Should I strive for inputs less than 5?
2. Market changes.
How do I measure the market change? Market moves from trending to
consolidating or volatility changes? In my five examples, the OOS profit
peaks occur on 3/29/2011, 4/5/2011, 5/10/2011 and 6/28/2011. I cannot tell
that the market (SPY) changes.
3. Extend the length of the build.
How many bars do I need? I have 4900 (30m) bars for TNA and 2600 (30m) bars
for SPY.
4. Change bar size.
Use smaller or larger bar size or both?
5. OOS testing in real time.
This is going to take several months to get enough trades to be
statistically significant. This discards the nice feature of OOS in the
Builder. There may be an effective alternative. I like your answer to
John4Bucks on Oct. 19, 2011 that optimizing the strategy parameter values on
OOS data after the strategy performs well in OOS. Right now, we can do this
in TS. But TS optimizes the strategy based on net profit only not the
fitness. Can we do the fitness optimization on parameter values of selected
strategy in Builder?

[Mark]

Hi Rocky,

I've not yet built a successful trading system, but I had a friend
that used to do this for a living.

When I was asking him for suggestions, he mentioned that for a system
to be statistically significant he would look for 1000+ trades.

I remember him saying that even 500 trades is borderline significant.

With a tool like adaptrade, I think this rule is important, because
you can quickly come up with a trading system that is profitable based
on a small sample set and/or very few trades. In your case, many
trades is possibly even more important because of your very large
search space (you have a very large number of inputs!).

Having more trades = more statistically significant.

Once you find a profitable system using adaptrade I would still put it
through a walk forward optimizer to find out how it operates in a
"real life" situation where you are constantly tuning it.

[Rocky]

Hi! Mark,
Many thanks for your advice – “more trades ( 1000+ trades) = more
statistically significant”.
How to achieve this goal of 1000+ trades in strategy building?
1. Including more bars in the data? Minimum of 10000 bars?
2. Set number of trades target to 1000+ and weight > 1 in Build Goal?
What number of inputs is not too large?
I have number of inputs from 5, 12, 18 and 21 in my 5 examples.
Do you set complexity target in Build Goal and what number is
recommended?
I would like to put the strategy through walk forward optimizer as you
recommended when it can pass beyond OOS period.
Thanks for your comments and advice.

> > > Mike Bryant- Hide quoted text -
>
> - Show quoted text -

[mandelmus]

+1 vote from me on Mark's request.

I was wondiering the same thing ... how can I get Builder to only go
long when above some MA and only go short when below another MA

[Michael R. Bryant]

I understand what you're asking, but that tends to go against the idea of
the program. It's the program's job to find the logic; it's your job to
specify what you want based on a higher-lever description (i.e.,
performance). If a moving average filter is suitable to give you the
performance you're specifying, the program should do that.

Having said that, I'm working to add flexibility to the program so it will
be possible to optionally specify more things at the level of strategy
logic. At some point, I expect to make it possible to specify arbitrary
trading logic as a starting point, which would satisfy your request.

Mike Bryant

Subject: Re: Strategy to perform beyond out-of-sample period.

[Mark Knecht]

Hi Mike,

On Sun, Jan 8, 2012 at 2:34 PM, Michael R. Bryant
<m...@breakoutfutures.com> wrote:
> I understand what you're asking, but that tends to go against the idea of
> the program. It's the program's job to find the logic; it's your job to
> specify what you want based on a higher-lever description (i.e.,
> performance). If a moving average filter is suitable to give you the
> performance you're specifying, the program should do that.
>

If we only wanted to test one contiguous time frame then yes, the
program _should_ be able to do that. However it might not find it, and
why use the computer time _hoping_ the program finds the filter we're
looking for. For clarity, I used the moving average example earlier
because it's a pretty typical filter for looking at end-of-day
trading, but not because I'm interested in that specific filter.

In my case there are two things that come up in this area that today
the program simply cannot do:

1) My filters use many bits of math (descriptive statistics, calculus,
etc.) that are far outside of the standard indicator type math that
the program understands today.

2) I absolutely do _not_ want to look at contiguous time frames. In my
case I might want to model operation in Builder over a recent 1-2
month time period, but then I want to evaluate overa much longer
timeframe anytime the filter identifies that type of trading period
where I expect, according to my filter, that it might have occurred 40
times in the last 10 years, etc..

> Having said that, I'm working to add flexibility to the program so it will
> be possible to optionally specify more things at the level of strategy
> logic. At some point, I expect to make it possible to specify arbitrary
> trading logic as a starting point, which would satisfy your request.

I do appreciate your considering this as time goes forward, along with
some of the other requests we've made, most specifically for me the
ability to optimize based on different sizing formulas, most
specifically being able to target specific Kelly & Optimal f values of
interest.

Thanks & cheers,
Mark

[mandelmus]

ya, you are right Mike ... the MA crossover thing was a bad example
for my case, it's already built into Builder ... perhaps a better
example would be some of the things Mark mentioned ... and, what if
instead of using a moving average, I wanted to use some complicated
alternative timing strategy, where my own "magical" indicators said
that a strategy was optimal. For example, as someone previously
mentioned, what if I wanted Builder to only look for strategies during
trending, ranging, or statistically significant periods based on when
my "magical" trend-range finder said to start trading. Those
conditions would produce either a 1 (zone open for trading) or 0 (zone
closed for trading) signal.

I guess I could get TradeStation to display/save trade data only
during the "zone open for trading" time frames just like I currently
do when exporting restricted sessions and feed that to Builder.

Mark is right on with the statistical angles. Most of my recent few
years of investigation has been in the realm of PDFs, CDFs, Markov
propagations, Hurst exponents, etc. I have found some great trading
strategies based on those and I was hoping that Builder would extend
my edge a bit in that area.

Mike, as far as money management integration with Builder ... do you
see a way to integrate that with Builder or would it be the same as
exporting directly to MSA?

[Michael R. Bryant]

You could implement what you're talking about using a custom indicator, such
as with a Boolean return value. If Builder finds it useful, it will be used.

Yes, position sizing is on my list.

Mike Bryant

Subject: Re: Strategy to perform beyond out-of-sample period.

ya, you are right Mike ... the MA crossover thing was a bad example