Strategies quickly fade
Mark
trader... I think it's clean this way having cash at the end of the
day.
I'm more interested in getting the methodology right - that I'm not
curve fitting in the method I apply to getting a working strategy.
I notice that most of the strategies I build - last only about one
month before they go bad.
So to prove myself right, I wrote a harness to apply my build
methodology to about 50 ETFs...
In summary it is:
* 10 minute bars from 2009 to 2011-09-30
* Default build goals and options
* Trade from 9:40 to 3:50
* Sample size of 2000 with 5 generations
* Retain top 400
From that group I got 19 working strategies.
NOW - to choose which strategy to trade I simply chose that with the
highest significance from the OOS set.. Am I curve fitting here ??
I ran the 19 through portfolio backtester in parallel and found that
they were good for one month.
As an aside, I repeated the exercise removing the "Trade from 9:40 to
3:50" and that group lasted til the end of December before going
bad...
What am I doing wrong? I'm thinking maybe I don't have a significant
number of trades?
Keith Brightfield
Mark Knecht
Hi Mark,
No answers, but a few comments for you to consider:
1) By my count on TradeStation you have something like 27,000 bars of
data. That might be too much data allowing Builder to find some really
sweet trades which end up dominating the results. What happens to the
returns when you throw out the top and bottom 5% of trades and look at
the 90% remaining?
2) In my experience I'd reduce population size and up the generations.
Maybe 500/20 instead? Should be roughly the same amount of compute
time.
3) You don't say whether you had and IS-OOS setting in Builder. If you
didn't, try it. If you did then what was the date where you changed
and did you reset the runs based on any criteria?
4) I think the real thing you need to determine is whether the results
fell off after 1 month because the model died or did the character of
the market change? What would happen if you built new models using
data from 1/1999-1/2001 and then looked at the results over the last
11 years? Would there be times where the models excelled and other
times where they failed?
Good luck,
Mark
Mark
Thanks for the feedback above, I think I have pinpointed what I'm
doing wrong with the product.
I have built a harness in AutoIt to download intraday data from
eSignal, export it, then run a Builder template on that data.
Most of the symbols can be generated for - which is good. Up to this
point, it's basically a technical problem of automating the whole
process because I don't want to be clicking away and then waiting for
the build process to complete when I could be at the gym or sailing.
Here comes the tricky part.
I have also automated the selection of the strategy. As per Mark or
Keith's suggestion, from the OOS resultset I pick the one with the
highest correlation or significance. It must also meet certain
profit, average trade, number of trades and profit factor criteria.
From there, I take the 25 or so chosen strategies and then walk them
forward on new data in Portfolio Optimizer in MultiCharts.
Using data up to 2011-06, I've got it to work from period's 2011-07 to
2011-11 then fail.
Using data up to 2011-11 Ive not got it to work from 2011-11 to now.
So I've spent a few days pondering this problem.
The only step that I can see is introducing bias is the selection bit
- where I sort the generated strategies and pick which one to trade.
I see this as introduced selection bias or survivorship bias because
in a real life situation you cannot pick the outcome that eventuates.
I can see a positive out of this however.
I am at the point now where I see the out of sample results as a large
walk forward pool (which really, is what it is).
If you have 100 strategies in there, and only one is profitable,
likely that is an outlier and the generation process cannot fit a
strategy to the signal part of the symbol.
However, if 90% of them are profitable, then I believe from this point
forward, you will have a 90% chance of success trading forward.
** NOTE for this to work you must only generate up to the point before
strategies start to converge, I think about 7 generations ***
If you take strongly trending stocks or instruments, like Swiss Franc
- US Dollar, Australian - US Dollar or Apple Computer Stock, it is
possible to generate for those and have good walk forward outcomes
because they are strongly trending. Whether they continue to do so is
always unknown, but Builder is smart enough to generate a BTFD
strategy for those.
If you take very random stocks like the QQQ tracker fund it has very
poor walk forward results - it is trying to fit something to basically
noise.
So in summary to make this product work for me
* I automated the generation process and work on a very large universe
of instruments
* I believe I introduced selection bias in the "cherry picking" of the
best strategy from the OOS pool
* I now believe the OOS pool is to be used as like a Monte Carlo
simulation of possible outcomes, the higher percentage of success
there the higher success you will have
Rick
fitting. I've tried in the past another product but I couldn't get any
startegies that worked under real conditions.
What type of systems do you get in general? Do they involve
indicators? Do these indicators have parameters that are optimized by
the program?
I think choosing the best startegies from an OSS run introduces
survivorship bias.
Lawrence Lewis
If you have to select from a bunch of strategies, all of which rated highly during the training period, you are using insight that was not available or discoverable during the training period. A no no.
You might as well not have an OOS period. The question you have to ask is, "Is there some piece of information that was known during the training period that could have discriminated between the good OOS results and the bad OOS results.
Rick
>
> - Show quoted text -
Steve LeBel
I have taken a different approach with the OOS. Instead of discounting it,
I have reserved an additional set of OOS data, an OOS-2, if you will.
As an example, I am using five and a half years of 30 minute bars for the
strategy. I am using 4 years for the IS, and 1 year for the OOS. If (and
only if) I find something I like (I have about 12-15 things I look at before
I decide if I like something), then I will look at the OOS-2 period from the
last 6 months. If that 6 months fails, then the strategy is rejected. No
exceptions.
In spite of my rigor here, I have still terminated (for poor performance)
about 1/3 of the strats that made it through this process. This has given
me about 5-6 strategies that seem to be working for me.
I can't say I have seen the strategies "quickly fade", although in the 6
months I have been working with them, there have been some ebs and flows. I
do monitor them individually by reviewing their equity curves, maximum
drawdowns, maximum consecutive losses, total profit, average profit per
trade, etc. If they fail to perform in these areas, then I can and do
suspend them.
I also have a few additional tests I run on the strategies before letting
them do actual trading for me. I have found many of the AB strategies are
incomplete (to me), and they do not have certain protective stops, profit
taking exits, time exits, etc. So I run some tests to see if I can limit my
risk or increase my gain by adding some extra code in these areas. I have
created simple strategy modules (similar to the TS strategy modules that do
just one type of exit), and I add them to the chart that has the AB strategy
I am testing. So far this has been helpful, and I sometimes keep one of the
modules active with the AB strategy.
I was also testing some market condition filters to see if there are times
when a trade should not be taken. When I was working with strategies that
produced a higher number of trades, I saw some potential here, but the 5
years of 30 minute bars have not given me the confidence to insert any of
the filters because the number of trades would be reduced to the point where
I am concerned about over-fitting.
Steve
*********************************************************
Strategy Traders Forum - A place where TradeStation users
can share ideas and resources for their mutual benefit.
www.strategytraders.org
*********************************************************
Mark
> You might as well not have an OOS period. The question you have to ask is,
> "Is there some piece of information that was known during the training
> period that could have discriminated between the good OOS results and the
> bad OOS results.
I agree with you. The problem is that the inputs to the process are
not strategy code, but the build objectives.
So whether Adaptrade is curve fitting or generating a predictive model
is unknown until we run it in *real life*.
Good stuff, but where does this get us?
I've been thinking more on this, and I think the process to overcome
the problem of selection bias would be similar to a walk forward
optimizer.
Fix your build options and goals, generate strategy, walk forward one
month with the best strategy.
Move the data forward one month, rebuild, walk forward with the best
strategy.
Because realistically that is what will happen - we will continually
generate new strategies and run with the best one.
We can almost ditch the out of sample set that way, we simply choose
the fittest member from the in sample set and choose to run with it.
There are flaws with this walkforward approach to strategy validation
too.. Because if we pick Apple or AUDUSD as the instrument to trade
we are using knowledge not known to the developer at build time; that
these two instruments are strongly trending (up). Hence it is always
better to test strategies in groups of symbols rather than one at a
time so that we don't cherry pick what we already know will perform
well.
So I guess my thinking is:
* Choosing from the OOS data renders this data as part of the in-
sample set, because in real life you do not have that choice,
introducing *possible* selection/survivorship bias
* The only way to realistically see how this performs "going forward"
is to mimic how the product will be used in a real life situation,
similar to a walk forward optimizer
Mark
findings.
My methodology was to pick the best looking strategy built using 4
months of data that validated in the Adaptrade OOS results, and then
get the P/L in MultiCharts for the following month.
Here were the settings:
• Symbol: TNA
• Window: Rolling 4 months stepping forward 1 month per cycle
• Other settings are 15 minute bars, Intraday first trade at 9:45 and
last at 3:45, $15 R/T
The P/L is as follows I've prefaced the profit months with a (*):
*Apr 11, Profit $1894.00
*May 11, Profit $512.00
Jun 11, Profit -$243.00
*Jul 11, Profit $2163.00
*Aug 11, Profit $1863.00
Sep 11, Profit -$151.00
Oct 11, Profit -$940.00
*Nov 11, Profit $3363.00
Dec 11, Profit -$908.00
Jan 12, Profit -$385.00
Feb 12, Profit -$902.00
Notably, Dec to Feb still is not profitable (suggesting that the
market conditions had changed during this time).
Overall, $6266 profit was made over 149 trades averaging $42.05 per
trade. I was trading in $10,000 blocks. No slippage or commission
included for the walkforward P/L from multicharts.
This is a simple walk forward process that has:
• Inputs as builder goals and strategy options
• Outputs as the validated strategy
• Real world recorded results as the walk forward month in Multi
Charts
From what I can see this resembles what real world outcomes are in the
world of automated trading - feast or famine with some profit to be
made / similar to that described by Steve above.
*** The question now becomes *** Not how to get the best looking
strategy in Adaptrade.. But what symbols out there have repeatable
patterns over time, that are *not* rapidly changing - ie have
consistent out of Adaptrade performance (like OOS-2 or walkforward).
Also, if I had the time I would like to repeat this experiment to
prove that the profit made here was not an outlier, but that is for
another day...
Rick
sample set, because in real life you do not have that choice,
introducing *possible* selection/survivorship bias
experienced and understand. But, I think that selecting from OSS is
just used to rule out spurious correlations, not to increase
probability of profitable future performance. There is no selection
criterion that can be used to guarantee future performance.
* The only way to realistically see how this performs "going forward"
is to mimic how the product will be used in a real life situation,
similar to a walk forward optimizer
the most out of past data. You still have the problem of selecting the
system with the highest probability of profitable future performance.
Lawrence Lewis
a system. It is so tempting to use a combination of good correlation
and equity performance...that is, a low drawdown, smooth, upward
sloping equity curve. However, by now, we should understand that that
IS NOT necessarily the best predictor of a low drawdown, smooth,
upward sloping equity curve in the future.
So, what might be? Well, there are lots of other metrics that
characterize trading systems. The question is, which ones are
predictive, not which ones look nice in retrospect. Take a look at
some of the work at meyersanalytics.com
Rick
On Mar 6, 10:04 am, Lawrence Lewis <l...@usa.com> wrote:
> One way to approach this is to evaluate which metrics we use to select
> a system. It is so tempting to use a combination of good correlation
> and equity performance...that is, a low drawdown, smooth, upward
> sloping equity curve. However, by now, we should understand that that
> IS NOT necessarily the best predictor of a low drawdown, smooth,
> upward sloping equity curve in the future.
>
> So, what might be? Well, there are lots of other metrics that
> characterize trading systems. The question is, which ones are
> predictive, not which ones look nice in retrospect. Take a look at
> some of the work at meyersanalytics.com
win rate.
Excellent point you made here. But keep in mind that most metrics are
related to one another.
mandelmus
Forward Performance Metric Explorer" ... http://meyersanalytics.com/wfme.php
Exactly. Most of us have powerful enough computers. Rather than
guessing which combination of metrics is the best predictor, why not
have Builder calculate/suggest the most robust combination of metrics
and weights.
Additionally, Builder should provide us with an in-sample window,
validation window and a true out-of-sample window to speed up the
process a bit.
Michael R. Bryant
in order the determine what he/she actually wants. If you say you want it to
automatically find the metrics that give the most robust performance, you
have to define "robust". Any reasonable definition would most likely involve
some definition of performance, which would require input from the user as
to what constitutes good performance, leaving you back to where we started.
I noticed on the page for WFPME, for example, they talk about
user-selectable criteria for deciding this, much like in Builder (except
seemingly less flexible). There's no free lunch. Having said that, I have
several ideas in mind for streamlining the build metric selection process.
When I get closer to working on them, I'll have more to say on the topic.
Mike Bryant
-----Original Message-----
Subject: Re: Strategies quickly fade
Steve LeBel
I can tell you a "robustness" feature I would like to see.
I would like to see all of the strategy combinations coded in some way so
they can be recognized as a unique strategy. I am not talking about the
parameter settings. I know this sounds overwhelming, but even if a strategy
combination was labeled as "ACEFG" or longer, it would let us answer some
questions about the viability of the parameters. For example, if on
separate IS and OOS summary pages, we found that 52 versions (i.e.,
different parameters) were used on a single strategy, we would have more
confidence the strategy was not just finding a peak performance point.
Perhaps an easier approach would be to have AB scan the strategy list and
identify how many different strategies there are and how many parameter
variations there are.
This would make it easier to pull out the robust strategies. It would also
give us something we can filter and manipulate in excel so those of us with
different outcome criteria can view things differently. It would also make
it much easier to take the more robust of the core strategies and test their
optimization ranges. Those that look good can easily be run through a WFO.
Steve
Lawrence Lewis
"structure" and "parameters".
In fact, I think it might be interesting to only evolve structure and
use more typical optimization for the parameters.
Keith Brightfield
To: adaptrad...@googlegroups.com
Sent: Wednesday, March 7, 2012 4:39 PM
Subject: RE: Strategies quickly fade
Mark
etc, we used a few rules to determine what was a robust strategy.
1. Number of trades - must be high enough... A friend of mine used to
say 1000+
2. OOS testing combined with a Walk Forward Optimising
3. Lastly, I think to test for robustness you might test in other
markets. **** < --- this one I will discuss on
For example, if you are building a S&P system, to prove you have not
curve fitted, you would test it on NQ data.. You don't expect the
same results, but similar. Similarly, if you are developing something
for EURUSD you would test it also for GBPUSD or similar.
I think the more generations you do the more refined / fit the
strategies are to the given data. This is bad - becase we all know
that over fitting / optimization = bad out of sample performance.
So I started doing a build with a difference:
Three symbols - SPY, TNA and IWM... I would build for 2 generations,
then save, then switch to the next symbol and **restart** from where
it left off. You can change the underlying data mid cycle I think if
the date ranges and granularity stay the same. I changed the point
value also so that I am making similarly sized trades.
What I found is that by cycling through different symbols over many
generations, the strategies converge (after maybe 10 cycles), but the
remaining strategies are generally more robust. Strategies that are
not robust over a number of instruments are simply discarded as part
of the generation process. I automated the switching of instruments
with a windows automation tool so that I just let it loop until I'm
satisfied that things have converged and generated sufficiently.
It went like this:
Load SPY data, build 2 generations
Load TNA data, restart build for 2 generations
Load IWM data, restart build for 2 generations
Load SPY data, restart build for 2 generations
and so on
I don't think this is the holy grail - I really cannot get profitable
performance from the three months 2011-dec to 2012-feb, but I did get
very very good performance between 2011-feb and 2011-oct **completely
out of sample** for SPY and TNA. IWM didn't walk forward successfully
within builder so that was discarded. I'm assuming that volatilities
have changed around November last year.
There is a great feature in Adaptrade that I've only started using
recently - Evaluate and Evaluate All ... It lets you validate the
strategies on new data after the generation process... This too I
guess would be another test of robustness without using future data.
If you can't look into the future, there's nothing stopping you from
going sideways with other symbols.
Mark
I really am getting to like your product, I only recently got a chance
to spend lots of time with it.
I have two thoughts on mechanical methods to test for robustness.
I used to work in CRM analytics and one measure of success we used to
have was a control group. For example, if you were to market to 100
customers you would leave 10 as a control group to see how they
behaved without recieving any marketing.
I think this concept could also be extended to Adaptrade beyond the
OOS feature. You could build on data for symbol X and have symbols A
B and C as a control group. These symbols are similar to X but not
used in the strategy generation. Adaptrade will test the final
strategies on A B and C as well as X. It may prove that the strategy
is not overly fit to symbol X. If the strategy works on X and A B and
C and in the OOS period, I think that is using all possible data to
determine that the strategy has the best chance of success going
forward.
Additionally you could run a Null hypothesis test.
For example, if the strategy made 10 long trades, a Null hypothesis
test would place 10 long trades at random. If the end result is the
same as the strategy, then the strategy has no edge it is no better
(or worse) than placing trades at random. I think this would be
particularly useful in stocks like Apple where any long trade would
have made money!
Hope you are well, look forward to what ever features are in the next
release.
Kind regards,
Mark.
On Mar 8, 9:39 am, "Michael R. Bryant" <m...@BreakoutFutures.com>
wrote:
> in order the determine what he/she actually wants. If you say you want it to
> automatically find the metrics that give the most robust performance, you
> have to define "robust". Any reasonable definition would most likely involve
> some definition of performance, which would require input from the user as
> to what constitutes good performance, leaving you back to where we started.
> I noticed on the page for WFPME, for example, they talk about
> user-selectable criteria for deciding this, much like in Builder (except
> seemingly less flexible). There's no free lunch. Having said that, I have
> several ideas in mind for streamlining the build metric selection process.
> When I get closer to working on them, I'll have more to say on the topic.
>
> Mike Bryant
>
>
>
>
>
>
>
> -----Original Message-----
> Subject: Re: Strategies quickly fade
>
> Lawrence, thanks for the link. I like the concept behind this, "Walk
Michael R. Bryant
I’ve talked about this a bit before on the forum, I believe. If it were only a matter of updating the metrics, it’d be done. Unfortunately, adding constraints for the build metrics will require a major reworking of parts of the build algorithm and will require some research/experimentation. Those familiar with optimization theory and algorithms know that constrained optimization is probably an order of magnitude more difficult than unconstrained optimization. It’s on my list, though.
The problem with adding multiple items on the reset on OOS performance option is that the more you add, the closer the OOS segment comes to being in-sample data. Keeping it less restrictive means you’re violating the separation of OOS and in-sample less.
Mike Bryant
Subject: Re: Strategies quickly fade
As you update the build metrics, please consider allowing the capability to specify "greater than or equal to" and "less than or equal to"
Mark Knecht
<m...@breakoutfutures.com> wrote:
> I’ve talked about this a bit before on the forum, I believe. If it were only
> a matter of updating the metrics, it’d be done. Unfortunately, adding
> constraints for the build metrics will require a major reworking of parts of
> the build algorithm and will require some research/experimentation. Those
> familiar with optimization theory and algorithms know that constrained
> optimization is probably an order of magnitude more difficult than
> unconstrained optimization. It’s on my list, though.
>
>
>
> The problem with adding multiple items on the reset on OOS performance
> option is that the more you add, the closer the OOS segment comes to being
> in-sample data. Keeping it less restrictive means you’re violating the
> separation of OOS and in-sample less.
>
>
>
> Mike Bryant
>
>
A couple of ideas for you to consider down the road:
1) A program I bought years ago called Trading Solutions actually
divided the data into 3 regions. I forget the name of the regions but
the use was (more or less) as follows, keeping in mind that I haven't
used the program in at least 6 years:
a) In-sample data for model generation
b) A second area used for correlation. IIRC they used this second area
like you use your OOS area today when you do resets, but also checked
that the results correlated in some way with the IS region. (I.e. -
not that they just made money.)
c) A third area that was truly OOS.
I think someone else here (mandelmus maybe?) suggested 3 areas. Maybe
a model like that above would make sense for Builder some day.
The other thing I'd like to see, which I cannot imagine would be all
that difficult, is to make the OOS reset driven by some _percentage_
of whatever the IS parameter was. I.e. - Instead of resetting if
profit < 0, reset if OS profit/trade is less than 75% of whatever the
profit/trade was IS, or even that the OOS fitness value is < 80% of
the IS fitness value, etc.. To me a good model OOS is one that
performs more or less like IS. Maybe a feature like that wouldn't be
as hard to integrate into your current setup?
And still, please consider something like Kelly Ratio as a single
Build metric. Targeting a Kelly Ratio range of 15%-20% for instance
could be pretty powerful.
Just some ideas.
Cheers,
Mark
Rick
Build metric. Targeting a Kelly Ratio range of 15%-20% for instance
could be pretty powerful."
You can set the metrics so you get the desired ratio.
To : Mike Bryant
Hi Mike. Did you delete a post of mine where I gave a link to how PAL
does the multi-symbol control group in response to Mark's comments?
II think it's good to know what others do; systems that work on just
one symbol are more than often artifacts of survivorship and selection
bias. I also think you shouldn’t worry about competition too much.
Usually people who have money to buy a copy of AB may also buy a copy
of PAL and work with both.
Mark Knecht
> "And still, please consider something like Kelly Ratio as a single
> Build metric. Targeting a Kelly Ratio range of 15%-20% for instance
> could be pretty powerful."
>
> This ratio is equal to: (avg. trade/avg. win) = 0.15 - 0.20
>
> You can set the metrics so you get the desired ratio.
>
On paper yes, but in real life no, that doesn't actually work very
well in Builder today.
Yes, you can set Builder to target those two values. Let's assume you
choose Avg. Trade = $200 & Avg. Win at $300. Builder unfortunately
emphasizes very heavily ANY solution that _exactly_ hits _one_ of
those values, creates a high fitness number, and then eliminates
solutions that have come close to both values but not exactly hit
either one.
This would be my first use of the ">=" request in the metrics section.
If I told Builder I'd be happy with ANY solution that had an average
trade >= $175 AND an average win >= $250 then I've effectively set a
minimum ratio that I'm happy with and Builder can keep the solutions
that are most appropriate for my interests.
Cheers,
Mark
Michael R. Bryant
Mike Bryant
-----Original Message-----
Subject: Re: Strategies quickly fade
Rick
control group. "
bias substantially. I think 2 - 3 symbols in the control group may be
enough. Actually, I'm currently testing the demo of PAL and this is
what it does as an option. You can specify the profit factor of the
results in the control group to make this as robust as possible. See
how they do it: http://tinyurl.com/7lqjgpx
mandelmus
prefer comparing against a monte carlo distribution of random trades
traderjohn
again, but are you sure that’s the right formula? I have code a Kelly
optimization metric into NinjaTrader and Multicharts, but the formula
I used was Kelly f = W –((1-W)/R, where W = %wins, and R = avgWin/
avgLoss.
Actually, this is not quite correct since, technically, this formula
only applies to situations where the payoff is a Bernoulli
distribution, with all wins being the same size and all losses the
same size. You find this some gambling situations but not in
financial markets where profits and losses are all over the place. If
you want to target geometric growth when trading the markets, you are
supposed to use Optimal f. But I suspect this is a bit academic
since I don’t think the calculated numbers are all that different – at
least not enough to matter if you’re using a fractional Kelly bet size
anyway that is well below the calculated value to be on the safe side
of the expected drawdowns (e.g. 20-50% of the calculated Kelly f).
So I think the above much simpler formula might still be used as an
approximation.
But Mike is the Money Management expert here, so please correctly me
if I’m wrong, Mike.
Anyway, Mike, the reason I think this might be a good metric for
Builder is that my experience in using it for optimization in
NinjaTrader is that it produces much straighter equity curves than
what you get from max Net Profit, max ProfitFactor, max
ExpectancyScore, or any other optimization metric I’ve tried (although
max ExpectatncyScorePerDay come closest). This seems to make sense,
since the parameters that produce a straight equity curve should be
similar to the ones most suitable for compounding. But I don’t know
whether a Kelly metric would come close to the performance of
Builder’s Correlation and Significance metrics in the straightness
department. I also don’t know whether this kind of optimization
metric would have higher or lower curve fitting risks than non-MM
metrics in a GP program like Builder. If it is viable, I’d almost
prefer having a way (beyond the platform workarounds that Mike
describes in the newsletters) to enter it myself as a custom metric,
rather than risk the effects of a directly releasing this Kracken on
the retail market!
Any comments, Mike?
Rick
On Mar 9, 12:23 pm, traderjohn <johnraint...@hotmail.com> wrote:
> Rick, glad to see the Kelly fraction coming up in the feature requests
> again, but are you sure that’s the right formula? I have code a Kelly
> optimization metric into NinjaTrader and Multicharts, but the formula
> I used was Kelly f = W –((1-W)/R, where W = %wins, and R = avgWin/
> avgLoss.
the Kelly metric by the way? Did you tarhet a range, a min value or a
max value?
>
> Actually, this is not quite correct since, technically, this formula
> only applies to situations where the payoff is a Bernoulli
> distribution, with all wins being the same size and all losses the
> same size. You find this some gambling situations but not in
> financial markets where profits and losses are all over the place. If
> you want to target geometric growth when trading the markets, you are
> supposed to use Optimal f. But I suspect this is a bit academic
> since I don’t think the calculated numbers are all that different – at
> least not enough to matter if you’re using a fractional Kelly bet size
> anyway that is well below the calculated value to be on the safe side
> of the expected drawdowns (e.g. 20-50% of the calculated Kelly f).
> So I think the above much simpler formula might still be used as an
> approximation.
>
> But Mike is the Money Management expert here, so please correctly me
> if I’m wrong, Mike.
>
> Anyway, Mike, the reason I think this might be a good metric for
> Builder is that my experience in using it for optimization in
> NinjaTrader is that it produces much straighter equity curves than
> what you get from max Net Profit, max ProfitFactor, max
> ExpectancyScore, or any other optimization metric I’ve tried (although
> max ExpectatncyScorePerDay come closest). This seems to make sense,
> since the parameters that produce a straight equity curve should be
> similar to the ones most suitable for compounding. But I don’t know
> whether a Kelly metric would come close to the performance of
> Builder’s Correlation and Significance metrics in the straightness
> department. I also don’t know whether this kind of optimization
> metric would have higher or lower curve fitting risks than non-MM
> metrics in a GP program like Builder. If it is viable, I’d almost
> prefer having a way (beyond the platform workarounds that Mike
> describes in the newsletters) to enter it myself as a custom metric,
> rather than risk the effects of a directly releasing this Kracken on
> the retail market!
and R and as a result when optimizing for it is equivalent to
optimizing for those other two parameters. But when optimizing for W
and R is equivalent to optimizing wrt profit factor. Hence, there is
no need for a new metric IMO.
traderjohn
On Mar 10, 12:11 pm, Rick <rh4...@gmail.com> wrote:
> On Mar 9, 12:23 pm, traderjohn <johnraint...@hotmail.com> wrote:
> The two forms are equivalent if you do the algebra. How did you use
> the Kelly metric by the way? Did you tarhet a range, a min value or a
> max value?
posted in Builder. I’ve only used it as an objective function for
conventional optimizations in NinjaTrader and Multicharts. To do
that I maximize the Kelly function in a fixed-lot optimization to find
the parameters that give the highest Kelly number. I’ve had bots that
traded very infrequently and gave a Kelly number as high as 70% but
most are in the teens, but I would never trade that fraction of my
account, of course. However, I think this is useful as a relative
metric for finding bots (settings) that are relatively safe for
compounding (i.e. those with the highest Kelly numbers, which are
seldom the highest Max Profit solutions btw). Then, within my actual
trading bots, I’ve added a position sizing function that automatically
calculates a fixed percentage position size for each trade based on
some fraction of the “optimum” Kelly number, using the max MAE value
from the fixed lot backtest in place of the “stoploss” value that is
normally used in such position sizing algorithms. I hope that answers
your question. These are two different ways of using the Kelly
fraction outside of Builder.
Btw, I think it would take some pretty creative algebra to show that
your formula and my formula are equivalent! I just don't see how they
can be. If you are basing your statement on the Kelly.pdf on that
website, well I looked at it and I don’t I find an algebraic proof
that the two are equivalent, only an unconvincing assertion that they
are. (He SETS the two as being equal, but he doesn’t actually derive
the equivalence algebraically as far as I can see.) I’m not an expert
on the Kelly formula and my algebra is not as nimble as it used to be,
but this looks fishy to me. Anyway, I just wanted to give you an
honest reaction, this is not the place to debate Kelly formulas! Lol.
However, if Mike has the time to step in and give us the benefit of
his extensive experience with money management algroithms and comment
on whether he sees any role for a Kelly metric in Builder, I'd be all
ears!
Rick
You claimed that you used: f = W –(1-W)/R
This is equivalent to: f = W – [(1-W)avgLoss]/avgWin
Multiply through by avgWin: f x AvgWin = W x AvgWin - (1-W) X AvgLoss
But W x AvgWin - (1-W) X AvgLoss = expectancy = AvgTrade (Easy to
show)
Thus: f = AvgTrade/AvgWin
Q.E.D.
Actually some here are giving Mike a hard time about his program
asking for a metric based on Kelly but that sounds funny to me because
it is trivial to see that optimizing for Kelly is equivalent to
optimizing for R and W. The equations say so. Some people in forums
who have elevated Kelly to a special optimization metric basically
haven't done their homework and I wound't be surprised if they don't
understand what they are talking about, as usual of course.
Michael R. Bryant
(MSA) software. After doing a little arithmetic, it's the same as the
formula you present (f = W - (1 - W)/R). Yes, it assumes all wins the same
size and all losses the same size, but that doesn't reflect an error in the
formula. That's one of the assumptions of the Kelly formula.
With regard to whether it might be useful as a metric or whether we could
accomplish the same thing by maximizing both the win/loss ratio and the
percentage of wins, the result is likely to be different if you maximize
Kelly. If you simply include the win/loss ratio and the percentage of wins
as build metrics in Builder, they'll be added together in a linear
combination, which is how the fitness is computed. This will produce a much
different function of those two metrics than the Kelly formula provides.
Therefore, maximizing the Kelly formula may give different results than
maximizing the fitness composed from the two metrics.
I'll be adding a bunch of new metrics to the next release, including Kelly.
Mike Bryant
-----Original Message-----
Subject: Re: Strategies quickly fade
Rick, glad to see the Kelly fraction coming up in the feature requests
again, but are you sure that's the right formula? I have code a Kelly
optimization metric into NinjaTrader and Multicharts, but the formula
I used was Kelly f = W -((1-W)/R, where W = %wins, and R = avgWin/
avgLoss.
Actually, this is not quite correct since, technically, this formula
only applies to situations where the payoff is a Bernoulli
distribution, with all wins being the same size and all losses the
same size. You find this some gambling situations but not in
financial markets where profits and losses are all over the place. If
you want to target geometric growth when trading the markets, you are
supposed to use Optimal f. But I suspect this is a bit academic
since I don't think the calculated numbers are all that different - at
traderjohn
news about the new metrics. Builder just goes from strengh to
strengh!
John
On Mar 11, 4:30 pm, "Michael R. Bryant" <m...@BreakoutFutures.com>
wrote:
Mark Knecht
<m...@breakoutfutures.com> wrote:
<SNIP>
>
> I'll be adding a bunch of new metrics to the next release, including Kelly.
>
> Mike Bryant
>
>
>
Really great news from my perspective. Thanks Mike! I look forward to
trying it out.
Without committing to anything, what's your best guess at a time frame
on the next release? Weeks? Months?
Cheers,
Mark
James Tann
On Tuesday, February 28, 2012 4:10:58 AM UTC-8, Mark wrote:
Spent a few days (weeks) now with this product - I'm an end of day
trader... I think it's clean this way having cash at the end of the
day.
I'm more interested in getting the methodology right - that I'm not
curve fitting in the method I apply to getting a working strategy.
I notice that most of the strategies I build - last only about one
month before they go bad.
So to prove myself right, I wrote a harness to apply my build
methodology to about 50 ETFs...
In summary it is:
* 10 minute bars from 2009 to 2011-09-30
* Default build goals and options
* Trade from 9:40 to 3:50
* Sample size of 2000 with 5 generations
* Retain top 400
From that group I got 19 working strategies.
NOW - to choose which strategy to trade I simply chose that with the
highest significance from the OOS set.. Am I curve fitting here ??
I ran the 19 through portfolio backtester in parallel and found that
they were good for one month.
As an aside, I repeated the exercise removing the "Trade from 9:40 to
3:50" and that group lasted til the end of December before going
bad...
What am I doing wrong? I'm thinking maybe I don't have a significant
number of trades?
mandelmus
Michael R. Bryant
I don’t believe anyone is suggesting using Kelly f for position sizing, which, as you note, is impractical to say the least. The idea is to use it as measure of strategy performance or fitness. For example, if you calculate the Kelly f value for a strategy based on trading single lots, a strategy with a Kelly f of 50% is presumably much better than one with a Kelly f value of 10%. Nothing to do with position sizing.
Mike Bryant
From: adaptrad...@googlegroups.com [mailto:adaptrad...@googlegroups.com] On Behalf Of James Tann
Sent: Sunday, March 11, 2012 7:46 PM
To: adaptrad...@googlegroups.com
Subject: Re: Strategies quickly fade
Am I missing something here? I have looked at the Kelley formula and would not use that as a fitness function or for any other criteria since the assumptions it uses are for essentially a normal curve and as we all know, this is far from the case. It is fun to see how much money one would make if they use Optimal F or a maximized Kelley, but my experience over the last 40 years is that is a certain way for your "risk of ruin" to be 100%. Anyone trading at anything near the optimal f or Kelley will surely blow out his account sooner or later. At least that is what I have read, studied and always thought was true. If I am mistaken, sure would like to understand this and why.
Michael R. Bryant
Mike Bryant
Rick
On Mar 12, 3:33 am, "Michael R. Bryant" <m...@BreakoutFutures.com>
wrote:
"For example, if you calculate the Kelly f value for a strategy based
on trading single lots, a strategy with a Kelly f of 50% is presumably
much better than one with a Kelly f value of 10%. Nothing to do with
position sizing."
that for yourself?
traderjohn
certainly was not arguing for direct use of the Kelly fraction to
determine position size. I agree, that is a fools errand. Forget
about that. The less said about it in this forum the better.
For me the main value of the Kelly number is that it has some
interesting properties as an optimization metric. My observations on
this are impirical rather than theoretical. Having played around with
several differnt optimization metrics in testing many NinjaTrader
bots, I kept coming back to the Kelly script simply because it gave
the nicest equity curves. I'm talking about straightness, not
height. These optimizations were run on fixed lot position sizing,
since that's the only what you can get something comparable between
bots. Think of this as a RELATIVE metric. What I found was that the
settings that returned higher Kelly numbers also gave straighter
equity curves. Since I am an extremely risk aversive trader, I like
straight equity curves. And this make perfect sense. The strategies
that are safer for compounding are the ones with straighter equtiy
curves. I still suspect Mike Correlation and
As regards the secondary issue of position sizing, if you already use
a modest fixed % risk then you and I are doing pretty much the same
thing as far as position sizing goes. The difference is that I am
simply applying a conservative fixed % position sizing algorithm to
strategies that have been screened for high Kelly value. FRACTIONAL
KELLY is a completely different animal from full Kelly position
sizing, since the volatility risk drops off much faster than the
profits as you use smaller fractions. If you are want to pursue this,
I'd recommend the books by Pound and Ziemba. That's all I'm going to
say about it, since I think it would be a mistake for this forum to
get sidetracked into that discussion. There be dragons there .
Mike, that's wonderful news about the timing of the next upgrade!
Rick, do you really think Mike hasn't tested these ideas? Chill dude,
traderjohn
even though I like the Kelly metric for its straightness, I suspect
Mike Correlation and Significance metrics will do better in that
department. But as Mike has pointed out, you could get a very
straight but low profit result from these metrics also, so you have to
add some profitabilty metric. What I'm excited about is the
possibility of using the Kelly metric for that purpose instead of max
profit or such.
Rick
"Rick, do you really think Mike hasn't tested these ideas? Chill
dude"
other places. Everything is under control here.
I sent an email yesterday to MIchael Harris (PAL) telling him that I'm
interested in his program but I would like to have a Kelly
maximization option. He said he would post something in his blog. His
response is at this link:
http://www.priceactionlab.com/Blog/2012/03/maximizing-kelly-is-equivalent-to-maximizing-win-rate/
Michael R. Bryant
ratio" by maximizing the win rate. That basically makes my point. The
win/loss ratio is never a given, so optimizing the Kelly f value is not the
same as simply saying "I want the highest win rate for a given win/loss
ratio." The strategy that gives you the maximum Kelly f value is the one
defined by the trade-off between win/loss ratio and win rate as defined by
the Kelly f formula and constrained by the space of possible strategy
solutions available during the build process. It's true that the solution,
whatever it happens to be, will have the highest win rate for whatever
win/loss ratio it has, but the win/loss ratio is not given ahead of time.
Put another way, if you plot a three-dimensional plot of Kelly f on the z
axis versus win rate and win/loss ratio on the x and y axes, the surface
slopes upward along both x and y axes, with the absolute maximum at the
maximum values of both the x and y axes. However, only a few of the values
on that chart are possible given the constraints of the strategy logic in
combination with the market and how the strategy trades on that market. Only
one of those "feasible" values will be the highest point on the surface.
That's the optimum if you maximize Kelly f.
I don't want to over-emphasize any value there may be in including Kelly f
as part of the optimization process. My personal opinion is that the
bottom-line result may be no better than a simple weighted average of the
same metrics, which is what Builder currently offers. However, depending on
the metrics you choose for optimization, the math clearly shows that it's
different than optimizing a weighted average, which is the only point I was
making.
Mike Bryant
Mark Knecht
<m...@breakoutfutures.com> wrote:
<SNIP>
> That's the optimum if you maximize Kelly f.
>
Not that it matters but I agree with everything you pointed out. The
only thing I'd consider in addition is that (in my mind) this isn't
about _only_ maximizing an individual metric like Kelly. For instance
one might look maximize for profit with a higher weighting while
minimizing Kelly with a lower weighting, etc. simply to investigate
what the solution space looks like and not necessarily to even find a
strategy to trade. If lower Kelly ratios produce better OOS results
then in general we gain info about where to put targets.
Where I'm personally interested is in what the solution space looks
like when you allow us to target ranges of build metrics. What do the
solutions look like with a Kelly target between 10-15% vs the
solutions between 20-30%, etc., giving no preference to any value
within the range. I.e. - 10% is as valid as 12.5% is as valid as 15%.
Now, way down the road, if you were couple targeting of ranges with
the ability to do resets based on something like a percentage of the
OOS vs. in-sample reset metric (i.e - Kelly OOS was 16% while Kelly
in-sample was 18%, etc.) I suspect we might move toward solutions
that begin to address the title of this thread, but I have no data to
back that up.
And thanks much for the release target dates. I'll consider it an
early, on-time or even late birthday (4/5) present! ;-)
Cheers,
Mark
Rick
On Mar 12, 1:48 pm, "Michael R. Bryant" <m...@BreakoutFutures.com>
wrote:
> ratio" by maximizing the win rate. That basically makes my point. The
> win/loss ratio is never a given, so optimizing the Kelly f value is not the
> same as simply saying "I want the highest win rate for a given win/loss
> ratio."
optimize just for win rate W1 and that results in some R1 you get a
value of Kelly f1 but then when you optimize for both W and R, you get
W2 and R2 such that W2 < W1 and I assume then R2 > R1, and with f2 >
f1. Why choosing the strategy with the lowest win rate that maybe
gives a higher value for Kelly? Lower win rate means higher risk of
ruin, I believe. I would not trade off higher R for lower W. Higher R
is more risky.
Am I missing something here?
Michael R. Bryant
Kelly f is one way to make that choice. Whether that choice is best for you
is up to you. Other measures of fitness will give you different values
representing different points along the curve. You may want a different
point on the curve than someone else.
Mike Bryant
mandelmus
mandelmus
Michael R. Bryant
Let’s see… I’m updating Builder, working on a cure for cancer, solving global warming, and ending world hunger. … Oh wait, I had to drop those last three because I’ve been busy with Builder.
Mike Bryant
Subject: Re: Strategies quickly fade
May I ask, which other projects are you working on?
Rick
under control here.
I contacted Michael Harris yesterday and told him that I'm interested
in his PAL program but I'd like it to have a Kelly metric included in
the search. In his reply he told me that he would post a response in
his blog today. Here is what he posted FYI: http://tinyurl.com/6saxntz
mandelmus
Rick
On Mar 17, 4:39 am, mandelmus <gmb1...@gmail.com> wrote:
> I knew it!! I knew you were on to something amazing! Now, if you
> really could find the cure for cancer, I'd excuse you from working
> on Builder forever. :D
AB is just too high. Just take a look at this forum: it is virtually
dead. It doesn't look to me like the market for this type of software
is big enough to justify the opportunity risk. If Mike used his
knowledge in other fields, like in oil & gas exploration for example,
he would be making multiples in comparison I believe.
The majority of traders do not have the required background to use
properly software like AB and at the same time they are too lazy to
learn.
>
>
>
> On Tuesday, March 13, 2012 12:39:00 PM UTC-5, MikeBryant wrote:
> > Let’s see… I’m updating Builder, working on a cure for cancer, solving
> > global warming, and ending world hunger. … Oh wait, I had to drop those
> > last three because I’ve been busy with Builder.
>
> > Mike Bryant
>
>
> > May I ask, which other projects are you working on?- Hide quoted text -