A little frustrating
Bill
data from 9/1997 to 04/2012. Every time I change the weights in the
Build Metrics slightly I get a different system, often a completely
different one. I use No. of trades, pct win, prof fact, win/loss
ratio, max. DD as the metrics.
Can anyone here give me some guidance as to the best set of metrics to
use? I need to use a fixed set of metrics otherwise this process will
lead to more frustration and possibly nowhere. If getting a good
system depends on selecting the proper set of metrics then what we are
doing here is just moving from the level of chaotic manual system
testing to chaotic build metrics setting.
FYI, I have been developing trading systems for more than 10 years and
I have used Metastock, Amibroker, Metatrader, Ninjatrader, eSignal and
some other platforms and a few different data services.
Mark Knecht
> I downloaded the demo and I'm trying to build a system with ES daily
> data from 9/1997 to 04/2012. Every time I change the weights in the
> Build Metrics slightly I get a different system, often a completely
> different one. I use No. of trades, pct win, prof fact, win/loss
> ratio, max. DD as the metrics.
>
In fact Bill that's _exactly_ what should be happening. I often run
the exact same settings 5 or times to look at what Builder finds and
pretty much every time the output is different and sometimes _very_
different.
> Can anyone here give me some guidance as to the best set of metrics to
> use? I need to use a fixed set of metrics otherwise this process will
> lead to more frustration and possibly nowhere. If getting a good
> system depends on selecting the proper set of metrics then what we are
> doing here is just moving from the level of chaotic manual system
> testing to chaotic build metrics setting.
>
I'd suggest you try something like 1997-2005 or 2006 as in-sample,
then 2005 or 2006 to maybe 2009 out-of-sample but reset the population
if the OOS period doesn't make money. This would allow you to reserve
2009 to present to discover whether you found something that actually
worked. Remember, the way Builder does the OOS reset really means
there is a selection bias all the way to the end of the data it used
so 2009 to present would be the only truly OOS data.
> FYI, I have been developing trading systems for more than 10 years and
> I have used Metastock, Amibroker, Metatrader, Ninjatrader, eSignal and
> some other platforms and a few different data services.
>
Welcome.
Cheers,
Mark
Bill
On Apr 15, 5:49 pm, Mark Knecht <markkne...@gmail.com> wrote:
> > I downloaded the demo and I'm trying to build a system with ES daily
> > data from 9/1997 to 04/2012. Every time I change the weights in the
> > Build Metrics slightly I get a different system, often a completely
> > different one. I use No. of trades, pct win, prof fact, win/loss
> > ratio, max. DD as the metrics.
>
> In fact Bill that's _exactly_ what should be happening. I often run
> the exact same settings 5 or times to look at what Builder finds and
> pretty much every time the output is different and sometimes _very_
> different.
>
> > Can anyone here give me some guidance as to the best set of metrics to
> > use? I need to use a fixed set of metrics otherwise this process will
> > lead to more frustration and possibly nowhere. If getting a good
> > system depends on selecting the proper set of metrics then what we are
> > doing here is just moving from the level of chaotic manual system
> > testing to chaotic build metrics setting.
>
> I'd suggest you try something like 1997-2005 or 2006 as in-sample,
> then 2005 or 2006 to maybe 2009 out-of-sample but reset the population
> if the OOS period doesn't make money. This would allow you to reserve
> 2009 to present to discover whether you found something that actually
> worked. Remember, the way Builder does the OOS reset really means
> there is a selection bias all the way to the end of the data it used
> so 2009 to present would be the only truly OOS data.
metrics that work best. Slight variations in either the weights of the
metrics or in the metric set result in wide variations in the build
output. This is the definition of a chaotic system.
Mark Knecht
OK, so that's as hit & miss as anything, but I'll tell you there is no
'best set' TTBOMK when it comes to doing anything in Builder. So much
depends on whether you're making a swing trading system holding for
weeks or a scalping system holding for minutes. It's very hard to even
suggest what you would consider 'best' because I don't know your
tolerance for holding through a drawdown vs tight stops, etc.
You might ask yourself the question what you would consider
'reasonable' results for your in-period time. You might then target
that number for the total returns against some number of trades based
on how much you think you want to trade, and maybe target linearity
with a high correlation coefficient. I'd suggest fewer build metrics,
not more.
In my experience a good way to start is just playing and look at the
results your getting. Sometime very small changes end up giving you
very different results. That's the nature of this sort of genetic tool
I think.
I'm somewhat under the weather today so I don't have the energy to go
download data but sometime next week maybe I'll try to put something
together and see what I come up with. I've done ES daily bar systems
before. They were relatively easy to get something to at least
consider but I've never traded one myself.
HTH,
Mark
Bill
On Apr 15, 6:44 pm, Mark Knecht <markkne...@gmail.com> wrote:
during the weekend but nothing useful came out. Virtually all good in-
sample results fail out-of-sample and some even ruin the initial
account. I will spend a couple more days testing with TY data. If I
don't get anything that looks good to me I will give up. Process
appears too chaotic to me.
>
> HTH,
> Mark- Hide quoted text -
>
> - Show quoted text -
Mark Knecht
<SNIP>
>> I'm somewhat under the weather today so I don't have the energy to go
>> download data but sometime next week maybe I'll try to put something
>> together and see what I come up with. I've done ES daily bar systems
>> before. They were relatively easy to get something to at least
>> consider but I've never traded one myself.
>
> I would really appreciate that Mark. I tried hundreds of combinations
> during the weekend but nothing useful came out. Virtually all good in-
> sample results fail out-of-sample and some even ruin the initial
> account. I will spend a couple more days testing with TY data. If I
> don't get anything that looks good to me I will give up. Process
> appears too chaotic to me.
>
Dunno Bill. It took me about 15 minutes to come up with at least one
reasonably interesting result for @ES.D from 1997-2012. It made some
money, is pretty linear, and the OOS results are reasonably similar to
the IS results.
One problem with these Google Forums, unlike Yahoo Forums, is there's
no place to store files for sharing with other members. I'll forward
you the gpstrat file off list. Maybe Mike can look into some sort of
member based storage solution so that I could post the gpstrat file
and anyone who cared to look at it could grab a copy down the road.
Respectfully, I ask others to not request this gpstrat from me
directly as I cannot respond to every list member independently. Also,
as has been happening lately, I'd additionally like people to _not_
ask me for data files off list. The TradeStation data agreements do
not allow me to give away copies of their data. Thanks in advance.
Cheers,
Mark
Bill
On Apr 16, 1:40 pm, Mark Knecht <markkne...@gmail.com> wrote:
>
> <SNIP>
>
> >> I'm somewhat under the weather today so I don't have the energy to go
> >> download data but sometime next week maybe I'll try to put something
> >> together and see what I come up with. I've done ES daily bar systems
> >> before. They were relatively easy to get something to at least
> >> consider but I've never traded one myself.
>
> > I would really appreciate that Mark. I tried hundreds of combinations
> > during the weekend but nothing useful came out. Virtually all good in-
> > sample results fail out-of-sample and some even ruin the initial
> > account. I will spend a couple more days testing with TY data. If I
> > don't get anything that looks good to me I will give up. Process
> > appears too chaotic to me.
>
> Dunno Bill. It took me about 15 minutes to come up with at least one
> reasonably interesting result for @ES.D from 1997-2012. It made some
> money, is pretty linear, and the OOS results are reasonably similar to
> the IS results.
project you have specified to save the best 100 strategies, the
screenshot of the equity you included in your email was #343. If one
generates so many strategies, then the probability of one or several
testing positive in OOS is very high. Regardless, I have a couple more
important observations to make about the results:
Observation 1: When I run AB using in-sample of 85% of original data
without including the OOS, AB generates too many trades, about 4,500
of them for highest net profit and 1,604 for highest profit factor. I
checked the EL code and it shows the long entry condition always set
to True.
Observation 2: When I run AB on the combined data file after
specifying the IS and OOS proportions of 85%/15% in the program, I get
only 405 trades in the IS for highest net profit with negative
performance in OOS. This is very strange. I ran the project many times
and I got the same result. This made me suspicious that maybe AB is
looking at the OOS when selecting systems from the IS results. I
cannot imagine this is what is happening but the reduction in the
number of trades by a factor of 10 when OOS is included is very
peculiar to say the least.
Mark Knecht
>
>
> On Apr 16, 1:40 pm, Mark Knecht <markkne...@gmail.com> wrote:
>> On Mon, Apr 16, 2012 at 3:30 AM, Bill <billcheno...@gmail.com> wrote:
>>
>> <SNIP>
>>
>> >> I'm somewhat under the weather today so I don't have the energy to go
>> >> download data but sometime next week maybe I'll try to put something
>> >> together and see what I come up with. I've done ES daily bar systems
>> >> before. They were relatively easy to get something to at least
>> >> consider but I've never traded one myself.
>>
>> > I would really appreciate that Mark. I tried hundreds of combinations
>> > during the weekend but nothing useful came out. Virtually all good in-
>> > sample results fail out-of-sample and some even ruin the initial
>> > account. I will spend a couple more days testing with TY data. If I
>> > don't get anything that looks good to me I will give up. Process
>> > appears too chaotic to me.
>>
>> Dunno Bill. It took me about 15 minutes to come up with at least one
>> reasonably interesting result for @ES.D from 1997-2012. It made some
>> money, is pretty linear, and the OOS results are reasonably similar to
>> the IS results.
>
> Thank you for the project file. I noticed that although in your
> project you have specified to save the best 100 strategies, the
> screenshot of the equity you included in your email was #343. If one
> generates so many strategies, then the probability of one or several
> testing positive in OOS is very high.
OK, there's something about Builder and the way Mike Bryant
architected it that you'll just have to get your head around. It's not
what I might have done if I was a programmer, but then again, I'm not.
(Well, I program LOTS of EasyLanguage, but not C++) Anyway, the ONLY
strategy that Builder cares about is the ONE strategy that has the
highest fitness value. From generation to generation, as I understand
the program, it's really only working to get higher fitness values.
Creating a large population does statistically give you more chance of
getting something that work OOS, but isn't that what this is all
about?
> Regardless, I have a couple more
> important observations to make about the results:
>
> Observation 1: When I run AB using in-sample of 85% of original data
> without including the OOS, AB generates too many trades, about 4,500
> of them for highest net profit and 1,604 for highest profit factor. I
> checked the EL code and it shows the long entry condition always set
> to True.
>
Then play with your build metrics and entry/exit types. Are you
allowing it to exit end-of-day? There were only 3500 or so bars. (15
years @ 250 days/years?) I suspect you are looking for something
that's more of a swing system but Builder is creating something that's
exiting every night?
> Observation 2: When I run AB on the combined data file after
> specifying the IS and OOS proportions of 85%/15% in the program, I get
> only 405 trades in the IS for highest net profit with negative
> performance in OOS. This is very strange. I ran the project many times
> and I got the same result. This made me suspicious that maybe AB is
> looking at the OOS when selecting systems from the IS results. I
> cannot imagine this is what is happening but the reduction in the
> number of trades by a factor of 10 when OOS is included is very
> peculiar to say the least.
>
Don't know. There isn't enough info there for me to take any real action.
Note that Builder isn't some solution that makes you rich over night.
It's just a tool that generates code. My experience has been
reasonably positive however honestly I have never traded a Builder
generated system live. The biggest problem I have isn't getting
systems that appear to make money inside of Builder but rather getting
the result in Builder and TradeStation to match.
Also, and I've found this reasonably important, if you are trying to
exit end-of-day then what Builder does is use SetExitOnClose which
actually makes the trade at the open the next day in real life, but I
believe it uses today's closing price to make its profit calculations.
This can cause major differences.
One other thing you might try: 3500-4000 bars is a bit small from my
experience. You're essentially trying to find a single system that has
performed flawlessly for 15 years but with only 250 bars/year. That's
very restrictive. You might try building systems for more consistent
markets - a bull model, a bear model - and then try running those
models when you suspect the market is actually trending up or down,
etc.
Good luck,
Mark
MikeBryant
On Apr 16, 4:02 pm, Mark Knecht <markkne...@gmail.com> wrote:
>
> > Thank you for the project file. I noticed that although in your
> > project you have specified to save the best 100 strategies, the
> > screenshot of the equity you included in your email was #343. If one
> > generates so many strategies, then the probability of one or several
> > testing positive in OOS is very high.
>
to see how many were profitable.
> OK, there's something about Builder and the way Mike Bryant
> architected it that you'll just have to get your head around. It's not
> what I might have done if I was a programmer, but then again, I'm not.
> (Well, I program LOTS of EasyLanguage, but not C++) Anyway, the ONLY
> strategy that Builder cares about is the ONE strategy that has the
> highest fitness value. From generation to generation, as I understand
> the program, it's really only working to get higher fitness values.
> Creating a large population does statistically give you more chance of
> getting something that work OOS, but isn't that what this is all
> about?
>
is to develop a profitable trading strategy, and it's based on the
fitness, so the results are always ranked by fitness. It doesn't limit
you to the fittest population member, and you can save all members of
the population if you wish. Every member of the population is used in
each generation. The only time it uses only the fittest member is to
trigger a complete rebuild of the population based on OOS results,
which, of course, is optional.
> > Regardless, I have a couple more
> > important observations to make about the results:
>
> > Observation 1: When I run AB using in-sample of 85% of original data
> > without including the OOS, AB generates too many trades, about 4,500
> > of them for highest net profit and 1,604 for highest profit factor. I
> > checked the EL code and it shows the long entry condition always set
> > to True.
>
> Then play with your build metrics and entry/exit types. Are you
> allowing it to exit end-of-day? There were only 3500 or so bars. (15
> years @ 250 days/years?) I suspect you are looking for something
> that's more of a swing system but Builder is creating something that's
> exiting every night?
>
> > Observation 2: When I run AB on the combined data file after
> > specifying the IS and OOS proportions of 85%/15% in the program, I get
> > only 405 trades in the IS for highest net profit with negative
> > performance in OOS. This is very strange. I ran the project many times
> > and I got the same result. This made me suspicious that maybe AB is
> > looking at the OOS when selecting systems from the IS results. I
> > cannot imagine this is what is happening but the reduction in the
> > number of trades by a factor of 10 when OOS is included is very
> > peculiar to say the least.
>
except to decide whether to start over from scratch if you select the
"rebuild on OOS performance" option.
> Don't know. There isn't enough info there for me to take any real action.
>
> Note that Builder isn't some solution that makes you rich over night.
> It's just a tool that generates code. My experience has been
> reasonably positive however honestly I have never traded a Builder
> generated system live. The biggest problem I have isn't getting
> systems that appear to make money inside of Builder but rather getting
> the result in Builder and TradeStation to match.
>
Builder live, but others have done so successfully, as they have said
on this forum.
If you're still having trouble getting your Builder results to match
those in TS, that's news to me, Mark. I'm always eager to see any
examples of that so I can hunt down any possible source of
discrepancy.
> Also, and I've found this reasonably important, if you are trying to
> exit end-of-day then what Builder does is use SetExitOnClose which
> actually makes the trade at the open the next day in real life, but I
> believe it uses today's closing price to make its profit calculations.
> This can cause major differences.
>
a good way to fit a strategy to the market noise, rather than signal.
There simply isn't enough information on one bar of data (o, h, l, c
is all you get) to make such a trade anything but random. Also, if
your stops are too tight, there can be ambiquity over whether the stop
is hit. If you want to develop a day trading strategy, you should use
intraday data. Otherwise, you should remove the exit end-of-day exit
type.
Mike Bryant
Adaptrade Software
Mark Knecht
>
>
> On Apr 16, 4:02 pm, Mark Knecht <markkne...@gmail.com> wrote:
>
>> OK, there's something about Builder and the way Mike Bryant
>> architected it that you'll just have to get your head around. It's not
>> what I might have done if I was a programmer, but then again, I'm not.
>> (Well, I program LOTS of EasyLanguage, but not C++) Anyway, the ONLY
>> strategy that Builder cares about is the ONE strategy that has the
>> highest fitness value. From generation to generation, as I understand
>> the program, it's really only working to get higher fitness values.
>> Creating a large population does statistically give you more chance of
>> getting something that work OOS, but isn't that what this is all
>> about?
>>
>
> I really don't know what you're referring to here, Mark. The objective
> is to develop a profitable trading strategy, and it's based on the
> fitness, so the results are always ranked by fitness. It doesn't limit
> you to the fittest population member, and you can save all members of
> the population if you wish. Every member of the population is used in
> each generation. The only time it uses only the fittest member is to
> trigger a complete rebuild of the population based on OOS results,
> which, of course, is optional.
>
OK, first let me extend my apologies. The comments above weren't meant
to be a dig against you or Builder in any way but I think it sounded a
bit stronger than maybe I really meant and it certainly wasn't meant
negatively. I'm sorry if it came off that way.
Again, my apologies.
With best regards,
Mark
Michael R. Bryant
Mike Bryant
Bill
On Apr 16, 7:02 pm, Mark Knecht <markkne...@gmail.com> wrote:
When I run on the IS only, I get something like 4,500 trades. There
are 4,012 bars in the file. I just run the build again and got 3,348
trades for highest net profit. I have not even included commission and
slippage.
Update: I just added r/t commission of $3. Now I get only 94 trades,
this is a completely different system, 65% profitable with profit
factor 4.7. That is too few trades. Just to clarify things the IS
covers years 1993 to end of 2008 and OOS starts there to 03/30/2012.
The OOS test produced $322 of profit with 19 trades only.
Next I used the combined IN+OOS. The opposite happened this time
around: the number of trades increased in the combined samples to
1,669 and the OOS part had negative performance. Again, this was with
commission of $3.
Then I just ran the build without changing anything, I got a
completely different system with 1,312 trades in the IS and with
negative performance in the OOS. This system has both long and short
entry conditions set to true and plays with the exits only.
My conclusion: this program generates strategies but with very high
unpredictability, which depends on the values of several parameters
like population size, number of generation, commission rate, whether
you use the IS only and then the OOS or both together, you name it. I
give up for now. I may come back to it at some point. Thanks a lot for
the help. I appreciated your willingness to help me.
Lawrence Lewis
experimenting with any substantial piece of software to get "good"
results. OTOH, it's very unlikely that software which provides easy,
fast results will actually work in the long run. That said, you have
to pick your battles, and decide where you going to invest your time.
Unfortunately, there's no guarantee that any particular piece of
software will be useful to you, and there's always the possibility it
won't and the time you invested is gone forever. That's life.
With respect to AB, one of the things I notice is that the way I
interpret you build parameters, is they are inherently "unstable".
When you only apply a $3 commission to trades, where the bid/ask
spread is itself $12.50, you are clearly searching a huge space of
trading systems where most of the systems are in the noise. IMHO, the
goal is to find a trading system which takes advantage of some real,
non-random underlying process. Because there are an infinite number of
systems, the first goal must be to weed out as many as you can that
are not profitable, except for chance, before you start your search,
or in the case of AB, to limit the search significantly. Only then can
you have any hope that a random selection process has a shot of
finding something real. For instance, I have had some success using a
commission of $25.
Bill
On Apr 17, 9:11 am, Lawrence Lewis <l...@usa.com> wrote:
> In my experience (which is considerable), it takes quite a long time
> experimenting with any substantial piece of software to get "good"
> results. OTOH, it's very unlikely that software which provides easy,
> fast results will actually work in the long run. That said, you have
> to pick your battles, and decide where you going to invest your time.
> Unfortunately, there's no guarantee that any particular piece of
> software will be useful to you, and there's always the possibility it
> won't and the time you invested is gone forever. That's life.
>
> With respect to AB, one of the things I notice is that the way I
> interpret you build parameters, is they are inherently "unstable".
> When you only apply a $3 commission to trades, where the bid/ask
> spread is itself $12.50, you are clearly searching a huge space of
> trading systems where most of the systems are in the noise. IMHO, the
> goal is to find a trading system which takes advantage of some real,
> non-random underlying process. Because there are an infinite number of
> systems, the first goal must be to weed out as many as you can that
> are not profitable, except for chance, before you start your search,
> or in the case of AB, to limit the search significantly. Only then can
> you have any hope that a random selection process has a shot of
> finding something real. For instance, I have had some success using a
> commission of $25.
commission rate. Here is what I got for maximum net profit:
Build on IS only: 15 long trades, win rate = 80%. Manual OOS produces
no trades.
Build on IS+OSS: 364 trades in IS, win rate = 76.65%, failed in OOS.
The troubling issue here is the significant increase in trades in the
IS when the OOS is added. This cannot mean anything else but that AB
takes into consideration the OOS at some point. How could the huge
difference arise otherwise? This is what troubles me most and I have
seen it occurring in the hundreds of tests I run in the last three
days.
Please keep in mind that I am using a project file Mark provided with
dally ES data from 11/1993 to 3/2012.
Mark Knecht
<m...@breakoutfutures.com> wrote:
> Not necessary, but thanks.
>
> Mike Bryant
>
Nahh, it really was and is required. I've been in bed sick for a week
and really shouldn't even have been posting the last couple of dayst.
In rereading my post I don't like the tone of it and feel that I owe
you at least that much if not more.
I'll be back in a week or two when hopefully I feel more healthy.
Until then, over and out.
Cheers,
Mark