Price Action Lab (formerly APS)
mandelmus
but just found them again. I remember seeing this software years
ago. Like Builder, it also outputs code to TradeStation ... and
Metastock, TeleChart, NinjaTrader, Wealth-Lab, MetaTrader, Amibroker,
etc. It seems to be based on finding recurrent price patterns across
multiple securities. The price is between $7,795 and $9,995!
Ref:
http://www.priceactionlab.com/index.html
http://www.trade2win.com/boards/trading-software/107654-price-action-lab.html
Rick
info on their website it seems that this program does searches but
also scans and it has an indicator feature. They claim that:
"Price Action Lab™ does not employ genetic optimizers, genetic
programming or neural networks, methods that are known to produce
fitted and optimized trading systems that may survive out of sample
testing due to chance alone. Unlike some other programs that claim
automatic system generation, Price Action Lab™ does not search for
random pre-defined pattern formations and it does not perform
permutations or combinations of rules and indicators. No lagging
indicators, no curve-fitting, no optimization. The algorithm used for
the search, scan and p-indicator functions was designed carefully to
minimize data mining and data snooping bias."
Also they claim to find patterns that have been profitable in a number
of markets.
Do you think one could feed those patterns from PAL to AB and have it
find the best system configuration? I think that would be interesting
if it could be done.
Rick
On Feb 29, 3:59 am, mandelmus <gmb1...@gmail.com> wrote:
> I had long forgotten about Price Action Lab software (formerly APS),
> but just found them again. I remember seeing this software years
> ago. Like Builder, it also outputs code to TradeStation ... and
> Metastock, TeleChart, NinjaTrader, Wealth-Lab, MetaTrader, Amibroker,
> etc. It seems to be based on finding recurrent price patterns across
> multiple securities. The price is between $7,795 and $9,995!
is for a lifetime license with no limitations on number of
installations and executables loaded. If the results are good, I will
upgrade to that license by paying the difference and then run 16
copies on 4 quad core machines with the aim of identifying edges
across a large number of symbols.
By the way PAL is a true white-box. It also outputs the code of the
patterns in a text file and you can use those as inputs to an NN or GP
engine. Scroll down to v2.0 release here:
http://www.priceactionlab.com/Support/versions/versions.html
I think it is a great idea to use the output of PAL as an input to
another engine. I wonder if this could be done with AB.
>
> Ref:http://www.priceactionlab.com/index.htmlhttp://www.trade2win.com/boards/trading-software/107654-price-action-...
Lawrence Lewis
1) The question first: After a short review of PAL manual, it seems
that they are primarily looking for a single type of pattern: a
combination of comparisons of Open, High, Low, Close with prior values
of Open, High, Low, Close. I realize they also have some indicators as
filters, but that's the primary thing, right?
2) The comments
They claim:
"Price Action Lab™ does not employ genetic optimizers, genetic
programming or neural networks, methods that are known to produce
fitted and optimized trading systems that may survive out of sample
testing due to chance alone. Unlike some other programs that claim
automatic system generation, Price Action Lab™ does not search for
random pre-defined pattern formations and it does not perform
permutations or combinations of rules and indicators. No lagging
indicators, no curve-fitting, no optimization. The algorithm used for
the search, scan and p-indicator functions was designed carefully to
minimize data mining and data snooping bias."
BS. No curve fitting, no optimization? They how are you selecting the
good trading systems? Astrology? Direct word from God? They don't use
pre-defined pattern formations? It certainly seems that they do.
Restriction to comparison between four arbitrarily selected price/time
points (O, H, L, C) certainly seems like a predefined pattern
selection. They don't use genetic optimizers, neural networks, etc.,
because they "may" produce systems which don't perform out of sample.
Well, unlike PAL, there is a large amount of academic research,
conferences, and peer reviewed journals for these techniques, as well
as proven results in a variety of fields.
Am I missing something here? Of course, just because they have bad
marketing blurbs doesn't mean their technique is flawed. But, without
getting confirmation of my question, it does seem that their search
space is very restrictive. For instance, P(0) > P(-1) may not be
predictive, but P(0) > c * P(-1) may be. Yet I don't recall seeing
that simple addition to their patterns.
Rick
On Mar 20, 9:38 am, Lawrence Lewis <lel...@gmail.com> wrote:
> A couple of comments and a question:
>
> 1) The question first: After a short review of PAL manual, it seems
> that they are primarily looking for a single type of pattern: a
> combination of comparisons of Open, High, Low, Close with prior values
> of Open, High, Low, Close. I realize they also have some indicators as
> filters, but that's the primary thing, right?
>
> 2) The comments
>
> They claim:
>
> "Price Action Lab™ does not employ genetic optimizers, genetic
> programming or neural networks, methods that are known to produce
> fitted and optimized trading systems that may survive out of sample
> testing due to chance alone. Unlike some other programs that claim
> automatic system generation, Price Action Lab™ does not search for
> random pre-defined pattern formations and it does not perform
> permutations or combinations of rules and indicators. No lagging
> indicators, no curve-fitting, no optimization. The algorithm used for
> the search, scan and p-indicator functions was designed carefully to
> minimize data mining and data snooping bias."
>
> I have to say these claims are real red flags for me. Basically, IMHO,
> BS. No curve fitting, no optimization? They how are you selecting the
> good trading systems? Astrology? Direct word from God?
that this program finds system building-blocks, like indicators, but
they are price patterns. They are selected based on win rate, number
of trades, consecutive losers, etc. I mean the philosophy is different
from that of AB. This program tries to find repeatable patterns.
There is OOS testing capability. It is up to the user to properly
combine those patterns into a system.
> They don't use
> pre-defined pattern formations? It certainly seems that they do.
> Restriction to comparison between four arbitrarily selected price/time
> points (O, H, L, C) certainly seems like a predefined pattern
> selection. They don't use genetic optimizers, neural networks, etc.,
> because they "may" produce systems which don't perform out of sample.
> Well, unlike PAL, there is a large amount of academic research,
> conferences, and peer reviewed journals for these techniques, as well
> as proven results in a variety of fields.
>
> Am I missing something here? Of course, just because they have bad
> marketing blurbs doesn't mean their technique is flawed. But, without
> getting confirmation of my question, it does seem that their search
> space is very restrictive. For instance, P(0) > P(-1) may not be
> predictive, but P(0) > c * P(-1) may be. Yet I don't recall seeing
> that simple addition to their patterns.
provide a list but I may have missed it. You are correct anyway. The
search space is restricted but they claim they do that to avoid what
they call TYPE-I curve-fitting: http://tinyurl.com/7zg8f3f
I see this program as a preprocessor for AB or TSL, or maybe my own
custom GP engine.
Mark Knecht
>
>
> On Mar 20, 9:38 am, Lawrence Lewis <lel...@gmail.com> wrote:
>> They claim:
>>
>> "Price Action Lab™ does not employ genetic optimizers, genetic
>> programming or neural networks, methods that are known to produce
>> fitted and optimized trading systems that may survive out of sample
>> testing due to chance alone. Unlike some other programs that claim
>> automatic system generation, Price Action Lab™ does not search for
>> random pre-defined pattern formations and it does not perform
>> permutations or combinations of rules and indicators. No lagging
>> indicators, no curve-fitting, no optimization. The algorithm used for
>> the search, scan and p-indicator functions was designed carefully to
>> minimize data mining and data snooping bias."
>>
>> I have to say these claims are real red flags for me. Basically, IMHO,
>> BS. No curve fitting, no optimization? They how are you selecting the
>> good trading systems? Astrology? Direct word from God?
>
> Funny you said that because I was also curious about it. It appears
> that this program finds system building-blocks, like indicators, but
> they are price patterns. They are selected based on win rate, number
> of trades, consecutive losers, etc. I mean the philosophy is different
> from that of AB. This program tries to find repeatable patterns.
> There is OOS testing capability. It is up to the user to properly
> combine those patterns into a system.
>
There is an option in Builder's Indicators section to enable or
disable price pattern detection. Has anyone played with only using
that indicator and no others to determine what patterns Builder looks
for? Possibly it more a matter of determining what patterns we'd like
Mike to add in the future and not so much a start difference between
Builder and some other program?
Or is it that the people looking at this other program believes that
program generates & mines patterns specific to that data set
specifically? I'm not clear about that.
This whole discussion leads me toward a point/comment/question about
all this stuff that sort of goes like this:
1) One argument technical analysis is thought to work at all is because:
a) Humans operate with a herd mentality
b) The herd isn't that sophisticated and is observing standard
indicators with standard settings.
c) If we watch the same indicators with the same standard settings we
have a chance to move with the herd
By optimizing indicators we introduce a problem. We might find better
values over the short term but that's curve fitting defined. Over the
long run the herd is watching standard indicators with standard
values.
QUESTION: Would a program searching for solutions to trade a market
(Builder or any other program) produce better LONG term results (i.e.
- OOS results that don't fade as per a recent thread) if it looked for
the best (or just a good) set of standard indicators without any
optimization at all, or at most very minimal optimization? I.e. -
maybe the @ES moves because that market trades with MACD while APPL
moves because that market trades with momentum.
NOTE: Please understand that I get I've grossly oversimplified that
argument just to make the question clear. The world isn't that black &
white, and we all know it, but I wanted the question to be both simple
and clear.
Thanks,
Mark
Mark Knecht
<SNIP>
> Thanks,
> Mark
Sorry. In rereading the previous posts I believe I had PAL's claims
backwards in terms of pre-defined price patterns. That means my
previous post was really off-topic. Sorry. (To my small credit, at
least I reread!) ;-)
Cheers,
Mark
traderjohn
want to explain why I don't think you need to worry much about whether
you characterized PAL correctly. For me that has become an
increasingly "off topic" subject. I always read your posts because
they're usually on-topic and quite insightful, even when posted in an
"off-topic" context. For me, and I suspect also for you, the main on-
topic categories are: 1) Adaptrade Builder per se, 2) the extensive
body of insightful Mike Bryant articles and code demonstrations out of
which Builder emerged, and 3) ocassional references to features in
other (exogenous) software that are of potential relevance to Builder.
The 3rd category has about as much weighting for me as "mutations" in
the default Builder settings, i.e. much lower than the other
settings. But it's important to include this category because and
mutations can SOMETIMES (i.e. rarely) be beneficial. In biology,
mutations are often fatal and/or cancerous, so the organism need to
protect itself from that kind of mutation. Reliance on the
reproductive process is a safer way of introducing change because it
(by definition) preserves the continuity of the organism and the
genetic line, whereas mutations have the potential to disrupt that
essential continuity. So, in the context of this discussion group,
how do we recognize a harmful mutant post or thread when we see one?
Well, off the top of my head I'd say one characteristic would be that
it keeps bringing the focus back, not to Builder, but to the exogenous
software itself (especially if it is so persistent in doing this that
it begins to raise suspicions that the poster's actual intent is to
promote a competing software product. I'm not saying this is
definitely the case here, I'm only reporting that it has begun to
raise suspicions. If you want to learn more about this category of
internet behavior, look up "concern trolling").
Now for the interesting part of your post:
> QUESTION: Would a program searching for solutions to trade a market
> (Builder or any other program) produce better LONG term results (i.e.
> - OOS results that don't fade as per a recent thread) if it looked for
> the best (or just a good) set of standard indicators without any
> optimization at all, or at most very minimal optimization? I.e. -
> maybe the @ES moves because that market trades with MACD while APPL
> moves because that market trades with momentum.
Mike has, in a way, dealt with in his article on "Trading Rules from
Statistical Grouping of Indicators" (Dec 2006 Newsletter,
breakoutfutures.com). He didn't answer your question directly but I
think he has given us a tool (StatBins1) for looking into it. I
haven't had the time to do much with it yet, but I think it could be
applied to your question (i.e. to characterizing markets according to
their affinity for certain indicator groups). So I'm just bringing
this to the attention the group as a potential "on-topic" subject for
our collective swarm intelligence.
By the way, it's always puzzling to me when new people join the forum
and put so much energy into exploring expensive exogenous software and
so little into exploring the very extensive, coherent, and progressive
set of FREE software tools that Mike has already given us. I've read
all the articles on both of his websites from start to finish and all
I can say is: It's been a real education. I agree with the comment
by mandelmus in another thread that this tends to be a very high
quality discussion group, but I think it could be much better if more
people did the homework.
Mark Knecht
<SNIP>
John,
Thanks for the positive personal comments. I do appreciate that my
posts might be helpful some at times. I come from about 15 years of
Linux Open Source where on-topic, off-topic, nuclear war-of-words,
threats and then return to on-topic is the norm. I understand that
sometimes threads go awry. We can all try to gently nudge them back at
times. Like you I do get frustrated at times with some conversations
but for the most part I just try to stay quiet unless I have an
opportunity to make a positive contribution.
Also, be assured that my interest here in this forum is really &
truly (1) the most effective use of Builder for myself and others
coupled with (2) an attempt to gently drive the future development in
directions that hopefully I and others will find useful and more
effective over time. As I stated sometime ago, I purchased my license
with an eye to the very long term and that hasn't changed. I've been a
customer of Mike's for MSA for a long time and while I don't know Mike
I am sure he is one the people I want on my side.
As for trolling, yeah, consider yourself welcome to join the wild
west world of Linux if you really wanna see that in effect. There are
many people that I suspect are simply paid by M$ to spread FUD and
nothing else. Sometimes it's hard to tell who the good guys are. (Not
really if you stick around and really study, but moment by moment it
can get confusing. :-) Also, note that while I don't write my full
name here it should be trivial for anyone interested in finding me to
get a private email to me at any time. (I ain't hard to find if some
hedge fund out there wants to hire me play with Builder or write
EasyLanguage for them. Lord knows I could use the diversion and a
little extra income!) ;-)
OK, all that said, thanks very much for the pointer to StatBins1.
I've spent a little time playing with it this morning and can see
exactly how it could address if not begin to answer at least the study
of the sort of question I raised about the use of standard (or
non-standard) indicators. I posted 3 different Performance Results
charts from TS here:
http://www.flickr.com/photos/29328985@N03/
Note that the one that's least impressive (actually ugly) was a quick
test where I set the Max/Min binning values to a larger range than
StatBins told me to. That really didn't work. The other two were just
examples of two date ranges for in-sample (1999-1/2002 & 1999-1/2004)
each of which produced at least reasonable results out-of-sample.
A few observations and one heads-up for anyone who plays with these:
1) As per instructions SfMax/Min are set to the in-sample maximums I
am unsure what the strategy does when OOS produces values outside of
those ranges. It may work fine, or it may not trade, or it might even
die. Don't know.
2) While it might cause the number of bins to jump quite a lot I think
that had I been insightful enough to have done this on my own (which I
wasn't so props to Mike for contributing it) I might have chosen
either 3 or 5 bins instead of 4. With a _hard_ divide at 0 I'm a
little troubled by the idea that for an oscillator 0.000000001 falls
into a different bin than -0.00000001. Consider possibly +/- 1 StdDev
as a center bin and then >1, <-1, >2, <-2 StdDev as another way to
approach the binning. Granted, if I get serious about that I (we?) can
do that so don't take this as anything more than an observation.
3) To me a next step that almost begs study is the effect of
walk-forward analysis on binning choices but that's for sometime in
the future I think.
HEADS-UP: If you or others run into an error message about trying to
reference outside the bounds of an array my experience so far says
I've got too much in-sample data, or more likely too many trades or
something, so I've filled the fixed array sizes and tried to go where
there ain't no storage. I haven't verified that or tried to see where
the code could be modified. Just documenting what I had to do to get
it to work.
Now, how might we use StatBins1 WRT Builder. It sort of begs the
rhetorical question 'How does Builder choose the indicators it's going
to use in the first place and how does Builder create and choose to do
crossovers & mutations?' While I wouldn't want Mike to ever expose his
secret sauce this whole process does make me wonder about taking
arbitrarily complex Builder strategies, breaking the entry and exit
conditions down into their component 'indicators' and then exposing
each of those indicators to something like StatBins1 for some study on
when they work & why they work. The goal being to understand why a
strategy is fading and what to do to keep it on track. Huge project
though...
Anyway, I much appreciate the pointer and thanks for allowing my to
wax technical...
Cheers,
Mark
traderjohn
as I’m concerned! I completely agree, it’s best to stay quiet until
you see an opportunity to make a positive contribution. That’s what
prompted my post to you about StatBins1, but I never expected it to
yield such a quick return Thanks for the encouraging proof of concept
work here. I realize this is just the beginning of what you probably
will do with this. If this were a moderated forum, this is probably
the place where the moderator would step in and move the StatBinsX
discussion to a new thread of its own. (Is there any way to do that?)
Anyway, I don’t have much to contribute right now – other than my
hearty thanks for the work you’ve done already -- but I hope you will
keep us informed of further developments. :)
I enjoy working with evolutionary concepts from the biological
(biomimetic) side, but I am much less capable on the programming side
than you, as I’ve only started learning about 4 years ago with
NinjaTrader and am now further disadvantaged by trying to switch
horses in midstream as I move from NinjaScript to EasyLanguage. It’s
not that EasyLanguage isn’t easier than Ninjascript (C#) but I don’t
dream in EasyLanguage yet. This is important because to get out of
the frequent coding conundrums I’ve found myself in, I have usually
relied on waking up dreaming about the solution. These days, it seems
I’m much smarter when I’m asleep! LOL. Since that hasn’t started
happening yet for me in EL (well actually PLA, as I am using
Multicharts), I’m really glad to make the acquaintance of a
programming wizard, and I hope maybe I can get some help from you if
I get really stuck at some point down the line! ; )
I trade forex, so the ES example is not as immediately relevant to me
as many others here, and I haven’t had much luck applying the AB
sample codes straight out of the box. But what I’ve thought I could
do with StatBinsX is to use the basic function to test a whole slew of
indicators to find out which ones get used for different types of
markets or what I like to call market niches, and maybe even, in an
adapted form, to discover what those niches are. So I think the basic
approach is probably a great research tool for a lot of interrelated
questions, and I have a lot of work cut out for me doing the work up
needed to apply AB effectively in trading forex pairs. The forex
market sometimes follows the equity markets but it seems to have a
very different rhythm overall, with three major sessions in a 24 hour
period. Finding the sweet spots in those markets is what I hope
eventually to use StatBinsX to sort out. It will be slow going for a
while, but once I've defined them I hope I’ll be able to stand back
and say “Release the Krachen!” and turn Builder loose on those
niches. I expect I'll have to do that periodically to keep everything
up to date.
Cheers,
John
On Mar 21, 1:00 pm, Mark Knecht <markkne...@gmail.com> wrote:
Mark Knecht
> Wow Mark, nice work! You can “wax technical” anytime you want as far
> as I’m concerned! I completely agree, it’s best to stay quiet until
> you see an opportunity to make a positive contribution. That’s what
> prompted my post to you about StatBins1, but I never expected it to
> yield such a quick return Thanks for the encouraging proof of concept
> work here. I realize this is just the beginning of what you probably
> will do with this. If this were a moderated forum, this is probably
> the place where the moderator would step in and move the StatBinsX
> discussion to a new thread of its own. (Is there any way to do that?)
> Anyway, I don’t have much to contribute right now – other than my
> hearty thanks for the work you’ve done already -- but I hope you will
> keep us informed of further developments. :)
>
No reason you can't start a new thread. However before you do, as this
is the 'Adaptrade Builder' group, let's see if we can't find a way to
more directly relate it to Builder so that it's potentially of
interest to more people.
I wanted to raise one (big) caveat before people go off chasing equity
curves in StatBins. READ THE CODE! UNDERSTAND IT! While the ES curves
were pretty unfortunately on the hours the strategy was in the
positions they don't conform with my personal trading requirements.
Nor should they as Mike wrote it for his own purposes. Anyone who
wants to go down the path of this sort of investigate should get
inside the code and modify it (or hire someone to modify it) to meet
their own trading requirements before spending too much time looking
at curves. At least that's the way I feel.
Cheers,
Mark
Rick
On Mar 21, 4:00 pm, Mark Knecht <markkne...@gmail.com> wrote:
only close to 350 trades in about 13.5 years or about 81,000 bars if
my estimates are correct. This doesn't look to me like a significant
sample but again that depends as you said on the logic of the system.
Do you know the average number of bars for the trades? What was the
win rate? Could you post the first equity curve with a time axis
intead of trade number? Thanks.
>
> Cheers,
> Mark
Mark Knecht
<SNIP>
>>
>> I wanted to raise one (big) caveat before people go off chasing equity
>> curves in StatBins. READ THE CODE! UNDERSTAND IT! While the ES curves
>> were pretty unfortunately on the hours the strategy was in the
>> positions they don't conform with my personal trading requirements.
>> Nor should they as Mike wrote it for his own purposes. Anyone who
>> wants to go down the path of this sort of investigate should get
>> inside the code and modify it (or hire someone to modify it) to meet
>> their own trading requirements before spending too much time looking
>> at curves. At least that's the way I feel.
>
> Mark, only the first equity curve appears acceptable but there are
> only close to 350 trades in about 13.5 years or about 81,000 bars if
> my estimates are correct. This doesn't look to me like a significant
> sample but again that depends as you said on the logic of the system.
> Do you know the average number of bars for the trades? What was the
> win rate? Could you post the first equity curve with a time axis
> intead of trade number? Thanks.
>
>
Hi Rick,
Again, with the caveat that this curve is just what StatBins does
by default and that none of the curves meet my personal needs, the
average number of bars was roughly 18.5 60 minute bars and was fairly
equal for both winning and losing trades. Keep in mind that these are
60 minute bars so we're talking about being in any trade for less than
1 day on average. This is mainly due (I believe) to the sample code
using SetExitOnClose for session 1 so no matter when you enter you're
out at 15:00. I don't think a trade could ever be longer than 24 bars.
And as my previous post stated, the code Mike provided is there for
anyone to look at. This wasn't a Builder system and wasn't optimized
in any way in terms of the 5 indicators it used or the settings of
those indicators. (I don't think anyway.) The indicators in the code
aren't rocket science type indicators. It was just an example that
Mike wrote to get people thinking.
Please keep in mind that beyond all the stuff above the real
question I was curious about was out-of-sample results fading over
time, as well as whether they roughly approximated in-sample results,
which they sort of did in the case of the curve that made
approximately $50K. I.e. - there were about 150 trades over the 5
years of in-sample data and approximately 200 trades over the 8 years
out of sample. If you viewed the equity curve by date (as you request)
instead of by trade number it remained reasonably linear.
I've posted that here:
http://www.flickr.com/photos/29328985@N03/7003938561/in/photostream
On more thing: Assuming you have access to 60 minute ES data this
process is _completely_ deterministic and you or any other reader
should be able to recreate the exact curve that I've posted.
So, at the risk of being overly clear: THIS AIN'T A TRADING SYSTEM.
IT'S JUST A STUDY. :-) ;-) However maybe it will lead to something
that is tradable. I've started rewriting the code to be of more
interest to my needs.
HTH,
Mark
Daniel Cai
Lawrence Lewis
http://www.priceactionlab.com/Literature/pal30manual/pal30manual.html#process41
As I said, I'm not sure if this is representative. I'll leave that to
others who have more time than I do for this.
I read the short blurb on curve fitting. I have to say, it makes no
real sense to me. He classifies trading system optimization methods
into 3 types: 1) Those where the optimization affects both the entry
and exit points, 2) Those where the optimization only affects the
entry points, and 3) those where the optimization only affects the
exit points. OK. So what.
My perspective is you want your optimization method to curve fit.
That's the whole point. If I have a bunch of observed data on the
trajectory of canon balls, I want my curve fitter to come up with x =
x0 + v0*t +.5*32*t*t. I want to find the underlying process. It's just
you want to fit the signal and not the noise. What you don't want is
overfiiting. There are a large number of techniques that attempt to
prevent overfitting when using things like neural networks or
evolutionary algorithms. The main ones use some form of in sample/out
of sample. As we have seen, however, if you hand select your out of
sample results, you invalidate the whole thing, because you are using
a selection method (your own brain) which looks into the future (the
out of sample period).
Rick
On Mar 20, 12:34 pm, Lawrence Lewis <l...@usa.com> wrote:
> I looked at some sample AmiBroker code:
>
> As I said, I'm not sure if this is representative. I'll leave that to
> others who have more time than I do for this.
and they match the PAL results. In the case of TS it generates the
full code, not just the conditions.
>
> I read the short blurb on curve fitting. I have to say, it makes no
> real sense to me. He classifies trading system optimization methods
> into 3 types: 1) Those where the optimization affects both the entry
> and exit points, 2) Those where the optimization only affects the
> entry points, and 3) those where the optimization only affects the
> exit points. OK. So what.
>
> My perspective is you want your optimization method to curve fit.
> That's the whole point. If I have a bunch of observed data on the
> trajectory of canon balls, I want my curve fitter to come up with x =
> x0 + v0*t +.5*32*t*t.
you know the initial conditions (velocity and position) and the system
equations you know the path of the system. If you have data from a
canon ball path then this is a good application for a GP or NN to
determine the equations of motion and the dynamics of the system.
The market is not a deterministic system. Given the same initial
conditions the path/dynamics may be different. Thus, if you fit one
path you will get equations that may not describe a new path. This
implies that curve-fitting is not only inappropriate but that it
should not be used at all. OOS does not solve this problem because the
OSS path is just another path where the fitted dynamics may work well
by chance. It introduces selection bias.
>I want to find the underlying process. It's just
> you want to fit the signal and not the noise. What you don't want is
> overfiiting. There are a large number of techniques that attempt to
> prevent overfitting when using things like neural networks or
> evolutionary algorithms. The main ones use some form of in sample/out
> of sample. As we have seen, however, if you hand select your out of
> sample results, you invalidate the whole thing, because you are using
> a selection method (your own brain) which looks into the future (the
> out of sample period).
introduced invalidates the process. IMO, what one should do instead is
to use a large set of randomly generated values for the system
parameters and the central limit theorem to determine the mean of the
distribution of returns. If the mean is not sufficiently far away from
zero then the probability of failure is high.
I think OOS is for determinig the validity of the process that
generates the systems and not for selecting the systems.
>
> - Show quoted text -
mandelmus
Rick
On Mar 23, 3:57 pm, mandelmus <gmb1...@gmail.com> wrote:
> Are you using this method? With any particular software?
consulting. I have applied it to hundreds of systems and all but two
failed the test when a typical range of parameter values was used. The
two systems that passed the test had four parameters, two for long and
two for short entries. They are both trend-following systems and do
not suit my style.
The conclusions I have reached are:
(1) Optimization leads to selection bias towards a set of parameters
that may have worked due to chance along a specific path.
(2) OOS is useless because of selection bias. It is rather a method
for validating a methodology of trading system development.
(3) If you need to specify parameter values (or calculate them) to
determine the timing of trades, then the system performance will most
likely depend on the path and the probability of ruin is very high (it
depends also on risk).
You can use excel and do a t-test.
>
>
>
> On Friday, March 23, 2012 12:25:05 PM UTC-5, Rick wrote:
> > I agree. This is essentially what happens. The selection bias that is
> > introduced invalidates the process. IMO, what one should do instead is
> > to use a large set of randomly generated values for the system
> > parameters and the central limit theorem to determine the mean of the
> > distribution of returns. If the mean is not sufficiently far away from
Lawrence Lewis
Regarding the use of NN/GP and "curve fitting" for non-deterministic
systems...]
I agree my simple example was deterministic, but people have used
successfully used NN and GP to determine things like the distribution
for non-deterministic systems as well.
Obviously, if a system is completely non-deterministic - a random
variable with a flat distribution, nothing works. But, if there is a
non-deterministic component that would allow you to assign non-equal
probabilities to the possible paths, then a NN or GP could
theoretically discover it and compute those probabilities. You could
then take action on the most likely path. In this case, the NN or GP
is curve-fitting the underlying probability distribution rather the an
underlying deterministic process.
However, one of the known problems with NN and GP is the difficulty in
computing confidence intervals, so you get predictions, but no
confidence interval for that prediction. From what I know, the way
people produce confidence intervals in this case is with cross
validation...which uses up lots of data.
Rick
On Mar 26, 2:48 pm, Lawrence Lewis <lel...@gmail.com> wrote:
> Rick -
>
> Regarding the use of NN/GP and "curve fitting" for non-deterministic
> systems...]
>
> I agree my simple example was deterministic, but people have used
> successfully used NN and GP to determine things like the distribution
> for non-deterministic systems as well.
>
> Obviously, if a system is completely non-deterministic - a random
> variable with a flat distribution, nothing works. But, if there is a
> non-deterministic component that would allow you to assign non-equal
> probabilities to the possible paths, then a NN or GP could
> theoretically discover it and compute those probabilities. You could
> then take action on the most likely path. In this case, the NN or GP
> is curve-fitting the underlying probability distribution rather the an
> underlying deterministic process.
"seen" by the GP or NN and it actually fits a normal to it. Even worse
is the case that the distribution does not exist at all and the fit is
random. In additon, some specific types of distributions cannot be
fitted at all with any significance. But I agree with your overview in
principle. It's a good one.
>
> However, one of the known problems with NN and GP is the difficulty in
> computing confidence intervals, so you get predictions, but no
> confidence interval for that prediction. From what I know, the way
> people produce confidence intervals in this case is with cross
> validation...which uses up lots of data.
used. This is the problem. Future data must be drawn from the same
distribution for the cross validation to make sense. If not -->
disaster hits.:)
