Re: Strategy Check Article (Chime in please)

[mandelmus]

Read the first 100 or so posts by Acrary.  He analyzed, via Monte Carlo analysis, all kinds of different metrics and convinced himself that ... "From this past test we can see that if we kept the profit factor the same but changed the win % and expectancy, we'd have the same confidence level as we started with 80%. From this we can tell the win % and expectancy are not critical to consistency.  One of the keys that is important is the expected profit factor. The higher the profit factor, the more likely we are to achieve consistent profitability."

I have not tested his theories myself, yet.

[mandelmus]

Oh, Market System Analyzer (MSA) will also help you analyze metrics via it's "Parameter Studies" function using both the "Current sequence of trades" and the "Monte Carlo analysis".  Perhaps Mike will add some more metrics to the "Parameter Studies" function.

[Lawrence Lewis]

I have to say that I am not a great fan of Monte Carlo analysis when
used in this way to evaluate trading systems. To be useful, the
possible alternatives generated by the Monte Carlo procedure must have
some possibility of occurring in "nature". That is, they must actually
exist in the distribution you are randomly sampling from, the
distribution generated by the true underlying process plus noise.

In the case of trading systems, I highly doubt that any random
sequence of trades, as assumed by this Monte Carlo procedure is
actually possible. Could you truly have the data generated by the same
underlying process plus different noise generate all the losing trades
first followed by all the winning trades? Would it be reasonable to
assume the same underlying process could generate the trades in
reverse?

I think a much more fruitful analysis, which takes into account the
actually thing happening here is to look at what happens when a little
noise is added to the data. Do 1000 runs with different noise in the
input data.

[DaveA]

On Jun 24, 4:02 pm, Hammer <hugh2...@gmail.com> wrote:
> Here is what I found out in this article, It states in short.... you should
> take* (Profit Factor) x( Avr Win Loss Ratio) x (Percentage of winning
> trades) =  anything above 1.2 to trade strategy so what do you think about
> that Mike or anyone?*

What is the justification for the formula, empirical or mathematical?
>
>  Here is the use of a statistic I call the Cardinal profitability construct
> *(CPC) index*. It is my cardinal rule. Without this I could not tell how
> one system would compare to another. To figure a system’s CPC index, you
> read three performance statistics from the system performance report:
> profit factor, percent profitable, and ratio of the average win to average
> loss. You multiply these three numbers together and come up with a single
> one that you use to evaluate all systems. If this CPC index is less than
> 1.2, you don’t trade the system.
>
> In testing systems recently, we found several systems that looked good at
> first glance. The net profit was large, the equity curve looked good, and
> the average trade was even pretty large. But the CPC index came out to
> numbers fewer than 1.0. In fact, many of them were in the 0.2–0.5 range.
> The problem was in the ratio of the average win to average loss.

CPC equal to 0.2 requires a win rate of 80% to break even. In that
case the profit factor is 1 and the average win to average loss ratio
is 0.25, or the average loser is 4 times the average winner. For a
profit factor equal to 2, which is a a reasonable target, the required
win rate for a CPC equal to 0.2 is 95% and the ratio of average win to
average loss is equal to 0.1, or the average loser is 10 times the
average winner!!!!!!!!!!!!!!!!!!!!!!!!

So it seems to me that those systems that "looked good at first
glance" did so because you guys did not pay attention to the
unrealistic parameters but you instead looked at net profit and equity
curves, something that newbies do. Sorry to say this, no personal
offense intended. Trading system development takes years to master.

>
> In each failing case, the ratio was small. This means the wins might have
> been $1,000, but the losses were also approaching $1,000. To have a good
> system, you want this ratio to approach 2.0, or even exceed it.
>
>  Analyst John Bollinger said that good systems need the ratio and the
> profit factor to be 2.0 or greater. I don’t think it needs to be that
> strict, which is why I came up with my CPC index. It allows one of the
> result to be overall profitable.* //// So does anyone have a better
> parameter check list to validate strategies////*

[Howard B]

Hi Lawrence, and all --

Monte Carlo analysis is a valid technique.  Michael's articles describe the process and its justification very well, and his software implements the process correctly.  Trades that actually occurred are excellent sources of data from which to draw the Monte Carlo sequences.  Using a sliding window of recent trades, together with Monte Carlo analysis, provide efficient and accurate tools for monitoring system health and determining position size. 

Unfortunately, the True underlying process is unknowable.  The best we can do is use recent real or paper trades and assume they are representative.  Lacking real trades, the next best source is trades produced by the system during out-of-sample validation runs made using the data sources that will be used during trading. 

There is some value in adding random noise to the price series fed into the trading system as part of the robustness analysis.  But the signal to noise ratio of financial data as processed by trading systems is already very low, and I do not recommend diluting it further than necessary.  In my opinion, there is no value in adding random noise to the trade results before generating the Monte Carlo sequences.

Regarding the metrics --

Every trader is faced with the same question every time the system generates a signal -- how much confidence do I have that the system is working, and that the signal will lead to a profitable trade.  Systems that have low percentages of winning trades produce longer runs of losing trades than systems with high percentages of winning trades.  One of the tests that can be used to determine system health is the percentage of the most recent trades that are profitable.  If a system has a 60% win rate, there will probably be between 9 and 15 winning trades out of the most recent 20.  If there are 8 or few winners, the system is probably broken and should be taken offline while it is being evaluated.  Compare with a system that has a 30% win rate.  That system could have as few as 3 winning trades in the most recent 20 and still be performing within its normal expectations. 

There is no doubt that expectancy must be positive in order to have long term profitability.  But expectancy alone is a poor metric.  It is very difficult for traders with small accounts to trade systems that have low win rate even though the ratio of amount won to amount lost is high.  Consistency is very important in tradability.  A system with a win rate of 65% and a gain to loss ratio of 1 to 1 is very easy to trade, and very easy to determine system health.  I recommend performing Monte Carlo analysis of several hypothetical trading systems, then evaluating which you will be confident trading from your account.   

Thanks for listening,
Howard

[Michael R. Bryant]

Thanks for commenting, Howard. I hope everyone appreciates your reasoned responses as much as I do. For those of you who don’t know Mr. Bandy, he’s authored several books on trading, including one sitting on my desk right now titled “Modeling Trading System Performance”, which I highly recommend (disclaimer: he discusses my MSA software in the book).

 

Mike Bryant

 

Subject: Re: Strategy Check Article (Chime in please)

[Hugh Hambruch]

Thanks Howard!

 

I was trying to find a weed out tool or technique when I posted this article.I know nothing about the Monte Carlo other than it was a nice car.

 

Has anyone compared the MSA to the Walk Forward Optimizer Analysis at Tradestation

[DaveA]

Howard B "Trades that actually occurred are excellent sources of data

from which to draw the Monte Carlo sequences"

Actually they may not be and they are not in the case of trend-
following systems where trade serial dependence plays an important
role. Actual trades are not a good source of data also in the case
that the distribution is not representative of the actual trade
distribution (ex. a lucky streak of winners). Obviously, actual trades
are not a good source of data when the distribution does not exist.

BTW, I would not call the process of sampling trades that have
actually occured MCS. For this you need the distribution of trades
which is unknown in general or even non-existent.

On Jun 25, 7:46 pm, "Michael R. Bryant" <m...@BreakoutFutures.com>
wrote:

> > "Parameter Studies" function.- Hide quoted text -
>
> - Show quoted text -

[Howard B]

Hi David, and all --

You make an excellent point with regard to serial correlation.  If the trades have serial correlation, then random selection with replacement, the technique often used to create trade sequences which are in turn used to create distributions of performance results, will break the correlation and result in inaccurate results.  My recommendation is to test for serial correlation first.  If it is present, recognition of the correlation can often be incorporated into the logic of the system, both improving results and removing the serial correlation.  If it is not possible to modify the logic, an alternative is to use daily equity changes rather than trades.  They may have less serial correlation than the trades.  If that does not help, consider units consisting of multiple trades or of calendar periods. 

If more accurate results are required, the effect of the correlation may be determined by creating test samples, similar except for serial correlation -- one without serial correlation, the other with the same amount of serial correlation as the trade data.  Performing Monte Carlo tests on the two samples and comparing the results will allow an estimate of the effect.

Yes, try to correct for serial correlation.  In the final analysis, even a flawed Monte Carlo simulation may provide insight into the characteristics of the system that are not obtainable by any other means. 

----

In my opinion, there is little difference between out-of-sample trades resulting from an automatic walk forward process, paper trades recorded from a system that has passed validation and is about to be put into live trading, and actual trades taken by a system.  Each of these sets of trades is a sample that represents an equally knowable or unknowable distribution.  And the Monte Carlo process is equally applicable to all of them.

Serial correlation aside, if a streak of winning trades is recorded, I consider that to be just one instance of a trade sequence.  If I am to draw nine cards in sequence from a small deck consisting of ace through nine, the sequence 123456789 will occur exactly as often as the sequence 813462597 or any other sequence.  If a sequence of winning trades did occur, that is simply evidence that it can occur and the distribution should contain a representation of it.  I should not discard it, or adjust my representation of the sample to account for it, merely because it is more memorable.   

Assuming a trading system has been properly developed and is being traded.  The trading signals it generates are based on the logic of the model recognizing profitable trading opportunities.  The system is recognizing the signal portion of the data among the noise of the data.  When a series of winning trades results, I interpret that to be a high degree of synchronization between the logic and the data.  That is what I was hoping for.  I do not penalize the system or reduce my position size in anticipation of a series of losing trades.  Rather, to use the gambling phrase, I "bet the run of the table" and take advantage of it.  Perhaps it is only one lucky random sequence; or perhaps the distribution of trade results has changed.  As trades occur, the best I can do is use a sliding window of recent results (perhaps biased by some prudent subjective judgment.) 

Thanks for listening,
Howard

[Michael R. Bryant]

I’d just add that even when serial correlation exists, if often just makes the Monte Carlo results more conservative. For example, if you use the MC analysis to estimate the worst-case peak-to-valley drawdown, the result will likely be larger if you don’t take any correlation into account. If, on the other hand, you group the trades into consecutive sets, the size of which depends on the measured correlation, then, unless the groups of trades are biased towards consecutive losses, the drawdown estimated by MC will probably be smaller than if the trades were not grouped.

 

Mike Bryant

 

Subject: Re: Strategy Check Article (Chime in please)

 

Hi David, and all --

[DaveA]

Hi ZigZag - The paper in tradingblox forum contains several
inaccuracies and fundamental flaws. To start with, what he is doing is
not MC simulation but just some arbitrary simulation. MC simulation
involves introducing randomness to the system input. You cannot claim
MC simulation by resampling the win rate and other output parameters.
More importantly, the example with the balls fixes the probability of
win by fixing the number of possible events in the probability space.
This is not true with trading systems. The win rate can vary
significantly depending on market conditions and this is why
introducing randomness as part of the input is necessary for a
complete MC simulation. The results without such randomness
underestimate significantly the risk of ruin and I disagree with Mike
that in the case of serial dependence the results will be
conservative. Actually, they will underestimate risk in many cases
significantly.

The conclusion is that no simulation of this kind can offer useful
information as future random components of the input are unknown.


On Jun 26, 8:59 pm, ZigZag <leoja...@yahoo.com> wrote:
> Monte Carlo (“MC”) technique has been around for about 60 years and is
> widely regarded as a robust and reliable simulation method that may be
> applied to many fields.  There are tons of free literature and research
> done on it readily available on the internet.
>
> More specifically, as applied to rules-based trading systems, I have seen
> MC sampling and “Bootstrapping” (“BS”) being used interchangeably as well
> as MC to mean random sampling from commonly known probability distributions
> and BS to mean random sampling from the population sample.  I have also
> read in trading blogs and sites people advocating sampling with replacement
> (which does not preserve the statistical characteristics – like, say,
> %winners and win:loss ratio - of the original sample) and without (which
> does preserve the statistical characteristics of the original sample) – any
> thoughts on this?  Maybe adding “some” variability to the metrics of the
> samples drawn would reflect future probable trades P&L distributions that
> are more likely to happen?  Also, most of the discussions on MC/BS that I
> have seen so far implicitly assume that: 1) observations are independent
> from each other and as such, a sample length of 1 observation is
> appropriate and 2) 10,000 or so re-samplings are enough for convergence to
> the “true” population distribution.  This interesting and publicly
> available paper *http://www.tradingblox.com/Files/MC_resampling_Nbars.pdf*<http://www.tradingblox.com/Files/MC_resampling_Nbars.pdf>(warning: it’s a 7.2Mb file) addresses some of these topics in a very
> pragmatic and interesting way.
>
> Even though there are several MC software packages commercially available
> (CrystalBall, @Risk, Sim, RiskAMP, etc), for my own system development and
> trading, I use Mike Bryant’s MSA software.

>
>
>
> On Tuesday, June 26, 2012 2:36:36 PM UTC-4, MikeBryant wrote:
>
> >  I’d just add that even when serial correlation exists, if often just
> > makes the Monte Carlo results more conservative. For example, if you use
> > the MC analysis to estimate the worst-case peak-to-valley drawdown, the
> > result will likely be larger if you don’t take any correlation into
> > account. If, on the other hand, you group the trades into consecutive sets,
> > the size of which depends on the measured correlation, then, unless the
> > groups of trades are biased towards consecutive losses, the drawdown
> > estimated by MC will probably be smaller than if the trades were not
> > grouped.
>
> > Mike Bryant
>

> > *Subject:* Re: Strategy Check Article (Chime in please)

>
> > Hi David, and all --
>
> > You make an excellent point with regard to serial correlation.  If the
> > trades have serial correlation, then random selection with replacement, the
> > technique often used to create trade sequences which are in turn used to
> > create distributions of performance results, will break the correlation and
> > result in inaccurate results.  My recommendation is to test for serial
> > correlation first.  If it is present, recognition of the correlation can
> > often be incorporated into the logic of the system, both improving results
> > and removing the serial correlation.  If it is not possible to modify the
> > logic, an alternative is to use daily equity changes rather than trades.
> > They may have less serial correlation than the trades.  If that does not
> > help, consider units consisting of multiple trades or of calendar periods.
>
> > If more accurate results are required, the effect of the correlation may
> > be determined by creating test samples, similar except for serial
> > correlation -- one without serial correlation, the other with the same
> > amount of serial correlation as the trade data.  Performing Monte Carlo
> > tests on the two samples and comparing the results will allow an estimate
> > of the effect.
>
> > Yes, try to correct for serial correlation.  In the final analysis, even a
> > flawed Monte Carlo simulation may provide insight into the characteristics
> > of the system that are not obtainable by any other means.
>

> > ----- Hide quoted text -

[DaveA]

ZigZag - time does not allow getting into details but re-sampling is
not MC simulation. Re-sampling tests the null hypothesis of the
returns were sampled from a distribution with zero mean. MCS tests the
null hypothesis that the ordering of trades was random and not the
outcome of an intelligent process. In order to accomplish this, one
must include the segments where the system was flat. In the absence of
such information, re-sampling can get grossly optimistic about the
system robustness.

These are just a few of the considerations. Also, asymmetry of risk/
reward impacts MCS in addition to serial dependence. My take on this:
you either spend a lot of time doing it properly or it would be truly
a waste of time.

On Jun 28, 8:31 pm, ZigZag <leoja...@yahoo.com> wrote:
> Dave
>
> Thanks for your post – always insightful and pointed.  For the rest of that
> piece, beyond the possibly (?) flawed balls example, the author conducts MC
> experiments as follows (an excerpt):
>
> “Now we examine the rate of convergence for an equity curve resampling
> problem.  Starting with the daily returns from a historical simulation of
> the EMA XO(300,50) system… …we resample the returns (in 1-bar segments) and
> construct 100 new equity curves.  The new equity curves have the same
> distribution of (1-bar) returns as the historical simulation, but the order
> of the returns have been randomly scrambled.”
>
> The MC procedure described above sounds very similar to the MC procedure
> implemented by Mike Bryant’s MSA, the way I understand it, and which also
> seems to be very similar to MC procedures widely accepted by the community.
>
> Do you have an opinion on the MC procedure as applied to the trading
> systems (not the one in the balls example) described in the TradingBlox
> paper?
>
> Also, how exactly would you implement a MC simulation applied to trading
> systems?  When you say “You cannot claim
> MC simulation by re-sampling the win rate and other output parameters.”, do
> you mean that one cannot run a MC simulation which uses, for instance, a
> series of trades profits/losses (an output in back-tests) as input?  How
> exactly would you introduce randomness to a MC simulation?  Would it be by
> using randomly re-sampling with replacement? Or by shocking price data by
> some random amount previously to submitting it to the trading rules at
> hand? Etc?...
>
> Thank you.
>
> ZigZag

> > > - Show quoted text -- Hide quoted text -