Re: Suggestion: Metric Sets, Metric Set Library
mandelmus
at this point, i export my strategies to excel and then look at which metrics were most correlated with the top out-of-sample strategies ... this takes a little time, but gave me some ideas to pursue
since, we are using builder to generate strategies, it might be nice if builder would do the metrics/OOS performance correlation (sensitivity) analysis for us ... perhaps, builder would review the list of strategies and see which metrics contributed the most and least to our successful strategies ... we could then, for example, add, or increase the weight of, metrics that contribute the most to increasing profit while removing, or lowering the weight, of non-correlated metrics. it would nice to know the optimum combination of metrics and weights that produced certain types of strategies on a particular market/symbol ... and, save those combinations for use on future builds
Michael R. Bryant
I think this suggestion is more applicable to the prior versions, which didn’t include build conditions. If your main focus is in choosing the build conditions (i.e., constraints), then the build objectives – metrics to minimize or maximize – can be kept pretty simple. If that’s the case, I’m not sure a sensitivity analysis would reveal much about the relationship between in-sample metrics and out-of-sample results. In fact, relying on any correlations you might find could be misleading since, as we all know, correlation does not imply causation.
The premise of the build process is that the out-of-sample results will tend to improve with successive generations (up to a point) if the build process is valid. In other words, provided the build metrics (objectives and conditions) reflect your intent, you shouldn’t need to analyze your own choices. I discussed this process in a prior newsletter article, and a future version of Builder will directly incorporate the ideas in that article about build, test, and validation periods and the relationship between the build and test periods in particular.
Mike Bryant
Subject: Re: Suggestion: Metric Sets, Metric Set Library
mandelmus
I discussed this process in a prior newsletter article, and a future version of Builder will directly incorporate the ideas in that article about build, test, and validation periods and the relationship between the build and test periods in particular.Mike Bryant
Was this "future version of Builder" the version with Monte Carlo or do you still have plans to release a version with a validation period?
Michael R. Bryant
My plans have not changed in that regard.
Mike Bryant
Subject: Re: Suggestion: Metric Sets, Metric Set Library
On Friday, March 1, 2013 4:41:02 PM UTC-6, MikeBryant wrote: