Reverse engineering a known system

[LGTrader]

I've been playing around a bit with the idea for reverse engineering a known system that Mike presented in one of his studies:

http://www.adaptrade.com/Newsletter/NL-ReverseEngr.htm

The basic idea for those that haven't read this paper is that given a reasonably clear TradeStation (or other platform) performance report one can direct Builder to attempt to duplicate those results. One typical result I got doing this process against the daily system Mike presents in his write-up above is shown here:

http://www.flickr.com/photos/29328985@N03/7039380231/in/photostream

I do not show OOS results as 1) I have nothing to compare and 2) I'm not that interested in trading swing systems at this time.

For kicks I then went looking on the web for day trading systems that had enough information to try the same reverse engineering ideas. One system I found was based on 5 minute @ES bars and had reasonably interesting in-sample results so I spent a Sunday afternoon seeing what I could come up with. Those reverse engineering results are shown here:

http://www.flickr.com/photos/29328985@N03/7039354315/in/photostream

Here I do show OOS results as I might be interested in trading the system if the results were good. They are 'interesting' as it turns out in that they make money, but they don't make so much money as to get me overly interested. Unfortunately the web site that sells this strategy doesn't show OOS results so again, no real comparison is possible.

Anyway, for those struggling with getting Builder to do something useful you might want to read the first link above and go through this exercise on your own. I find it at least thought provoking. Keep in mind that many systems use multiple data streams which Builder does not currently handle. (But might at some future date...)

One thought I have is the possibility of some sort of 'group think' experiment where as a group we look for systems on the web, as a group pick a system to work on, and then pool results as a learning exercise. Note that personally I don't start from a POV that trading needs to be a zero-sum game so my interests here might not align exactly with 'typical thinking'... ;-)

Cheers,
Mark

[Rick]

IMO Mike's logic is questionable when he claims that:

"Since I know that MiniMax has held up well out-of-sample, my approach
is to reverse engineer the strategy over the entire reported period,
without saving any data for out-of-sample (OOS) testing."

Actually what he is doing is curve-fitting to the MinMax system
metrics. Since the two systems are not structurally equivalent in
general, the OOS performance may not be the same.

IMO, the way of doing this correctly is to use 3/4 of the data sample
to reverse engineer the system and then the remaining 1/4 to see is
the OOS performance of the result is identical to the original system.

Otherwise, this is just curve-fitting and selection bias and I am
surprised that Mike thinks this is sound practice. It is not IMO.

[Michael R. Bryant]

There's always a trade-off between saving data for OOS testing and using
more data to increase the statistical significance. In this case, my
approach was to use as much data as possible for building the strategy in
order to get as close a correspondence as possible between the copy and
MiniMax. I felt there was particular value in the more recent data for
MiniMax because the results over that period are OOS. As I usually point out
in these types of articles, the final arbiter is always the real-time
tracking results post-optimization and post-OOS testing. You should always
do that kind of tracking in my opinion, regardless of how you come up with a
strategy.

The point you seem to be missing is that by using a reverse-engineering
approach, you're building against realistic targets, as opposed to just
trying to maximize or minimize the metrics. Of course I was curve-fitting
MiniMax; that's the whole point. Doing so is much, much less likely to
result in an over-fit strategy precisely because the targets and metrics are
based on good OOS results from a sound strategy.

Mike Bryant

[Rick]

On Apr 3, 2:32 pm, "Michael R. Bryant" <m...@BreakoutFutures.com>
wrote:

> There's always a trade-off between saving data for OOS testing and using
> more data to increase the statistical significance. In this case, my
> approach was to use as much data as possible for building the strategy in
> order to get as close a correspondence as possible between the copy and
> MiniMax. I felt there was particular value in the more recent data for
> MiniMax because the results over that period are OOS. As I usually point out
> in these types of articles, the final arbiter is always the real-time
> tracking results post-optimization and post-OOS testing. You should always
> do that kind of tracking in my opinion, regardless of how you come up with a
> strategy.
>
> The point you seem to be missing is that by using a reverse-engineering
> approach, you're building against realistic targets, as opposed to just
> trying to maximize or minimize the metrics. Of course I was curve-fitting
> MiniMax; that's the whole point. Doing so is much, much less likely to
> result in an over-fit strategy precisely because the targets and metrics are
> based on good OOS results from a sound strategy.

The issue is if the fit results in a system that has the same
distribution of returns with the original system, not just metrics. If
the distributions are not the same, performances will diverge in the
future. This is where OOS helps but just a little. It serves as a
validation of the method of fitting used wrt to the sample
distributions. Otherwise you might have just generated spurious
correlations that only matched performance during the in sample to
great accuracy.

>
> Mike Bryant
>
>
>
> -----Original Message-----
> Subject: Re: Reverse engineering a known system
>
> IMO Mike's logic is questionable when he claims that:
>
> "Since I know that MiniMax has held up well out-of-sample, my approach
> is to reverse engineer the strategy over the entire reported period,
> without saving any data for out-of-sample (OOS) testing."
>
> Actually what he is doing is curve-fitting to the MinMax system
> metrics. Since the two systems are not structurally equivalent in
> general, the OOS performance may not be the same.
>
> IMO, the way of doing this correctly is to use 3/4 of the data sample
> to reverse engineer the system and then the remaining 1/4 to see is
> the OOS performance of the result is identical to the original system.
>
> Otherwise, this is just curve-fitting and selection bias and I am

> surprised that Mike thinks this is sound practice. It is not IMO.- Hide quoted text -
>
> - Show quoted text -

[Michael R. Bryant]

Granted, but that's why I like the reverse engineering approach. If you do
it correctly, you do get the same distribution of returns, which you can
confirm with real-time tracking.
-----------

[aaron bartelson]

Where do I unsubscribe from these group emails?

Thanks,
Aaron

[Michael R. Bryant]

There should be an option when you log in to the group.

 

Mike Bryant

 

Subject: RE: Reverse engineering a known system

 

Where do I unsubscribe from these group emails?

Thanks,
Aaron

[Mark Knecht]

On Tue, Apr 3, 2012 at 6:03 AM, Rick <rh4...@gmail.com> wrote:
> IMO Mike's logic is questionable when he claims that:
>
> "Since I know that MiniMax has held up well out-of-sample, my approach
> is to reverse engineer the strategy over the entire reported period,
> without saving any data for out-of-sample (OOS) testing."
>
> Actually what he is doing is curve-fitting to the MinMax system
> metrics. Since the two systems are not structurally equivalent in
> general, the OOS performance may not be the same.
>
> IMO, the way of doing this correctly is to use 3/4 of the data sample
> to reverse engineer the system and then the remaining 1/4 to see is
> the OOS performance of the result is identical to the original system.
>
> Otherwise, this is just curve-fitting and selection bias and I am
> surprised that Mike thinks this is sound practice. It is not IMO.
>

Hi Rick,
Sorry for the delay answering. I've been travelling due to family
issues and am just back and getting started on email responses.

First, thanks for your comments. (And to Mike for his responses
also.) I suppose I see this problem a little differently so let's
talk.

1) While I agree conceptually with your comment about OOS data, I
don't personally agree with using a portion of the target system's IS
Data as OOS unless I know the distributions of returns for the first
3/4 vs the last 1/4 are very similar. I suspect that to match Mike's
IS results I should use all the same data he used as IS.

2) In the case of a _specific_ target system like MiniMax in this case
Mike & I would be operating with different data. He can look at those
distributions, I cannot. However he can also see how the system has
operated way OOS also. I cannot do either. Lacking that info my view
is:

a) Attempt to match the IS performance report.
b) See if it works OOS.
c) If it does then consider trading it, if not move on.

3) While Mike suggests that in the case of MiniMax he got similar code
from Builder, I ran the reverse engineering process 5 times and I
didn't get the same code all 5 times. Lacking the real MiniMax code I
have no way of knowing if any one of those is even similar to Mike's
system or whether they are all different. However, if I had MiniMax
code why would I reverse engineer what I already have? Mike did it,
IMO, to explore the reverse engineering idea and not to necessarily
promote that it's the right way to approach system development.

4) Last, for now, this whole process is really just another way to
decide what build metrics to use in Builder. If I start with no
information about a real system then my build metrics are really just
goals I might have. So much profit, so many trades, some values for
other things. They aren't real. They are just things I'd like. On the
other hand, if I start with known results from a real system then, if
Builder is successful, I end up with Builder code that is _possibly_ a
little closer to something other people have developed elsewhere. In
neither case do I really know anything about how it will work OOS.
That's a different problem in my mind.

Anyway, I'll continue to think on the issue and report back results
as I get a chance. Maybe others will want to investigate the idea. (Or
maybe not!) ;-)

Cheers,
Mark

[Rick]

Hi Mark, you are making some good points here but I still think that a
pure set of performance metrics cannot be matched to a given system
uniquely but represent a large class of possible systems most of which
will fail forward.

>
>    Anyway, I'll continue to think on the issue and report back results
> as I get a chance. Maybe others will want to investigate the idea. (Or
> maybe not!) ;-)
>
> Cheers,

> Mark- Hide quoted text -

[Mark Knecht]

On Sat, Apr 7, 2012 at 10:09 AM, Rick <rh4...@gmail.com> wrote:
<SNIP>

>
> Hi Mark, you are making some good points here but I still think that a
> pure set of performance metrics cannot be matched to a given system
> uniquely but represent a large class of possible systems most of which
> will fail forward.

<SNIP>

Rick,
I __COMPLETELY__ agree. But on the other hand how is this any
different than doing a system from scratch? I don't think, but it's
just my opinion, that Mike ever suggested that using this process will
cause Builder to duplicate the code in any system. He as much as said
it didn't duplicate his system. Based on that I have to agree with
you. Future performance of a 'reverse engineered' system must
certainly be different from the system itself. Therefore we're left
with the same old problem. When do I trade this system, from code,
from Builder using reverse engineering, from Builder from scratch.
It's always down to that problem in my mind.

Cheers,
Mark

[Michael R. Bryant]

Just to clarify, Rick, are you using Builder? While this group is not
limited to licensees, I think it's helpful to know if an active contributor
is actually using the program.

Mike Bryant