"Build process failed"

[Paul Tozour]

I've been seeing some issues when building strategies for the SPY ETF.  I bought a new Lenovo specifically to run Adaptrade, with an i9 processor and 64 GB of memory, but I'm still seeing timeouts and build cancellations with this ETF:

[6/25/2020 12:58:19 PM]  Strategy evaluation took too long (timed out), possibly from too many trades, Monte Carlo samples, or bars.

[6/25/2020 12:58:28 PM]  Build process failed. Results from completed generation with highest fitness on test segment displayed.

I've reduced the number of Monte Carlo iterations to 5 and the number of "fittest strategies" to save each generation to 20, and still seeing the problem.

Now, I'm using data for this ETF since 1993.  That's a lot of data, but in my experience, more data means significantly more reliable strategies that work much better out-of-sample.  I could certainly use a shorter data series but I'd rather not have to do that.

I'm also using other symbols as series 2-8, just to figure out if any of those help build better strategies.  Not sure if that's contributing to the problem or not.

I'm curious if there's any sort of setting I could change that would prevent this timeout from triggering?  I tend to leave this PC running in the background, so I really don't care how long it takes.

Paul

[Mike Bryant]

The more data you’re using, the longer each evaluation will take and that includes secondary series, which are also part of the evaluation process for any strategy that includes those series. Also, for N Monte Carlo samples, each evaluation will take N times as long (and that’s with parallel processing). How many trades do your typical strategies have? More trades means a longer evaluation time. If you don’t know how many trades you’re getting from strategies, try running it without the stress testing/Monte Carlo and examine the population. If numerous strategies have more than, say, 10000 trades, that’s a potential factor.

 

The build process only fails if it can’t evaluate a strategy within the specified time-out interval, as set on Program Options (View menu), after the specified number of attempts (also set on Program Options). In other words, if it fails the first time, it will build another strategy, and try again, and so on until it reaches the specified number of attempts. This implies you can try increasing the time-out interval and/or increasing the number of attempts, but that will slow the whole build process down considerably. It’s almost always better to identify the underlying problem, as suggested by the hints in the error message, and correct it.

 

Limiting the number of saved strategies is unrelated to the time it takes to evaluate a strategy, which is what caused the build process failure you’re seeing. Just to be clear, it also has nothing to do with memory, only with processing time.

 

There’s a section in the user’s guide with more information about this (Build Time, Usage Topics), but your basic options are (1) reduce the amount of data you’re using (2) change settings to reduce to the number of trades in strategies (only useful if you’re getting a large number of trades, as explained above), (3) run without stress testing, or (4) increase the time-out interval and/or number of retry attempts.

 

There’s one other not-so-basic option that may or may not help: you can try to optimize the number of cores you’re allocating to the program. As discussed in a number of threads on this forum, the optimal number of processing threads will depend on the particular build settings and data you’re using. Builder allows you to specify the number of threads you’re using, so you can do some test runs with your settings and data to see if there’s a more optimal number than you’re currently using.

 

Mike Bryant

Adaptrade Software

[Paul Tozour]

Ah, that's what it was; thank you!

I just needed to do View > Program Options > Time-Out Interval and crank that up to something larger than 20 seconds.

[Paul Tozour]

OK, processing time is no longer an issue thanks to cranking up the Time-Out Interval, but I'm now running into an odd problem with the same ETF (SPY) where it reliably auto-cancels after 4-5 generations with an "Available memory exceeded" error.  It's strange because I've run this same ETF many times before, and only occasionally encountered this problem.  I'm running essentially the same .gpstrat as before, just with a few more indicators turned off.

I'm running on a new PC with 64 GB of memory, 79 GB free on the hard drive, and no other apps running at the same time as Adaptrade.

I've been trying the following to reduce the amount of memory used, but nothing is making a difference:

-Changing "Starting Date" from from 1993 all the way up to 2010

-Eliminating unnecessary build symbols (I used to have up to 8 additional "series" but am now down to just 4 besides SPY)

-Reducing "Population Size" from 1800 down to just 400

-Reducing Monte Carlo Iterations from 10 down to 4

-"After building, save <X> fittest strategies" - reduced from 30 down to 18

-Shutting down and re-opening Adaptrade

It seems odd that it's still having these sorts of memory issues at this point; 64 GB ought to be more than enough for this ...

PT

--
You received this message because you are subscribed to the Google Groups "Adaptrade Builder" group.
To unsubscribe from this group and stop receiving emails from it, send an email to adaptrade-buil...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/adaptrade-builder/b6c79502-6231-455f-a2f3-d2fedd43d0a2o%40googlegroups.com.

[Mike Bryant]

You shouldn’t have any problems with 64GB. Did you check the size of Top Strategies? If you’re not very selective with your metrics for Top Strategies, the number of members can build up quickly, and it’s up to you to delete them from the population; there’s no automatic deletion of Top Strategies by design. The other issue you didn’t mention is the number of trades in your strategies. If your strategies have tens of thousands of trades each, that can take up a lot of memory, not to mention processing time for each strategy. FYI, Monte Carlo iterations has little to do with memory, only processing time. Lastly, what about any other running programs? The memory bar in Builder looks at available memory on your computer, regardless of which program is consuming it.

[Paul Tozour]

Yes, that was the problem -- thank you.  I had been assuming it would clear out the Top Strategies when I started a build with "Reset on Build" checked, but it does not.

Deleting them all manually (Top Results tab, right-click, "Delete All Strategies") solved the problem.

I'd love to see an option in the GP Settings dialog (or the Program Options) to reset the Top Strategies when starting a new build ...

PT

--

You received this message because you are subscribed to the Google Groups "Adaptrade Builder" group.
To unsubscribe from this group and stop receiving emails from it, send an email to adaptrade-buil...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/adaptrade-builder/002a01d6555b%2471fded60%2455f9c820%24%40BreakoutFutures.com.

[Mike Bryant]

The Top Strategies population works that way by design. If it were to automatically delete the population with each build, you wouldn’t be able to collect strategies from successive builds, which is part of the point of it. Also, if you’re getting a large number of strategies there, it means you’re not setting the metrics properly for that population. The whole idea of the Top Strategies population it to filter out just the very best strategies, regardless of generation or build. Filtering out a large number defeats the purpose.

[Paul Tozour]

The motivation makes sense, but from an end-user perspective,

A) you really only care about a certain number of strategies -- I typically only look at the first 10 or 20, or when I'm doing a lot of testing, maybe the first 50-100;

B) and when the number of Top Strategies gets huge, as it did in my case, and the user doesn't quite realize what's going on, it's indistinguishable from a memory leak.

Admittedly it's on me for not fully understanding the accumulation of Top Strategies, though I did read the documentation pretty thoroughly.

At the very least, I'd think you'd want to have an option to say "Limit Size of Top Strategies to <N>", and have it unchecked by default and fill in a value of 1000 or something like that.  And then users who want the current functionality unchecked can leave it turned off, and users like me can turn it on and set it to maybe 1000 and then not have to worry about it.

Not a huge deal since you can reset them manually, but I think a feature like that could only help.

PT

--

You received this message because you are subscribed to the Google Groups "Adaptrade Builder" group.
To unsubscribe from this group and stop receiving emails from it, send an email to adaptrade-buil...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/adaptrade-builder/004c01d65575%245316dc30%24f9449490%24%40BreakoutFutures.com.