Neural Network output backtesting vs live - possible rounding discrepency?

[Nick]

Mike,

I'm noticing some differences in backtesting vs live trading on very specific trades when entry is based ONLY on a neural network output.  Based on multiple strategies I've seen in builder, entries for long trades are always >= 0.5 and entries for short trades are always <=0.5.  This aligns with your user guide explanation as well.  I'm curious if there could be some differences in rounding between back testing and live data that is causing the discrepancy between back testing and live trading?  I'm 99% positive I've eliminated all other factors that normally cause discrepancy.  Again, I'm not talking about the overall performance results, but actual SPECIFIC trades.  Where in back testing a trade is NOT entered, but in live trading it is, and the neutral network is the only variable placing a buy stop order to go long and the price movement easily moves past the order so slippage is not a concern.  Rounding in the NN is the only other variable I can think of that I can't debug.

Thanks!

Nick

[Mike Bryant]

Given that the neural network indicator in Builder can involve a large number of calculations, it’s certainly possible that there could be rounding errors, but I don’t know why rounding errors would be different in live trading versus back-testing. More likely, the difference is in the live trading data versus back-testing data.

 

Mike Bryant

Adaptrade Software

[Nick]

I'm not sure that's the case in this instance.  I have the strategy running on two identical charts, one is in sim mode and the other is Live trading.  Both are getting the same data feed and forming the same bars.  If the trade triggers and goes long like it did today, a bar later I can turn off the strategy on the sim account, turn it back on and that trade will never have existed.  I can also verify that by running a regular back test in Ninjatrader instead of the backtest on the actual chart with the active strategy.  For some reason, there are still discrepancies on a small number of trades even when everything else is equal.

[Mike Bryant]

I’m not sure what you’re suggesting. Calculations are calculations. It’s not as if the strategy is somehow going to calculate things differently on the same data because it’s live instead of a back-test. The fact that sim mode is different than live trading using the same data is hardly surprising. Sim mode is not exactly the same as live trading, despite the intent. Also, it’s not uncommon to rerun a strategy (e.g., turn it off and on again) and get a slightly different results on occasion. That’s usually because the data has been updated, so when you rerun it, it starts using the updated data. All it takes is one tick difference is some cases.

[Nick]

Not suggesting anything really, trying to rule out and inevitably fix the cause of the discrepancy.  The only thing I don't have visibility into is the NN calculations.  I understand your view point on the data variations, but I can verify the OHLC and volume of the trigger bar is the same in both backtesting and live trading.  Time doesn't matter as these are range bars.  The only input to the neural network is AS_Consecutive_Bars_Down so that's obvious to see discrepancies both on a chart and in the indicator.  So the only thing left to rule out would be something related to the NN calulation, hence my question of how feasible it is that a rounding error would create a difference in backtesting vs live trading.  

[Mike Bryant]

Again, I can’t see any way that the same strategy can cause two different results with the same input. It’s just computer calculations. Input determines output. Simple as that. And, just to clarify, the rounding error will be exactly the same if the data are exactly the same – it’s computer after all.

[Nick]

Ok, understood. So if we agree on the calculation having to be the same both in sim and live, can you explain a little further about the difference in data processing between live and sim that you are referring to?  Maybe I'm not comprehending this properly.  I understand that every data feed might be slightly different and only a tick can throw off a strategy.  But the strategies are based on the actual chart, back testing based on last bar and live based on first tick of bar, correct?  So if the trigger bar and bars leading up to the trigger bar are exactly the same in live datafeed and a backtest, I'm slightly confused on how there can still be a discrepancy assuming OHLC and volume are identical?  What am I missing?

[Mike Bryant]

I was just referring to my past experience. You’ll have to look into the specifics of how your platform handles sim vs. real-time to find the differences. Obviously, sim is not live, so there is SOME difference.

[Nick]

Mike,

Some follow up... I might be on to something here.  I added some debug code to print out the value of the NN Output incase a discrepancy occurred again.  I was unfortunate to have another one hit today (Friday) but it showed there is a clear difference in the NN_Output, more than just a rounding issue.  Note my platform is NT7.

Live trading  - NNOutput= 0.67724112293824   2/11/2022 9:39:38 AMFriday, February 11, 2022

Backtest - NNOutput= 0.372221999142828   2/11/2022 9:39:38 AMFriday, February 11, 2022

 

The NN Inputs are coming from two variables: 

VarL3.Set(Typical[BTAdjust]);

and

AS_ConsecBarsDn(Low)[BTAdjust];

Obviously typical price is going to be the same because the bar OHLC is all the same on the trigger bar.  I DID however notice something interesting with the output of ConsecBarsDn.  On my live chart, the value is always seems to be close to some level about 1000x more than it should be.  More importantly, I can see clear areas where it's not actually calculating the bars down properly and skips one.  On the back test chart, the ConsecBarsDn is calculating properly.  Now, if I add ANOTHER AS_ConsecBarsDn(Low) to my live chart manually, it plots properly.  So it seems something about the way the strategy is enabled live is making it different.  

I'm not sure if this is causing the NN output difference during live trading, the only other explanation is some sort of issue with the neural network itself which is obviously much harder, if not impossible to debug.

[Mike Bryant]

There are certainly some differences in NT between live and back-test. I’m still trying to understand all of them, but it’s not surprising that you’d find some differences in results. I’ve never seen the consecutive up/down indicators give results in the 1000 range, but I’ll see if I can duplicate what you’re seeing. Have you seen the same in NT 8? I can’t promise that I’ll be supporting NT 7 indefinitely. I’ll add that there’s something suspicious about the NinjaScript framework if indicator code gives correct results in back-test but not in live trading. A good framework should be insensitive to simple things like that.

 

Mike Bryant

Adaptrade Software

 

From: adaptrad...@googlegroups.com <adaptrad...@googlegroups.com> On Behalf Of Nick
Sent: Friday, February 11, 2022 12:27 PM
Subject: Re: Neural Network output backtesting vs live - possible rounding discrepency?

 

Mike,

 

Some follow up... I might be on to something here.  I added some debug code to print out the value of the NN Output incase a discrepancy occurred again.  I was unfortunate to have another one hit today (Friday) but it showed there is a clear difference in the NN_Output, more than just a rounding issue.  Note my platform is NT7.

 

Live trading  - NNOutput= 0.67724112293824   2/11/2022 9:39:38 AMFriday, February 11, 2022

Backtest - NNOutput= 0.372221999142828   2/11/2022 9:39:38 AMFriday, February 11, 2022

 

 

 

The NN Inputs are coming from two variables: 

VarL3.Set(Typical[BTAdjust]);

and

AS_ConsecBarsDn(Low)[BTAdjust];

 

Obviously typical price is going to be the same because the bar OHLC is all the same on the trigger bar.  I DID however notice something interesting with the output of ConsecBarsDn.  On my live chart, the value is always seems to be close to some level about 1000x more than it should be.  More importantly, I can see clear areas where it's not actually calculating the bars down properly and skips one.  On the back test chart, the ConsecBarsDn is calculating properly.  Now, if I add ANOTHER AS_ConsecBarsDn(Low) to my live chart manually, it plots properly.  So it seems something about the way the strategy is enabled live is making it different.  

 

I'm not sure if this is causing the NN output difference during live trading, the only other explanation is some sort of issue with the neural network itself which is obviously much harder, if not impossible to debug.

 

--
You received this message because you are subscribed to the Google Groups "Adaptrade Builder" group.
To unsubscribe from this group and stop receiving emails from it, send an email to adaptrade-buil...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/adaptrade-builder/27d7e7d8-f0ef-49d8-8f48-b87bb020fea6n%40googlegroups.com.

[Nick]

I have not yet had a chance to run it live on NT8, primarily because there hasn't been much quiet time in the markets since we started debugging this and I don't want to turn my account off on NT7 to connect on NT8 and miss a move.  I have seen the same behavior in the backtest part of NT8 however so that at least lines up with NT7 although not that helpful.  I'm very curious if you see the same behavior on your rig for NT7.  My goal is to migrate to NT8 by end of the year, but some things are just easier in NT7 for coding purposes so just being an old man reluctant to change...

Have a good weekend!

[Nick]

Mike,

Any luck reproducing this issue or diagnosis the root cause?

Thanks!

Nick

[Mike Bryant]

I’ll let you know if I find anything.

 

Mike Bryant

Adaptrade Software

 

From: adaptrad...@googlegroups.com <adaptrad...@googlegroups.com> On Behalf Of Nick
Sent: Thursday, February 17, 2022 5:08 AM
Subject: Re: Neural Network output backtesting vs live - possible rounding discrepency?

 

Mike,

 

Any luck reproducing this issue or diagnosis the root cause?

[antuan silvester]

Dear All.

I'm developing a strategy with futures 15 minutes in ES from Neural Network. and Ninja Trader 8

I'm experiencing the same problems with strategies generated with Neural networks. The strategy starts well and after several days (in the market replay that is similar to real, data on each tick or price change) the strategy does not work well. 

In the case of using Backtesting that works only on bar close, the strategy works well. (I have tested the strategy with market replay data on bar close and the results are similar to on price change and on each tick) 

attached you can find one example problem of my strategy:

                VarL1[0] = Blai10Modif1.PlotMarron[Shift1 + BTAdjust];
                VarL2[0] = Blai10Modif1.PlotMarron[BTAdjust];
                VarL3[0] = MAX(VarL2, N1)[Shift2];
                VarL4[0] = BlaiModif1.PlotMarron[BTAdjust];
                VarL5[0] = BlaiModif1.PlotMarron[Shift3 + BTAdjust];
               CondL1 = VarL1[0] == VarL3[0];
               CondL2 = VarL4[0] == X1;
CondL3 = VarL5[0] > X2;
                CondL4 = CondL2 == CondL3;
               EntCondL = CondL1 == CondL4;

to fulfill  EntCondL with double precision after several calculations is not possible due to rounding in the case of real data, but in the case of backtesting, it's possible.

some possible solutions are:

To change AS_NNCompute to Decimal precision.

And the outputs, round to the first decimal or unit.

any comments, please.

Thanks in advance

[Mike Bryant]

The equality operator (“==” in NinjaTrader) works best with logical operands, such as in your CondL4 and EntCondL statements below where it’s comparing true/false conditions. When applied to real values, it works best as a way to exclude, rather than include; i.e., when comparing two real numbers, it should be the case that it will be false most of the time, which will tend to mitigate the effects of rounding. I would take that into account when screening strategies and possibly exclude the strategy if it looks like it’s relying on that operator for a true result when equating two real numbers.

 

Mike Bryant

Adaptrade Software

 

From: adaptrad...@googlegroups.com <adaptrad...@googlegroups.com> On Behalf Of antuan silvester
Sent: Tuesday, March 5, 2024 5:05 AM
Subject: Re: Neural Network output backtesting vs live - possible rounding discrepency?

 

Dear All.

[antuan silvester]

Thanks for your answer Mike.

Finally the problem is the AS_Orderprocessing method. After one trade, the strategy continues working but not send more orders to the market.

I have change the order unmanaged for managed limit orders and the strategy works fine. 

I remove from strategy

//            else if (State == State.Realtime)
//            {
//                AS_OnChangeToRealtime();
//            }
//            else if (State == State.Terminated)
//            {
//                AS_CancelEntriesOnTerminate();
//            }
        }

        // Manage trading orders; called every time an order changes state.
//        protected override void OnOrderUpdate(Order order, double limitPrice, double stopPrice,
//                                              int quantity, int filled, double averageFillPrice,
//                                              OrderState orderState, DateTime time, ErrorCode error, string comment)
//        {
//            AS_ManageOrders(order);
//        }

//LongOrderOnFirstBar == null

and change BTAdjust to 0 instead of 1 for real time.

 
and 

//                    AS_EnterLongMarket(NShares);
   EnterLong(NShares);

 //                   AS_ExitLongStop(LStop);
 ExitLongStopMarket(LStop);

 //                   AS_CancelOrders();
        ExitLong();

Regards

[Logan]

I didn't want to create a new conversation as it relates to Neural Network too. I notice the builder sets the weights directly, well, yes it trains and sets it over the training set and is fixed in the code.

Is there a way the weights be backpropagated((overwrite new adjusted weights and biases from updated arrays for example) in the built code itself, be it MQL4 or NT8? 

Just wondering, coz otherwise, we'd build it everyday at a certain time and re-implement the top strategies in the market say every day or couple of days. 

To have a generic backpropagate function in the builder code itself is something good to have. 

Any thoughts from experts out there, possibly this could be introduced in newer updates. Please share.

Sincerely,

Logan