Hidden Nodes

[David Pedersen]

Mike, I did not pay much attention to the parameters going into the Neural Network, but recently I noticed that virtually all of the strategies from builder have the same number of Inputs as Weights.  As you know, this means that there are no hidden nodes within that particular strategy based on the NNCompute function.  Question is, how are hidden nodes determined and should I be seeing more hidden nodes in the strategies that builder produces?  I read the help file, but it did not really explain the details.  I also noticed that NNCompute says that it is a special case when there are no Hidden nodes and the output is only based on the weighted inputs.  Perhaps a more detailed explanation of Hidden nodes would be helpful.  Thanks in advance, Dave

[Michael R. Bryant]

I don’t know what I could add over and above the section in the user’s guide “Neural Networks”. This section gives the details down to the equations. Here are some of the relevant sections, with the parts I think address your question highlighted in red:

The output from the network is defined by the following equations:

H₁ = tanh(w₁₁ * I₁ + w₂₁ * I₂ + … + w_n1 * I_n)

H₂ = tanh(w₁₂ * I₁ + w₂₂ * I₂ + … + w_n2 * I_n)

…

H_m = tanh(w_1m * I₁ + w_2m * I₂ + … + w_nm * I_n)

Output = tanh(w_nm+1 * H₁ + w_nm+2 * H₂ + … + w_nm+m * H_m)
 

The I_i represent the inputs. In the context of trading, these could be anything that might have some predictive value for trading, such as momentum, stochastics, ADX, moving averages, etc. The H_j represent the hidden nodes, the weights are given by the w_kl, and the output value by Output. The hyperbolic tangent function, tanh, returns a value in the range -1 to +1, so the output will lie in this range. The inputs are typically scaled so that they also lie between -1 and +1.

…

In Builder, the inputs to the network are chosen by the genetic programming process and evolved along with the entry and exit conditions using the same processes of crossover and mutation used to evolve the entry and exit conditions. …

In addition to evolving the inputs to the network, Builder also evolves the number of nodes in the hidden layer and the weight values. The initial number of inputs is a user-selectable option, along with the maximum number of nodes in the hidden layer. The number of inputs may change from strategy to strategy over successive generations as the inputs from different strategies are combined using crossover.

The total number of weights in the network will be given by (n + 1) * m, where n is the number of inputs and m is the number of nodes in the hidden layer, provided m is at least one. If there is no hidden layer (i.e., m is zero), the number of weights is the same as the number of inputs. In the resulting strategy code, each weight is listed as a strategy input. Other strategy inputs may result from the network inputs, such as the look-back length of a moving average.

 

So, provided you’re selecting non-zero values for the maximum number of hidden nodes and initial number of inputs, you should be seeing hidden nodes in at least some cases. If not, just increase the maximum number of hidden nodes and/or the initial number of inputs.

 

Mike Bryant

Adaptrade Software

[David Pedersen]

Mike, thank you for the reply and highlighting the section in the user's guide.  I have 6 and 4 set for the NN for initial inputs and Hidden Nodes respectfully.  I guess I was surprised to notice that it is very rare that I see even one hidden node appear in the evolution process.  I guess my question simply could have been should I be seeing more hidden nodes?  I seem to see it as hidden nodes are more of an exception rather than the norm and wanted to see if that assumption is valid.  I do realize the answer depends on how I have the other parameters set which tells Builder how to evolve the strategies.  Thanks again for your reply, Dave

[Michael R. Bryant]

I would say that hidden nodes are the rule rather than the exception. Initially, the number of hidden nodes is chosen randomly between 0 and the value you specify. You can therefore increase the number of hidden nodes by increasing the maximum allowable value chosen as input. During evolution of the strategies, your other settings and metrics will determine whether networks with more or fewer hidden nodes will rise to the top. If you’ve selected “complexity” as a metric, that could be biasing the results to ones with simpler neural networks. It also might simply be the nature of your market and other metrics. If you want to see the full range of diversity in the initially chosen neural networks, just run it with zero generations.

 

Mike Bryant