Add input node to already trained neural network

[Benji]

Hi,

Could anyone tell me if it is possible to add an input node to an
already trained neural network?

Thanks
Benji

[Stephen Wolstenholme]

On Wed, 21 Mar 2012 09:57:01 -0700 (PDT), Benji <bkok...@gmail.com>
wrote:

That depends of the application. It's not allowed in my applications
because it would invalidate any existing configuration and training.
On the other hand adding and removing hidden layers and hidden nodes
is possible. It is possible to have a hidden layer with frozen weights
connected to the inputs so that the another hidden node could be added
without invalidating the training. I'm not sure what help that would
be.

Steve

--
Neural Network Software. http://www.npsl1.com
EasyNN-plus. Neural Networks plus. http://www.easynn.com
SwingNN. Forecast with Neural Networks. http://www.swingnn.com
JustNN. Just Neural Networks. http://www.justnn.com

[John Ladasky]

If you write your own software, it is certainly possible. If you
depend on software written by other people, it might be harder.

As to which architectures would handle that new input node WELL,
mathematically speaking, I would guess that the Fahlman Cascade
Correlation network would be a good choice.

A CasCor network trains by adding one node at a time, freezing the
weights of the older nodes as they accumulate. So to add an input,
you could first add a new input weight to each existing node, train
it, and then in subsequent steps go back to training the CasCor
network using its standard algorithm.

[Tomasso]

"Stephen Wolstenholme" <st...@npsl1.com> wrote in message news:cf2km79l6a2k2o9ih...@4ax.com...

> On Wed, 21 Mar 2012 09:57:01 -0700 (PDT), Benji <bkok...@gmail.com>
> wrote:
>
>>Hi,
>>
>>Could anyone tell me if it is possible to add an input node to an
>>already trained neural network?
>>
>>Thanks
>>Benji
>
> That depends of the application. It's not allowed in my applications
> because it would invalidate any existing configuration and training.
> On the other hand adding and removing hidden layers and hidden nodes
> is possible. It is possible to have a hidden layer with frozen weights
> connected to the inputs so that the another hidden node could be added
> without invalidating the training. I'm not sure what help that would
> be.
>
> Steve

An input with frozen weights of value zero (in each case) would be a start. Zero means the input would have no effect on
hidden or output nodes. Frozen would keep it at zero.

When you want to "add" another input, unfreezing it would let it change from zero via backpropagation.

I've used NN systems where this can be organised, but never tried it.

Normally, however, if I had N inputs and I wanted to assess K (<N) of the inputs in an NN model I wouldn't try to cobble
something together initialised to a different model (with K-1 inputs, say, plus the trick above). I suspect you would be
likely to lock yourself into a smaller weight space, and end up much more likely to hit (poor), restricted basins of
attraction, and local minima.

You may be interested in Shapley Modelling. This includes an attempt to assess the importance of each input
variable/predictor. It builds models from every possible combination of predictors and subtracts suitably weighted
amounts of the R^2 for model fits to isolate the single contribution of each variable (even if some multi-colinearity is
present). If done with MLPs, it would need best model of a collection for each combo of inputs.

Tom.

[Tomasso]

"Tomasso" <t...@sso.com.au> wrote in message news:a_idneq4uY53k_HS...@westnet.com.au...

>
> "Stephen Wolstenholme" <st...@npsl1.com> wrote in message news:cf2km79l6a2k2o9ih...@4ax.com...
>> On Wed, 21 Mar 2012 09:57:01 -0700 (PDT), Benji <bkok...@gmail.com>
>> wrote:
>>
>>>Hi,
>>>
>>>Could anyone tell me if it is possible to add an input node to an
>>>already trained neural network?
>>>
>>>Thanks
>>>Benji
>>
>> That depends of the application. It's not allowed in my applications
>> because it would invalidate any existing configuration and training.
>> On the other hand adding and removing hidden layers and hidden nodes
>> is possible. It is possible to have a hidden layer with frozen weights
>> connected to the inputs so that the another hidden node could be added
>> without invalidating the training. I'm not sure what help that would
>> be.
>>
>> Steve
>
> An input with frozen weights of value zero (in each case) would be a start. Zero means the input would have no effect
> on hidden or output nodes. Frozen would keep it at zero.
>
> When you want to "add" another input, unfreezing it would let it change from zero via backpropagation.
>
> I've used NN systems where this can be organised, but never tried it.
>
> Normally, however, if I had N inputs and I wanted to assess K (<N) of the inputs in an NN model I wouldn't try to
> cobble something together initialised to a different model (with K-1 inputs, say, plus the trick above). I suspect you
> would be likely to lock yourself into a smaller weight space, and end up much more likely to hit (poor), restricted
> basins of attraction, and local minima.

By "smaller weight space", I meant "smaller sub-manifold of the weight space". After training on K-1 inputs,
you get a solution which is optimised for those inputs. When you incorporate the extra input, you are starting
from an initial state which is biased against using information from that input (if that info had been available in
other ways from the K-1 inputs).

T.

[Benji]

On Mar 23, 7:49 am, "Tomasso" <t...@sso.com.au> wrote:
> "Stephen Wolstenholme" <st...@npsl1.com> wrote in messagenews:cf2km79l6a2k2o9ih...@4ax.com...
> > On Wed, 21 Mar 2012 09:57:01 -0700 (PDT), Benji <bkoks...@gmail.com>
> > wrote:
>
> >>Hi,
>
> >>Could anyone tell me if it is possible toaddaninputnodeto an

> >>already trained neural network?
>
> >>Thanks
> >>Benji
>
> > That depends of the application. It's not allowed in my applications
> > because it would invalidate any existing configuration and training.
> > On the other hand adding and removing hidden layers and hidden nodes
> > is possible. It is possible to have a hidden layer with frozen weights

> > connected to the inputs so that the another hiddennodecould be added

> > without invalidating the training. I'm not sure what help that would
> > be.
>
> > Steve
>

> Aninputwith frozen weights of value zero (in each case) would be a start. Zero means theinputwould have no effect on

> hidden or output nodes. Frozen would keep it at zero.
>

> When you want to "add" anotherinput, unfreezing it would let it change from zero via backpropagation.

>
> I've used NN systems where this can be organised, but never tried it.
>
> Normally, however, if I had N inputs and I wanted to assess K (<N) of the inputs in an NN model I wouldn't try to cobble
> something together initialised to a different model (with K-1 inputs, say, plus the trick above). I suspect you would be
> likely to lock yourself into a smaller weight space, and end up much more likely to hit (poor), restricted basins of
> attraction, and local minima.
>
> You may be interested in Shapley Modelling. This includes an attempt to assess the importance of eachinput
> variable/predictor. It builds models from every possible combination of predictors and subtracts suitably weighted
> amounts of the R^2 for model fits to isolate the single contribution of each variable (even if some multi-colinearity is
> present). If done with MLPs, it would need best model of a collection for each combo of inputs.
>
> Tom.

Thanks Tom!