Training ANNs with Levenberg-Marquardt & Early stopping
jonarg
I have a dataset with about 10000 data points and I am trying to
construct an ANN with 2 inputs and 1 output.
I am using the Levenberg-Marquadt training algorithm with early
stopping technique
and I initially set the training goal at 1e-6 and the max number of
epochs at 2000 all other parameters are set to default values (in
matlab).
I started with 1 hidden layer varying the number of neurons from 1 to
30.
The Training and Validation MSE drops from about 0.0020 in 2-1-1
configuration to 1e-6 in to 2-10-1 configuration
and remains 1e-6 in all network configurations with more than 10
neurons.
When setting the training goal to 1e-9 I get MSE=1e-9 in 2-15-1
configuration and this figure remains
to all ANNs with more than 15 neurons in the hidden layer.
Questions!:
1) If I choose training goal of 1e-6, does the 2-10-1 configuration is
the optimal one?
2) Do I have to try ANNs with more than 1 hidden layer?
3) What training goal should I set?
3) Since the results are more than excellent Is there any problem with
overfitting?
Thank you in advance!
Greg Heath
> Hi!
> I have a dataset with about 10000 data points and I am trying to
> construct an ANN with 2 inputs and 1 output.
>
> I am using the Levenberg-Marquadt training algorithm with early
> stopping technique and I initially set the training goal at 1e-6
This has no meaning without reference to the variance
(scaling) of the target variable. Set MSEgoal = var(t)/100.
Then, at convergence, R^2 > ~ 0.99
However, since you are using Early Stopping, the algorithm
may stop before it converges due to training error.
> and the max number of
> epochs at 2000 all other parameters are set to default values
> (in matlab).
>
> I started with 1 hidden layer varying the number of neurons
> from 1 to 30.
> The Training and Validation MSE drops from about 0.0020
> in 2-1-1 configuration to 1e-6 in to 2-10-1 configuration
> and remains 1e-6 in all network configurations with more than
> 10 neurons.
>
> When setting the training goal to 1e-9 I get MSE=1e-9 in
> 2-15-1 configuration and this figure remains
> to all ANNs with more than 15 neurons in the hidden layer.
>
> Questions!:
> 1) If I choose training goal of 1e-6, does the 2-10-1
> configuration is the optimal one?
It seems to be. This can be verified with additional trials
with different initial random weights.
> 2) Do I have to try ANNs with more than 1 hidden layer?
NO.
> 3) What training goal should I set?
var(t)/100 works fine for me
> 3) Since the results are more than excellent Is there any
> problem with overfitting?
That's why you used Early Stopping. Look at the plot of
validation error.
In addition, consider the ratio of Neq (No. training equations
to Nw (No. of unknown weights).
Search Google Groups using
greg-heath Neq Nw
for more details.
Hope this helps.
Greg
jonarg
I really appreciate your help!
Could you please elaborate a little more on choosing the MSEgoal with
regard to variance?
E.g. the variance of my target data is about 180. According to var/
100, I should use a MSEgoal
of 1.8. Isnt this number a little odd compared to usual targets found
in journal papers 1e-6,1e-9 etc...??
Thanks again
Greg Heath
> Hi Greg,
> I really appreciate your help!
> Could you please elaborate a little more on choosing the MSEgoal with
> regard to variance?
The R^2 statistic (AKA "explained variance") is the fraction of the
total variance that is represented by the model.
R^2 = 1- SSE / TSS
= 1- ( N * MSE ) / [ (N -1) * VAR]
~ 1- MSE/VAR
MSE < ~VAR/100 ==> R^2 > ~ 0.99
==> > ~ 99% of the total variance is represented by the model.
>
> E.g. the variance of my target data is about 180. According to var/
> 100, I should use a MSEgoal
> of 1.8. Isnt this number a little odd compared to usual targets found
> in journal papers 1e-6,1e-9 etc...??
Maybe they didn't know what a practical number was and
chose those values
or
Maybe they did know what a practical number was but chose
those values to to impress referees, editors and readers..
How many real world models can you find with R^2 > 0.99?
Hope this helps.
Greg