Neural Network - Pattern Recognition....!

[Sriram]

I used the neural network toolbox ( nprtool ) for classifying my detected objects into either of 3 classes. I used 14 parameters (image moments) for all the 3 classes of input for training. As of now, I was able to collect only few data for each classes say around

Class 1 - 17 * 14

Class 2 - 11 * 14

Class 3 - 48 * 14

Total

Inputclass - 14*76 && Outputclass - 3 * 76

I arranged these data in column wise to feed into nprtool box.... I used the default toolbox functions like - hidden as 10 , training - 70% (54 samples) , validation - 15% (11 samples), testing - 15% (11 samples) and started the training.

Total number of Iterations was - 20

Performance (MSE) was - 0.0731

Gradient was 0.0617

Validation checks was - 6

Once the net has been created, I tried to use some data in "sim(net,input)" to check my networks performance. For certain inputs from the trained data set, the network's performance was fine but for many it was very bad. (unexpected results).

This is my status and problem. Now I need suggestions -

In what all ways I can improve the performance of the network.

1. Increasing the inputclass database will improve but suggest me something other than that.

2. Increasing the number of hidden layers from 10 to many doesn't seem to make much difference :(....

Through the documentation of Neural Network toolbox - I found the default nprtool in Matlab take cares of input and output processing (ex: mapminmax) and also it uses trainscg function for training.... Should I use some other efficient training algorithms such as trainlm ? But here how can I decide logically (not by trying all algorithms) which training function will be useful for me.?

I have just started to work on neural network after exploring some basics...Kindly help me on improving it -- making me a transition from advance beginner to expert :P

Thanks for your time....!

[Greg Heath]

"Sriram " <sri...@live.in> wrote in message
<kc3b9p$q3k$1...@newscl01ah.mathworks.com>...

> I used the neural network
toolbox ( nprtool ) for classifying my detected objects into either of 3
classes. I used 14 parameters (image moments) for all the 3 classes of
input for training. As of now, I was able to collect only few data for
each classes say around
> > Class 1 - 17 * 14
> > Class 2 - 11 * 14
> >Class 3 - 48 * 14
> > Total Inputclass - 14*76 && Outputclass - 3 *76
> > I arranged these data in column wise to feed into nprtool box....
I used the default toolbox functions like - hidden as 10 , training - 70%
(54 samples) , validation - 15% (11 samples), testing - 15% (11 samples)
and started the training.
> > Total number of Iterations was - 20
> >Performance (MSE) was - 0.0731
> > Gradient was 0.0617
> > Validation checks was - 6
> > Once the net has been created, I tried to use some data in "sim(net,input)" to
> >check my networks performance. For certain inputs from the trained data set, the
> >network's >performance was fine but for many it was very bad. (unexpected results).

Neither nonquantitative terms like "fine" and "bad" nor the quantitative
result MSE = 0.0731 helps much without references. With target matrix

t = [ repmat( [1 0 0 ]',1,17) repmat( [0 1 0 ]',1,11) repmat( [0 0 1 ]',1,48) ];

a reasonable reference is MSE00 = mean(var(t',1)) = 0.17671
so that the normalized MSE is NMSE = MSE/MSE00 = 0.41368 and the
resulting R^2 statistic ( see "coefficient-of-determination in Wikipedia)
is R2 = 1-MSE = 0.58632 which is not necessarily good.

Even with a reference MSE, a classifier should be graded on error rate; not MSE.

What is needed is NMSE and PCTerr for each class (1-3) and each data
split (trn/val/tst)

> >This is my status and problem. Now I need suggestions -
> > In what all ways I can improve the performance of the network.

1. Duplicate copies of class1 and class2 so that you have 48 vectors of
each class (N = 3*48) = 142
2. Standardize the I = 14 input variables to zero-mean/unit-variance (zscore or
mapstd)
3. The columns of the target matrix should be columns of the O=3-dimensional
unit matrix eye(3). For example, t = repmat( eye(3), 1, 48 ). The corresponding
classindices are obtained from classind = vec2ind(t);
4. For a 0.7/0.15/0.15 data split Ntrn = N-round(0.15*N) = 100 resulting in
Ntrneq = Ntrn*O = 300 training equations
5. For an I-H-O network topology, there will be Nw =(I+1)*H+(H+1)*O = 3 + 18*H
unknown weights to estimate with Ntrneq equations. This results in Ndof = Ntrneq-Nw
estimation degrees of freedom. It is desirable that Ntrneq >> Nw or H << 297/18 =
16.5. However, larger values of H can be tolerated by using validation stopping
or regularization using trainbr.
6. The normalizing factor for the MSE should be the average variance of the target
matrix variables

MSE00 = mean(var(t',1)) % Biased (divided by N) or
MSE00a = mean(var(t',0)) % "a"djusted to be unbiased (divided by N-1)

whereas the biased and unbiased training MSEs are

MSE = sse(t-y)/Ntrneq
MSEa = sse(t-y)/Ndof

7. The recommended training function for classification is 'trainscg'. A reasonable
training goal is MSEgoal = 0.01*(Ndof/Ntreq)*MSE00a

8. Try a number of values for H and run 10 or more random weight initialization trials
for each value of H..

>> 1. Increasing the inputclass database will improve but suggest me something other than that.
> > 2. Increasing the number of hidden layers from 10 to many doesn't seem to make much difference :(....
> > Through the documentation of Neural Network toolbox - I found the default
nprtool in Matlab take cares of input and output processing (ex: mapminmax) and
also it uses trainscg function for training.... Should I use some other
efficient training algorithms such as trainlm ? But here how can I decide
logically (not by trying all algorithms) which training function will be
useful for me.?
> > I have just started to work on neural network after
exploring some basics...Kindly help me on improving it -- making me a
transition from advance beginner to expert :P
> > Thanks for yourtime....!

Hope this helps

Greg

[Greg Heath]

"Greg Heath" <he...@alumni.brown.edu> wrote in message <kc6bvc$9mu$1...@newscl01ah.mathworks.com>...

144

> 2. Standardize the I = 14 input variables to zero-mean/unit-variance (zscore or
> mapstd)
> 3. The columns of the target matrix should be columns of the O=3-dimensional
> unit matrix eye(3). For example, t = repmat( eye(3), 1, 48 ). The corresponding
> classindices are obtained from classind = vec2ind(t);
> 4. For a 0.7/0.15/0.15 data split Ntrn = N-round(0.15*N) = 100 resulting in
> Ntrneq = Ntrn*O = 300 training equations

Ntrn = 122, Ntrneq = 366

> 5. For an I-H-O network topology, there will be Nw =(I+1)*H+(H+1)*O = 3 + 18*H
> unknown weights to estimate with Ntrneq equations. This results in Ndof = Ntrneq-Nw
> estimation degrees of freedom. It is desirable that Ntrneq >> Nw or H << 297/18 =
> 16.5.

It is desirable that Ntrneq > Nw or H < floor(363/18) = 20

> However, larger values of H can be tolerated by using validation stopping
> or regularization using trainbr.
> 6. The normalizing factor for the MSE should be the average variance of the target
> matrix variables
>

> MSE00 = mean(var(t',1)) % Biased (divided by N*O) or
> MSE00a = mean(var(t',0)) % "a"djusted to be unbiased (divided by (N-1)*O)
>

For the output y, the biased and unbiased training MSEs are

MSE = sse(t-y)/Ntrneq, NMSE = MSE/MSE00
MSEa = sse(t-y)/Ndof, NMSEa = MSEa/MSE00a

resulting in

R2 = 1-NMSE
R2a= 1-NMSEa

>
> 7. The recommended training function for classification is 'trainscg'. A reasonable

training goal is R2a > 0.99 or MSEgoal = 0.01*(Ndof/Ntrneq)*MSE00a