Excellent results on learning examples, Poor results on other examples...... WHY ?
Leonid H2009
Hello all !
This is my first experience with neural networks, and I find the
subject very interesting, but I have some problems and I really hope
to get some answers here :-)
I wrote a simple neural network that should tell me of the input for
the network is Type_A or Type_B, the network is build of two layers,
15x15 neurons in each one, when the first one is getting the inputs,
and another layer (layer number 3) which I use as the output of the
network, and it got only 2 neurons -
If the output of the First neuron is bigger than the output of the
Second neuron, then the answer is Type_A
If the output of the Second neuron is bigger than the output of the
First neuron, then the answer is Type_B.
The Input for my network are images of 15x15 pixels, each image can
have 15 Horizontal lines (this is my "Type_A" examples) , Or images of
15 Vertical lines (this is my "Type_B" examples) each line in the
image is made of 15 pixels that have the same value, which is a random
floating number with a value between 0 and 1, for example it can be
0.2875, or 0.8749 and so on.... so I want that my network will tell me
if the input image is Type_A (15 Horizontal lines) or Type_B (15
Vertical lines).
The Weight for each neuron connection is a floating value between -1.0
and +1.0, the Bias also have a velue in the same range, each neuron
use for it's output the formula:
NeuronOutput = 1.0 / ( 1.0 + exp( -SumOfAllInputs ) );
For the learning process I'm using a set of 100 images, 50 with
vertical lines, and 50 with horizontal lines. So now for my questions
-
1. The learning process of my network looks good (but not perfect as I
wanted...) after some learning process it manage to detect right about
90% - 95% of the images in the learning set, BUT, when I'm trying to
show it a new set of 100 images (50 images with horizontal lines and
50 images with vertical lines) the result drops down to about 55% -
60% :-(
Why does it happens? what am I doing wrong?
My learning process is made of about 250 - 300 learning cycles, in
each cycle I shows the network my 100 examples and then I change the
weights and the biases values and start another cycle. Should I maybe
use a New set of (100) examples for each learning cycle? or should I
continue using the same (100) examples for All the learning cycles?
Or maybe I should add some random noise to the images at each cycle?
2. Is there a benefit for adding more layers to the network? I do
understand that the learning process will be longer, but should the
final result be better? (for example I know that one of the reasons
for the power of our brains is that they have 6 layers of neurons in
the cerebral cortex, comparing to only 2-3 layers in other
animals....)
3. Should I maybe use an easier examples instead of the vertical and
horizontal lines images? do you have an idea for a better learning
images for the learning? the code of the application is made of about
850 lines of code (in C++) and maybe I'm still having a bug
somewhere.... what is the best Type_A and Type_B examples to use for
testing my network and see if it's working right? please suggest.
I hope to get some good answers soon,
Thanks very much!
Leon.
Leonid H2009
> wanted...) after some learning process it manage to detect right about
> 90% - 95% of the images in the learning set, BUT, when I'm trying to
> show it a new set of 100 images (50 images with horizontal lines and
> 50 images with vertical lines) the result drops down to about 55% -
> 60% :-(
>
> Why does it happens? what am I doing wrong?
Actually it's usually doesn't get better than 88% - 90% on the
learning examples.... and it's much worse on other examples (after the
learning prosess) then I get only about 55-60% success.....
Can you please help me with this? how can I fix it? (I want to get at
least 97-98% for this images after the learning, is it too much to
ask?)
Thanks!
Leonid H2009
Can someone please help ? anyone ?
To make it more clear, my Vertical images (15x15 pixels) looks like
this:
V01, V02, V03 ............. V15
V01, V02, V03 ............. V15
V01, V02, V03 ............. V15
V01, V02, V03 ............. V15
V01, V02, V03 ............. V15
V01, V02, V03 ............. V15
V01, V02, V03 ............. V15
V01, V02, V03 ............. V15
V01, V02, V03 ............. V15
V01, V02, V03 ............. V15
V01, V02, V03 ............. V15
V01, V02, V03 ............. V15
V01, V02, V03 ............. V15
V01, V02, V03 ............. V15
V01, V02, V03 ............. V15
and my Horizontal images (also 15x15 pixels) looks like this:
V01, V01, V01 .............. V01
V02, V02, V02 .............. V02
V03, V03, V03 .............. V03
V04, V04, V04 .............. V04
V05, V05, V05 .............. V05
V06, V06, V06 .............. V06
V07, V07, V07 .............. V07
V08, V08, V08 .............. V08
V09, V09, V09 .............. V09
V10, V10, V10 .............. V10
V11, V11, V11 .............. V11
V12, V12, V12 .............. V12
V13, V13, V13 .............. V13
V14, V14, V14 .............. V14
V15, V15, V15 .............. V15
when each Vxx is a RANDOM float number between 0.0 to 1.0, for
example V08 can be 0.286541, and V14 can be 0.782954 and so on....
I don't have much experience with neural networks but it looks to me
like an easy task for such network, why doesn't my network learn this
examples properly? should I make any manipulatings with this data
before I insert it into the network?
Please give me any advice on this, Thanks very much!
Leon.
Leonid H2009
OK I see that it's very hard to get answers here, can you at least
give me some link to another neural-networks group or forum where I
can get some answers?
Thanks.
Greg
wrote:
> OK I see that it's very hard to get answers here, can you at least
> give me some link to another neural-networks group or forum where I
> can get some answers?
The information that separates the two classes is
contained in the horizontal and vertical gradient.
Hope this helps.
Greg
Leonid H2009
> contained in the horizontal and vertical gradient.
So what should be the input for my network? what should I change?
Leonid H2009
As suggested here I changed my network to work with tanh instead of
sigmoid activation function for all nodes (see near the last image in
this page) -
and I also changed my inputs to be between -1.0 to +1.0, but it didn't
help much.... my network still don't want to learn this basic
pattern....
Why? what is wrong with my network?
Greg
wrote:
> > The information that separates the two classes is
> > contained in the horizontal and vertical gradient.
>
> So what should be the input for my network?
Horizontal and vertical gradients.
Hope this helps.
Greg
Greg
wrote:
> As suggested here I changed my network to work with tanh instead of
> sigmoid activation function for all nodes (see near the last image in
> this page) -
>
>
> and I also changed my inputs to be between -1.0 to +1.0, but it didn't
> help much.... my network still don't want to learn this basic
> pattern....
>
> Why? what is wrong with my network?
Your inputs are not feeding the net with the best info for
classification.
Your row and column values are random; however, the actual values
are useless. It is their differences that are important.
Hope this helps.
Greg
Leonid H2009
"> So what should be the input for my network?
Horizontal and vertical gradients"
What do you mean? how do I calculate it? what mathematic formula?
I understand that this networks are very good at patterns recognition,
so why can't the network recognize this pattern as it is?
> Your inputs are not feeding the net with the best info for
> classification.
> Your row and column values are random; however, the actual values
> are useless. It is their differences that are important.
So how do I calculate this differences? how many inputs should my
network have? only 1? "gradients" and that's it?
Thanks.
Greg
wrote:
> Hi Greg thanks for the help!
>
> "> So what should be the input for my network?
>
> Horizontal and vertical gradients"
>
> What do you mean? how do I calculate it? what mathematic formula?
numerical gradient = difference-in-value/difference-in-distance
For neighboring rows and columns, difference-in-distance = 1
> I understand that this networks are very good at patterns recognition,
> so why can't the network recognize this pattern as it is?
Garbage in, Garbage out.
> > Your inputs are not feeding the net with the best info for
> > classification.
> > Your row and column values are random; however, the actual values
> > are useless. It is their differences that are important.
>
> So how do I calculate this differences? how many inputs should my
> network have? only 1? "gradients" and that's it?
For an image with r rows and c columns
the horizontal gradient image will yield r*(c-1) inputs from r rows
and c-1 columns
the verttcal gradient image will yield (r-1)*c inputs from r-1 rows
and c columns
Think about whether you need both gradients or just one.
Hope this helps.
Greg
Leonid H2009
>
> numerical gradient = difference-in-value/difference-in-distance
>
> For neighboring rows and columns, difference-in-distance = 1
What do you mean? "why = 1" ? how did you calculate 1?
> > I understand that this networks are very good at patterns recognition,
> > so why can't the network recognize this pattern as it is?
>
> Garbage in, Garbage out.
I still don't understand why can't the network learn the data just As-
It-Is, after all it's a very simple pattenn of vertical and horizontal
lines, why is it so complicated for it to learn it just as it is? just
as I look at this picture with my eyes and my brain right away knows
if it's vertical or horizontal lines image? (Type_A or Type_B)
> > > Your inputs are not feeding the net with the best info for
> > > classification.
> > > Your row and column values are random; however, the actual values
> > > are useless. It is their differences that are important.
But if the rows and columns values are random, so their differences
will be Random values TOO, so why does it matters?
> For an image with r rows and c columns
>
> the horizontal gradient image will yield r*(c-1) inputs from r rows
> and c-1 columns
> the verttcal gradient image will yield (r-1)*c inputs from r-1 rows
> and c columns
Sorry but I still didn't understand the calculation.... how many final
values do I get for each image?
Should my calculations produce 2 values for each image? "Gradient_X"
and "Gradient_Y" ?
Should my code looks something like this?
void CalculateImageGradient( int *My_Gradient_In_X, int
*My_Gradient_In_Y )
{
int My_Gradient_In_X = 0;
int My_Gradient_In_Y = 0;
for ( y = 0; y < NumberOfRows; y++ ) // Calculate gradient in X
{
for ( x = 0; x < ( NumberOfColumns - 1 ); x++ )
{
My_Gradient_In_X += abs( MyImage[x][y] - MyImage[x + 1][y] );
}
}
for ( x = 0; x < NumberOfColumns; x++ ) // Calculate gradient in Y
{
for ( y = 0; y < ( NumberOfRows - 1 ); y++ )
{
My_Gradient_In_Y += abs( MyImage[x][y] - MyImage[x][y + 1] );
}
}
}
Should I calculate one single value for all the Absolute differences
in X, and one single value for all the Absolute differences in Y?
should I collect the total differences between each pixel and the
pixel next to it, first in X and then in Y ?
Please explain,
Thanks!
Leonid H2009
> numerical gradient = difference-in-value / difference-in-distance
OK sorry, I missed this line on my last reading.... but I don't
understand, it looks like "difference-in-distance" will allways be 1
isn't it? I thought that I allways check the difference only for
neighboring pixels, am I wrong? should I also calculate the "numerical-
gradient" for pixels that are not located next to each other? for
example pixel[30][70] and pixel[30][78] ....?
Another thing that I didn't understand, for each original image that I
have of 15x15 pixels should I get one 15x15 image of horizontal
gradient, and one 15x15 image of vertical gradient? or should I get
One Single Value for "horizontal gradient" and another one single
value for "vertical gradient"?
If posible please show me a short pseudo-code for this, something like
-
Horizontal gradient calculation -
for ( y = 0; y < NumberOfRows; y++ ) // Calculate gradient in X
{
for ( x = 0; x < ( NumberOfColumns - 1 ); x++ )
{
MyImage[x][y] = ??? // What should I write here ?
}
}
Horizontal gradient calculation -
.....
Thanks.
Greg
wrote:
> > > What do you mean? how do I calculate it? what mathematic formula?
>
> > numerical gradient = difference-in-value/difference-in-distance
>
> > For neighboring rows and columns, difference-in-distance = 1
>
> What do you mean? "why = 1" ? how did you calculate 1?
For forward differences, the distance between any row and the
previous row is 1 row.
> > > I understand that this networks are very good at patterns recognition,
> > > so why can't the network recognize this pattern as it is?
>
> > Garbage in, Garbage out.
>
> I still don't understand why can't the network learn the data just As-
> It-Is, after all it's a very simple pattenn of vertical and horizontal
> lines,
If the values in each image are random, the pattern structure
for the collection of images is certainly not simple.
> why is it so complicated for it to learn it just as it is? just
> as I look at this picture with my eyes and my brain right away knows
> if it's vertical or horizontal lines image? (Type_A or Type_B)
If you are implying that the patterns must be simple because "your"
eye-brain network can classify them, I refuse to get sucked into any
other response except
THINK WHAT YOU ARE SAYING!
> > > > Your inputs are not feeding the net with the best info for
> > > > classification.
> > > > Your row and column values are random; however, the actual values
> > > > are useless. It is their differences that are important.
>
> But if the rows and columns values are random, so their differences
> will be Random values TOO,
STOP.
THINK.
Create a few 3X3 examples.
THINK AGAIN
Get the picture (no pun intended) now?
> so why does it matters?
>
> > For an image with r rows and c columns
>
> > the horizontal gradient image will yield r*(c-1) inputs from r rows
> > and c-1 columns
> > the verttcal gradient image will yield (r-1)*c inputs from r-1 rows
> > and c columns
>
> Sorry but I still didn't understand the calculation.... how many final
> values do I get for each image?
Sorry, I can't make it any plainer than that.
Try carefully rereading the thread.
> Should my calculations produce 2 values for each image? "Gradient_X"
> and "Gradient_Y" ?
> Should my code looks something like this?
>
> void CalculateImageGradient( int *My_Gradient_In_X, int
> *My_Gradient_In_Y )
> {
> int My_Gradient_In_X = 0;
> int My_Gradient_In_Y = 0;
>
> for ( y = 0; y < NumberOfRows; y++ ) // Calculate gradient in X
> {
> for ( x = 0; x < ( NumberOfColumns - 1 ); x++ )
> {
> My_Gradient_In_X += abs( MyImage[x][y] - MyImage[x + 1][y] );
> }
> }
>
> for ( x = 0; x < NumberOfColumns; x++ ) // Calculate gradient in Y
> {
> for ( y = 0; y < ( NumberOfRows - 1 ); y++ )
> {
> My_Gradient_In_Y += abs( MyImage[x][y] - MyImage[x][y + 1] );
> }
> }
>
> }
>
> Should I calculate one single value for all the Absolute differences
> in X, and one single value for all the Absolute differences in Y?
> should I collect the total differences between each pixel and the
> pixel next to it, first in X and then in Y ?
>
> Please explain,
1. Create two 3X3 random images for each class.
2. Create the four corresponding gradient images.
3. Compare the results by eye.
You should be able to take it from there.
Hope this helps.
Greg
Leonid H2009
Hi,
> 1. Create two 3X3 random images for each class.
> 2. Create the four corresponding gradient images.
> 3. Compare the results by eye.
I'm sorry but I didn't understand how to calculate number 2, you told
me that I have to to calculate the gradient in the following way -
Gradient = ( Difference-in-value / Difference-in-distance )
but how can "Difference-in-distance" ever be other value than 1 ? I
thought that I should calculate this only for pixels that are next to
each other, this value can be different than 1 ONLY if you say that I
have to calculate it also for pixels that are NOT next to each other,
sorry but I'm a little confused about it.
Maybe a a small example will help, if I have this original image:
a1, a2, a3,
a4, a5, a6,
a7, a8, a9
if I understand you right your new Gradient image in X will be -
b1, b2,
b3, b4,
when:
b1 = abs( a1 - a2 ) / 1
b2 = abs( a2 - a3 ) / 1
b3 = abs( a4 - a5 ) / 1
b4 = abs( a5 - a6 ) / 1
Am I right or am I wrong?
And I do the same on the other axis (Y) to get the Gradient image for
Y. If that's right, then the new values will ALSO be random, so what's
the point....?
I'm really trying to (think and) understand you, but the explanation
is not so clear to me, don't get so mad.
Thanks.
Leonid H2009
Sorry in my last example it should be (if I understand it right) like
the following -
b1, b2,
b3, b4,
b5, b6,
when:
b1 = abs( a1 - a2 ) / 1
b2 = abs( a2 - a3 ) / 1
b3 = abs( a4 - a5 ) / 1
b4 = abs( a5 - a6 ) / 1
b5 = abs( a7 - a8 ) / 1
b6 = abs( a8 - a9 ) / 1
Am I right or am I wrong?
Thanks.
Leonid H2009
Can someone PLEASE answer my simple question?
Or is this group abandoned ?
Tomasso
> I don't understand, IS THIS GROUP ABANDONED or what ??
>
> Why is it SO HARD to get here an answer for simple question?
>
> Isn't there anyone in this place that can answer my questions?
>
> Come on this is so ridiculous.
I need to understand the questions. Please organise them better.
From what I read, you are carrying out intricate pre-processing
on data (which we can't see) and may have lost your original
learning problem.
I'm not sure about this because I don't fully know what you are
doing or asking.
Tomasso.
Leonid H2009
> I'm not sure about this because I don't fully know what you are
> doing or asking.
>
> Tomasso.
Tomasso the input to my network are images of 15x15 values, each
image
can have 15 vertical lines, OR 15 horizontal lines, I want my network
to tell me if a given image is of horizontal lines? of vertical
lines?
(Type_A or Type_B)
Each line in an image are 15 equal values between -1.0 to +1.0, for
example one line in an images can be 15 values of 0.275134, another
one
will have 15 values of -0.8254790.
So far my network was able (after the learning process) to recognize
ONLY about 60% of the given images, which is very poor result....
Greg told me to use Gradient, like this -
Gradient = ( Difference-in-value / Difference-in-distance )
1. I didn't understand how to use this formula and for which pixels
to
use it.
2. Why does his formula contain the variable "Difference-in-distance"
if it's ALLWAYS equal to "1" ? are there cases where it's other
number?
3. This Gradient will ALSO be a random value, so what's the point in
doing this new calculation of the gradient images? (if I understand
it
right for each input image I have to make two gradient images one for
X axis and one for Y axis.
I hope that this description is clearer.
Thanks.
Tomasso
OK, I understand the data, I think.
You have a grid of 15 rows or 15 columns. You want to tell if it's row
or col. A row (or col) is 15 identical values, and there are 15 or them,
or they are transposed you they are in columns (and fall into the other
class). Values (for a line) are uniform in -1 to +1.
Eg.
aaaaaaaaaaaaaaa
bbbbbbbbbbbbb
...
ooooooooooooo
where a, b, ... o are random.
And if it's cols, you transpose, so row1 is
ab... ...o.
The random values are chosen for each pattern.
The dimensionality of this kind of problem is high because of
the variability of a,b, ... ...o.
Ie, an NN would need a HUGE amount of data to solve this problem.
However, a rule for solving this problem would be pretty simple:
X1 = X2 => row lines
Y1 = Y1 => col lines
A multi-layer NN which extracted row variability and col variability
would handle this easily.
Can a generic NN (225 inputs and 1 classification output, and one or two
hidden layers) handle this? It would need an enormous amount of
data without some hints in the design.
This is the kind of classification problem where intermediate hints would
help. Something that does feature detection help. You don't need to
know the features before you detection them (this is not part of
supervised learning, but about information gain).
Tomasso.
Leonid H2009
> OK, I understand the data, I think.
Yes it looks like you understood it right, I have 2 types of images
1. Image of 15 Vertical lines (Type_A).
2. Image of 15 Horizontal lines (Type_B).
Each line are 15 pixels with the exactly same value, a random value
that can be between -1.0 to +1.0, so when I show an image to my
network I want it to tell me if it's Type_A (Vertical lines image) or
Type_B (Horizontal lines image).
So tell me, if I have this original image:
a1, a2, a3,
a4, a5, a6,
a7, a8, a9
The Gradient Image for X axis will be -
b1, b2,
b3, b4,
b5, b6,
where:
b1 = abs( a1 - a2 )
b2 = abs( a2 - a3 )
b3 = abs( a4 - a5 )
b4 = abs( a5 - a6 )
b5 = abs( a7 - a8 )
b6 = abs( a8 - a9 )
and the Gradient Image for Y axis will be -
c1, c2, c3
c4, c5, c6
c1 = abs( a1 - a4 )
c2 = abs( a4 - a7 )
c3 = abs( a2 - a5 )
c4 = abs( a5 - a8 )
c5 = abs( a3 - a6 )
c6 = abs( a6 - a9 )
Am I right or am I wrong?
So, at the end I have 2 Gradient images, one for for 'X' with the
zise
of 2x3 values, and one for 'Y' with size of 3x2 values, now, in what
way should I insert this two images to the network? Please explain.
Thanks!
Greg
> > Please explain,
>
> 1. Create two 3X3 random images for each class.
> 2. Create the four corresponding gradient images.
> 3. Compare the results by eye.
>
> You should be able to take it from there.
I'll do it with one 3 x 3 sample image from each class
MATLAB Notation:
Semicolons are row separators
Apostrophes represent transposition
Commas are column separators
Class A
A = [a a a; b b b; c c c]
HGA = [0 0; 0 0; 0 0]
VGA = [b-a b-a b-a; c-b c-b c-b]
VGA = [r1 r1 r1; r2 r2 r2] (r1,r2 random)
VGA'= [r1 r2; r1 r2; r1 r2]
NN input
IA = [ HGA , VGA']
= [ 0 0 r1 r2; 0 0 r1 r2; 0 0 r1 r2]
Dominant Class A characteristic:
First 3 X 2 submatrix is zero
NN target output = 1 (probabililiy of Class A)
-----------------------------------------------------
Class B
B = [d e f; d e f; d e f]
HGB = [e-d f-e; e-d f-e; e-d f-e]
HGB = [r3 r4; r3 r4; r3 r4] (r3,r4 random)
VGB = [0 0 0; 0 0 0 ]
VGB' = [0 0; 0 0; 0 0]
NN Input
IB = [HGB , VGB']
= [r3 r4 0 0; r3 r4 0 0; r3 r4 0 0]
Dominant Class B characteristic:
Second 3 X 2 submatrix is zero
NN target output = 0 (probabililiy of Class A)
Hope this helps.
Greg
Leonid H2009
Greg thanks for the answer I appreciate it, but the syntax of Matlab
is confusing me, can you instead take a look at MY example and tell me
if it's right?
If I have this original image (which can be rows OR columns image, I
want my network to tell me that!) -
a1, a2, a3,
a4, a5, a6,
a7, a8, a9
The Gradient Image for X axis will be -
b1, b2,
b3, b4,
b5, b6,
where:
b1 = abs( a1 - a2 )
b2 = abs( a2 - a3 )
b3 = abs( a4 - a5 )
b4 = abs( a5 - a6 )
b5 = abs( a7 - a8 )
b6 = abs( a8 - a9 )
and the Gradient Image for Y axis will be -
c1, c2, c3
c4, c5, c6
where:
c1 = abs( a1 - a4 )
c2 = abs( a4 - a7 )
c3 = abs( a2 - a5 )
c4 = abs( a5 - a8 )
c5 = abs( a3 - a6 )
c6 = abs( a6 - a9 )
Am I right or am I wrong?
So, at the end I have 2 Gradient images, one for for 'X' with the zise
of 2x3 values (b1-b6), and one for 'Y' with size of 3x2 values (c1-
c6), now, in what way should I insert this two images to the network?
for this example should my network have 12 inputs? (6 + 6) Please
explain.
Thanks!
TomH488
like. Here are my problems:
On Sep 8, 2:01 pm, Leonid H2009 <one4d5581804...@getonemail.com>
wrote:
> Hello all !
>
> This is my first experience with neural networks, and I find the
> subject very interesting, but I have some problems and I really hope
> to get some answers here :-)
>
> I wrote a simple neural network that should tell me of the input for
> the network is Type_A or Type_B, the network is build of two layers,
OK, a 2 layer network: Input layer and output Layer.
> 15x15 neurons in each one,
Hummm: 225 inputs and 225 outputs???
> when the first one is getting the inputs,
"when the first one"... first... what? Layer?
That doesn't make any sense.
Do you mean that sometimes the second one gets the inputs???
Your phrase makes no sense to me at this point.
Only the Input layer (first layer) "gets the inputs"
> and another layer (layer number 3) which I use as the output of the
> network, and it got only 2 neurons -
>
OK, so we don't have a 2 Layer network now, right? Its a 3 layer
network?
Do you now mean 225 Inputs, 225 Hiddens, and 2 Outputs?
> If the output of the First neuron is bigger than the output of the
> Second neuron, then the answer is Type_A
What is a "first neuron" and what layer is it in?
>
> If the output of the Second neuron is bigger than the output of the
> First neuron, then the answer is Type_B.
What is a "second neuron" and what layer is it in?
>
> The Input for my network are images of 15x15 pixels, each image can
> have 15 Horizontal lines (this is my "Type_A" examples) , Or images of
> 15 Vertical lines (this is my "Type_B" examples) each line in the
> image is made of 15 pixels that have the same value, which is a random
> floating number with a value between 0 and 1, for example it can be
> 0.2875, or 0.8749 and so on.... so I want that my network will tell me
> if the input image is Type_A (15 Horizontal lines) or Type_B (15
> Vertical lines).
this actually makes sense for the input layer one.
>
> The Weight for each neuron connection is a floating value between -1.0
> and +1.0, the Bias also have a velue in the same range, each neuron
> use for it's output the formula:
>
> NeuronOutput = 1.0 / ( 1.0 + exp( -SumOfAllInputs ) );
I think this is conceptually OK, however, you make no mention of your
weight values in this equation.
>
> For the learning process I'm using a set of 100 images, 50 with
> vertical lines, and 50 with horizontal lines. So now for my questions
> -
>
OK
If your network is 225 Inputs, 225 Hiddens, and 2 Outputs, then:
1) 225 is an astronomically large number of hidden nodes.
2) you are using the two outputs as flags for A and B?
Example: Output values: 1, 0 = A and 0, 1 = B?
If true, you can cut your model in half by a single Output where 1 =
A, and 0 (or -1) = B
3) training on H or V lines implies you are interested in merely
detecting H or V lines.
If this is the case, you only need look at any 3 corners, however if
you are concerned with noise in a line, you might want to sample the
line and look at any 3 edges.
You could also include the 4th corner and the 4th edge for again,
further sampling.
But in its most fundamental state, you have the 3 corners
a b
c _
where a=b is H and a=c is V
This is a very small problem
But if it is about handling noise, you can still sample all your rows/
columns reducing the Input layer down to 5x5 or 10x10 which would be
again, a huge reduction in your model.
Hopefully, my lack of comprehension is not a result of grammar issues
(!)
Good Luck with your project,
Tom
Greg
wrote:
> > MATLAB Notation:
>
> Greg thanks for the answer I appreciate it, but the syntax of Matlab
> is confusing me, can you instead take a look at MY example and tell me
> if it's right?
No. I would prefer if you have specific questions about what I have
posted.
Greg
Tomasso
Leonid's problem looks technically trivial and can be solved with a very low connectivity
network. Indeed it can be solved analytically.
Assuming no noise, and only two classes (horizontal lines and vertical lines).
The primitive decision is an equality test between two adjacent cells in same row (or same
col).
This can be achieved with a single 2-1 NN which learns x1=x2 -> 1, otherwise -> 0, USING
a gaussian (or similar). It's really a ONE parameter learning problem in that case. If
using logistic (or other monotonic transfer functions) needs one small hidden layer.
If two cells are identified as not equal (in same row), then the image can't be class
horizontal lines.
If equal, there is a small probability of that it's vertical lines which co-incidentally
has that pair of row pixels equal. To resolve this possibility, you can aggregate multiple
primitive decisions. Or there would be other ways as well (like 3-way equality tests,
etc). Ultimately, a uniform 15*15 image is both horizontal lines AND vertical lines.
I have no idea why Leonid wants to use all 15*15 inputs for this with 225 hidden neurons
and full connectivity.
The presence of noise should not make this any harder, just introduces a small error rate.
In the presence of noise AND a third class (neither horizontal nor vertical lines) it
could be harder...
T.
Observation: The retina does line and edge detection (with orientation), which ONLY uses
very local fan-in (over about 3 layers). The edge detection information (and intensity and
colour) flows from the retina through the optic nerve and is processes in the lateral
geniculate nucleii and the visual cortex (back of brain) where various kinds of
aggregation and integration happen, identifying higher order features.
Leonid H2009
Greg I'm not going to learn Matlab now just in order to understand
your answers, Thank you.
Tom, my network is a very basic one, which have to decide if an input
image is of horizontal lines or vertical lines. My question was what
is the best way to insert this image into the network, I read your
answer, and yes I indeed want my network to be able to handle images
with some noise.
Tommaso I'll read your answer again I'm not sure that I understoond it
all, and what's wrong with 15x15 inputs? it's a very small image, for
example if I'll want my network to recognoze faces I'll need at least
80x80 inputs....so 15 x15 is not so much.
Thanks.
TomH488
please answer the following simple questions:
1) How many layers?
2) How many nodes in each of the layers?
Thanks
Tom
Tomasso
You're assuming that one image = one training record. For your posed
problem this isn't necessary.
The 15x15 "input" image can be organised into 2*14*15 records looking
at adjacent pairs of pixels. Sampling ONE of these pairs and testing for
equality solves your classification problem except for rare cases where two
adjacent lines have the same value. You can re-sample pairs to confirm.
Think of a filter which scans over the 15x15 image. The filter is very simple
and detects adjacent pixels which are equal. The filter can be a NN which
you train, but it can be hard coded.
Other kinds of filters work this way in image processing (to do things like
sharpening, noise reduction, deconvolutions, etc). Sometimes with NNs
or other kinds of adaptive filters. Sometimes hard coded.
However you have now raised the task of face recognition.
That is a quite different problem than the one you proposed first, with
its rather odd Subject: line.
There are hundreds, probably thousands of publication on face recognition.
I suggest you do some reading. I have taught undergraduates who have
done face recognition as a minor project and most did reasonably well
after a little thought and a little reading.
I have had other students who didn't think much and wanted to know
why their project didn't work. I'm afraid I was unable to offer help.
Learning is a process that uses some kind of re-inforcement, and in
their case, there was nothing to re-inforce! :-) or :-(
Tomasso.
Leonid H2009
> 2) How many nodes in each of the layers?
>
> Thanks
> Tom
Tom I can change the number of layers and nodes in each layer as I
want, and I play with it to test which structure gives me the best
result, usually I try 3 layers, the first one which get the inputs
from the image, have 15x15 nodes (one node for each pixel in the
image) , the second one have also 15x15 nodes, and the last layer have
1 node which is the Output of the network, if this output is smaller
than 0.0 then it's one type, if it's equal of bigger then it's the
other type (each image can be of Type_A or Type_B).
But again the inner structure of my network is not the main issue
here, the main issue is how to insert each image to the network, and
how to prepare the image data BEFORE putting it into the first layer.
Look at the network as a black box, you can change only the number of
nodes in the first layer (the input layer) and the input data itself.
> The 15x15 "input" image can be organised into 2*14*15 records looking
> at adjacent pairs of pixels. Sampling ONE of these pairs and testing for
> equality solves your classification problem except for rare cases where two
> adjacent lines have the same value. You can re-sample pairs to confirm.
>
> Think of a filter which scans over the 15x15 image. The filter is very simple
> and detects adjacent pixels which are equal. The filter can be a NN which
> you train, but it can be hard coded.
Tomasso I understand from your answer that my network can have ONLY 2
inputs, and all it have to do is to check if they are equal or not,
I'm not sure that I want such a solution, I want a solution that check
ALL the pixels in the image and check statistically what are the
chances that it's Type_A (image with vertical lines) or Type_B (image
with horizontal lines) becouse in the next step I'm going to add noise
to the image, so checking only 1-2 pairs of pixels is not good enough
here.
Tomasso
I used the word "scan". This means that the filter (network) looks at two pixels
(eg positions (1,1) and (1,2) then can look at (1,2) and (1,3), and keep going
to (1,14) and (1,15), then (2,1) and (2,2), until (14,15) and (15,15).
And then the transpose (row-col swap).
It should not need to look at all adjacent pairs. It should have a highly reliable
classification after looking at only a few pairs.
Think of the NN as something that only has to look at a ocal number of pixels,
then move on to another locality.
--
Jumping forward to harder problems like face recognition. The basic fillters
should look for things like outline, eyebrows, hairline, etc and once that
information is know, it can be used.
Our visual system does not process like your model. The focea and the edge
and corner detectors in the retina pass very interpreted information to the
visual cortex. Learning from nature is almost always a good idea.
T.
Leonid H2009
Tomasso I still didn't understand how to insert the image data into
the network.
Let's say that instead of 15x15 image I'm using only 4x4 image -
a01, a02, a03, a04
a05, a06, a07, a08
a09, a10, a11, a12
a13, a14, a15, a16
where the values can be between -1 and 1, for example -
a01 = 0.827561
a02 = 0.539875
a03 = -0.27594
.
.
.
a16 = -0.729872
1. How exactly should I insert this data into the network?
2. How many nodes should my network have in it's input layer?
Please explain.
Tomasso
1. Input:
rows tests:
1: a01 and a02
2. a02 and a03
3. a03 and a04
4. a05 and a06
5 a06 and a07
6. a07 and a08
...
12 a15 and a16
transpose (column tests):
13: a01 and a05
14 a05 and a09
15 a09 and a13
...
24. a12 and a16.
Output:
Either 0 or 1 depending on whether the values are equal or not (or approx equal).
The row tests identify constant values in rows. The col tests (records 13 thru 24)
identify constant values in cols.
This solves the horizontal vs vertica lines classification.
2. NN architecture:
If you have a Gaussian activation function then 2 inouts one output, no hidden neurons.
If you use logistic (or other monotonic). I'll leave that part to you as an exercise. You
need to do something for yourself! It will be instructive.
T.
Leonid H2009
> 1. Input:
>
> rows tests:
>
> 1: a01 and a02
> 12 a15 and a16
>
> transpose (column tests):
>
> 13: a01 and a05
> 24. a12 and a16.
Do you mean like this?
Input 1 = abs( a01 - a02 )
Input 2 = abs( a02 - a03 )
...
Input 24 = abs( a12 - a16 )
So for a 4x4 pixels image my network should have 24 input nodes, that
each one of them get the absolute value of a subtraction between 2
nearby pixels? is that what you meant?
>
> If you have a Gaussian activation function then 2 inputs one output, no hidden neurons.
This is very confusing, at the begining you showed me that my network
should have 24 inputs, and now you are telling me that it should have
only 2, I really don't understand now.
> If you use logistic (or other monotonic)....
The activation function for each node in my network is tanh -
NodeOutput = tanh( SumAllNeuronInputs );
Is it a Gaussian activation function? or is it a logistic function? I
really don't know....
Please explain.
Tomasso
>> 1. Input:
>>
>> rows tests:
>>
>> 1: a01 and a02
>> ...
>> 12 a15 and a16
>>
>> transpose (column tests):
>>
>> 13: a01 and a05
>> ...
>> 24. a12 and a16.
>
> Do you mean like this?
>
> Input 1 = abs( a01 - a02 )
> Input 2 = abs( a02 - a03 )
> ...
> Input 24 = abs( a12 - a16 )
>
> So for a 4x4 pixels image my network should have 24 input nodes, that
> each one of them get the absolute value of a subtraction between 2
> nearby pixels? is that what you meant?
No
One 4x4 image = 24 input records.
>> If you have a Gaussian activation function then 2 inputs one output, no hidden neurons.
>
> This is very confusing, at the begining you showed me that my network
> should have 24 inputs, and now you are telling me that it should have
> only 2, I really don't understand now.
Think on this.
Look at one image.
Look at a01.
Compare it to a02.
Compare it to a05.
If one equals a01 and the other doesn't, you are finished.
>> If you use logistic (or other monotonic)....
>
> The activation function for each node in my network is tanh -
>
> NodeOutput = tanh( SumAllNeuronInputs );
>
> Is it a Gaussian activation function? or is it a logistic function? I
> really don't know....
>
> Please explain.
Tanh and logistic functions are monotonic. Gaissian is not. It has a maximum
value for net input = 0.
Leonid H2009
>
> One 4x4 image = 24 input records.
What are "records" ? I read some things about neural networks, I know
that they have layers, they have nodes, they have inputs and outputs,
activation functions and so on... I don't understand what are
"records", and how they are connected to my network?
> Look at one image.
>
> Look at a01.
>
> Compare it to a02.
>
> Compare it to a05.
>
> If one equals a01 and the other doesn't, you are finished.
What do you mean by "Compare" ? what if the input image contain noise
and the values are not allways equal to each other? also as I said I
want to check All the pixels and not just few.
> Tanh and logistic functions are monotonic
So what does it say about the number of inputs that my network should
have?
It's very unclear to me, should my network have 2 inputs? which will
test only 3 nearby pixels? what about the other pixels?
TomH488
wrote:
> > No
>
> > One 4x4 image = 24 input records.
>
> What are "records" ? I read some things about neural networks, I know
> that they have layers, they have nodes, they have inputs and outputs,
> activation functions and so on... I don't understand what are
> "records", and how they are connected to my network?
>
This is really getting bogged down in semantics and possibly
translation into some other foreign language.
input records = input cases = inputs
An Input is usually stored as a matrix where each row is an input
case.
You train on many input cases.
Your training cases are stored in a training matrix where each row is
a case.
There are many ways to prepare your pixel information into a row or
record or case.
In the beginning you took all the pixels and made them a row. 1 image
= 1 case = 1 record. each row has N columns where N = number of
pixels.
The comparison method, however, subdivides an N column image case into
24 two column inputs = imput cases = input records. In this case you
get 24 input records for each image instead of a single record per
image when you were looking at all N pixels at once.
Leonid H2009
OK, so that was terms confusing, but still not all is clear.
> The comparison method, however, subdivides an N column image case into
> 24 two column inputs = imput cases = input records. In this case you
> get 24 input records for each image instead of a single record per
> image when you were looking at all N pixels at once.
It's 24 only for the example I gave of 4x4 pixels image, but for the
original question of 15x15 pixels it will be much more records.... so
I don't understand how does it makes easiest on the network to process
much more data....
> 1: a01 and a02
> 2. a02 and a03
> 3. a03 and a04
> 4. a05 and a06
> 5 a06 and a07
> 6. a07 and a08
> ...
> 12 a15 and a16
> ...
> ...
> Input 24 = abs( a12 - a16 )
What does it mean "a01 and a02" I still didn't understand that,
specially when I said that the values are not precise and can include
noise, for example if a01=0.25874 and a02=0.25878 what should be the
value of the first column of the record? should it be -0.00004 ?
( 0.25874 - 0.25878 ).
For the example of the 4x4 pixels image, how many input nodes should
my network have?
TomH488
wrote:
HaHa, I think I got confused with what I was saying too!
I believe the method is "a01 and a02" now becomes a column of which
there are 24 columns or "inputs" as explained.
Obviously a column is not a row. An Input usually means an Input Case
which is a row.
So instead of a row of pixels for an input case:
a1, a2, ..., an where n = 24 for example,
you have a row of absolute delta pixel calculations for an input case:
abs(a1-a2), abs(a2-a3), ..., abs ( a[n-1]) - an )
All the deltas are on the same row because all of them must be known
to define a particular image at the pixels.
Leonid H2009
>
> a1, a2, ..., an where n = 24 for example,
>
> you have a row of absolute delta pixel calculations for an input case:
>
> abs(a1-a2), abs(a2-a3), ..., abs ( a[n-1]) - an )
OK that's better, and if I understand it right half of the values will
be absolute delta for the pixels in 'X' axis, and half of them will be
absolute delta for pixels in the 'Y' axis, right?
What I still don't understand is why is it better than just giving the
pixels values as they are? why this method should be easier for the
network? as I said before the image may contain noise, so for for a
rows image, pixels in each line will not be 100% equal to each other,
so in the new method I'll get series of random values (deltas) for
nearby pixels , in what way is it better comparing to the method of
giving the image as it is?
Tomasso
I suggest you implement both methods and report back to this news group.
T.
Greg
wrote:
> Greg I'm not going to learn Matlab now just in order to understand
> your answers, Thank you.
Sorry, didn't mean to scare you by using the "MATLAB"
word.
Perhaps I should have written
Common matrix notation:
-Apostrophes represent transposition
-Commas are column separators
Greg Heath shorthand:
-Semicolons are row separators
Therefore
A = [ a a a
b b b
c c c ]
= [ a a a; b b b; c c c ]
which, by the way, can also be written
A = [ [a b c ]', [a b c ]',[a b c ]'].
Similarly,
B = [ d e f
d e f
d e f ]
= [ d e f; d e f; d e f ]
which, by the way, can also be written
= [ [d d d]', [e e e]', [f f f]' ].
Now you can unfold the shorthand in any line
that I wrote and see how straightforward the
use of horizontal and vertical gradients aids
in separating the two classes.
Hope this helps.
Greg
Greg
> On Sep 20, 1:53 pm, Leonid H2009 <one4d5581804...@getonemail.com>
> wrote:
> So instead of a row of pixels for an input case:
>
> a1, a2, ..., an where n = 24 for example,
>
> you have a row of absolute delta pixel calculations for an input case:
>
> abs(a1-a2), abs(a2-a3), ..., abs ( a[n-1]) - an )
>
> All the deltas are on the same row because all of them must be known
> to define a particular image at the pixels.
For a 4X4 matrix, the horizontal gradient matrix is 3X4 and the
vertical gradient matrix is 4X3. The total number of NN inputs is
3X4 + 4X3 = 24.
However, there is no need to take absolute values.
Greg
Greg
Why the need for absolute values?
Greg
Tomasso
That was TomH, not me.
TomO.
Greg
wrote:
The important information is not values; they are random.
The important information is differences in values; the
most important ones are zero and, therefore, independent
of the coreesponding values.
As long as the SNR is not too low, noise will have little effect
on performance because classification will not depend on
certain gradients being exactly zero. Sufficiently small
gradients will be adequate.
Inputting the original image will work. However, more
hidden nodes, weights and training data will be needed.
Hope this helps.
Greg
Leonid H2009
Greg thanks for the clarifications, by the way the word "MATLAB" did
not scare me, what I didn't understand was for example what's the
meaning of VGA and VGA' , or IA = [ HGA , VGA'] or HGB and all your
other shortcuts.... just now I understand that "VGA" is Vertical....
something, and that "HGB" is Horizontal.... something? and I'm still
not sure what is the VGA' and what does it mean....I also don't
understand what are r1, r2, r3, r4.....? where did they came from?
what are they?
Too many shortcuts just making the explanation unclear.
Greg
wrote:
Class A
A = [a a a; b b b; c c c] % The "row" class; a,b,c
random
HGA = [0 0; 0 0; 0 0] % Horizontal Gradient of A
VGA = [b-a b-a b-a; c-b c-b c-b] % Vertical gradient of A
VGA = [r1 r1 r1; r2 r2 r2] (r1,r2 random) % r1 = b-a, r2 = c-b
VGA'= [r1 r2; r1 r2; r1 r2] % transpose of VGA
NN input
IA = [ HGA , VGA'] % Neural net input
= [ 0 0 r1 r2; 0 0 r1 r2; 0 0 r1 r2]
Dominant Class A characteristic:
First 3 X 2 submatrix is zero
NN target output = 1 (probabililiy of Class A)
-----------------------------------------------------
etc;
Hope this helps.
Greg
Leonid H2009
OK Greg, thanks.