Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss
SVD for PCA: The right most rotation matrix
2 views
Skip to first unread message
Paul
Oct 28, 2012, 8:28:37 PM10/28/12
to
My apologies if this appears twice. The posting of this message seems
to have been held up.
I am trying to understand SVD in the context of PCA. I have looked at
Leskovec (http://search.yahoo.com/r/
_ylt=A0oG7t31r41QSHsAFA9XNyoA;_ylu=X3oDMTE0YmlrMDI5BHNlYwNzcgRwb3MDMQRjb2xvA2FjMgR2dGlkA01BUDAwNl83MQ--/
SIG=13fl10gvd/EXP=1351491701/**http%3a//www.cs.cmu.edu/~guestrin/Class/
10701-S06/Handouts/recitations/recitation-pca_svd.ppt) and Shlen
(http://search.yahoo.com/r/
_ylt=A0oG7t0dsI1Qj3oAG1ZXNyoA;_ylu=X3oDMTE0YmlrMDI5BHNlYwNzcgRwb3MDMQRjb2xvA2FjMgR2dGlkA01BUDAwNl83MQ--/
SIG=11r2sjgrs/EXP=1351491741/**http%3a//www.snl.salk.edu/~shlens/
pca.pdf) for intution.
The scenario I use is a lab experiment in which m sensors
syncrhonously sample data at n points in time, yielding a data matrix
X with m rows and n columns. Each row contains the readings from a
single sensor/instrument, and each column contains the readings from
an instant in time. I suppose that the rows could also be key words
in a data mining exercise, and the columns could be documents in which
we try to find these key words in (as per Leskovec above), but that
scenario is a bit foggier for me because it deals with "concepts", the
number of which matches neither m nor n. So as a first step, stick
with the scenario for lab sensor/instrument. Also, consider only real
data, so the data covariance matrices are diagonalizable with
orthonormal eigenvectors corresponding to simple rotations of the data
in m-space.
http://en.wikipedia.org/wiki/Principal_component_analysis#Details
diagonalizes the data set X by factoring it into X=(W)(Sigma)(Vt)
where:
* For W, the columns of this (m)x(m) matrix are the orthonormal
eigenvectors of covariance matrix (X)(Xt) {Xt is the transpose of X}.
* Specifically, (X)(Xt) contain the covariances from pairing the m
sensors/instruments rather than from pairing the n samples of m
measurements. The former is of interest to us while for the life of
me, I can't see the relevance of the latter.
* Xt = Is the transpose of X.
* Vt is the transpose of (n)x(n) matrix V. The columns of V are the
orthonormal eigenvectors of the covariance matrix (Xt)(X) --
specifically, the covariances from pairing the n samples of m
measurements. This relevance of this matrix is what I can't see the
relevance of (intuitively).
* Sigma is the diagonal matrix of square roots of eigenvalues of (X)
(Xt), which are the same as for (Xt)(X).
I am trying to eek out some intuition from X=(W)(Sigma)(Vt). I find
it curious and interesting that the covariances (X)(Xt) are viewed as
a linear transformation, and the eigenvectors in W become the
orthogonal directions in which the scalings differ. Hence, they form
the basis vectors that are aligned with the principal components.
Then it becomes obvious that Sigma is simply the anisotropic axial
scaling.
If X is viewed as some kind of linear tranformation (and I'm not sure
if I'm actully suppose to do that), than Vt can be seen as a rotation
so that the princpal component aligns with the 1st axis, the 2nd
principal component aligns with the 2nd, etc., prior to the scaling by
Sigma. Finally, I would expect W to rotate the data back to its
original orientation, thus yielding X on the LHS.
Following Shlen's tutorial, I find the above picture is easier to see
if we rewrite the SVD formula as (Wt)(X)=(Sigma)(Vt), where the /rows/
of Wt are the eigenvectors of covariance (X)(Xt) between sensors/
instruments. Treating them as basis vectors, then multiplying them by
the columns of X simply projects the m-value samples from each
measurement instance onto the principle components, which yields the
rotation of the data points so that the principle components align
with the axes. Conversely, X=(W)[(Sigma)(Vt)] takes the data points
in the rotated state (principle components aligned with axes) and
unrotates themm so that it matches the orientation of the measured
data points.
One of the most disturbing things I haven't been able to figure out is
what V (or Vt) corresponds to in the real world. I mean, if X was a
transformation, then Vt is simply a rotation in n-space. But X
*isn't* a transformation. And n-space is meaningless because we would
never treat the vector of data from a single sensor as a data point
(i.e., each measurement instance in time as a dimension) and plot it
in n-dimensional space. So even though V or Vt somehow corresponds to
a geometric rotation of sorts, it's in an space that is nonsensical
and has no bearing in the real world.
I realize that Leskovec describes SVD differently, as documents versus
search terms, with concepts as an intermediate thing that is
determined by the SVD. The left and right singular vectors then
represent the correlation of documents versus concepts and search
terms versus concepts. However, he doesn't really delve into why the
math corresponds to that. Also, I'm much more interested in the lab
sensor/instrument scenario, where the size of the diagonal matrix
corresponds to the size of the data set (at least before dimensional
reduction).
So when I look at the mockingly simple SVD formula, I have developed a
phobia of the mysterious rotation matrix at the tail end. It has
defied my endless attempts (no joke) to try to understand
intuitively. Thank you anyone for imparting some clear intution to
this.
to have been held up.
I am trying to understand SVD in the context of PCA. I have looked at
Leskovec (http://search.yahoo.com/r/
_ylt=A0oG7t31r41QSHsAFA9XNyoA;_ylu=X3oDMTE0YmlrMDI5BHNlYwNzcgRwb3MDMQRjb2xvA2FjMgR2dGlkA01BUDAwNl83MQ--/
SIG=13fl10gvd/EXP=1351491701/**http%3a//www.cs.cmu.edu/~guestrin/Class/
10701-S06/Handouts/recitations/recitation-pca_svd.ppt) and Shlen
(http://search.yahoo.com/r/
_ylt=A0oG7t0dsI1Qj3oAG1ZXNyoA;_ylu=X3oDMTE0YmlrMDI5BHNlYwNzcgRwb3MDMQRjb2xvA2FjMgR2dGlkA01BUDAwNl83MQ--/
SIG=11r2sjgrs/EXP=1351491741/**http%3a//www.snl.salk.edu/~shlens/
pca.pdf) for intution.
The scenario I use is a lab experiment in which m sensors
syncrhonously sample data at n points in time, yielding a data matrix
X with m rows and n columns. Each row contains the readings from a
single sensor/instrument, and each column contains the readings from
an instant in time. I suppose that the rows could also be key words
in a data mining exercise, and the columns could be documents in which
we try to find these key words in (as per Leskovec above), but that
scenario is a bit foggier for me because it deals with "concepts", the
number of which matches neither m nor n. So as a first step, stick
with the scenario for lab sensor/instrument. Also, consider only real
data, so the data covariance matrices are diagonalizable with
orthonormal eigenvectors corresponding to simple rotations of the data
in m-space.
http://en.wikipedia.org/wiki/Principal_component_analysis#Details
diagonalizes the data set X by factoring it into X=(W)(Sigma)(Vt)
where:
* For W, the columns of this (m)x(m) matrix are the orthonormal
eigenvectors of covariance matrix (X)(Xt) {Xt is the transpose of X}.
* Specifically, (X)(Xt) contain the covariances from pairing the m
sensors/instruments rather than from pairing the n samples of m
measurements. The former is of interest to us while for the life of
me, I can't see the relevance of the latter.
* Xt = Is the transpose of X.
* Vt is the transpose of (n)x(n) matrix V. The columns of V are the
orthonormal eigenvectors of the covariance matrix (Xt)(X) --
specifically, the covariances from pairing the n samples of m
measurements. This relevance of this matrix is what I can't see the
relevance of (intuitively).
* Sigma is the diagonal matrix of square roots of eigenvalues of (X)
(Xt), which are the same as for (Xt)(X).
I am trying to eek out some intuition from X=(W)(Sigma)(Vt). I find
it curious and interesting that the covariances (X)(Xt) are viewed as
a linear transformation, and the eigenvectors in W become the
orthogonal directions in which the scalings differ. Hence, they form
the basis vectors that are aligned with the principal components.
Then it becomes obvious that Sigma is simply the anisotropic axial
scaling.
If X is viewed as some kind of linear tranformation (and I'm not sure
if I'm actully suppose to do that), than Vt can be seen as a rotation
so that the princpal component aligns with the 1st axis, the 2nd
principal component aligns with the 2nd, etc., prior to the scaling by
Sigma. Finally, I would expect W to rotate the data back to its
original orientation, thus yielding X on the LHS.
Following Shlen's tutorial, I find the above picture is easier to see
if we rewrite the SVD formula as (Wt)(X)=(Sigma)(Vt), where the /rows/
of Wt are the eigenvectors of covariance (X)(Xt) between sensors/
instruments. Treating them as basis vectors, then multiplying them by
the columns of X simply projects the m-value samples from each
measurement instance onto the principle components, which yields the
rotation of the data points so that the principle components align
with the axes. Conversely, X=(W)[(Sigma)(Vt)] takes the data points
in the rotated state (principle components aligned with axes) and
unrotates themm so that it matches the orientation of the measured
data points.
One of the most disturbing things I haven't been able to figure out is
what V (or Vt) corresponds to in the real world. I mean, if X was a
transformation, then Vt is simply a rotation in n-space. But X
*isn't* a transformation. And n-space is meaningless because we would
never treat the vector of data from a single sensor as a data point
(i.e., each measurement instance in time as a dimension) and plot it
in n-dimensional space. So even though V or Vt somehow corresponds to
a geometric rotation of sorts, it's in an space that is nonsensical
and has no bearing in the real world.
I realize that Leskovec describes SVD differently, as documents versus
search terms, with concepts as an intermediate thing that is
determined by the SVD. The left and right singular vectors then
represent the correlation of documents versus concepts and search
terms versus concepts. However, he doesn't really delve into why the
math corresponds to that. Also, I'm much more interested in the lab
sensor/instrument scenario, where the size of the diagonal matrix
corresponds to the size of the data set (at least before dimensional
reduction).
So when I look at the mockingly simple SVD formula, I have developed a
phobia of the mysterious rotation matrix at the tail end. It has
defied my endless attempts (no joke) to try to understand
intuitively. Thank you anyone for imparting some clear intution to
this.
Ray Koopman
Oct 29, 2012, 2:42:14 AM10/29/12
to
On Oct 28, 5:28 pm, Paul <paul.domas...@gmail.com> wrote:
>
> ... I am trying to understand SVD in the context of PCA. ...
Try Jolliffe's Principal Component Analysis (2nd ed).
>
> ... I am trying to understand SVD in the context of PCA. ...
Try Jolliffe's Principal Component Analysis (2nd ed).
Art Kendall
Oct 29, 2012, 7:05:32 AM10/29/12
to
SVD and PCA are among topics that are frequently discussed at the
Classification Society over the last few decades.
There are abstracts from meetings, etc., at
http://www.classification-society.org/clsoc/clsoc.php
We have a discussion list at
http://lists.sunysb.edu/index.cgi?A0=CLASS-L
where you might ask your question.
Art Kendall
Social Research Consultants
Classification Society over the last few decades.
There are abstracts from meetings, etc., at
http://www.classification-society.org/clsoc/clsoc.php
We have a discussion list at
http://lists.sunysb.edu/index.cgi?A0=CLASS-L
where you might ask your question.
Art Kendall
Social Research Consultants
Art Kendall
Oct 29, 2012, 7:09:46 AM10/29/12
to
also see
http://public.lanl.gov/mewall/kluwer2002.html
http://math.stackexchange.com/questions/3869/what-is-the-intuitive-relationship-between-svd-and-pca
www.math.ucf.edu/~xli/SVD_PCA.pdf
Art Kendall
Social Research Consultants
On 10/28/2012 8:28 PM, Paul wrote:
http://public.lanl.gov/mewall/kluwer2002.html
http://math.stackexchange.com/questions/3869/what-is-the-intuitive-relationship-between-svd-and-pca
www.math.ucf.edu/~xli/SVD_PCA.pdf
Art Kendall
Social Research Consultants
On 10/28/2012 8:28 PM, Paul wrote:
Paul
Oct 29, 2012, 1:31:03 PM10/29/12
to
On Oct 29, 7:09 am, Art Kendall <A...@DrKendall.org> wrote:
> also seehttp://public.lanl.gov/mewall/kluwer2002.html
>
> http://math.stackexchange.com/questions/3869/what-is-the-intuitive-re...
Thanks, Art. I came across those references in my web wanderings. I
think my questions are rather specific for general tutorial material.
I haven't found the explanation in various books that I've delved into.
> also seehttp://public.lanl.gov/mewall/kluwer2002.html
>
> http://math.stackexchange.com/questions/3869/what-is-the-intuitive-re...
>
> www.math.ucf.edu/~xli/SVD_PCA.pdf
>
> Art Kendall
> Social Research Consultants
>
> On 10/28/2012 8:28 PM, Paul wrote:
>
>
>
> > My apologies if this appears twice. The posting of this message seems
> > to have been held up.
>
> > I am trying to understand SVD in the context of PCA. I have looked at
> > Leskovec (http://search.yahoo.com/r/
> > _ylt=A0oG7t31r41QSHsAFA9XNyoA;_ylu=X3oDMTE0YmlrMDI5BHNlYwNzcgRwb3MDMQRjb2xvA2FjMgR2dGlkA01BUDAwNl83MQ--/
> www.math.ucf.edu/~xli/SVD_PCA.pdf
>
> Art Kendall
> Social Research Consultants
>
> On 10/28/2012 8:28 PM, Paul wrote:
>
>
>
> > My apologies if this appears twice. The posting of this message seems
> > to have been held up.
>
> > I am trying to understand SVD in the context of PCA. I have looked at
> > Leskovec (http://search.yahoo.com/r/
> > SIG=13fl10gvd/EXP=1351491701/**http%3a//www.cs.cmu.edu/~guestrin/Class/
> > 10701-S06/Handouts/recitations/recitation-pca_svd.ppt) and Shlen
> > (http://search.yahoo.com/r/
> > _ylt=A0oG7t0dsI1Qj3oAG1ZXNyoA;_ylu=X3oDMTE0YmlrMDI5BHNlYwNzcgRwb3MDMQRjb2xvA2FjMgR2dGlkA01BUDAwNl83MQ--/
> > 10701-S06/Handouts/recitations/recitation-pca_svd.ppt) and Shlen
> > (http://search.yahoo.com/r/
think my questions are rather specific for general tutorial material.
I haven't found the explanation in various books that I've delved into.
Art Kendall
Oct 29, 2012, 1:45:42 PM10/29/12
to
Since many people in the Classification Society deal with and teach SVD
and PCA, perhaps if you posted you question there you might get helpful
responses.
Art Kendall
Social Research Consultants
and PCA, perhaps if you posted you question there you might get helpful
responses.
Art Kendall
Social Research Consultants
Paul
Oct 29, 2012, 3:16:07 PM10/29/12
to
Paul
Oct 29, 2012, 3:25:33 PM10/29/12
to
I just got a password for the site. However, I am apparenly not
allowed to post to the list even after logging in.
allowed to post to the list even after logging in.
Ray Koopman
Oct 29, 2012, 3:31:16 PM10/29/12
to
Art Kendall
Oct 29, 2012, 4:12:29 PM10/29/12
to
This is a LISTSERV list. I.e, an email discussion list.
After you subscribe you just send an email to
CLA...@LISTS.SUNYSB.EDU
Give that a try and let me know if you still have a problem.
Art Kendall
Social Research Consultants
After you subscribe you just send an email to
CLA...@LISTS.SUNYSB.EDU
Give that a try and let me know if you still have a problem.
Art Kendall
Social Research Consultants
Paul
Oct 29, 2012, 5:33:40 PM10/29/12
to
On Oct 29, 3:31 pm, Ray Koopman <koop...@sfu.ca> wrote:
> On Oct 29, 12:16 pm, Paul <paul.domas...@gmail.com> wrote:
>
> > On Oct 29, 2:42 am, Ray Koopman <koop...@sfu.ca> wrote:
> >> On Oct 28, 5:28 pm, Paul <paul.domas...@gmail.com> wrote:
> >>> ... I am trying to understand SVD in the context of PCA. ...
>
> >> Try Jolliffe's Principal Component Analysis (2nd ed).
>
> > I'll see what I can do about getting the book. Thanks.
>
> http://kolxo3.tiera.ru/M_Mathematics/MV_Probability/MVsa_Statistics%2...
> On Oct 29, 12:16 pm, Paul <paul.domas...@gmail.com> wrote:
>
> > On Oct 29, 2:42 am, Ray Koopman <koop...@sfu.ca> wrote:
> >> On Oct 28, 5:28 pm, Paul <paul.domas...@gmail.com> wrote:
> >>> ... I am trying to understand SVD in the context of PCA. ...
>
> >> Try Jolliffe's Principal Component Analysis (2nd ed).
>
> > I'll see what I can do about getting the book. Thanks.
>
Thanks, Ray.
It's much denser than I'm accustomed to, but Section 2.2 does describe
the interpretation of high-dimension ellipsoids of constant
probability, if the PDF of the data is mulitvariate normal. This is
pretty well the intuition from ignoring the right most orthonormal
rotation in my original post. Since the mathematical formulation is
quite dense and differs from SVD formulation that I referenced, I'm
having a hard time seeing the connection (specifically to the
rightmost orthormal rotation).
Rich Ulrich
Oct 29, 2012, 6:18:48 PM10/29/12
to
On Sun, 28 Oct 2012 17:28:37 -0700 (PDT), Paul
<paul.d...@gmail.com> wrote:
[snip, much]
For a different sort of reference -- If I recall correctly,
H. Harman's textbook on factor analysis went into more
geometrical explanations than most books I've seen. IIRC,
he talked a lot about rotations. The earlier editions were
more theoretical than the 3rd edition. He draws examples
from scales and testing.
Maybe you need some reassurance that you are, mainly,
doing okay. But you are getting hung up on details.
I no longer have trouble in saying that a vector is a data
point in n-dimensional space. It is a definition, isn't it?
When the dimensions span time, there are apt to be build-in
autocorrelations, but it is still a space.
In contrast to your examples, my own examples of PCA
have been from psychology, using people and items. The
usual process is analyze items, which is called R-type
analysis. Analyzing to get factors derived from correlations
between people is called Q-type analysis (which seems to
have disappeared from most practice and discussion).
I mention P and Q to emphasize that you *can* analyze
in either direction. Discussions in psychology (more than the
practice) also inolve time or "occasions" as a way to orient
analyses.
Hope this helps.
--
Rich Ulrich
<paul.d...@gmail.com> wrote:
[snip, much]
>
>One of the most disturbing things I haven't been able to figure out is
>what V (or Vt) corresponds to in the real world. I mean, if X was a
>transformation, then Vt is simply a rotation in n-space. But X
>*isn't* a transformation. And n-space is meaningless because we would
>never treat the vector of data from a single sensor as a data point
>(i.e., each measurement instance in time as a dimension) and plot it
>in n-dimensional space. So even though V or Vt somehow corresponds to
>a geometric rotation of sorts, it's in an space that is nonsensical
>and has no bearing in the real world.
[snip]
>One of the most disturbing things I haven't been able to figure out is
>what V (or Vt) corresponds to in the real world. I mean, if X was a
>transformation, then Vt is simply a rotation in n-space. But X
>*isn't* a transformation. And n-space is meaningless because we would
>never treat the vector of data from a single sensor as a data point
>(i.e., each measurement instance in time as a dimension) and plot it
>in n-dimensional space. So even though V or Vt somehow corresponds to
>a geometric rotation of sorts, it's in an space that is nonsensical
>and has no bearing in the real world.
For a different sort of reference -- If I recall correctly,
H. Harman's textbook on factor analysis went into more
geometrical explanations than most books I've seen. IIRC,
he talked a lot about rotations. The earlier editions were
more theoretical than the 3rd edition. He draws examples
from scales and testing.
Maybe you need some reassurance that you are, mainly,
doing okay. But you are getting hung up on details.
I no longer have trouble in saying that a vector is a data
point in n-dimensional space. It is a definition, isn't it?
When the dimensions span time, there are apt to be build-in
autocorrelations, but it is still a space.
In contrast to your examples, my own examples of PCA
have been from psychology, using people and items. The
usual process is analyze items, which is called R-type
analysis. Analyzing to get factors derived from correlations
between people is called Q-type analysis (which seems to
have disappeared from most practice and discussion).
I mention P and Q to emphasize that you *can* analyze
in either direction. Discussions in psychology (more than the
practice) also inolve time or "occasions" as a way to orient
analyses.
Hope this helps.
--
Rich Ulrich
Paul
Oct 29, 2012, 7:02:03 PM10/29/12
to
On Oct 29, 6:18 pm, Rich Ulrich <rich.ulr...@comcast.net> wrote:
> On Sun, 28 Oct 2012 17:28:37 -0700 (PDT), Paul
>
> On Sun, 28 Oct 2012 17:28:37 -0700 (PDT), Paul
>
it.
I don't need assurance that I'm doing the right thing, I just want an
intuitive understanding of the steps in PCA. To me, math is ideally
just a codification of an intuitive understanding. Of course one can
take the opposite of the intuition that motivates the problem -- that
is, in the data matrix X of my original post, treat the data samples
(each sample containing a set of readings, one per sensor from a
common instance in time) as points in 1000-dimensional space (assuming
that samples are taken from 1000 instances in time). But at an
intuitive level, I don't see the relevance of these covariances to the
aim of finding correlation between sensors, even if time-domain
structure is revealed. It must play some role, and mathematically,
that's clear. It's just not clear intuitively what is going on.
Especially when much ado is made about the eigenvectors of the
1000x1000 covariance matrix being an orthonormal rotation -- in 1000-
dimensional space, which isn't relevant to the problem.
There is also the fact that the view of that orthonormal rotation is
only valid if the transformation is applied to some vector(s) in the
1000-dimension space -- and it isn't. X and its factors are not
transformations at all because it consists of data -- it isn't applied
to anything (at least as far as the sources I've read).
Paul
Oct 29, 2012, 11:05:29 PM10/29/12
to Art Kendall
On Oct 29, 4:11 pm, Art Kendall <A...@DrKendall.org> wrote:
> This is a LISTSERV list. I.e, an email discussion list.
> After you subscribe you just send an email to
> CLA...@LISTS.SUNYSB.EDU
>
> Give that a try and let me know if you still have a problem.
>
> Art Kendall
> Social Research Consultants
OK, done. Thanks.
> This is a LISTSERV list. I.e, an email discussion list.
> After you subscribe you just send an email to
> CLA...@LISTS.SUNYSB.EDU
>
> Give that a try and let me know if you still have a problem.
>
> Art Kendall
> Social Research Consultants
Funny, but http://lists.sunysb.edu/index.cgi?A0=CLASS-L generates an
error "Server not found" even when accessed from
http://www.classification-society.org/clsoc/clsoc.php .
Art Kendall
Oct 30, 2012, 8:03:21 AM10/30/12
to
Stony Brook is on Long Island . Right now Long Island and the rest of
the New York City area is a major disaster area due to hurricane Sandy.
Many places there are without power.
I suggest you try again in a couple days.
Art Kendall
Social Research Consultants
the New York City area is a major disaster area due to hurricane Sandy.
Many places there are without power.
I suggest you try again in a couple days.
Art Kendall
Social Research Consultants
Gottfried Helms
Nov 1, 2012, 1:21:23 PM11/1/12
to
Hi Paul,
if I understood you correctly, you setup the SVD on your
X-data such that (let Z denote Sigma and W and Vt the unique rotations
which cause that Z becomes diagonal)
Z = W * X * Vt
The X-data contain n samples along the columns taken with m
sensors defining the rows (I assume, they are centered, and for
the example below that they are also standardized)
Then I understood your question that you ask, what relevance has
Vt and especially in terms of a multidimensional euclidean model.
If I understand you correctly so far, then the following might be
helpful concerning Vt.
The key is, that the n samples define m vectors in an n-dimensional
euclidean space; simply each column of X can be seen as a spatial
dimension. In that n-dimensional space there are m vectors, where
the number m is smaller than n.
Any rotation in that space repositions the vectors, but *not* the
relation, or better: the angles, between them
So we can rotate the vector model X (columnwise) first such, that
sensor 1 defines the x-axes,
sensor 2 and 1 define the x-y-plane
sensor 3 to 1 define the x-y-z-space
and so on.
In effect, that rotation provides a matrix X1 which is triangular
with as many nonzero-columns as the rank of the matrix is (and we
assume for simplicityness, that it equals m)
Then the matrix X1 can be rotated to the position of their principal
components (we're talking already of the nonzero columns only), let's
call this X2
That two rotations together form your matrix Vt.
After that, X2 can be rotated by rotation of its rows to diagonal
form - this is your rotation-matrix W, which rotates for the
principal components with respect of the rows in X2 (and which
is the same as the rotation with respect of the rows in X).
I've done this step-by-step with my matrix-calculator MatMate and
show the matrices where we have only (m=)3 sensors and (n=)6 samples.
We generate a random dataset for 3 sensors, and 6 samples, centered and
standardized normal distributed data in matrix X
[22] set randomstart=41
[23] X =zvaluezl(abwzl( randomn(3,6)))
X :
0.2096 0.2276 -2.0787 -0.1263 0.9337 0.8340
0.0668 0.0848 0.8837 -1.6367 -0.7827 1.3842
0.4677 -0.4803 0.8500 0.2395 0.9091 -1.9860
Each row defines the coordinates of one vector in the n=6 dimensional space.
Now wet get the rotation-matrix t1, which rotates X to triangular form, preserving
the angles (=cosines, correlations) between the vectors:
[24] t1 = gettrans(X,"Drei")
t1 :
0.0856 0.0449 0.3898 0.6802 -0.4701 -0.3937
0.0929 0.0538 -0.1865 -0.1958 0.3348 -0.8963
-0.8486 0.1986 0.1513 0.2615 0.3856 -0.0206
-0.0516 -0.6916 -0.5339 0.4630 0.1392 0.0151
0.3812 -0.2498 0.5843 0.1452 0.6459 0.1125
0.3405 0.6441 -0.4049 0.4418 0.2858 0.1685
and we apply that rotation to X to get the sensor's data in that rotated coordinates in matrix X1
[25] X1 = X*t1
X1 :
2.4495 0.0000 0.0000 0.0000 0.0000 0.0000
-0.4790 2.4022 0.0000 -0.0000 0.0000 0.0000
-1.0680 -1.5080 1.6079 0.0000 -0.0000 0.0000
******************************************** | -------------------------------------------
we see, that we need only a m=3-dimensional space to account for coordinates
of 3 (linearly independent) vectors.
Now we rotate that coordinates into the "principal components" position, which is defined
such that the first column defines an axis on which the maximal possible sum of squares
of coordinates occur. The required rotation-matrix t2 has significant entries only in
the top-left mxm submatrix:
[26] t2 = gettrans(X1,"pca",1..3,1..3])
t2 :
0.4781 -0.8458 0.2367 0.0000 0.0000 0.0000
0.7858 0.5323 0.3149 0.0000 0.0000 0.0000
-0.3923 0.0354 0.9191 0.0000 0.0000 0.0000
0.0000 0.0000 0.0000 1.0000 0.0000 0.0000
0.0000 0.0000 0.0000 0.0000 1.0000 0.0000
0.0000 0.0000 0.0000 0.0000 0.0000 1.0000
We aply that rotation by postmultiplication to get the coordinates for the sensors in X2,
where the axis are such that they represent the principal components:
[27] X2 = X1*t2
X2 :
1.1712 -2.0718 0.5797 0.0000 0.0000 0.0000
1.6586 1.6839 0.6431 -0.0000 0.0000 0.0000
-2.3264 0.1575 0.7503 0.0000 -0.0000 0.0000
We check the sum-of-squares of the coordinates along each column:
[32] SSQ = sqsumsp(X2)
SSQ :
9.5348 7.1526 1.3126 0.0000 0.0000 0.0000
and indeed the axes 1 to 3 have the required relation.
Now we reproduce the matrix Vt of your question by chaining the two rotations which is
just a matrix-multiplication. Vt is thus also a rotation!
[28] Vt = t1*t2
Vt :
-0.0768 -0.0347 0.3927 0.6802 -0.4701 -0.3937
0.1599 -0.0566 -0.1325 -0.1958 0.3348 -0.8963
-0.3090 0.8289 0.0008 0.2615 0.3856 -0.0206
-0.3586 -0.3435 -0.7207 0.4630 0.1392 0.0151
-0.2433 -0.4347 0.5486 0.1452 0.6459 0.1125
0.8278 0.0406 -0.0888 0.4418 0.2858 0.1685
... and apply this directly to X to get Y:
[29] Y = X*Vt
Y :
1.1712 -2.0718 0.5797 0.0000 0.0000 0.0000
1.6586 1.6839 0.6431 -0.0000 0.0000 0.0000
-2.3264 0.1575 0.7503 0.0000 -0.0000 0.0000
which gives now the same result as before ( Y = X2).
Now we get the rotation-matrix W, which rotates Y (or X, which were the same) to its
principal components position, but row-wise:
[30] W = gettrans(Y',"pca")'
W :
-0.3793 -0.5371 0.7534
0.7747 -0.6296 -0.0589
0.5060 0.5613 0.6549
and find, that the complete matrix-product results in a diagonal matrix (in the relevant
leftmost columns only) :
[31] Z = W*Y
Z :
-3.0879 -0.0000 0.0000 0.0000 -0.0000 -0.0000
0.0000 -2.6744 0.0000 0.0000 0.0000 0.0000
0.0000 0.0000 1.1457 -0.0000 -0.0000 0.0000
Note, that the rotation-criterion "principal components" ("pca") is another expression
for the goal, that the columns (or rows) become mutually orthogonal.
Gottfried Helms
if I understood you correctly, you setup the SVD on your
X-data such that (let Z denote Sigma and W and Vt the unique rotations
which cause that Z becomes diagonal)
Z = W * X * Vt
The X-data contain n samples along the columns taken with m
sensors defining the rows (I assume, they are centered, and for
the example below that they are also standardized)
Then I understood your question that you ask, what relevance has
Vt and especially in terms of a multidimensional euclidean model.
If I understand you correctly so far, then the following might be
helpful concerning Vt.
The key is, that the n samples define m vectors in an n-dimensional
euclidean space; simply each column of X can be seen as a spatial
dimension. In that n-dimensional space there are m vectors, where
the number m is smaller than n.
Any rotation in that space repositions the vectors, but *not* the
relation, or better: the angles, between them
So we can rotate the vector model X (columnwise) first such, that
sensor 1 defines the x-axes,
sensor 2 and 1 define the x-y-plane
sensor 3 to 1 define the x-y-z-space
and so on.
In effect, that rotation provides a matrix X1 which is triangular
with as many nonzero-columns as the rank of the matrix is (and we
assume for simplicityness, that it equals m)
Then the matrix X1 can be rotated to the position of their principal
components (we're talking already of the nonzero columns only), let's
call this X2
That two rotations together form your matrix Vt.
After that, X2 can be rotated by rotation of its rows to diagonal
form - this is your rotation-matrix W, which rotates for the
principal components with respect of the rows in X2 (and which
is the same as the rotation with respect of the rows in X).
I've done this step-by-step with my matrix-calculator MatMate and
show the matrices where we have only (m=)3 sensors and (n=)6 samples.
We generate a random dataset for 3 sensors, and 6 samples, centered and
standardized normal distributed data in matrix X
[22] set randomstart=41
[23] X =zvaluezl(abwzl( randomn(3,6)))
X :
0.2096 0.2276 -2.0787 -0.1263 0.9337 0.8340
0.0668 0.0848 0.8837 -1.6367 -0.7827 1.3842
0.4677 -0.4803 0.8500 0.2395 0.9091 -1.9860
Each row defines the coordinates of one vector in the n=6 dimensional space.
Now wet get the rotation-matrix t1, which rotates X to triangular form, preserving
the angles (=cosines, correlations) between the vectors:
[24] t1 = gettrans(X,"Drei")
t1 :
0.0856 0.0449 0.3898 0.6802 -0.4701 -0.3937
0.0929 0.0538 -0.1865 -0.1958 0.3348 -0.8963
-0.8486 0.1986 0.1513 0.2615 0.3856 -0.0206
-0.0516 -0.6916 -0.5339 0.4630 0.1392 0.0151
0.3812 -0.2498 0.5843 0.1452 0.6459 0.1125
0.3405 0.6441 -0.4049 0.4418 0.2858 0.1685
and we apply that rotation to X to get the sensor's data in that rotated coordinates in matrix X1
[25] X1 = X*t1
X1 :
2.4495 0.0000 0.0000 0.0000 0.0000 0.0000
-0.4790 2.4022 0.0000 -0.0000 0.0000 0.0000
-1.0680 -1.5080 1.6079 0.0000 -0.0000 0.0000
******************************************** | -------------------------------------------
we see, that we need only a m=3-dimensional space to account for coordinates
of 3 (linearly independent) vectors.
Now we rotate that coordinates into the "principal components" position, which is defined
such that the first column defines an axis on which the maximal possible sum of squares
of coordinates occur. The required rotation-matrix t2 has significant entries only in
the top-left mxm submatrix:
[26] t2 = gettrans(X1,"pca",1..3,1..3])
t2 :
0.4781 -0.8458 0.2367 0.0000 0.0000 0.0000
0.7858 0.5323 0.3149 0.0000 0.0000 0.0000
-0.3923 0.0354 0.9191 0.0000 0.0000 0.0000
0.0000 0.0000 0.0000 1.0000 0.0000 0.0000
0.0000 0.0000 0.0000 0.0000 1.0000 0.0000
0.0000 0.0000 0.0000 0.0000 0.0000 1.0000
We aply that rotation by postmultiplication to get the coordinates for the sensors in X2,
where the axis are such that they represent the principal components:
[27] X2 = X1*t2
X2 :
1.1712 -2.0718 0.5797 0.0000 0.0000 0.0000
1.6586 1.6839 0.6431 -0.0000 0.0000 0.0000
-2.3264 0.1575 0.7503 0.0000 -0.0000 0.0000
We check the sum-of-squares of the coordinates along each column:
[32] SSQ = sqsumsp(X2)
SSQ :
9.5348 7.1526 1.3126 0.0000 0.0000 0.0000
and indeed the axes 1 to 3 have the required relation.
Now we reproduce the matrix Vt of your question by chaining the two rotations which is
just a matrix-multiplication. Vt is thus also a rotation!
[28] Vt = t1*t2
Vt :
-0.0768 -0.0347 0.3927 0.6802 -0.4701 -0.3937
0.1599 -0.0566 -0.1325 -0.1958 0.3348 -0.8963
-0.3090 0.8289 0.0008 0.2615 0.3856 -0.0206
-0.3586 -0.3435 -0.7207 0.4630 0.1392 0.0151
-0.2433 -0.4347 0.5486 0.1452 0.6459 0.1125
0.8278 0.0406 -0.0888 0.4418 0.2858 0.1685
... and apply this directly to X to get Y:
[29] Y = X*Vt
Y :
1.1712 -2.0718 0.5797 0.0000 0.0000 0.0000
1.6586 1.6839 0.6431 -0.0000 0.0000 0.0000
-2.3264 0.1575 0.7503 0.0000 -0.0000 0.0000
which gives now the same result as before ( Y = X2).
Now we get the rotation-matrix W, which rotates Y (or X, which were the same) to its
principal components position, but row-wise:
[30] W = gettrans(Y',"pca")'
W :
-0.3793 -0.5371 0.7534
0.7747 -0.6296 -0.0589
0.5060 0.5613 0.6549
and find, that the complete matrix-product results in a diagonal matrix (in the relevant
leftmost columns only) :
[31] Z = W*Y
Z :
-3.0879 -0.0000 0.0000 0.0000 -0.0000 -0.0000
0.0000 -2.6744 0.0000 0.0000 0.0000 0.0000
0.0000 0.0000 1.1457 -0.0000 -0.0000 0.0000
Note, that the rotation-criterion "principal components" ("pca") is another expression
for the goal, that the columns (or rows) become mutually orthogonal.
Gottfried Helms
Paul
Nov 1, 2012, 2:43:29 PM11/1/12
to
On Oct 30, 8:03 am, Art Kendall <A...@DrKendall.org> wrote:
> Stony Brook is on Long Island . Right now Long Island and the rest
> of the New York City area is a major disaster area due to hurricane
> Sandy. Many places there are without power. I suggest you try
> again in a couple days.
I had my head out of the news until recently. Wow.
> Stony Brook is on Long Island . Right now Long Island and the rest
> of the New York City area is a major disaster area due to hurricane
> Sandy. Many places there are without power. I suggest you try
> again in a couple days.
>On 10/29/2012 11:05 PM, Paul wrote:
>>On Oct 29, 4:11 pm, Art Kendall <A...@DrKendall.org> wrote:
>>> This is a LISTSERV list. I.e, an email discussion list. After you
>>>
>>> Give that a try and let me know if you still have a problem.
>>
>> OK, done. Thanks.
>>> Give that a try and let me know if you still have a problem.
>>
>>
>> Funny, buthttp://lists.sunysb.edu/index.cgi?A0=CLASS-Lgenerates an
Paul
Nov 1, 2012, 3:53:39 PM11/1/12
to
I might be missing some linear algebra theory here, but I looked up
gettrans() and I'm not sure what is meant by a column rotation in that
context.
Please see below for further comments.
On Nov 1, 1:21 pm, Gottfried Helms <he...@uni-kassel.de> wrote:
> Hi Paul,
>
> if I understood you correctly, you setup the SVD on your X-data such
> that (let Z denote Sigma and W and Vt the unique rotations which
> cause that Z becomes diagonal)
>
> Z = W * X * Vt
>
> The X-data contain n samples along the columns taken with m sensors
> defining the rows (I assume, they are centered, and for the example
> below that they are also standardized)
>
> Then I understood your question that you ask, what relevance has Vt
> and especially in terms of a multidimensional euclidean model.
>
> If I understand you correctly so far, then the following might be
> helpful concerning Vt.
>
> The key is, that the n samples define m vectors in an n-dimensional
> euclidean space; simply each column of X can be seen as a spatial
> dimension. In that n-dimensional space there are m vectors, where
> the number m is smaller than n. Any rotation in that space
> repositions the vectors, but *not* the relation, or better: the
> angles, between them
I'm not sure why *any* rotation in n-space would not preserve angles.
I thought that a rotation is by definition a unitary transformation
(from a recent brush-up on linear algebra at Wikipedia e.g.
http://en.wikipedia.org/wiki/Orthogonal_matrix).
> ...So we can rotate the vector model X
columnwise". If you interpret each column of X as a point (or vector)
in n-space, we get what you describe (sensor 1 is the x-axis, sensor 2
is the y-axis, etc.). However, a rotation is not needed for this.
> In effect, that rotation provides a matrix X1 which is triangular
> with as many nonzero-columns as the rank of the matrix is (and we
> assume for simplicityness, that it equals m)
I think I'm missing something fundamental...the data matrix is not
triangular, though the (n)x(n) covariance matrix (Xt)(X) is symmetric.
> Then the matrix X1 can be rotated to the position of their principal
> components (we're talking already of the nonzero columns only),
> let's call this X2
I see that the data must be rotated so that the principal axes align
with the axes of m-space (not n-space), and then the diagonal matrix
Sigma performs the anisotropic axial stretching.
> That two rotations together form your matrix Vt. After that, X2 can
> be rotated by rotation of its rows to diagonal form - this is your
> rotation-matrix W, which rotates for the principal components with
> respect of the rows in X2 (and which is the same as the rotation
> with respect of the rows in X).
But W is not applied after Vt, Sigma is (the anisotropic axial
scaling). After that, however, I see that W does rotate the axially
scaled body of data points so that maximally stretched axis becomes
aligned with the principal component of the measured data, the 2nd
most stretched axis aligns with the 2nd principal component, etc.
So the rotation by W is very intuitive to me, while the rotation by Vt
is not. And as I described, it's all the more mysterious when you
consider that X isn't actually a transformation that is applied to
data -- it *is* the data. For this reason, I find it difficult to see
the decomposition of X as a series of transformations (rotate,
stretch, rotate) despite the intuitive appeal of the (W)(Sigma) (there
is no intuition on my part concerning Vt).
This inability to picture (X)(Sigma)(Vt) as a transformation shows up
particularly in my lack of intuition concerning Vt...it is the first
of the 3 decomposed transformations that gets applied to any data
point/vector that is subjected to the 3-step stransformation. The
question is "What is this data that gets subjected to this
transformation? And in n-space, no less". It seems that the 3-step
transformation and the data are the same!
Furthermore, when I am seeking correlation between the m sensors, it
confounds me to think about why one would picture the data points in n-
space. As an analogy, if I am doing simple linear regression on a
cloud of 1000 points in the x-y plane, I don't try to picture the data
points in 1000-dimension space.
> I've done this step-by-step with my matrix-calculator MatMate and
> show the matrices where we have only (m=)3 sensors and (n=)6
> samples.
>
> We generate a random dataset for 3 sensors, and 6 samples, centered
> and standardized normal distributed data in matrix X
> [22] set randomstart=41
> [23] X =zvaluezl(abwzl( randomn(3,6)))
> X :
> 0.2096 0.2276 -2.0787 -0.1263 0.9337 0.8340
> 0.0668 0.0848 0.8837 -1.6367 -0.7827 1.3842
> 0.4677 -0.4803 0.8500 0.2395 0.9091 -1.9860
>
> Each row defines the coordinates of one vector in the n=6
> dimensional space. Now wet get the rotation-matrix t1, which
> rotates X to triangular form, preserving the angles (=cosines,
> correlations) between the vectors:
> [24] t1 = gettrans(X,"Drei")
> t1 :
> 0.0856 0.0449 0.3898 0.6802 -0.4701 -0.3937
> 0.0929 0.0538 -0.1865 -0.1958 0.3348 -0.8963
> -0.8486 0.1986 0.1513 0.2615 0.3856 -0.0206
> -0.0516 -0.6916 -0.5339 0.4630 0.1392 0.0151
> 0.3812 -0.2498 0.5843 0.1452 0.6459 0.1125
> 0.3405 0.6441 -0.4049 0.4418 0.2858 0.1685
Sorry, I tried to google gettrans, but wasn't able to find much beyond
the fact that it is a column rotation. It's not clear to me what is
meant by that. Consequently, I wasn't able to follow the rest of the
example.
However, I appreciate the time that you took to compose the attempted
explanation.
gettrans() and I'm not sure what is meant by a column rotation in that
context.
Please see below for further comments.
On Nov 1, 1:21 pm, Gottfried Helms <he...@uni-kassel.de> wrote:
> Hi Paul,
>
> if I understood you correctly, you setup the SVD on your X-data such
> that (let Z denote Sigma and W and Vt the unique rotations which
> cause that Z becomes diagonal)
>
> Z = W * X * Vt
>
> The X-data contain n samples along the columns taken with m sensors
> defining the rows (I assume, they are centered, and for the example
> below that they are also standardized)
>
> Then I understood your question that you ask, what relevance has Vt
> and especially in terms of a multidimensional euclidean model.
>
> If I understand you correctly so far, then the following might be
> helpful concerning Vt.
>
> The key is, that the n samples define m vectors in an n-dimensional
> euclidean space; simply each column of X can be seen as a spatial
> dimension. In that n-dimensional space there are m vectors, where
> the number m is smaller than n. Any rotation in that space
> repositions the vectors, but *not* the relation, or better: the
> angles, between them
I thought that a rotation is by definition a unitary transformation
(from a recent brush-up on linear algebra at Wikipedia e.g.
http://en.wikipedia.org/wiki/Orthogonal_matrix).
> ...So we can rotate the vector model X
> (columnwise) first such, that
> sensor 1 defines the x-axes,
> sensor 2 and 1 define the x-y-plane
> sensor 3 to 1 define the x-y-z-space
> and so on.
I don't quite follow what you mean by "rotat[ing] the vector [model] X
> sensor 1 defines the x-axes,
> sensor 2 and 1 define the x-y-plane
> sensor 3 to 1 define the x-y-z-space
> and so on.
columnwise". If you interpret each column of X as a point (or vector)
in n-space, we get what you describe (sensor 1 is the x-axis, sensor 2
is the y-axis, etc.). However, a rotation is not needed for this.
> In effect, that rotation provides a matrix X1 which is triangular
> with as many nonzero-columns as the rank of the matrix is (and we
> assume for simplicityness, that it equals m)
triangular, though the (n)x(n) covariance matrix (Xt)(X) is symmetric.
> Then the matrix X1 can be rotated to the position of their principal
> components (we're talking already of the nonzero columns only),
> let's call this X2
with the axes of m-space (not n-space), and then the diagonal matrix
Sigma performs the anisotropic axial stretching.
> That two rotations together form your matrix Vt. After that, X2 can
> be rotated by rotation of its rows to diagonal form - this is your
> rotation-matrix W, which rotates for the principal components with
> respect of the rows in X2 (and which is the same as the rotation
> with respect of the rows in X).
scaling). After that, however, I see that W does rotate the axially
scaled body of data points so that maximally stretched axis becomes
aligned with the principal component of the measured data, the 2nd
most stretched axis aligns with the 2nd principal component, etc.
So the rotation by W is very intuitive to me, while the rotation by Vt
is not. And as I described, it's all the more mysterious when you
consider that X isn't actually a transformation that is applied to
data -- it *is* the data. For this reason, I find it difficult to see
the decomposition of X as a series of transformations (rotate,
stretch, rotate) despite the intuitive appeal of the (W)(Sigma) (there
is no intuition on my part concerning Vt).
This inability to picture (X)(Sigma)(Vt) as a transformation shows up
particularly in my lack of intuition concerning Vt...it is the first
of the 3 decomposed transformations that gets applied to any data
point/vector that is subjected to the 3-step stransformation. The
question is "What is this data that gets subjected to this
transformation? And in n-space, no less". It seems that the 3-step
transformation and the data are the same!
Furthermore, when I am seeking correlation between the m sensors, it
confounds me to think about why one would picture the data points in n-
space. As an analogy, if I am doing simple linear regression on a
cloud of 1000 points in the x-y plane, I don't try to picture the data
points in 1000-dimension space.
> I've done this step-by-step with my matrix-calculator MatMate and
> show the matrices where we have only (m=)3 sensors and (n=)6
> samples.
>
> We generate a random dataset for 3 sensors, and 6 samples, centered
> and standardized normal distributed data in matrix X
> [22] set randomstart=41
> [23] X =zvaluezl(abwzl( randomn(3,6)))
> X :
> 0.2096 0.2276 -2.0787 -0.1263 0.9337 0.8340
> 0.0668 0.0848 0.8837 -1.6367 -0.7827 1.3842
> 0.4677 -0.4803 0.8500 0.2395 0.9091 -1.9860
>
> Each row defines the coordinates of one vector in the n=6
> dimensional space. Now wet get the rotation-matrix t1, which
> rotates X to triangular form, preserving the angles (=cosines,
> correlations) between the vectors:
> [24] t1 = gettrans(X,"Drei")
> t1 :
> 0.0856 0.0449 0.3898 0.6802 -0.4701 -0.3937
> 0.0929 0.0538 -0.1865 -0.1958 0.3348 -0.8963
> -0.8486 0.1986 0.1513 0.2615 0.3856 -0.0206
> -0.0516 -0.6916 -0.5339 0.4630 0.1392 0.0151
> 0.3812 -0.2498 0.5843 0.1452 0.6459 0.1125
> 0.3405 0.6441 -0.4049 0.4418 0.2858 0.1685
the fact that it is a column rotation. It's not clear to me what is
meant by that. Consequently, I wasn't able to follow the rest of the
example.
However, I appreciate the time that you took to compose the attempted
explanation.
Gottfried Helms
Nov 2, 2012, 3:02:11 AM11/2/12
to
Hi Paul -
it seems I made my comment more complicated than the procedure is.
Am 01.11.2012 20:53 schrieb Paul:
> I might be missing some linear algebra theory here, but I looked up
> gettrans() and I'm not sure what is meant by a column rotation in that
> context.
No, gettrans is just a function-call in my MatMate-script-language,
which returns a rotation-matrix. For instance, by the command:
t1 = gettrans(X,"drei") // "drei" means "triangular"
t1 becomes the rotation-matrix, which is required to rotate columnwise
X to triangular shape. After that we can do the following with t1:
Y = X * t1 // Y is a lower triangular matrix, (with possibly empty columns to the right
Z = Y * t1' // Z equals now X, because t1*t1' = I (Identitymatrix)
or, for doing roation to principals components position:
t2 = gettrans(X,"pc") // "pc" means "principal components"
and then
B = X * t2 // the columns of B are now orthogonal, are the principal components
I've introduced that function "gettrans" additionally to the simple "rotate"-
function to have the rotation-matrix available for later manipulation, or
to be able to reverse a rotation later or to apply the same rotation to another
matrix etc. It can also be made to work only on certain columns and using
only certain rows for the criterion; This is then useful, if one uses
rotations, which are implemented as iterative procedures like "pc" or
"varimax" or similar.
>> The key is, that the n samples define m vectors in an n-dimensional
>> euclidean space; simply each column of X can be seen as a spatial
>> dimension. In that n-dimensional space there are m vectors, where
>> the number m is smaller than n. Any rotation in that space
>> repositions the vectors, but *not* the relation, or better: the
>> angles, between them
>
> I'm not sure why *any* rotation in n-space would not preserve angles.
> I thought that a rotation is by definition a unitary transformation
> (from a recent brush-up on linear algebra at Wikipedia e.g.
> http://en.wikipedia.org/wiki/Orthogonal_matrix).
My remark may be obfuscating here. There is the concept of "oblique
rotations" in factor analysis (as opposed to orthogonal rotations)
which do not preserve the angles - and I had the impulse to exclude this
case verbally... So this remark could just be deleted
>
>> ...So we can rotate the vector model X
>> (columnwise) first such, that
>> sensor 1 defines the x-axes,
>> sensor 2 and 1 define the x-y-plane
>> sensor 3 to 1 define the x-y-z-space
>> and so on.
>
> I don't quite follow what you mean by "rotat[ing] the vector [model] X
> columnwise". If you interpret each column of X as a point (or vector)
> in n-space, we get what you describe (sensor 1 is the x-axis, sensor 2
> is the y-axis, etc.). However, a rotation is not needed for this.
If we speak of the n-dimensional space, each column represent the
coordinates on one axis. Then each row represents one vector
(from the origin) to some point in this n-dimensional space: for
each sensor there is one wire from the origin into the n-space,
and the angles between that wires (more precisely: the cosines of that
angles) are expressed by the correlation-coefficients.
That view of statistical data may be somehow unusual - but it is
coherent with the operations of rotations and the finding of
principal components - and this is what your matrix Vt stands
for.
>
>> In effect, that rotation provides a matrix X1 which is triangular
>> with as many nonzero-columns as the rank of the matrix is (and we
>> assume for simplicityness, that it equals m)
>
> I think I'm missing something fundamental...the data matrix is not
> triangular, though the (n)x(n) covariance matrix (Xt)(X) is symmetric.
No, not the data matrix X. But after X is rotated to triangular
position by t1 then
X1 = X * t1
is lower triangular (with some empty columns due to the defective rank
of X)
>
>> Then the matrix X1 can be rotated to the position of their principal
>> components (we're talking already of the nonzero columns only),
>> let's call this X2
>
> I see that the data must be rotated so that the principal axes align
> with the axes of m-space (not n-space), and then the diagonal matrix
> Sigma performs the anisotropic axial stretching.
No, again we rotate in the columns/the n-space. Just we apply the
(costly because of iterations) rotation to orthogonality (which
gives principal components) only to the first m axes in X1 (which is
already triangular with only m significant columns)
X2 = X1 * t2
or equivalently
X2 = X * t1 * t2 = X * (t1 * t2) = X * Vt
After that X2 contains the coordinates of your sensor-measures
after rotation in the n-space in such a way that in the first
column the sum of squared coordinates is the maximum possible
and in the m'th column the least possible and because
X2 ' * X2 is diagonal we may say, that the columns are orthogonal
>
>> That two rotations together form your matrix Vt. After that, X2 can
>> be rotated by rotation of its rows to diagonal form - this is your
>> rotation-matrix W, which rotates for the principal components with
>> respect of the rows in X2 (and which is the same as the rotation
>> with respect of the rows in X).
>
> But W is not applied after Vt,
???
If we have
W * (X * Vt)
which is meant when I say that W is applied "after" the rotation by Vt
in my example....
>
> So the rotation by W is very intuitive to me, while the rotation by Vt
> is not. And as I described, it's all the more mysterious when you
> consider that X isn't actually a transformation that is applied to
> data -- it *is* the data.
This remark "... isn't actually a transformation..." confuses now
me. ;-) Well, I understood X as data as well, I have no idea, where
the idea of "being a transformation" comes from and what I am
possibly missing here. Very likely I didn't properly catch your way of
approaching the problem...
--------------------------------------------------------
(...)
just reflects a possibilitywhich occurs when we look at the correlation
matrix and its cholesky-factors. Say, with our m x n -datamatrix X
(I use the '-apostroph for transposition)
R = X * X' / n // R is the m x m correlation-matrix
then we have also with some rotation W
Z = W * R * W' // Z = Sigma = diagonal
but also, if we see R in its cholesky-factors L and L'
Z = W * (L * L') * W' // Z = Sigma = diagonal
and because any rotation-matrix t postmultiplied with its transpose is the
identity
Z = W * (L * I * L') * W' = W * (L * t * t' * L') * W'
Now L is usually taken as m x m matrix as well, but there is no
problem to expand it by empty columns to make a m x n matrix
out of it and then to assume t such that
L * t = X / sqrt(n)
and then rewrite:
Z = W * (L * t * t' * L') * W' = W * (X * t' * t * X')/n * W'
where again (X * t' * t * X')/n = X * X' /n = R shows the
identity of the solutions.
>> [24] t1 = gettrans(X,"Drei")
>> t1 :
>> 0.0856 0.0449 0.3898 0.6802 -0.4701 -0.3937
>> 0.0929 0.0538 -0.1865 -0.1958 0.3348 -0.8963
>> -0.8486 0.1986 0.1513 0.2615 0.3856 -0.0206
>> -0.0516 -0.6916 -0.5339 0.4630 0.1392 0.0151
>> 0.3812 -0.2498 0.5843 0.1452 0.6459 0.1125
>> 0.3405 0.6441 -0.4049 0.4418 0.2858 0.1685
>
> Sorry, I tried to google gettrans, but wasn't able to find much beyond
> the fact that it is a column rotation. It's not clear to me what is
> meant by that. Consequently, I wasn't able to follow the rest of the
> example.
With the given parameters X and "Drei" (="triangular") it calls the procedure,
which returns that rotation-matrix, which can rotate X to lower triangular shape.
Having it stored as an explicite matrix we can apply this rotation and also
revert it and furtherly do anything we want with it.
If you are using windows, you can even download that MatMate-program and
do the steps yourself (and possibly experiment further) See my software-pages
http://go.helms-net.de/sw/matmate . It's an amateurish program, however
working nice for me, but if some installation problems occur (which is
easily possible) let me know.
Gottfried
it seems I made my comment more complicated than the procedure is.
Am 01.11.2012 20:53 schrieb Paul:
> I might be missing some linear algebra theory here, but I looked up
> gettrans() and I'm not sure what is meant by a column rotation in that
> context.
which returns a rotation-matrix. For instance, by the command:
t1 = gettrans(X,"drei") // "drei" means "triangular"
t1 becomes the rotation-matrix, which is required to rotate columnwise
X to triangular shape. After that we can do the following with t1:
Y = X * t1 // Y is a lower triangular matrix, (with possibly empty columns to the right
Z = Y * t1' // Z equals now X, because t1*t1' = I (Identitymatrix)
or, for doing roation to principals components position:
t2 = gettrans(X,"pc") // "pc" means "principal components"
and then
B = X * t2 // the columns of B are now orthogonal, are the principal components
I've introduced that function "gettrans" additionally to the simple "rotate"-
function to have the rotation-matrix available for later manipulation, or
to be able to reverse a rotation later or to apply the same rotation to another
matrix etc. It can also be made to work only on certain columns and using
only certain rows for the criterion; This is then useful, if one uses
rotations, which are implemented as iterative procedures like "pc" or
"varimax" or similar.
>> The key is, that the n samples define m vectors in an n-dimensional
>> euclidean space; simply each column of X can be seen as a spatial
>> dimension. In that n-dimensional space there are m vectors, where
>> the number m is smaller than n. Any rotation in that space
>> repositions the vectors, but *not* the relation, or better: the
>> angles, between them
>
> I'm not sure why *any* rotation in n-space would not preserve angles.
> I thought that a rotation is by definition a unitary transformation
> (from a recent brush-up on linear algebra at Wikipedia e.g.
> http://en.wikipedia.org/wiki/Orthogonal_matrix).
rotations" in factor analysis (as opposed to orthogonal rotations)
which do not preserve the angles - and I had the impulse to exclude this
case verbally... So this remark could just be deleted
>
>> ...So we can rotate the vector model X
>> (columnwise) first such, that
>> sensor 1 defines the x-axes,
>> sensor 2 and 1 define the x-y-plane
>> sensor 3 to 1 define the x-y-z-space
>> and so on.
>
> I don't quite follow what you mean by "rotat[ing] the vector [model] X
> columnwise". If you interpret each column of X as a point (or vector)
> in n-space, we get what you describe (sensor 1 is the x-axis, sensor 2
> is the y-axis, etc.). However, a rotation is not needed for this.
coordinates on one axis. Then each row represents one vector
(from the origin) to some point in this n-dimensional space: for
each sensor there is one wire from the origin into the n-space,
and the angles between that wires (more precisely: the cosines of that
angles) are expressed by the correlation-coefficients.
That view of statistical data may be somehow unusual - but it is
coherent with the operations of rotations and the finding of
principal components - and this is what your matrix Vt stands
for.
>
>> In effect, that rotation provides a matrix X1 which is triangular
>> with as many nonzero-columns as the rank of the matrix is (and we
>> assume for simplicityness, that it equals m)
>
> I think I'm missing something fundamental...the data matrix is not
> triangular, though the (n)x(n) covariance matrix (Xt)(X) is symmetric.
position by t1 then
X1 = X * t1
is lower triangular (with some empty columns due to the defective rank
of X)
>
>> Then the matrix X1 can be rotated to the position of their principal
>> components (we're talking already of the nonzero columns only),
>> let's call this X2
>
> I see that the data must be rotated so that the principal axes align
> with the axes of m-space (not n-space), and then the diagonal matrix
> Sigma performs the anisotropic axial stretching.
(costly because of iterations) rotation to orthogonality (which
gives principal components) only to the first m axes in X1 (which is
already triangular with only m significant columns)
X2 = X1 * t2
or equivalently
X2 = X * t1 * t2 = X * (t1 * t2) = X * Vt
After that X2 contains the coordinates of your sensor-measures
after rotation in the n-space in such a way that in the first
column the sum of squared coordinates is the maximum possible
and in the m'th column the least possible and because
X2 ' * X2 is diagonal we may say, that the columns are orthogonal
>
>> That two rotations together form your matrix Vt. After that, X2 can
>> be rotated by rotation of its rows to diagonal form - this is your
>> rotation-matrix W, which rotates for the principal components with
>> respect of the rows in X2 (and which is the same as the rotation
>> with respect of the rows in X).
>
> But W is not applied after Vt,
If we have
W * X * Vt
we can also write
W * (X * Vt)
which is meant when I say that W is applied "after" the rotation by Vt
in my example....
>
> So the rotation by W is very intuitive to me, while the rotation by Vt
> is not. And as I described, it's all the more mysterious when you
> consider that X isn't actually a transformation that is applied to
> data -- it *is* the data.
me. ;-) Well, I understood X as data as well, I have no idea, where
the idea of "being a transformation" comes from and what I am
possibly missing here. Very likely I didn't properly catch your way of
approaching the problem...
--------------------------------------------------------
(...)
> Furthermore, when I am seeking correlation between the m sensors, it
> confounds me to think about why one would picture the data points in n-
> space. As an analogy, if I am doing simple linear regression on a
> cloud of 1000 points in the x-y plane, I don't try to picture the data
> points in 1000-dimension space.
Well, we might say, such a concept is superfluous, not needed. It
> confounds me to think about why one would picture the data points in n-
> space. As an analogy, if I am doing simple linear regression on a
> cloud of 1000 points in the x-y plane, I don't try to picture the data
> points in 1000-dimension space.
just reflects a possibilitywhich occurs when we look at the correlation
matrix and its cholesky-factors. Say, with our m x n -datamatrix X
(I use the '-apostroph for transposition)
R = X * X' / n // R is the m x m correlation-matrix
then we have also with some rotation W
Z = W * R * W' // Z = Sigma = diagonal
but also, if we see R in its cholesky-factors L and L'
Z = W * (L * L') * W' // Z = Sigma = diagonal
and because any rotation-matrix t postmultiplied with its transpose is the
identity
Z = W * (L * I * L') * W' = W * (L * t * t' * L') * W'
Now L is usually taken as m x m matrix as well, but there is no
problem to expand it by empty columns to make a m x n matrix
out of it and then to assume t such that
L * t = X / sqrt(n)
and then rewrite:
Z = W * (L * t * t' * L') * W' = W * (X * t' * t * X')/n * W'
where again (X * t' * t * X')/n = X * X' /n = R shows the
identity of the solutions.
>> [24] t1 = gettrans(X,"Drei")
>> t1 :
>> 0.0856 0.0449 0.3898 0.6802 -0.4701 -0.3937
>> 0.0929 0.0538 -0.1865 -0.1958 0.3348 -0.8963
>> -0.8486 0.1986 0.1513 0.2615 0.3856 -0.0206
>> -0.0516 -0.6916 -0.5339 0.4630 0.1392 0.0151
>> 0.3812 -0.2498 0.5843 0.1452 0.6459 0.1125
>> 0.3405 0.6441 -0.4049 0.4418 0.2858 0.1685
>
> Sorry, I tried to google gettrans, but wasn't able to find much beyond
> the fact that it is a column rotation. It's not clear to me what is
> meant by that. Consequently, I wasn't able to follow the rest of the
> example.
which returns that rotation-matrix, which can rotate X to lower triangular shape.
Having it stored as an explicite matrix we can apply this rotation and also
revert it and furtherly do anything we want with it.
If you are using windows, you can even download that MatMate-program and
do the steps yourself (and possibly experiment further) See my software-pages
http://go.helms-net.de/sw/matmate . It's an amateurish program, however
working nice for me, but if some installation problems occur (which is
easily possible) let me know.
Gottfried
Paul
Nov 4, 2012, 2:04:43 PM11/4/12
to
Hi, Gottfried,
Thanks again for taking the time to write such a detailed response. I
gave it a quice twice-over, but it needs more than that. I'll get
back to it soon, but I got some preliminary comments.
On Nov 2, 2:02 am, Gottfried Helms <he...@uni-kassel.de> wrote:
> it seems I made my comment more complicated than the procedure is.
>
> Am 01.11.2012 20:53 schrieb Paul:
>
>> I might be missing some linear algebra theory here, but I looked up
>> gettrans() and I'm not sure what is meant by a column rotation in
>> that context.
>
> No, gettrans is just a function-call in my MatMate-script-language,
> which returns a rotation-matrix. For instance, by the command:
>
> t1 = gettrans(X,"drei") // "drei" means "triangular"
>
> t1 becomes the rotation-matrix, which is required to rotate
> columnwise...
I am still not sure what is a columnwise rotation. Do you actually
switch columns around, or is it more like a geometric rotation?
> ...X to triangular shape. After that we can do the following
What is meant by rotating to triangular position? Do you mean
geometric position, or that X somehow becomes a triangular matrix by
rearranging its columns? What if there are not enough properly placed
zeros for that to be possible?
I got lost...the middle matrix should be Sigma, a diagonal matrix of
Eigenvalues.
>> So the rotation by W is very intuitive to me, while the rotation by
>> Vt is not. And as I described, it's all the more mysterious when
>> you consider that X isn't actually a transformation that is applied
>> to data -- it *is* the data.
>
> This remark "... isn't actually a transformation..." confuses now
> me. ;-) Well, I understood X as data as well, I have no idea, where
> the idea of "being a transformation" comes from and what I am
> possibly missing here. Very likely I didn't properly catch your way
> of approaching the problem...
That's the view of X = W Sigma Vt. Sigma is an anisotropic axial
stretch while W rotates these stretch axes to the principal components
of the data in m-space. What is never explained is what Vt rotates.
In order for the rotations and stretches to apply, X=W*Sigma*Vt must
be viewed as a transformation applied to a vector (or a collection of
column vectors). Which means Vt is first applied, then Sigma, then
W. W and Vt are orthogonal rotations.
However, X isn't a transformation that is applied to data vectors, and
it is hard to imagine what vectors Vt would apply to. They would have
to be in n-space, but n-space doesn't have much meaning in the context
of finding correlations between the m data sets (one from each
sensor).
> software-pages http://go.helms-net.de/sw/matmate. It's an amateurish
Thanks again for taking the time to write such a detailed response. I
gave it a quice twice-over, but it needs more than that. I'll get
back to it soon, but I got some preliminary comments.
On Nov 2, 2:02 am, Gottfried Helms <he...@uni-kassel.de> wrote:
> it seems I made my comment more complicated than the procedure is.
>
> Am 01.11.2012 20:53 schrieb Paul:
>
>> I might be missing some linear algebra theory here, but I looked up
>> gettrans() and I'm not sure what is meant by a column rotation in
>> that context.
>
> No, gettrans is just a function-call in my MatMate-script-language,
> which returns a rotation-matrix. For instance, by the command:
>
> t1 = gettrans(X,"drei") // "drei" means "triangular"
>
> t1 becomes the rotation-matrix, which is required to rotate
I am still not sure what is a columnwise rotation. Do you actually
switch columns around, or is it more like a geometric rotation?
> ...X to triangular shape. After that we can do the following
> with t1:
>
> Y = X * t1
> // Y is a lower triangular matrix, (with possibly empty columns
> // to the right
>
> Y = X * t1
> // Y is a lower triangular matrix, (with possibly empty columns
> Z = Y * t1'
> // Z equals now X, because t1*t1' = I (Identitymatrix)
>
> or, for doing roation to principals components position:
>
> t2 = gettrans(X,"pc") // "pc" means "principal components"
>
> and then
>
> B = X * t2
> // the columns of B are now orthogonal, are the principal
> // components
> // Z equals now X, because t1*t1' = I (Identitymatrix)
>
> or, for doing roation to principals components position:
>
> t2 = gettrans(X,"pc") // "pc" means "principal components"
>
> and then
>
> B = X * t2
> // the columns of B are now orthogonal, are the principal
geometric position, or that X somehow becomes a triangular matrix by
rearranging its columns? What if there are not enough properly placed
zeros for that to be possible?
Eigenvalues.
>> So the rotation by W is very intuitive to me, while the rotation by
>> Vt is not. And as I described, it's all the more mysterious when
>> you consider that X isn't actually a transformation that is applied
>> to data -- it *is* the data.
>
> This remark "... isn't actually a transformation..." confuses now
> me. ;-) Well, I understood X as data as well, I have no idea, where
> the idea of "being a transformation" comes from and what I am
> possibly missing here. Very likely I didn't properly catch your way
> of approaching the problem...
stretch while W rotates these stretch axes to the principal components
of the data in m-space. What is never explained is what Vt rotates.
In order for the rotations and stretches to apply, X=W*Sigma*Vt must
be viewed as a transformation applied to a vector (or a collection of
column vectors). Which means Vt is first applied, then Sigma, then
W. W and Vt are orthogonal rotations.
However, X isn't a transformation that is applied to data vectors, and
it is hard to imagine what vectors Vt would apply to. They would have
to be in n-space, but n-space doesn't have much meaning in the context
of finding correlations between the m data sets (one from each
sensor).
Gottfried Helms
Nov 4, 2012, 8:44:39 PM11/4/12
to
Hi Paul -
Am 04.11.2012 20:04 schrieb Paul:
> Hi, Gottfried,
(...)
we look at two columns of X only and apply a column-rotation by
the rotation-matrix t consisting of cos/sin-values:
| x1 y1 | | cos(phi) sin(phi) |
| x2 y2 | * | -sin(phi) cos(phi) |
| x3 y3 |
| ... ...|
| xm ym |
If phi is chosen such that the columns are then
| x1' 0 |
| x2' y2' |
| x3' y3' |
| ... ... |
| xm ym |
this is a rotation to "triangular position". Clearly, if we do
not look at two columns only, we can have a rotation of
X to triangular position/shape as given in an earlier example
> and we apply that rotation to X to get the sensor's data in that rotated coordinates in matrix X1
> [25] X1 = X*t1
> X1 :
> 2.4495 0.0000 0.0000 0.0000 0.0000 0.0000
> -0.4790 2.4022 0.0000 -0.0000 0.0000 0.0000
> -1.0680 -1.5080 1.6079 0.0000 -0.0000 0.0000
>******************************************** | -------------------------------------------
>we see, that we need only a m=3-dimensional space to account for coordinates
> of 3 (linearly independent) vectors.
Here X1 has a triangular shape (only the lower triangle is nonzero).
Because we talk of vectors in the n-dimensional space, and I take
the vectors as a "wire-model" I also say "triangular position" because
the wire-model was rotated such that the first wire (representing the
first sensor) lays on the x-axis, the second wire (representing the
second sensor) lays in the x-y plane and so on.
>
> What is meant by rotating to triangular position? Do you mean
> geometric position, or that X somehow becomes a triangular matrix by
> rearranging its columns? What if there are not enough properly placed
> zeros for that to be possible?
Is it now understandable?
Ok, I see: I've mixed the formulas. You've given
X=(W)(Sigma)(Vt)
So I should always have taken
Sigma = W^-1 * X * Vt^-1
or, because W and Vt are orthogonal/rotations :
Sigma = Wt * X * V
So this should be replaced in my examples. However, my goal
was only to make understandable the relevance of the n-space
and th rotation in the n-space, and that the idea of correlations
between the m sensors is just identical to the idea of that
wire-model in the n-space, having angles between them whose
cosines are just the correlations.
>
> That's the view of X = W Sigma Vt. Sigma is an anisotropic axial
> stretch while W rotates these stretch axes to the principal components
> of the data in m-space. What is never explained is what Vt rotates.
> In order for the rotations and stretches to apply, X=W*Sigma*Vt must
> be viewed as a transformation applied to a vector (or a collection of
> column vectors). Which means Vt is first applied, then Sigma, then
> W. W and Vt are orthogonal rotations.
>
> However, X isn't a transformation that is applied to data vectors, and
> it is hard to imagine what vectors Vt would apply to. They would have
> to be in n-space, but n-space doesn't have much meaning in the context
> of finding correlations between the m data sets (one from each
> sensor).
Hmm, perhaps you should begin to look from the cosine between
two vectors, say A and B.
If in a multidimensional, say 10-dimensional, space the point "a"
has the coordinates [4,5,1,4,2,-3,-3,-1,-5,-4] and the point "b" the
coordinates [2,3,7,9,1,-4,-6,-6,-4,-2] then the vectors A, pointing
from the origin to "a", and B, pointing to "b", have the angle
between them whose cosine is determined by
(A * Bt)
-------- = (4*2 + 5*3 + 1*7 + 4*9 + 2*1 +3*4 + 3*6 + 1*6 + 5*4 + 4*2)/(l1*l2)
l1 * l2
(where l1 and l2 are the "lengthes" of A and B.)
So we can put in the 10-dimensional space the two vectors/wires
coming from the origin, pointing to "a" and "b". That two vectors
can now freely be rotated in that space, for instance such, that
A matches the x-axis, and that B lies then in the x-y-plane. This
roation does not change the angle between them.
But the above formula is the same which we also use to
calculate the correlation, if the coordinates are taken as
datasets/measures. (Note, that in the example I've taken data, such
that their mean is zero for this example to work - the wire-model
/correlation analogy is only usable if the data are centered)
Gottfried Helms
Am 04.11.2012 20:04 schrieb Paul:
> Hi, Gottfried,
(...)
>> No, gettrans is just a function-call in my MatMate-script-language,
>> which returns a rotation-matrix. For instance, by the command:
>>
>> t1 = gettrans(X,"drei") // "drei" means "triangular"
>>
>> t1 becomes the rotation-matrix, which is required to rotate
>> columnwise...
>
> I am still not sure what is a columnwise rotation. Do you actually
> switch columns around, or is it more like a geometric rotation?
>
If you have a matrix X and postmultiply a rotation-matrix, for instance
>> which returns a rotation-matrix. For instance, by the command:
>>
>> t1 = gettrans(X,"drei") // "drei" means "triangular"
>>
>> t1 becomes the rotation-matrix, which is required to rotate
>> columnwise...
>
> I am still not sure what is a columnwise rotation. Do you actually
> switch columns around, or is it more like a geometric rotation?
>
we look at two columns of X only and apply a column-rotation by
the rotation-matrix t consisting of cos/sin-values:
| x1 y1 | | cos(phi) sin(phi) |
| x2 y2 | * | -sin(phi) cos(phi) |
| x3 y3 |
| ... ...|
| xm ym |
If phi is chosen such that the columns are then
| x1' 0 |
| x2' y2' |
| x3' y3' |
| ... ... |
| xm ym |
this is a rotation to "triangular position". Clearly, if we do
not look at two columns only, we can have a rotation of
X to triangular position/shape as given in an earlier example
> and we apply that rotation to X to get the sensor's data in that rotated coordinates in matrix X1
> [25] X1 = X*t1
> X1 :
> 2.4495 0.0000 0.0000 0.0000 0.0000 0.0000
> -0.4790 2.4022 0.0000 -0.0000 0.0000 0.0000
> -1.0680 -1.5080 1.6079 0.0000 -0.0000 0.0000
>******************************************** | -------------------------------------------
>we see, that we need only a m=3-dimensional space to account for coordinates
> of 3 (linearly independent) vectors.
Because we talk of vectors in the n-dimensional space, and I take
the vectors as a "wire-model" I also say "triangular position" because
the wire-model was rotated such that the first wire (representing the
first sensor) lays on the x-axis, the second wire (representing the
second sensor) lays in the x-y plane and so on.
>
> What is meant by rotating to triangular position? Do you mean
> geometric position, or that X somehow becomes a triangular matrix by
> rearranging its columns? What if there are not enough properly placed
> zeros for that to be possible?
X=(W)(Sigma)(Vt)
So I should always have taken
Sigma = W^-1 * X * Vt^-1
or, because W and Vt are orthogonal/rotations :
Sigma = Wt * X * V
So this should be replaced in my examples. However, my goal
was only to make understandable the relevance of the n-space
and th rotation in the n-space, and that the idea of correlations
between the m sensors is just identical to the idea of that
wire-model in the n-space, having angles between them whose
cosines are just the correlations.
>
> That's the view of X = W Sigma Vt. Sigma is an anisotropic axial
> stretch while W rotates these stretch axes to the principal components
> of the data in m-space. What is never explained is what Vt rotates.
> In order for the rotations and stretches to apply, X=W*Sigma*Vt must
> be viewed as a transformation applied to a vector (or a collection of
> column vectors). Which means Vt is first applied, then Sigma, then
> W. W and Vt are orthogonal rotations.
>
> However, X isn't a transformation that is applied to data vectors, and
> it is hard to imagine what vectors Vt would apply to. They would have
> to be in n-space, but n-space doesn't have much meaning in the context
> of finding correlations between the m data sets (one from each
> sensor).
two vectors, say A and B.
If in a multidimensional, say 10-dimensional, space the point "a"
has the coordinates [4,5,1,4,2,-3,-3,-1,-5,-4] and the point "b" the
coordinates [2,3,7,9,1,-4,-6,-6,-4,-2] then the vectors A, pointing
from the origin to "a", and B, pointing to "b", have the angle
between them whose cosine is determined by
(A * Bt)
-------- = (4*2 + 5*3 + 1*7 + 4*9 + 2*1 +3*4 + 3*6 + 1*6 + 5*4 + 4*2)/(l1*l2)
l1 * l2
(where l1 and l2 are the "lengthes" of A and B.)
So we can put in the 10-dimensional space the two vectors/wires
coming from the origin, pointing to "a" and "b". That two vectors
can now freely be rotated in that space, for instance such, that
A matches the x-axis, and that B lies then in the x-y-plane. This
roation does not change the angle between them.
But the above formula is the same which we also use to
calculate the correlation, if the coordinates are taken as
datasets/measures. (Note, that in the example I've taken data, such
that their mean is zero for this example to work - the wire-model
/correlation analogy is only usable if the data are centered)
Gottfried Helms
Paul
Nov 6, 2012, 2:10:37 AM11/6/12
to
>Am 04.11.2012 20:04 schrieb Paul:
>> we see, that we need only a m=3-dimensional space to account for
>> coordinates of 3 (linearly independent) vectors.
This is very...curious. I never knew that you could do that. I am
>> coordinates of 3 (linearly independent) vectors.
operating off of highschool algebra when it comes to linear
transformations e.g. http://en.wikipedia.org/wiki/Transformation_matrix#Rotation
. There, a rotation is effected by left multiplying a point in space
by a rotation matrix like the one you provide above (but transposed,
which is the inverse of a unitary matrix). Is there an online
reference for the theory that explains the right-multiplication and
the reason why it yields a triangular matrix? The right
multiplication by a linear transformation matrix is very unnatural and
unintuitive to me right now.
> Here X1 has a triangular shape (only the lower triangle is nonzero).
> Because we talk of vectors in the n-dimensional space, and I take
> the vectors as a "wire-model" I also say "triangular position"
> because the wire-model was rotated such that the first wire
> (representing the first sensor) lays on the x-axis, the second wire
> (representing the second sensor) lays in the x-y plane and so on.
>
>> What is meant by rotating to triangular position? Do you mean
>> geometric position, or that X somehow becomes a triangular matrix
>> by rearranging its columns? What if there are not enough properly
>> placed zeros for that to be possible?
>
> Is it now understandable?
some theory.
> - Show quoted text -
>
> Ok, I see: I've mixed the formulas. You've given
>
> X=(W)(Sigma)(Vt)
>
> So I should always have taken
>
> Sigma = W^-1 * X * Vt^-1
>
> or, because W and Vt are orthogonal/rotations :
>
> Sigma = Wt * X * V
>
> So this should be replaced in my examples. However, my goal
> was only to make understandable the relevance of the n-space
> and th rotation in the n-space, and that the idea of correlations
> between the m sensors is just identical to the idea of that
> wire-model in the n-space, having angles between them whose
> cosines are just the correlations.
Well that's exactly it...the correlation (or rather, the covariance)
> Ok, I see: I've mixed the formulas. You've given
>
> X=(W)(Sigma)(Vt)
>
> So I should always have taken
>
> Sigma = W^-1 * X * Vt^-1
>
> or, because W and Vt are orthogonal/rotations :
>
> Sigma = Wt * X * V
>
> So this should be replaced in my examples. However, my goal
> was only to make understandable the relevance of the n-space
> and th rotation in the n-space, and that the idea of correlations
> between the m sensors is just identical to the idea of that
> wire-model in the n-space, having angles between them whose
> cosines are just the correlations.
between two sensors is the dot product between the two n-length
sequences of data collected by two sensors. That is W. Not Vt. Vt
doesn't seem to have any intuitive meaning. Each entry is a
correlation between a set of readings between 2 points in time. Each
vector in the dot-product is a set of readings from all m sensors at a
point in time. If I was manually looking for correlation between
sensors, this correlation between samples in time seems to have no
relevance.
>> That's the view of X = W Sigma Vt. Sigma is an anisotropic axial
>> stretch while W rotates these stretch axes to the principal
>> components of the data in m-space. What is never explained is what
>> Vt rotates. In order for the rotations and stretches to apply,
>> X=W*Sigma*Vt must be viewed as a transformation applied to a vector
>> (or a collection of column vectors). Which means Vt is first
>> applied, then Sigma, then W. W and Vt are orthogonal rotations.
>>
>> However, X isn't a transformation that is applied to data vectors,
>> and it is hard to imagine what vectors Vt would apply to. They
>> would have to be in n-space, but n-space doesn't have much meaning
>> in the context of finding correlations between the m data sets (one
>> from each sensor).
>
> Hmm, perhaps you should begin to look from the cosine between
> two vectors, say A and B.
> If in a multidimensional, say 10-dimensional, space the point "a"
> has the coordinates [4,5,1,4,2,-3,-3,-1,-5,-4] and the point "b" the
> coordinates [2,3,7,9,1,-4,-6,-6,-4,-2] then the vectors A, pointing
> from the origin to "a", and B, pointing to "b", have the angle
> between them whose cosine is determined by
>
> -------- = ----------------------------------------
> l1 * l2 l1*l2
>
> (where l1 and l2 are the "lengthes" of A and B.)
>
> So we can put in the 10-dimensional space the two vectors/wires
> coming from the origin, pointing to "a" and "b". That two vectors
> can now freely be rotated in that space, for instance such, that
> A matches the x-axis, and that B lies then in the x-y-plane. This
> roation does not change the angle between them.
Yes, I realize that a unitary matrix preserves angles and distances
> (where l1 and l2 are the "lengthes" of A and B.)
>
> So we can put in the 10-dimensional space the two vectors/wires
> coming from the origin, pointing to "a" and "b". That two vectors
> can now freely be rotated in that space, for instance such, that
> A matches the x-axis, and that B lies then in the x-y-plane. This
> roation does not change the angle between them.
between vectors (from the origin to the point in space represented by
the vector). The thing is, for m sensors, the natural view to take is
to view the cloud of data points in m-space and look for a straight
line relationship for the first principal component. Not in n-space.
Hence my confusion about the role of Vt.
> But the above formula is the same which we also use to calculate the
> correlation, if the coordinates are taken as datasets/measures.
> (Note, that in the example I've taken data, such that their mean is
> zero for this example to work - the wire-model /correlation analogy
> is only usable if the data are centered)
(before PCA) is also scaled so that the standard deviation is 1.
Otherwise, the sensor with the greatest swing in readings will
dominate in determining the principal component. This is not
meaningful since the sensors might not even be making the same kind of
physical measurement.
0 new messages