Hat matrix questions

[Archie]

OK, these are things I never learned in linear algebra, and I can't
find them in any books.

The hat matrix H is, given a matrix X, the matrix X(XtX)^-1 Xt, where
Xt is X transpose and (XtX)^-1 is the inverse of X transpose times X,
provided that (XtX) is invertible.

Now, it turns out that the hat matrix H has entries that are all <= 1.
WHY IS THIS?

Many calculations I have done have shown that every hat matrix I could
devise had determinant(H) = 0.

Now, using a matrix X that is a 3 x 2 matrix, I also found that if one
reorders the matrices such that M = (XtX)(XtX)^-1 = I, where I is the
identity matrix, and we take the euclidean (or Frobenius) norm of M,
we get || M || = sqrt(2) for this 2 x 2 matrix, AND a hat matrix H,
which is 3 x 3, also has || H || = sqrt(2). Does this suggest a
solution as to why the hat matrix has entries all <=1?

Also, I notice that if I begin with a 2 x 3 matrix, then it seems that
the matrix XtX, which is a 3 x 3 matrix, is NOT invertible. I can't
manage to create an invertible 3 x 3 matrix from the transpose of a 2
x 3 matrix multiplied on the right by its transpose. WHY WOULD THIS
BE???

What theorems do I need to know to understand this? Nothing I have
been taught or read in textbooks seems to address these questions.

Thanks in advance,
Archie

[Stephen Montgomery-Smith]

This is a good question.

If you use the singular value decomposition you will see that a hat matrix has to be of the form

H = Ut J U

where U is orthogonal, Ut is its transpose/inverse, and J is a diagonal matrix whose diagonal entries are zero or one. Then see that any entry is of the form x.Jy for unit vectors x and y, and hence must be less than 1 in absolute value.

Many linear algebra courses don't spend a lot of time on the singular value decomposition. In fact, I never learned it in any course while I was an undergraduate at Cambridge (25+ years ago), so don't feel bad if you have never seen it before.

The Frobenius norm of H is the square root of the number of non-zero entries of J. If X is m by n (m >= n), then this must be sqrt(n). This is because otherwise Xt X has rank less than n, and as a n by n matrix it must be non-invertible. If X is m by n (m < n), then Xt X is a n by n matrix whose rank is at most m, so it cannot be invertible.

[Virgil]

In article
<73da7640-79aa-440f...@z8g2000yql.googlegroups.com>,

Archie <kohmo...@yahoo.com> wrote:

> Also, I notice that if I begin with a 2 x 3 matrix, then it seems that
> the matrix XtX, which is a 3 x 3 matrix, is NOT invertible. I can't
> manage to create an invertible 3 x 3 matrix from the transpose of a 2
> x 3 matrix multiplied on the right by its transpose. WHY WOULD THIS
> BE???

The product of two or ore matrices cannot have rank greater than any of
its factors.

A 2x3 or 3x2 matrix, X, has rank <= 2 so that the resulting Xt*X or
X*Xt will also have rank <= 2.

And an n by n matrix of rank < n is necessarily singular.
--

[Rod]

"Archie" <kohmo...@yahoo.com> wrote in message
news:73da7640-79aa-440f...@z8g2000yql.googlegroups.com...

> OK, these are things I never learned in linear algebra, and I can't
> find them in any books.
>
> The hat matrix H is, given a matrix X, the matrix X(XtX)^-1 Xt, where
> Xt is X transpose and (XtX)^-1 is the inverse of X transpose times X,
> provided that (XtX) is invertible.
>
> Now, it turns out that the hat matrix H has entries that are all <= 1.
> WHY IS THIS?
>
> Many calculations I have done have shown that every hat matrix I could
> devise had determinant(H) = 0.
>

Note H * H = H

So H is a projection matrix.

det(H)^2 = det(H)

only works if H = 1

or det(H) = 0

[Stephen Montgomery-Smith]

Also, H is symmetric, i.e. Ht = H. Hence it can be diagonalized with an orthonormal change of basis. The eigenvalues are either 1 or zero. This is another way to see that

H = Ut J U

where U is orthogonal, and J is diagonal with entries 0 or 1. And this avoids the singular value decomposition.

[dilettante]

"Virgil" <vir...@ligriv.com> wrote in message
news:virgil-9BE761....@bignews.usenetmonster.com...

One easy way to see this: The result of the multiplication of the 3x2 and
the
2x3 is the same as if you turned the original matrices into squares by
addition of a row of zeros to one and a column of zeros to the other, then
multiplied. Since those two square matrices have determinant zero, so does
their product.

> --
>
>

[Archie]

Thanks for your comments. I suppose, then, there is a theorem that states that all hat matrices have such a decomposition, since I see that any matrix can be given a singular value decomposition; what would be different for a hat matrix is the diagonal matrix, which would have entries either 1 or 0 -- is that it?

[Stephen Montgomery-Smith]

Yes and yes.