Regression with Multiple Predicted Variables

[Chris Merck]

Evening NYCMLers,

First off, thanks Paul for all your work in setting this up.

So, during the lectures, I kept asking myself how the algorithm would
change if we tried to predict more than one variable. This would mean,
in Ng's notation, that y becomes a vector, just like x became a vector
when we let there be more than one feature. Is this bothering anyone
else?

Cheers,
Chris

[Bnsh Bnnrj]

Chris, it's an interesting question, but I don't think it's really an issue: (I'm pretty sure anyway)

Here's why:
say if y is a vector (let's say of length v), then you'd simply be solving
   Transpose(X) X Theta = Transpose(X) Y
where Y would be the _matrix_ of y's (so, it's a matrix of (is n the number of samples? that's how I'm going to use it anyway) n x v)

This is really no different from solving it when v happens to be 1. It's just now your Theta is going to be a matrix of size ((m+1), v)

You'd solve it the same way...
  (Transpose(X) X)^-1 Transpose(X) Y = Theta again, and now,
your theta would be matrix of size ((m+1), v)..

I'll try to code up a demo in Octave.

Binesh

[Bnsh Bnnrj]

OK, here's the octave code that does it (sans data)
# First, load the data.

load("data.txt");
load("Y.txt");
[rows, columns] = size(data);
# We need to prepend ones to our data to get the X matrix.
X = [ ones(rows, 1) data];
# Now, we compute theta as pinv(X' * X) * X' * Y
theta = pinv(X' * X) * X' * Y

Now, if Y was an m x 2 matrix, and X was an m x 5 matrix

theta = pinv(X'*X) * X' * Y becomes a
pinv(5x5) * (5xm) * (mx2) matrix so a
5x5 * 5xm * mx2 matrix so a
5xm * mx2 matrix so a
5x2 matrix. (Theta would then be a 5x2 matrix)

Binesh