Linear regression
Jens
I have some n-dimensional observations in a m x (n+1) matrix A. Now if
the A is partitioned as A= [X y] I'll like to find a "relationship"
between X and y. From my college studies (for a long time since) I'll
recall that a linear system Xb= y could be solved from X'Xb=X'y.
Parameters b can be found if the columns of X are linearly independent.
I have solved above system with a test dataset and technically it seems
to work. However I read from some article that the observations should
be independent and identical distributed. Now I have few questions:
- the columns of X are linearly independent, but there might exist some
unknown "couplings" between them, what kind of ramifications could
emerge from such "couplings"?
- also I don't have any idea how random the columns are and how they
are distributed, should I need to investigate this more detailed
manner? i.e. what kind of ramifications could emerge from violating the
requirement of identical distributed columns?
- any suggestions of alternative ways to "understand" the
"relationship" between X and y?
TIA,
Jens
Reef Fish
Jens wrote:
> Dear all,
>
> I have some n-dimensional observations in a m x (n+1) matrix A. Now if
> the A is partitioned as A= [X y] I'll like to find a "relationship"
> between X and y. From my college studies (for a long time since) I'll
> recall that a linear system Xb= y could be solved from X'Xb=X'y.
> Parameters b can be found if the columns of X are linearly independent.
>
> I have solved above system with a test dataset and technically it seems
> to work. However I read from some article that the observations should
> be independent and identical distributed.
That's the usual assumption about the ERRORS of the conditional
distribution of Y given X. More specifically iid N(0, sigma^2) is
the
usual assumption.
> Now I have few questions:
> - the columns of X are linearly independent,
As basis vectors in linear algebra.
> but there might exist some
> unknown "couplings" between them, what kind of ramifications could
> emerge from such "couplings"?
By "couplings" I think you mean either statistical or deterministic
relations
some pairs of X's. Short answer, as long as the X's are linearly
independent,
there's nothing to worry about.
> - also I don't have any idea how random the columns are and how they
> are distributed, should I need to investigate this more detailed
> manner?
There's nothing to investigate until AFTER you've fitted some tentative
model. THEN, you should examine the residuals (observed errors) to
validate the iid assumptions.
> i.e. what kind of ramifications could emerge from violating the
> requirement of identical distributed columns?
Not identically distribution COLUMNS. No assumptions about those.
If the N(0, sigma^2) assumption is violated, then your statistical
inference
based on that assumption would not hold.
> - any suggestions of alternative ways to "understand" the
> "relationship" between X and y?
Yes there are. But if you understand linear regression models and
the wide range of model it accommodates, then perhaps you don't
need to look elsewhere.
You seem to have the common misconceptions and misunderstanding
between "linear indepdence" (linear algebra) and stochastic
(statistical) independence of the error terms.
There are several threads in sci.stat.math specifically about the
difference of these two concepts.
-- Bob.
Jens
<skip>
> That's the usual assumption about the ERRORS of the conditional
> distribution of Y given X. More specifically iid N(0, sigma^2) is
> the usual assumption.
<skip>
> There's nothing to investigate until AFTER you've fitted some tentative
> model. THEN, you should examine the residuals (observed errors) to
> validate the iid assumptions.
So my model should be Xb=y+e and e=Xb-y should be "close to" iid N(0,
sigma^2).
<skip>
> By "couplings" I think you mean either statistical or deterministic
> relations some pairs of X's. Short answer, as long as the X's are linearly
> independent, there's nothing to worry about.
By couplings I mean some unknown relations between the columns. Unknown
in the sence that no-one have investigated such relations properly, but
based on "intelligent" guess some relations surely exists.
<skip>
> You seem to have the common misconceptions and misunderstanding
> between "linear indepdence" (linear algebra) and stochastic
> (statistical) independence of the error terms.
> There are several threads in sci.stat.math specifically about the
> difference of these two concepts.
Yes indeed I need to try to understand this topic a better manner.
Actually my current observations are only a subset of a larger set.
I'll return later with more specific questions.
Best regards,
Jens
Reef Fish
Jens wrote:
> Thanks Bob for your prompt answer.
>
> <skip>
> > That's the usual assumption about the ERRORS of the conditional
> > distribution of Y given X. More specifically iid N(0, sigma^2) is
> > the usual assumption.
> <skip>
> > There's nothing to investigate until AFTER you've fitted some tentative
> > model. THEN, you should examine the residuals (observed errors) to
> > validate the iid assumptions.
> So my model should be Xb=y+e and e=Xb-y should be "close to" iid N(0,
> sigma^2).
It's Y = Xb + e, where e is nearly iid N(0, sigma^2).
Y = Xb is what you try to model or fit. What does NOT fit is lumped
into
the random error term e. e may be the results of hundreds and
thousands
of variables that actually have SOME effect on what you measure, but
you
simply (for simplity sake) assume it comprises ONE single random error.
> <skip>
> > By "couplings" I think you mean either statistical or deterministic
> > relations some pairs of X's. Short answer, as long as the X's are linearly
> > independent, there's nothing to worry about.
> By couplings I mean some unknown relations between the columns. Unknown
> in the sence that no-one have investigated such relations properly, but
> based on "intelligent" guess some relations surely exists.
But you're NOT interested in those unknown, uninvestigated between
the "columns" (on the RIGHT hand side -- the independent variables).
If you want to explore its relation with the others, put it on the LEFT
as
the dependent variable.
>
> <skip>
> > You seem to have the common misconceptions and misunderstanding
> > between "linear indepdence" (linear algebra) and stochastic
> > (statistical) independence of the error terms.
> > There are several threads in sci.stat.math specifically about the
> > difference of these two concepts.
> Yes indeed I need to try to understand this topic a better manner.
> Actually my current observations are only a subset of a larger set.
> I'll return later with more specific questions.
What you are asking are the most BASIC questions regarding a
Linear (Regression) model. The fact that you wrote you model as
> Xb=y+e and e=Xb-y
suggests that you haven't read any book on the subject. Try
Neter, Wasserman, Kutner & Nachsheim's book on "Applied
Linear Models" which is a standard textbook in many universities
and recommended by many (including myself) as a good book
on the subject.
-- Bob.
Richard Ulrich
> Thanks Bob for your prompt answer.
[snip, some]
Bob >
> <skip>
> > By "couplings" I think you mean either statistical or deterministic
> > relations some pairs of X's. Short answer, as long as the X's are linearly
> > independent, there's nothing to worry about.
Jens >
> By couplings I mean some unknown relations between the columns. Unknown
> in the sence that no-one have investigated such relations properly, but
> based on "intelligent" guess some relations surely exists.
Does that matter? Maybe, maybe not.
Reef Fish Bob belongs to a "black-box" school of regression --
the statistician has no knowledge of control of any of the
many variables going into his equation, and no care for what
it may "mean". [And, in his ignorance, Bob is exceeding proud
that his way is the only right way.]
That is hardly ever the attitude in the sciences, but it is the
way business schools can approach the problem of predicting
who is the good insurance risk, etc. What is needed, mainly,
is a large dataset, and no concern for *wide* generalization
(that is, going beyond datasets that are rather similar).
If you want "meaningful" regressions -- an idea that is anathema
to Bob -- you will have a rather small set of variables, which
you might "reduce" even further before taking into the regression.
What is your purpose? Are you exploring, verifying, developing
on someone else's work?
What is your N? ... your number of predictors, your expected
R-squared? Do you have a choice of what variables to use,
or what functional forms of them? With Large-N and large-P
and no choice in the variables, you should follow Bob's
lead -- run the regression and study the diagnostics....
[snip]
--
Rich Ulrich, wpi...@pitt.edu
http://www.pitt.edu/~wpilib/index.html
David Winsemius
news:r9ur72d1kphoe52qh...@4ax.com:
> That is hardly ever the attitude in the sciences, but it is the
> way business schools can approach the problem of predicting
> who is the good insurance risk, etc. What is needed, mainly,
> is a large dataset, and no concern for *wide* generalization
> (that is, going beyond datasets that are rather similar).
>
What sort of evidence do you have that business schools teach an atheoretic
approach to statistics? Or that businesses use only data driven approaches
to risk estimation? I ask because my experience within the health and life
insurance sectors has been quite the opposite.
--
David Winsemius
Jens
> What you are asking are the most BASIC questions regarding a
> Linear (Regression) model. The fact that you wrote you model as
>
> > Xb=y+e and e=Xb-y
>
> suggests that you haven't read any book on the subject. Try
> Neter, Wasserman, Kutner & Nachsheim's book on "Applied
> Linear Models" which is a standard textbook in many universities
> and recommended by many (including myself) as a good book
> on the subject.
I was able to locate Neter, Wasserman & Kutner book on "Applied Linear
Statistical Models", third ed, 1990. So that will keep me busy for a
while.
Thanks,
Jens
Jens
> Reef Fish Bob belongs to a "black-box" school of regression --
> the statistician has no knowledge of control of any of the
> many variables going into his equation, and no care for what
> it may "mean". [And, in his ignorance, Bob is exceeding proud
> that his way is the only right way.]
>
> That is hardly ever the attitude in the sciences, but it is the
> way business schools can approach the problem of predicting
> who is the good insurance risk, etc. What is needed, mainly,
> is a large dataset, and no concern for *wide* generalization
> (that is, going beyond datasets that are rather similar).
>
> If you want "meaningful" regressions -- an idea that is anathema
> to Bob -- you will have a rather small set of variables, which
> you might "reduce" even further before taking into the regression.
view where the idea is that the interface of some module is everything
you need to know in order to utilize the module, you don't need to know
anything how the module operates internally. However when implementing
the module the inner workings are essential. I think I'm pretty similar
position with my current problem (see below).
>
> What is your purpose? Are you exploring, verifying, developing
> on someone else's work?
>
> What is your N? ... your number of predictors, your expected
> R-squared? Do you have a choice of what variables to use,
> or what functional forms of them? With Large-N and large-P
> and no choice in the variables, you should follow Bob's
> lead -- run the regression and study the diagnostics....
I'll try to give better overview what I'm trying to achive.
In the highest level there is a process which transforms some substance
(material) to other one. We are able to measure some aspects of this
substance while its transforming.
The next level reveals that the whole process consists of several
(sub)processes p1...p5.
X1 X2 X3 X4 X5
+--+ ----- +--+ ----- +--+ --- +--+ ----- +--+
->|p1|->\s12/->|p2|->\s23/->|p3|->/s34\->|p4|->\s45/->|p5|->
+--+ --- +--+ --- +--+ ----- +--+ --- +--+
y1 y2 y3 y4 y5
(above diagram should be viewed with monospaced font)
The processes are connected by storages s12...s45. So the input of p1
is gradually transformed to the output of p5. Delays in storages s12,
s23 and s45 varies between known limits and the order of the "flow" is
preserved (sort of queue), however the "flow" order in s34 in not
preserved (sort of heap). In each process there are some observations X
(input) and y (output) associated. Typically in each process X consists
of variables describing inputs and some process control information and
y consists of 1 or several (Y) indicators of the process state (and
"substance state"). These observations are collected in various time
intervals (mins to hrs). The processes are basically runnig 24 hrs, but
some interruptions occurs (hrs to days). The X's for various processes
contains 10 to 30 variables and 2000 to 10000 observations.
Simply my main goal is to understand better the "fluctuations" in
observations y5, consequently I need to understand the "fluctuations"
in all observations y. At this point the overall "time series"
structure of the process chain seems to be hihgly complicated and
therefore I have planned first to investigate the processes separately.
So I'm assuming that such investigations will reveal something
"useful", but am I able to recognize such things is the current "hot"
question.
Thanks,
Jens
Reef Fish
Richard Ulrich wrote:
> On 31 May 2006 09:58:58 -0700, "Jens" <jens....@gmail.com> wrote:
>
> > Thanks Bob for your prompt answer.
>
> [snip, some]
>
> Bob >
> > <skip>
> > > By "couplings" I think you mean either statistical or deterministic
> > > relations some pairs of X's. Short answer, as long as the X's are linearly
> > > independent, there's nothing to worry about.
> Jens >
> > By couplings I mean some unknown relations between the columns. Unknown
> > in the sence that no-one have investigated such relations properly, but
> > based on "intelligent" guess some relations surely exists.
>
>
> Does that matter? Maybe, maybe not.
>
> Reef Fish Bob belongs to a "black-box" school of regression --
> the statistician has no knowledge of control of any of the
> many variables going into his equation, and no care for what
> it may "mean". [And, in his ignorance, Bob is exceeding proud
> that his way is the only right way.]
It is too much to expect Richard Ulrich to be on good behavior
for long. The monkey on his back is on his back again.
More later. Have a plane to catch.
-- Bob.
Reef Fish
Now I can give a better diagnosis of Ulrich's gratuituous, but very
familiar attack, based on HIS OWN lack of understanding of the
subject to which I gave jens my advice.
It should be clear from the post to which I replied jens that he was
asking the most basic questions about linear regression model, not
having ever read any book of article on the subject, from his
description of
> So my model should be Xb=y+e and e=Xb-y should be "close to" iid N(0,
> sigma^2).
which I corrected him to,
RF> It's Y = Xb + e, where e is nearly iid N(0, sigma^2).
responded to his simple questions simply, and then adviced that reading
a book such as Neter, Kutner, Wasserman, Nachsheim is textbook
should be of great help to him.
There was NO indication whatsoever that jens was speaking of a
designed experiment, or a controlled observational study; or that
jens is even familiar with the basic notions of a linear model!
That's
when Richard Ulrich thought he found the opening to insert his
favorite salemanship of Quackery and Snake Oil,
> > Reef Fish Bob belongs to a "black-box" school of regression --
> > the statistician has no knowledge of control of any of the
> > many variables going into his equation
After his by now familiar style of gratuitous attack, Ulrich didn't
have much go on his own other than:
RU> What is your N? ... your number of predictors, your expected
RU> R-squared? Do you have a choice of what variables to use,
RU> or what functional forms of them? With Large-N and large-P
RU> and no choice in the variables, you should follow Bob's
RU> lead -- run the regression and study the diagnostics....
Apart from MISQUOTING my advice to jens, Richard Ulrich had
nothing of his own to offer.
Now that jens has given him the description of his problem, more
than 12 hours ago, and Richard Ulrich had followed up on another
post about an hour or so ago (so he is around), Ulrich has NOTHING
to offer jens, after he wasted no time (May 31, 4:30 pm) attacking
my post of May 31, 2:05 pm to jens which jens thanked me for
my suggestion on June 1, 6:39 am,
jens> I was able to locate Neter, Wasserman & Kutner book on
jens> "Applied Linear Statistical Models", third ed, 1990. So that
jens> will keep me busy for a while.
and gave Richard Ulrich a description of his problem on June 1,
6:56 am, as Ulrich had asked.
The supreme irony of this particular situation is that jens KNOWS
he is unfamiliar with linear regression models, and had followed my
advice to read up on it.
Richard Ulrich, on the other hand, had made ALL THE ERRORS
that are POSSIBLE, for anyone who had taken a course using
the material in Neter el al -- as evidenced by what's in the archives
-- Richard Ulrich knew even LESS than jens about LINEAR MODELS,
and Ulrich had no idea what the regression assumption are, about
the X's, what the error assumption are, and when to test them, and
various misinformation and errors made by Ulrich about the sign,
and the meaning of the multiple regression coefficients ...
felt obliged to rise to this occasion, in the guise to help jens, to
proclaim his completely unwarranted attack, based on what I had
adviced jens!!
> > Reef Fish Bob belongs to a "black-box" school of regression --
> > the statistician has no knowledge of control of any of the
> > many variables going into his equation, and no care for what
> > it may "mean". [And, in his ignorance, Bob is exceeding proud
> > that his way is the only right way.]
Richard Ulrich, you are hopeless and beyond redemption.
The one line sums it all up, given THIS post in which I gave my
substantiated REASONS why.
-- Bob.
Richard Ulrich
Sorry, no evidence to speak of, that they *do* teach an
"atheoretic approach" very widely, or to the exclusion of
the wider approach.
I can see reasons why the data-driven approach should be
included for their purposes, and I see that it is the approach that
Reef Fish espouses, so his school included it when he taught
the course. Bob has posted that he ended up (in my words)
scornful of the environment of academe - his own faculty and others -
but I don't recall that he mentioned the content, outside of
"dumbing down."
Richard Ulrich
[snip, about 'black box'; my questions]
Okay. That is certainly a more complex problem than defining
a single linear regression.
As I read it --
There are several stages, each (potentially) with their own inputs
and their own, *several* outputs. Is it true that the relations will
have to be "linear", or is that a questionable assumption?
One reason for looking at the separate measurements is to make
sure that they are comprehensive and sensible to the problem --
Are the quantities measured *useful* to the prediction? Are they
going to be linear at the extremes? Are there quantities that should
be measured *relative* to each other (that is, ratio or difference)?
If two feedstocks need to be equal to maximize combination,
then the prediction equation might be simpler if it includes a
term for that difference or ratio.
Some of the data, for certain stages, are collected as time series.
It is not clear to me whether the "Process control information"
is a matter of setting of controls for a stage, or if it is a record/
report of what was taking place, also in time-series.
I'd guess, a big question of design is how much you can simplify.
Can you collapse a time series? What do you learn from them?
Does the production mainly depend on certain totals?
I can imagine where timely monitoring was performed in order
keep the process in control; the numbers are available, but since
they *were* always kept in control, they won't predict anything now.
Or, they are not expected to. Models may be far simpler of you
can reasonably squeeze out the time-series aspect.
However, it is not clear, right now, whether every set of outputs
does not represent a time-series, with some fuzzy, lagged relation
to the inputs.
Here is a big, possible simplification:
Can you expect that the "inputs" to stage 1 will independently
predict the outputs of stage 5? Or, 1 to 3? 3 to 5?
In other words, how important are the middle stages?
Can you *relate* a single starting state to a single ending state?
Do you have 5 sets of regressions that will end up separate, or
do you hope to describe one equation, with entries (perhaps) from
each stage, in order to predict the stage-5 outcomes?
Hope this is useful.
Richard Ulrich
<Large_Nass...@Yahoo.com> wrote:
>
> Reef Fish wrote:
> > Richard Ulrich wrote:
> > > On 31 May 2006 09:58:58 -0700, "Jens" <jens....@gmail.com> wrote:
> > >
> > > > Thanks Bob for your prompt answer.
> > >
> > > [snip, some]
> > >
> > > Bob >
> > > > <skip>
> > > > > By "couplings" I think you mean either statistical or deterministic
> > > > > relations some pairs of X's. Short answer, as long as the X's are linearly
> > > > > independent, there's nothing to worry about.
- The lines above are what I quoted as dubious, what still
seems to justify the term, "black box approach", and what
encouraged me to drop my usual overnight embargo on
responding to Reef Fish Bob's comments.
It seems unprofessional -- to use Bob's term, "quackery" --
for Bob to ignore the fact that there is a large mainstream of
statistical analysts who will tell you to check out your
univariate predictors. And Bob says, "There's nothing to
worry about."
[snip -- Bob reviewing his post, which was otherwise okay.
Tossing insults. A bunch of stuff that I generally anticipated
with my closing....]
RFU > >>
> > > it may "mean". [And, in his ignorance, Bob is exceeding proud
> > > that his way is the only right way.]
>
> Richard Ulrich, you are hopeless and beyond redemption.
>
> The one line sums it all up, given THIS post in which I gave my
> substantiated REASONS why.
Reasons why? I see a place or two where Bob claims that I
misrepresented *what he posted*, but he does not say that I
misrepresented his views. Nor does he offer a corrective.
As usual -
Argument by repetition (okay content, but irrelevant to
what I posted or why I posted it).
Argument by verbal abuse.
Argument by intimidation-through-abuse.
Reef Fish
Richard Ulrich wrote:
> On Wed, 31 May 2006 19:12:54 -0500, David Winsemius
> <doe_...@comcast.NOT> wrote:
>
> > Richard Ulrich <Rich....@comcast.net> wrote in
> > news:r9ur72d1kphoe52qh...@4ax.com:
> >
> > > That is hardly ever the attitude in the sciences, but it is the
> > > way business schools can approach the problem of predicting
> > > who is the good insurance risk, etc. What is needed, mainly,
> > > is a large dataset, and no concern for *wide* generalization
> > > (that is, going beyond datasets that are rather similar).
> > >
> > What sort of evidence do you have that business schools teach an atheoretic
> > approach to statistics? Or that businesses use only data driven approaches
> > to risk estimation? I ask because my experience within the health and life
> > insurance sectors has been quite the opposite.
The business schools of Chicago, MIT, and Carnegie-Mellon teach
statistics at a higher theoretical level than most non-business
schools,
while at the same time apply statistics to a much wider variety of
problems than Richard Ulrich speculated, out of his own ignorance.
>
> Sorry, no evidence to speak of, that they *do* teach an
> "atheoretic approach" very widely, or to the exclusion of
> the wider approach.
Then why did you make your unsubstantiated statement, as you do
on most of your statements -- unsubstantiated and without evidence
"to speak of".
> I can see reasons why the data-driven approach should be
> included for their purposes, and I see that it is the approach that
> Reef Fish espouses, so his school included it when he taught
> the course.
What is your point here? You know no more about what I "espouse"
than you do about business schools or other graduate schools.
What I teach, at ALL schools, INCLUDE (but not limited to -- because
much of which is of too high a level, both in THEORY and in PRACTICE
than this group can usefully absorb) EVERYTHING I have discussed
in this newsgroup -- which covered the most elementary of the
basics of statistics and data analysis.
At that low level, I have already found supposed educated
statisticians,
of whom Richard Ulrich is the most glaring example, who are
completely ill-equipped to understand either the theory, the proper
methodology, OR the correct approach to practice -- in ANY field in
which statistics is used!
> Bob has posted that he ended up (in my words)
> scornful of the environment of academe - his own faculty and others -
> but I don't recall that he mentioned the content, outside of
> "dumbing down."
That was mostly with reference to the FIRST or second course in
statistics at the UNDERGRADUATE level.
As for the senior undergrad and graduate level statistics courses
I taught, as in Data Analysis, I can safely say, based on Richard
Ulrich's record in sci.stat.math on those topics, that he would surely
have failed the course, whether it was taught at the Business
schools in which I taught the courses that included regression and
linear models and data analysis.
As for Richard Ulrich's blunders in Type I and Type II errors, the
use of p-values, the lack of knowledge about the theory and
methodogy behind SIMPLE regression models, and simple linear
models, I would say Richard Ulrich would have failed some of
my UNDERGRADUATE courses too -- and that he would have
been another one of those student whiners who blamed it all
on me, because everyone else in OTHER classes all get high
grades and pass -- to produce the kind of product Ulrich is
representing today -- the DUMBING down of statistical
education.
Richard Ulrich, if you can't speak for yourself for your
Ulrich> Sorry, no evidence to speak of
try to refrain from your misrepresentation of what I do or what I
teach, and everything else in which you had NO knowledge
except that you FAILED in more topics discussed in sci.stat.math
than anyone else -- including Luis A. Afonso, because even he
knew enough about his own ignorance to keep his mouth shut
beyond a couple of undergrad topics that occupied his history of
posting.
>
> --
> Rich Ulrich, wpi...@pitt.edu
I pity Pitt for having its name attached to yours, for the Quackery
and malpractice you have shown in statistics.
But I did notice you longer have a Pitt email address to post from.
You should be filling in your Ho and H1 in the CV thread instead
of saying here that what you said was based on "no experience
to speak of" as if we didn't already know.
-- Bob.