# LESSON 3 in Model Building: Practical Significance

0 views

### Reef Fish

Jun 29, 2005, 8:32:43 AM6/29/05
to
This is the lesson that is seldom, if ever, discussed in textbooks, on
regression problems, but SHOULD be, on every problem, on assessing the
PRACTICAL significance (or usefulness or uselessness) or the result, as
opposed to the statistical significance of the result.

Here's the model and its sgatistical significance, repeated from LESSON
2:

INVDEX = -197.51 + 0.018234 * GNP
(15.031) (.000667)
T=-13.14 T=27.33
p-value 10^(-12)

Multiple R-sq = 0.9726, MSE = 317.96.

All very impressive and highly statistically significant

I am going to demonstrate that is is COMPLETELY USELESS, in practice,
for
prediction purposes.

For this EXERCISE, I had saved the last observation, to check how the
model
would have done, on its PREDICTION interval.

Suppose we found the model in 1965, and used the result for prediction
in
1966 (pretending that the GNP for the year was available before the
INVDEX
was), we would have found a 95% prediction interval to be

(413.1, 498.2)

The width of this interval (relative to the annual variation of INVDEX)
would have dampened our happiness-index (on the basis of the
statistical
significance of T and other measures) considerably. In fact, one may
judge the prediction interval result to be so WIDE as to render the
result useless.

IDA, the interactive data analysis package, had a built-in command
(VALI)
to let users to selectively look at the predictive performance of any
number of rows in the data matrix, to empirically "validate" the
"prediction" interval results. (IN if the actual observation is within
the prediction interval. OUT otherwise). This would have been the
result for the last several rows:

Abs. Error Abs.% error Abs. % Error Length of 95% IN=1
ROW |Y - YPRED| 100|Error-Y| 100|Error-YPRED| Pred.Interval OUT=0

28 3.87 1.23 1.22 77.9 1
29 8.76 2.61 2.54 78.9 1
30 14.41 3.66 3.80 80.5 1
31 14.06 3.24 3.35 82.8 1
32 47.16 11.54 10.35 85.1 0

In spite of the widths of the prediction intervals, the prediction
interval would have MISSED the actual observed value of 408.5 for
1966!

Let's see how we might have done had we not used ANY predictor
variable,
and just use the ANNUAL CHANGE of INVDEX to predict the 1966 value.

The annual change has a mean of 16.28 (increase) for INVDEX. A 95%
Prediction Interval for the CHANGE would be

16.28 +- t(.975;22) (23.45) = (-32.35 , 64.91)

The above, added to the observed value of 433.1 in 1965 would have
yielded a prediction interval of

(400.8, 499.0)

WITHOUT the use of any independent variable for prediction. It is
wider than the corresponding prediction interval using the GNP
variable,
to be sure, but it INCLUDEED the actual value 408.5 observed in 1996,
whereas the prediction interval of the model with GNP,

(413.1, 498.2)

missed the actual observed value (408.5). This kind of post hoc
"validation" check, should give one, in this particular example of the
SPSS Multiple Regression data, a sobering thought about the USELESSNESS
of any regression model, in spite of the STATISTICAL significance of
the regression results.

This phenomenon is the result of the "spurious correlation" between
two increasing time series. It happens ALL the time. It gives still
another validation of Tukey's caveat that the use of ANY correlation
is "sweep dirt under the rug with a vengeance".

The final installment of this example will be LESSON 4, on Superious
Correlations, later today. There are many corollary discussions that

-- Bob.

### Art Kendall

Jun 29, 2005, 9:18:08 AM6/29/05
to
A very important point. Right on.

This is related to what social scientists, especially psychologists,
call the "so what" test. For example, in a given experiment, the
apparent difference between the mean for the group of cases randomly
assigned to one treatment and the mean for the cases randomly assigned
to the other treatment is not readily attributable to random variation
in the dependent variable. That is, the difference is "statistically
significant".

It often helps clients/student to say: Another way to put the idea of
statistical significance is that the observed
difference/association/relation/correlation is sufficiently large that
it is not very consistent with the
differences/associations/relations/correlations that would occur if
there were no difference/association/relation/correlation and many
random samples were used.

For at least the last thirty years, psychologists have been urged to
avoid the term "significant" without a qualifier. In more recent years,
there has been a strong and growing emphasis on including some estimate
of "effect size" to aid in figuring out
helps clients/students to express this concept in term of "meaningful".

Art
A...@DrKendall.org
Social Research Consultants
University Park, MD USA
(301) 864-5570

### Andy Spragg

Jun 29, 2005, 12:54:31 PM6/29/05
to
On 29 Jun 2005 05:32:43 -0700, "Reef Fish"

<Large_Nass...@Yahoo.com> wrote:
>The final installment of this example will be LESSON 4, on Superious
>Correlations, later today. There are many corollary discussions that

Are they correlations that are SUPERIOR, SERIOUS, SPURIOUS, or some
weird hybrid of two or more of the above?

### Reef Fish

Jun 29, 2005, 1:12:21 PM6/29/05
to

Thanks for catch my typo.

In the context, I hope everyone would understand the misspelling
was meant to be "spurious".

I'll make sure I spell it correctly in the SUBJECT of LESSON 4. :-)

-- Bob.

### Russell...@wdn.com

Jun 29, 2005, 1:22:27 PM6/29/05
to
It actually might be a reasonable term to introduce to the field-
superious: correlations that are really good but for the "wrong"
reason. :-)

Cheers,
Russell

### G Robin Edwards

Jun 29, 2005, 7:35:50 PM6/29/05
to

> (413.1, 498.2)

Agreed with all of these, though I do it graphically for simple
or polynomial models. There's also another datum that falls outside
the 95% range, the 1961 value. But that's confidence intervals for
you, I guess. One in 20 may well be outside the 95% CI without causing
any surprise.

> (400.8, 499.0)

> (413.1, 498.2)

Haven't got around to looking at that, yet. It is half past midnight
so I'm off to bed.

However, I've noticed that my original (and rather poor) model using
all the data except 1966 comfortably includes the 1966 value within its
95% CI. Does this mean anything worth noting? And I wonder what the
1976 figures would have told us about the models. Alas, we shall never
know.

I look forward to tomorrow's reading. Will post this tonight.

Robin

### Reef Fish

Jun 29, 2005, 8:03:12 PM6/29/05
to

G Robin Edwards wrote:
> Fish <Large_Nass...@Yahoo.com> wrote:
> > This is the lesson that is seldom, if ever, discussed in textbooks,
> > on regression problems, but SHOULD be, on every problem, on assessing
> > the PRACTICAL significance (or usefulness or uselessness) or the
> > result, as opposed to the statistical significance of the result.

> > The final installment of this example will be LESSON 4, on Superious
> > Correlations, later today. There are many corollary discussions that
>
> Haven't got around to looking at that, yet. It is half past midnight
> so I'm off to bed.

Unless you're clairvoyant, I haven't WRITTEN LESSON 4 yet. :-)

>
> However, I've noticed that my original (and rather poor) model using
> all the data except 1966 comfortably includes the 1966 value within its
> 95% CI. Does this mean anything worth noting? And I wonder what the
> 1976 figures would have told us about the models. Alas, we shall never
> know.

FIVE TIMES the MSE of mine or very approximately 2xsqrt(5) or 4.46
times the WIDTH of my prediction interval for INVDEX of 1966, which I
considered much too wide to have any PRACTICAL significance.

4 times that width should catch any INVDEX value in the last century.
:-)

In that respect, the fact that your model "comfortably includes
the 1996 value" should mean ... "nothing". :)

>
> I look forward to tomorrow's reading. Will post this tonight.
>
> Robin

-- Bob.

### Reef Fish

Jun 29, 2005, 1:46:12 PM6/29/05
to

Good idea! Even better, a "supersilly superious" correlation,
or a "supersillious superious" correlation as one that
a social scientist can thumbs his nose at non-believers that
a causation has been proven! :-)

-- Bob.

### Reef Fish

Jun 29, 2005, 1:36:31 PM6/29/05
to

Art Kendall wrote:
> A very important point. Right on.
>
> This is related to what social scientists, especially psychologists,
> call the "so what" test.
>

> It often helps clients/student to say: Another way to put the idea of
> statistical significance is that the observed
> difference/association/relation/correlation is sufficiently large that
> it is not very consistent with the
> differences/associations/relations/correlations that would occur if
> there were no difference/association/relation/correlation and many
> random samples were used.

I recently discussed the notion of "significantly correlated"
variables in the sense of rejecting the null hypothesis of Ho: rho = 0

"Do the critical values of linear correlation depend on sample size? "

There I gave the rule of thumb for LARGE samples that a sample
correlation r is statistically significant (at the 5% level) in
that sense if the absolute value of r is greater than 2/sqrt(n).

Thus, for a sample of aize 10,000,

the critical value for "significance" would have been .02.

I can safely challenge that NO ONE can tell the difference bwtween
the scatterplot of 10,000 points with r = 0 and r = .02.

In fact, it'll be difficult to tell the difference scatter for
even 100 points or r = 0 vs r = 0.2.

They would ALL look like the holes left from a shotfun blast
shot at a wall.

Try that experiment of generating bivariate normal samples of
say 1,000 with rho = 0.5, 0.4, ..., 0; (if you know how to
generate these) and look at their scatterplots.

>
> For at least the last thirty years, psychologists have been urged to
> avoid the term "significant" without a qualifier. In more recent years,
> there has been a strong and growing emphasis on including some estimate
> of "effect size" to aid in figuring out

> helps clients/students to express this concept in term of "meaningful

This is not always possible on measures like "correlation", as shown
in the examples I suggested. Not many statisticians, let alone
non-statisticians, have a feel of the random LOOK in a scatterplot
of a statisticall highly significant correlation, for most sample
sizes.

The differences in MEANS are much more tangible in terms of assessing
any PRACTICAL significance.

-- Bob.

### Andy Spragg

Jun 30, 2005, 6:01:07 AM6/30/05
to
On 29 Jun 2005 10:12:21 -0700, "Reef Fish"
<Large_Nass...@Yahoo.com> wrote:

>> >The final installment of this example will be LESSON 4, on Superious
>> >Correlations, later today. There are many corollary discussions that

>> Are they correlations that are SUPERIOR, SERIOUS, SPURIOUS, or some
>> weird hybrid of two or more of the above?

>In the context, I hope everyone would understand the misspelling

>was meant to be "spurious".

That's certainly what I assumed. But for all I know, you might be a
lover of neologistics. You might have coined "superious"
intentionally, to indicate "allegedly top quality, but actually
bogus", for example - would have been entirely consistent with your
recent contributions.

I wouldn't even have bothered to pick it up had it not been for the
frequent SHOUTING for no particularly good REASON. Bob, it's pretty
clear that many people who read this NG, myself certainly included,
have a great deal to learn from you about statistics - that's the only
limited Usenet experience has taught me that, by and large, excessive
use of capitalized words in the middle of sentences generally equates
to kookiness or worse.

Can I suggest that you learn a little about the fine art of rendering
your pearls of wisdom in more ASCII ways than just speaking and
SHOUTING? For example, you can use /italics/ for emphasis. Or you can
_underline_ things if they are really important. These are pretty
universal Usenet currency. Some people are quite fond of
*astericization*, though I'm not too keen myself. I even saw someone
suggest an ~underwater~ font once, for which I have searched in vain
for a use - until now. Why not take it down onto the reef with you and
see if you can't work it into your repertoire?

<reminder>
Oh, and remember the use of pseudo-tags. There's a whole world of fun
waiting to be explored there too.
</reminder>

### Reef Fish

Jun 30, 2005, 7:23:20 AM6/30/05
to

Andy Spragg wrote:
> On 29 Jun 2005 10:12:21 -0700, "Reef Fish"
> <Large_Nass...@Yahoo.com> wrote:
>
> >> >The final installment of this example will be LESSON 4, on Superious
> >> >Correlations, later today. There are many corollary discussions that
>
> >> Are they correlations that are SUPERIOR, SERIOUS, SPURIOUS, or some
> >> weird hybrid of two or more of the above?
>
> >In the context, I hope everyone would understand the misspelling
> >was meant to be "spurious".
>
> That's certainly what I assumed. But for all I know, you might be a
> lover of neologistics. You might have coined "superious"
> intentionally, to indicate "allegedly top quality, but actually
> bogus", for example - would have been entirely consistent with your
> recent contributions.

Your follow-up should have ended there, if you felt a follow-up
was at all warranted.

>
> I wouldn't even have bothered to pick it up had it not been for the
> frequent SHOUTING for no particularly good REASON.

The IDEA about use of caps for shouting is as OLD (and obsolete) as
when I started in newsgroups in the /1980s/. Here's a RECENT post
of mine, in September 2002, in reply to some Clueless Newbie just
like you. :-)

RF> You're new here, aren't you? Since May 2002?

RF> That's my STYLE, used as "italics" for emphasis, instead of wasting

RF> two extra keystrokes to write _errors_ or *errors*. Have used it
RF> that way for years!

RF> Besides, the use of CAPS as "italics" or "underscores" for emphasis

RF> an accepted part of the English language, even in publications
where
RF> italics are available. Check out some nationally syndicated
columns
RF> by James J. Kilpatrick, or Dave Barry, to name just a couple who
RF> frequently use CAPS for non-shouting purposes.

Kilpatrick is a wordsmith who had a weekly column on the proper usage
of English, "Crimes, Misdemeanors, Faux Pas, and Peccadillos," from
which I learned much.

> Bob, it's pretty
> clear that many people who read this NG, myself certainly included,
> have a great deal to learn from you about statistics - that's the only
> limited Usenet experience has taught me that, by and large, excessive
> use of capitalized words in the middle of sentences generally equates
> to kookiness or worse.

You're new here aren't you, since April 27, 2005. And your debut in
newsgroups was on April 5, 2005. Your "limited Usenet experiences"
showed.

>
> Can I suggest that you learn a little about the fine art of rendering
> your pearls of wisdom in more ASCII ways than just speaking and
> SHOUTING?

Of oourse you CAN suggest, and you did, like other Clueless Newbies
had, hundreds of times, and years before you.

> For example, you can use /italics/ for emphasis. Or you can
> _underline_ things if they are really important. These are pretty
> universal Usenet currency. Some people are quite fond of
> *astericization*, though I'm not too keen myself. I even saw someone
> suggest an ~underwater~ font once, for which I have searched in vain
> for a use - until now. Why not take it down onto the reef with you and
> see if you can't work it into your repertoire?

As Yogi would say, "it's deja vu all over again."

Here's an older reference, posted in April 1993:

RF> Let the OTHER people complain if they thought that was SHOUTING.

RF> I use CAPS instead of **caps** or caps for emphasis, as in
RF> italics, without wasting keystrokes or space.

>
> <reminder>
> Oh, and remember the use of pseudo-tags. There's a whole world of fun
> waiting to be explored there too.
> </reminder>

Me thinks you're in the wrong newsgroup, in your maiden voyage into
the sea of USENET.

-- Bob.

### Andy Spragg

Jun 30, 2005, 9:05:09 AM6/30/05
to
On 30 Jun 2005 04:23:20 -0700, "Reef Fish"
<Large_Nass...@Yahoo.com> wrote:

>You're new here aren't you, since April 27, 2005.

If you say so, I didn't mark the date. New interests, new newsgroups.
I lurked for a while, of course, prior to posting.

>And your debut in newsgroups was on April 5, 2005.

No. Nowhere near your vintage, but a few years anyway. I wonder how a
Usenetter of such experience can get it so wrong? I wonder if you made

> Your "limited Usenet experiences" showed.

me that you choose to go along only with the bits of netiquette that
you agree with.

>RF> I use CAPS instead of **caps** or caps for emphasis, as in
>RF> italics, without wasting keystrokes or space.

So you don't count Caps Lock on and off as keystrokes then? Or do you
prefer the method of holding down the Shift key? As for /space/, on my
monitor lower case takes up less space. You must use a fixed pitch
font. How quaint.

### Reef Fish

Jun 30, 2005, 10:35:45 AM6/30/05
to

Andy Spragg wrote:

For YOUR frivol, even though YOU cross-posted, this follow-up goes
only to sci.stat.math, to some some other clueless ones happier. :-)

> On 30 Jun 2005 04:23:20 -0700, "Reef Fish"
> <Large_Nass...@Yahoo.com> wrote:
>
> >You're new here aren't you, since April 27, 2005.
>
> If you say so, I didn't mark the date. New interests, new newsgroups.
> I lurked for a while, of course, prior to posting.

>
> >And your debut in newsgroups was on April 5, 2005.
>
> No. Nowhere near your vintage, but a few years anyway. I wonder how a
> Usenetter of such experience can get it so wrong? I wonder if you made

Of course. Google pointed to everything you posted in USENET since
1981. You could be Anon Bob or anyone else. Why should I care. You
are what you post. Looks clueless, sounds clueless.

>
> > Your "limited Usenet experiences" showed.
>
> You talked about netiquette in the cross-posting thread. It seems to
> me that you choose to go along only with the bits of netiquette that
> you agree with.

Brilliant deduction. Should I have argued against myself?

>
> >RF> I use CAPS instead of **caps** or caps for emphasis, as in
> >RF> italics, without wasting keystrokes or space.
>
> So you don't count Caps Lock on and off as keystrokes then?

What LOCK? Between my typing speed and the keyboard Devil lurking
behind me, unless you're a very careless observer, you'll notice
sometimes I may even skip a few letters.

some clueless non-shouters use all lower-cases. and short hand.
it's ok w me if u write like that.

like i said, u'r in da wrong ng.

-- Bob.

### Jerry Dallal

Jun 30, 2005, 10:54:30 AM6/30/05
to
Reef Fish wrote:
>
> For YOUR frivol, even though YOU cross-posted, this follow-up goes
> only to sci.stat.math, to some some other clueless ones happier. :-)

Once a thread is cross-posted to ssm and sse, I think it pretty much has
to stay cross-posted.

This is an excellent illustration of the potential problem. I'm now in
the habit of checking both groups looking for loose ends to threads I've
joined. I look, and what do I see? A post with the subject "Re: LESSON
3 in Model Building: Practical Significance", a thread that I am
following, posted to ssm but not sse.

--Jerry

### Reef Fish

Jun 30, 2005, 11:46:42 AM6/30/05
to

Jerry Dallal wrote:
> Reef Fish wrote:
> >
> > For YOUR frivol, even though YOU cross-posted, this follow-up goes
> > only to sci.stat.math, to some some other clueless ones happier. :-)
>
> Once a thread is cross-posted to ssm and sse, I think it pretty much has
> to stay cross-posted.

I STRONGLY disagree.

SPAMS may be cross-posted to dozens of ngs. If there's a point
one wishes to make to the primary group one is posting, the
thing to do is NOT to cross-post it to ANY of the other groups.

In fact that's how *I* discovered that google does NOT allow
anyone to cross-post to more than FIVE ngs. So, when I
which I follow-up on was cross-posted to more than 5 groups,
I typically erase ALL the group addresses except ONE.

I DON'T want whatever I post to appear in alt.dallal.for.king
or alt.weird.sex and other groups that were cross-posted for
whatever reason.

Cross-posting or NOT cross-posting is up to the person who

And that was EXACTLY why I don't see any need for the sci.stat.edu
group to see the frivol posted by Andy Spragg, which was bad enough
for just ONE group. But since he addressed it to MY post, I
followed up on it, ERASING his cross-posting address of sci.stat.edu.

>
> This is an excellent illustration of the potential problem. I'm now in
> the habit of checking both groups looking for loose ends to threads I've
> joined. I look, and what do I see? A post with the subject "Re: LESSON
> 3 in Model Building: Practical Significance", a thread that I am
> following, posted to ssm but not sse.
>
> --Jerry

been in both groups. This is part of that two-group thread.

This is only PAET of the problem in the can of worm you and Bruce
Weaver opened up.

I carefully labelled parts of my Moudling Building thread as
LESSON 1, ..., LESSON 4, just to keep various subtopics merely
easily pursued, in the same spirit as different news subgroups
were originally created.

Now that Andy Spragg has polluted this thread, and you jumped
in with your discussion of the CROSS-POSTING TOPIC, rather than
the statistical topic of the thread TITLE, it's not easy to

It's the tangled web we weave, when we can't let well alone.
If it ain't broke, don't fix it.

You are witnessing the disintegration of the wheel you and Bruce
Weaver (and a few in the silent population) were trying to fix.

-- Bob.

### Andy Spragg

Jun 30, 2005, 12:36:33 PM6/30/05
to
On 30 Jun 2005 07:35:45 -0700, "Reef Fish"
<Large_Nass...@Yahoo.com> wrote:

Tried and eschewed it. I prefer a dedicated newsreader application.

>Google pointed to everything you posted in USENET since 1981.

Bzzt. Sorry, can't blame Google there. It pointed to everything I
posted, from work, since a change of work email address a few weeks
ago. The SPURIOUS INFERENCE was all yours.

>like i said, u'r in da wrong ng.

No, don't think so. Wrong person to learn from, though. "It ain't what
you say, it's the way that you say it, /that's/ what gets results".
Small comfort for you being right if no-one CBA to listen. That just
leads to COMPULSIVE SHOUTING. Over and out.