LESSON 3 in Model Building: Practical Significance

0 views
Skip to first unread message

Reef Fish

unread,
Jun 29, 2005, 8:32:43 AM6/29/05
to
This is the lesson that is seldom, if ever, discussed in textbooks, on
regression problems, but SHOULD be, on every problem, on assessing the
PRACTICAL significance (or usefulness or uselessness) or the result, as
opposed to the statistical significance of the result.

Here's the model and its sgatistical significance, repeated from LESSON
2:

INVDEX = -197.51 + 0.018234 * GNP
(15.031) (.000667)
T=-13.14 T=27.33
p-value 10^(-12)

Multiple R-sq = 0.9726, MSE = 317.96.

All very impressive and highly statistically significant


I am going to demonstrate that is is COMPLETELY USELESS, in practice,
for
prediction purposes.

For this EXERCISE, I had saved the last observation, to check how the
model
would have done, on its PREDICTION interval.


Suppose we found the model in 1965, and used the result for prediction
in
1966 (pretending that the GNP for the year was available before the
INVDEX
was), we would have found a 95% prediction interval to be

(413.1, 498.2)

The width of this interval (relative to the annual variation of INVDEX)
would have dampened our happiness-index (on the basis of the
statistical
significance of T and other measures) considerably. In fact, one may
judge the prediction interval result to be so WIDE as to render the
result useless.

IDA, the interactive data analysis package, had a built-in command
(VALI)
to let users to selectively look at the predictive performance of any
number of rows in the data matrix, to empirically "validate" the
"prediction" interval results. (IN if the actual observation is within
the prediction interval. OUT otherwise). This would have been the
result for the last several rows:

Abs. Error Abs.% error Abs. % Error Length of 95% IN=1
ROW |Y - YPRED| 100|Error-Y| 100|Error-YPRED| Pred.Interval OUT=0

28 3.87 1.23 1.22 77.9 1
29 8.76 2.61 2.54 78.9 1
30 14.41 3.66 3.80 80.5 1
31 14.06 3.24 3.35 82.8 1
32 47.16 11.54 10.35 85.1 0

In spite of the widths of the prediction intervals, the prediction
interval would have MISSED the actual observed value of 408.5 for
1966!

Let's see how we might have done had we not used ANY predictor
variable,
and just use the ANNUAL CHANGE of INVDEX to predict the 1966 value.

The annual change has a mean of 16.28 (increase) for INVDEX. A 95%
Prediction Interval for the CHANGE would be

16.28 +- t(.975;22) (23.45) = (-32.35 , 64.91)

The above, added to the observed value of 433.1 in 1965 would have
yielded a prediction interval of

(400.8, 499.0)

WITHOUT the use of any independent variable for prediction. It is
wider than the corresponding prediction interval using the GNP
variable,
to be sure, but it INCLUDEED the actual value 408.5 observed in 1996,
whereas the prediction interval of the model with GNP,

(413.1, 498.2)

missed the actual observed value (408.5). This kind of post hoc
"validation" check, should give one, in this particular example of the
SPSS Multiple Regression data, a sobering thought about the USELESSNESS
of any regression model, in spite of the STATISTICAL significance of
the regression results.

This phenomenon is the result of the "spurious correlation" between
two increasing time series. It happens ALL the time. It gives still
another validation of Tukey's caveat that the use of ANY correlation
is "sweep dirt under the rug with a vengeance".

The final installment of this example will be LESSON 4, on Superious
Correlations, later today. There are many corollary discussions that
may follow afterwards.

-- Bob.

Art Kendall

unread,
Jun 29, 2005, 9:18:08 AM6/29/05
to
A very important point. Right on.

This is related to what social scientists, especially psychologists,
call the "so what" test. For example, in a given experiment, the
apparent difference between the mean for the group of cases randomly
assigned to one treatment and the mean for the cases randomly assigned
to the other treatment is not readily attributable to random variation
in the dependent variable. That is, the difference is "statistically
significant".

It often helps clients/student to say: Another way to put the idea of
statistical significance is that the observed
difference/association/relation/correlation is sufficiently large that
it is not very consistent with the
differences/associations/relations/correlations that would occur if
there were no difference/association/relation/correlation and many
random samples were used.

For at least the last thirty years, psychologists have been urged to
avoid the term "significant" without a qualifier. In more recent years,
there has been a strong and growing emphasis on including some estimate
of "effect size" to aid in figuring out
theoretical/policy/administrative/clinical significance. It sometimes
helps clients/students to express this concept in term of "meaningful".


Art
A...@DrKendall.org
Social Research Consultants
University Park, MD USA
(301) 864-5570

Andy Spragg

unread,
Jun 29, 2005, 12:54:31 PM6/29/05
to
On 29 Jun 2005 05:32:43 -0700, "Reef Fish"

<Large_Nass...@Yahoo.com> wrote:
>The final installment of this example will be LESSON 4, on Superious
>Correlations, later today. There are many corollary discussions that
>may follow afterwards.

Are they correlations that are SUPERIOR, SERIOUS, SPURIOUS, or some
weird hybrid of two or more of the above?

Reef Fish

unread,
Jun 29, 2005, 1:12:21 PM6/29/05
to

Thanks for catch my typo.

In the context, I hope everyone would understand the misspelling
was meant to be "spurious".

I'll make sure I spell it correctly in the SUBJECT of LESSON 4. :-)

-- Bob.

Russell...@wdn.com

unread,
Jun 29, 2005, 1:22:27 PM6/29/05
to
It actually might be a reasonable term to introduce to the field-
superious: correlations that are really good but for the "wrong"
reason. :-)

Cheers,
Russell

G Robin Edwards

unread,
Jun 29, 2005, 7:35:50 PM6/29/05
to
In article <1120048363....@g14g2000cwa.googlegroups.com>, Reef

> (413.1, 498.2)

Agreed with all of these, though I do it graphically for simple
or polynomial models. There's also another datum that falls outside
the 95% range, the 1961 value. But that's confidence intervals for
you, I guess. One in 20 may well be outside the 95% CI without causing
any surprise.

> (400.8, 499.0)

> (413.1, 498.2)

Haven't got around to looking at that, yet. It is half past midnight
so I'm off to bed.

However, I've noticed that my original (and rather poor) model using
all the data except 1966 comfortably includes the 1966 value within its
95% CI. Does this mean anything worth noting? And I wonder what the
1976 figures would have told us about the models. Alas, we shall never
know.

I look forward to tomorrow's reading. Will post this tonight.

Robin


Reef Fish

unread,
Jun 29, 2005, 8:03:12 PM6/29/05
to

G Robin Edwards wrote:
> In article <1120048363....@g14g2000cwa.googlegroups.com>, Reef
> Fish <Large_Nass...@Yahoo.com> wrote:
> > This is the lesson that is seldom, if ever, discussed in textbooks,
> > on regression problems, but SHOULD be, on every problem, on assessing
> > the PRACTICAL significance (or usefulness or uselessness) or the
> > result, as opposed to the statistical significance of the result.

> > The final installment of this example will be LESSON 4, on Superious
> > Correlations, later today. There are many corollary discussions that
> > may follow afterwards.
>
> Haven't got around to looking at that, yet. It is half past midnight
> so I'm off to bed.

Unless you're clairvoyant, I haven't WRITTEN LESSON 4 yet. :-)

>
> However, I've noticed that my original (and rather poor) model using
> all the data except 1966 comfortably includes the 1966 value within its
> 95% CI. Does this mean anything worth noting? And I wonder what the
> 1976 figures would have told us about the models. Alas, we shall never
> know.

This paragraph of yours is worth a comment. Your model had about
FIVE TIMES the MSE of mine or very approximately 2xsqrt(5) or 4.46
times the WIDTH of my prediction interval for INVDEX of 1966, which I
considered much too wide to have any PRACTICAL significance.

4 times that width should catch any INVDEX value in the last century.
:-)

In that respect, the fact that your model "comfortably includes
the 1996 value" should mean ... "nothing". :)


>
> I look forward to tomorrow's reading. Will post this tonight.
>
> Robin

It appears you are lagging by about 1 day in your reading. There
are both advantages and disadvantages in such delays.

-- Bob.

Reef Fish

unread,
Jun 29, 2005, 1:46:12 PM6/29/05
to

Good idea! Even better, a "supersilly superious" correlation,
or a "supersillious superious" correlation as one that
a social scientist can thumbs his nose at non-believers that
a causation has been proven! :-)

-- Bob.

Reef Fish

unread,
Jun 29, 2005, 1:36:31 PM6/29/05
to

Art Kendall wrote:
> A very important point. Right on.
>
> This is related to what social scientists, especially psychologists,
> call the "so what" test.
>

> It often helps clients/student to say: Another way to put the idea of
> statistical significance is that the observed
> difference/association/relation/correlation is sufficiently large that
> it is not very consistent with the
> differences/associations/relations/correlations that would occur if
> there were no difference/association/relation/correlation and many
> random samples were used.

I recently discussed the notion of "significantly correlated"
variables in the sense of rejecting the null hypothesis of Ho: rho = 0
in the thread http://tinyurl.com/arkvs

"Do the critical values of linear correlation depend on sample size? "

There I gave the rule of thumb for LARGE samples that a sample
correlation r is statistically significant (at the 5% level) in
that sense if the absolute value of r is greater than 2/sqrt(n).

Thus, for a sample of aize 10,000,

the critical value for "significance" would have been .02.

I can safely challenge that NO ONE can tell the difference bwtween
the scatterplot of 10,000 points with r = 0 and r = .02.

In fact, it'll be difficult to tell the difference scatter for
even 100 points or r = 0 vs r = 0.2.

They would ALL look like the holes left from a shotfun blast
shot at a wall.

Try that experiment of generating bivariate normal samples of
say 1,000 with rho = 0.5, 0.4, ..., 0; (if you know how to
generate these) and look at their scatterplots.

>
> For at least the last thirty years, psychologists have been urged to
> avoid the term "significant" without a qualifier. In more recent years,
> there has been a strong and growing emphasis on including some estimate
> of "effect size" to aid in figuring out

> theoretical/policy/administrat­ive/clinical significance. It sometimes


> helps clients/students to express this concept in term of "meaningful

This is not always possible on measures like "correlation", as shown
in the examples I suggested. Not many statisticians, let alone
non-statisticians, have a feel of the random LOOK in a scatterplot
of a statisticall highly significant correlation, for most sample
sizes.

The differences in MEANS are much more tangible in terms of assessing
any PRACTICAL significance.

-- Bob.

Andy Spragg

unread,
Jun 30, 2005, 6:01:07 AM6/30/05
to
On 29 Jun 2005 10:12:21 -0700, "Reef Fish"
<Large_Nass...@Yahoo.com> wrote:

>> >The final installment of this example will be LESSON 4, on Superious
>> >Correlations, later today. There are many corollary discussions that
>> >may follow afterwards.

>> Are they correlations that are SUPERIOR, SERIOUS, SPURIOUS, or some
>> weird hybrid of two or more of the above?

>In the context, I hope everyone would understand the misspelling


>was meant to be "spurious".

That's certainly what I assumed. But for all I know, you might be a
lover of neologistics. You might have coined "superious"
intentionally, to indicate "allegedly top quality, but actually
bogus", for example - would have been entirely consistent with your
recent contributions.

I wouldn't even have bothered to pick it up had it not been for the
combination of your relentlessly DIDACTIC tone and your devotion to
frequent SHOUTING for no particularly good REASON. Bob, it's pretty
clear that many people who read this NG, myself certainly included,
have a great deal to learn from you about statistics - that's the only
reason I didn't stop reading your stuff already. Even my relatively
limited Usenet experience has taught me that, by and large, excessive
use of capitalized words in the middle of sentences generally equates
to kookiness or worse.

Can I suggest that you learn a little about the fine art of rendering
your pearls of wisdom in more ASCII ways than just speaking and
SHOUTING? For example, you can use /italics/ for emphasis. Or you can
_underline_ things if they are really important. These are pretty
universal Usenet currency. Some people are quite fond of
*astericization*, though I'm not too keen myself. I even saw someone
suggest an ~underwater~ font once, for which I have searched in vain
for a use - until now. Why not take it down onto the reef with you and
see if you can't work it into your repertoire?

<reminder>
Oh, and remember the use of pseudo-tags. There's a whole world of fun
waiting to be explored there too.
</reminder>

Reef Fish

unread,
Jun 30, 2005, 7:23:20 AM6/30/05
to

Andy Spragg wrote:
> On 29 Jun 2005 10:12:21 -0700, "Reef Fish"
> <Large_Nass...@Yahoo.com> wrote:
>
> >> >The final installment of this example will be LESSON 4, on Superious
> >> >Correlations, later today. There are many corollary discussions that
> >> >may follow afterwards.
>
> >> Are they correlations that are SUPERIOR, SERIOUS, SPURIOUS, or some
> >> weird hybrid of two or more of the above?
>
> >In the context, I hope everyone would understand the misspelling
> >was meant to be "spurious".
>
> That's certainly what I assumed. But for all I know, you might be a
> lover of neologistics. You might have coined "superious"
> intentionally, to indicate "allegedly top quality, but actually
> bogus", for example - would have been entirely consistent with your
> recent contributions.

Your follow-up should have ended there, if you felt a follow-up
was at all warranted.

>
> I wouldn't even have bothered to pick it up had it not been for the
> combination of your relentlessly DIDACTIC tone and your devotion to
> frequent SHOUTING for no particularly good REASON.

The IDEA about use of caps for shouting is as OLD (and obsolete) as
when I started in newsgroups in the /1980s/. Here's a RECENT post
of mine, in September 2002, in reply to some Clueless Newbie just
like you. :-)

RF> You're new here, aren't you? Since May 2002?

RF> That's my STYLE, used as "italics" for emphasis, instead of wasting

RF> two extra keystrokes to write _errors_ or *errors*. Have used it
RF> that way for years!

RF> Besides, the use of CAPS as "italics" or "underscores" for emphasis

RF> an accepted part of the English language, even in publications
where
RF> italics are available. Check out some nationally syndicated
columns
RF> by James J. Kilpatrick, or Dave Barry, to name just a couple who
RF> frequently use CAPS for non-shouting purposes.

Kilpatrick is a wordsmith who had a weekly column on the proper usage
of English, "Crimes, Misdemeanors, Faux Pas, and Peccadillos," from
which I learned much.


> Bob, it's pretty
> clear that many people who read this NG, myself certainly included,
> have a great deal to learn from you about statistics - that's the only
> reason I didn't stop reading your stuff already. Even my relatively
> limited Usenet experience has taught me that, by and large, excessive
> use of capitalized words in the middle of sentences generally equates
> to kookiness or worse.

You're new here aren't you, since April 27, 2005. And your debut in
newsgroups was on April 5, 2005. Your "limited Usenet experiences"
showed.


>
> Can I suggest that you learn a little about the fine art of rendering
> your pearls of wisdom in more ASCII ways than just speaking and
> SHOUTING?

Of oourse you CAN suggest, and you did, like other Clueless Newbies
had, hundreds of times, and years before you.

> For example, you can use /italics/ for emphasis. Or you can
> _underline_ things if they are really important. These are pretty
> universal Usenet currency. Some people are quite fond of
> *astericization*, though I'm not too keen myself. I even saw someone
> suggest an ~underwater~ font once, for which I have searched in vain
> for a use - until now. Why not take it down onto the reef with you and
> see if you can't work it into your repertoire?

As Yogi would say, "it's deja vu all over again."

Here's an older reference, posted in April 1993:

RF> Let the OTHER people complain if they thought that was SHOUTING.

RF> I use CAPS instead of **caps** or caps for emphasis, as in
RF> italics, without wasting keystrokes or space.

>
> <reminder>
> Oh, and remember the use of pseudo-tags. There's a whole world of fun
> waiting to be explored there too.
> </reminder>

Me thinks you're in the wrong newsgroup, in your maiden voyage into
the sea of USENET.

-- Bob.

Andy Spragg

unread,
Jun 30, 2005, 9:05:09 AM6/30/05
to
On 30 Jun 2005 04:23:20 -0700, "Reef Fish"
<Large_Nass...@Yahoo.com> wrote:

>You're new here aren't you, since April 27, 2005.

If you say so, I didn't mark the date. New interests, new newsgroups.
I lurked for a while, of course, prior to posting.

>And your debut in newsgroups was on April 5, 2005.

No. Nowhere near your vintage, but a few years anyway. I wonder how a
Usenetter of such experience can get it so wrong? I wonder if you made
the mistake of limiting your search to my current Reply To address?

> Your "limited Usenet experiences" showed.

You talked about netiquette in the cross-posting thread. It seems to
me that you choose to go along only with the bits of netiquette that
you agree with.

>RF> I use CAPS instead of **caps** or caps for emphasis, as in
>RF> italics, without wasting keystrokes or space.

So you don't count Caps Lock on and off as keystrokes then? Or do you
prefer the method of holding down the Shift key? As for /space/, on my
monitor lower case takes up less space. You must use a fixed pitch
font. How quaint.

Reef Fish

unread,
Jun 30, 2005, 10:35:45 AM6/30/05
to

Andy Spragg wrote:

For YOUR frivol, even though YOU cross-posted, this follow-up goes
only to sci.stat.math, to some some other clueless ones happier. :-)


> On 30 Jun 2005 04:23:20 -0700, "Reef Fish"
> <Large_Nass...@Yahoo.com> wrote:
>
> >You're new here aren't you, since April 27, 2005.
>
> If you say so, I didn't mark the date. New interests, new newsgroups.
> I lurked for a while, of course, prior to posting.

I guess you haven't found Google as your working slave yet.

>
> >And your debut in newsgroups was on April 5, 2005.
>
> No. Nowhere near your vintage, but a few years anyway. I wonder how a
> Usenetter of such experience can get it so wrong? I wonder if you made
> the mistake of limiting your search to my current Reply To address?

Of course. Google pointed to everything you posted in USENET since
1981. You could be Anon Bob or anyone else. Why should I care. You
are what you post. Looks clueless, sounds clueless.


>
> > Your "limited Usenet experiences" showed.
>
> You talked about netiquette in the cross-posting thread. It seems to
> me that you choose to go along only with the bits of netiquette that
> you agree with.

Brilliant deduction. Should I have argued against myself?


>
> >RF> I use CAPS instead of **caps** or caps for emphasis, as in
> >RF> italics, without wasting keystrokes or space.
>
> So you don't count Caps Lock on and off as keystrokes then?

What LOCK? Between my typing speed and the keyboard Devil lurking
behind me, unless you're a very careless observer, you'll notice
sometimes I may even skip a few letters.

some clueless non-shouters use all lower-cases. and short hand.
it's ok w me if u write like that.

like i said, u'r in da wrong ng.

-- Bob.

Jerry Dallal

unread,
Jun 30, 2005, 10:54:30 AM6/30/05
to
Reef Fish wrote:
>
> For YOUR frivol, even though YOU cross-posted, this follow-up goes
> only to sci.stat.math, to some some other clueless ones happier. :-)

Once a thread is cross-posted to ssm and sse, I think it pretty much has
to stay cross-posted.

This is an excellent illustration of the potential problem. I'm now in
the habit of checking both groups looking for loose ends to threads I've
joined. I look, and what do I see? A post with the subject "Re: LESSON
3 in Model Building: Practical Significance", a thread that I am
following, posted to ssm but not sse.

--Jerry

Reef Fish

unread,
Jun 30, 2005, 11:46:42 AM6/30/05
to

Jerry Dallal wrote:
> Reef Fish wrote:
> >
> > For YOUR frivol, even though YOU cross-posted, this follow-up goes
> > only to sci.stat.math, to some some other clueless ones happier. :-)
>
> Once a thread is cross-posted to ssm and sse, I think it pretty much has
> to stay cross-posted.

I STRONGLY disagree.

SPAMS may be cross-posted to dozens of ngs. If there's a point
one wishes to make to the primary group one is posting, the
thing to do is NOT to cross-post it to ANY of the other groups.

In fact that's how *I* discovered that google does NOT allow
anyone to cross-post to more than FIVE ngs. So, when I
inadvertently hit SEND when I reply, not knowing the post to
which I follow-up on was cross-posted to more than 5 groups,
google would keep me from replying to all the groups. So,
I typically erase ALL the group addresses except ONE.

I DON'T want whatever I post to appear in alt.dallal.for.king
or alt.weird.sex and other groups that were cross-posted for
whatever reason.

Cross-posting or NOT cross-posting is up to the person who
POSTS his reply!

And that was EXACTLY why I don't see any need for the sci.stat.edu
group to see the frivol posted by Andy Spragg, which was bad enough
for just ONE group. But since he addressed it to MY post, I
followed up on it, ERASING his cross-posting address of sci.stat.edu.

>
> This is an excellent illustration of the potential problem. I'm now in
> the habit of checking both groups looking for loose ends to threads I've
> joined. I look, and what do I see? A post with the subject "Re: LESSON
> 3 in Model Building: Practical Significance", a thread that I am
> following, posted to ssm but not sse.
>
> --Jerry

But I am adding back sse, because YOUR view on the subject had
been in both groups. This is part of that two-group thread.

If your post following mine addressed ONLY the frivol of Andy
Spragg, I would have let your single-group address stand.

This is only PAET of the problem in the can of worm you and Bruce
Weaver opened up.

I carefully labelled parts of my Moudling Building thread as
LESSON 1, ..., LESSON 4, just to keep various subtopics merely
easily pursued, in the same spirit as different news subgroups
were originally created.

Now that Andy Spragg has polluted this thread, and you jumped
in with your discussion of the CROSS-POSTING TOPIC, rather than
the statistical topic of the thread TITLE, it's not easy to
get back on thread.

It's the tangled web we weave, when we can't let well alone.
If it ain't broke, don't fix it.

You are witnessing the disintegration of the wheel you and Bruce
Weaver (and a few in the silent population) were trying to fix.

-- Bob.

Andy Spragg

unread,
Jun 30, 2005, 12:36:33 PM6/30/05
to
On 30 Jun 2005 07:35:45 -0700, "Reef Fish"
<Large_Nass...@Yahoo.com> wrote:

>I guess you haven't found Google as your working slave yet.

Tried and eschewed it. I prefer a dedicated newsreader application.

>Google pointed to everything you posted in USENET since 1981.

Bzzt. Sorry, can't blame Google there. It pointed to everything I
posted, from work, since a change of work email address a few weeks
ago. The SPURIOUS INFERENCE was all yours.

>like i said, u'r in da wrong ng.

No, don't think so. Wrong person to learn from, though. "It ain't what
you say, it's the way that you say it, /that's/ what gets results".
Small comfort for you being right if no-one CBA to listen. That just
leads to COMPULSIVE SHOUTING. Over and out.

Reply all
Reply to author
Forward
0 new messages