Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss
Please critique my scheme for re-weighting source data
6 views
Skip to first unread message
Jennifer Murphy
Feb 23, 2012, 11:27:19 AM2/23/12
to
I have a table of several thousand words showing how many times each
word occurs in a corpus of several hundred million words.
The table has 7 columns. Here is some sample data.
Word Total A B C D E
aardvark 30 3 9 8 8 2
aback 990 112 542 135 145 56
abacus 119 9 47 25 26 12
abalone 180 0 34 66 59 21
abattoir 116 4 22 24 3 63
abbess 171 1 125 7 6 32
abbey 376 35 138 78 29 96
abnormality 1261 153 37 387 83 601
acculturation 1613 1 2 23 18 1569
coefficient 4499 7 23 77 7 4385
covariate 668 0 0 0 0 668
curricular 1714 7 3 29 17 1658
operand 186 0 0 0 0 186
subscale 4160 1 0 3 0 4156
Columns A-E represent tallies from different types of sources:
Col Source
A Spoken sources (TV, radio, movies)
B Fiction (books)
C Popular magazines
D Newspapers
E Academic journals
The Total column represents the arithmetic sum of columns A-E.
The problem is that the sources contain very different types of words.
The biggest problem is the Academic genre. Those sources tend to use
highly technical terms and jargon and they use some common words in
somewhat unusual ways. There are over 17,000 words with academic tallies
that are at least double the average of the other four genres, over
4,000 that are at least 10 times higher, over 900 that are at least 100
times higher, and almost 500 that are only in the academic genre.
Several examples are included in the table above.
The Spoken genre is also skewed by slang and casual terminology, but to
a much lower degree.
I could just eliminate those two columns, but I would prefer to keep
them in the mix, but at a lower weight. I would like to come up with
some scheme for assigning weighting factors to each column.
One scheme is to assign each column a relative weight. Let's say I want
to give column A 3/4 weight (0.75) and column E 1/4 weight (0.25) as
compared to the other columns. If I assigned weighting factors of 0.75
1.0 1.0 1.0 0.25, I could multiply each score in each column by the
corresponding weighting factors.
This actually produces better results, but it reduces the totals by 4/5.
To keep the overall totals about the same, I could multiply the result
by 5/4 to compensate for reducing the overall weight from 5 to 4.
I would appreciate any comments on this method and any suggestions for a
better one.
Specifically,
1. Is my discounting scheme a reasonable one?
2. Is my readjustment solution appropriate?
3. Is there a better way to do this?
word occurs in a corpus of several hundred million words.
The table has 7 columns. Here is some sample data.
Word Total A B C D E
aardvark 30 3 9 8 8 2
aback 990 112 542 135 145 56
abacus 119 9 47 25 26 12
abalone 180 0 34 66 59 21
abattoir 116 4 22 24 3 63
abbess 171 1 125 7 6 32
abbey 376 35 138 78 29 96
abnormality 1261 153 37 387 83 601
acculturation 1613 1 2 23 18 1569
coefficient 4499 7 23 77 7 4385
covariate 668 0 0 0 0 668
curricular 1714 7 3 29 17 1658
operand 186 0 0 0 0 186
subscale 4160 1 0 3 0 4156
Columns A-E represent tallies from different types of sources:
Col Source
A Spoken sources (TV, radio, movies)
B Fiction (books)
C Popular magazines
D Newspapers
E Academic journals
The Total column represents the arithmetic sum of columns A-E.
The problem is that the sources contain very different types of words.
The biggest problem is the Academic genre. Those sources tend to use
highly technical terms and jargon and they use some common words in
somewhat unusual ways. There are over 17,000 words with academic tallies
that are at least double the average of the other four genres, over
4,000 that are at least 10 times higher, over 900 that are at least 100
times higher, and almost 500 that are only in the academic genre.
Several examples are included in the table above.
The Spoken genre is also skewed by slang and casual terminology, but to
a much lower degree.
I could just eliminate those two columns, but I would prefer to keep
them in the mix, but at a lower weight. I would like to come up with
some scheme for assigning weighting factors to each column.
One scheme is to assign each column a relative weight. Let's say I want
to give column A 3/4 weight (0.75) and column E 1/4 weight (0.25) as
compared to the other columns. If I assigned weighting factors of 0.75
1.0 1.0 1.0 0.25, I could multiply each score in each column by the
corresponding weighting factors.
This actually produces better results, but it reduces the totals by 4/5.
To keep the overall totals about the same, I could multiply the result
by 5/4 to compensate for reducing the overall weight from 5 to 4.
I would appreciate any comments on this method and any suggestions for a
better one.
Specifically,
1. Is my discounting scheme a reasonable one?
2. Is my readjustment solution appropriate?
3. Is there a better way to do this?
Rich Ulrich
Feb 23, 2012, 1:56:58 PM2/23/12
to
You give no hint, that I notice, of what it is that you
are trying to accomplish.
For most purposes of inference that come to my mind,
the extreme cases -- the ones that you seem to propose
to drop -- are the most informative and most interesting.
So I conclude that your interests are probably the opposite
(in some fashion) from what my naive interests would be.
I repeat-- What are you trying to do?
--
Rich Ulrich
are trying to accomplish.
For most purposes of inference that come to my mind,
the extreme cases -- the ones that you seem to propose
to drop -- are the most informative and most interesting.
So I conclude that your interests are probably the opposite
(in some fashion) from what my naive interests would be.
I repeat-- What are you trying to do?
--
Rich Ulrich
Jennifer Murphy
Feb 23, 2012, 2:20:05 PM2/23/12
to
On Thu, 23 Feb 2012 13:56:58 -0500, Rich Ulrich
<rich....@comcast.net> wrote:
>You give no hint, that I notice, of what it is that you
>are trying to accomplish.
>
>For most purposes of inference that come to my mind,
>the extreme cases -- the ones that you seem to propose
>to drop -- are the most informative and most interesting.
>So I conclude that your interests are probably the opposite
>(in some fashion) from what my naive interests would be.
>
>I repeat-- What are you trying to do?
I am trying to calculate for each word the relative likeliness that it
<rich....@comcast.net> wrote:
>You give no hint, that I notice, of what it is that you
>are trying to accomplish.
>
>For most purposes of inference that come to my mind,
>the extreme cases -- the ones that you seem to propose
>to drop -- are the most informative and most interesting.
>So I conclude that your interests are probably the opposite
>(in some fashion) from what my naive interests would be.
>
>I repeat-- What are you trying to do?
would be encountered by an average well-educated person in their daily
activities: reading the paper, listening to the news, attending classes,
talking to other people, reading books, etc.
The raw scores that I have already do that, but I question the
weighting.I do not think that the average person encounters the types of
words typically found in academic journals at the same frequency as they
would those found in newspapers or magazines. Therefore, I want to
re-weight the five sources to reflect a more average experience.
Gus Gassmann
Feb 23, 2012, 4:07:30 PM2/23/12
to
On Feb 23, 3:20 pm, Jennifer Murphy <JenMur...@jm.invalid> wrote:
> On Thu, 23 Feb 2012 13:56:58 -0500, Rich Ulrich
>
> On Thu, 23 Feb 2012 13:56:58 -0500, Rich Ulrich
>
information` that the `average well-educated person` would encounter
in their `daily activities`. So, I guess, those should be your
weights. In other words, you need to assess the volume of spoken
sources, etc. the average person you are interested in is exposed to
in each of the five groups. You could use that as a basis for your
weighting. Whether it beats the raw (equal) weights is not immediately
clear.
Ray Vickson
Feb 23, 2012, 6:19:31 PM2/23/12
to
On Feb 23, 11:20 am, Jennifer Murphy <JenMur...@jm.invalid> wrote:
> On Thu, 23 Feb 2012 13:56:58 -0500, Rich Ulrich
>
> On Thu, 23 Feb 2012 13:56:58 -0500, Rich Ulrich
>
journal. Whether or not a well-educated person will read a novel (or
fiction in general) will depend strongly on whether that person is
male or female.
RGV
JohnF
Feb 24, 2012, 5:35:15 AM2/24/12
to
Jennifer Murphy <JenM...@jm.invalid> wrote:
That is, define a person by a "state vector"
p = <w_A,w_B,...,w_E>
representing his inclination/weight to read each
kind of source. You're now kind of using p=<.2,.2,.2,.2,.2>.
Is that really "average"? Or maybe you can't define
a single average person. College-educated will probably have
a different vector than high-school dropouts.
So you ultimately have a five-dimensional (that is,
#sources-dimensional) people space, with each point in that
space having its own "likelihood distribution" for coming
across your words. ... Or something like that. The basic
point, again, being to weight the people.
--
John Forkosh ( mailto: j...@f.com where j=john and f=forkosh )
> Rich Ulrich <rich....@comcast.net> wrote:
>
>> What are you trying to do?
>
> I am trying to calculate for each word the relative likeliness that it
> would be encountered by an average well-educated person in their daily
> activities: reading the paper, listening to the news, attending classes,
> talking to other people, reading books, etc.
>
> The raw scores that I have already do that, but I question the
> weighting.I do not think that the average person encounters the types of
> words typically found in academic journals at the same frequency as they
> would those found in newspapers or magazines. Therefore, I want to
> re-weight the five sources to reflect a more average experience.
Don't weight the sources, weight the people.
>
>> What are you trying to do?
>
> I am trying to calculate for each word the relative likeliness that it
> would be encountered by an average well-educated person in their daily
> activities: reading the paper, listening to the news, attending classes,
> talking to other people, reading books, etc.
>
> The raw scores that I have already do that, but I question the
> weighting.I do not think that the average person encounters the types of
> words typically found in academic journals at the same frequency as they
> would those found in newspapers or magazines. Therefore, I want to
> re-weight the five sources to reflect a more average experience.
That is, define a person by a "state vector"
p = <w_A,w_B,...,w_E>
representing his inclination/weight to read each
kind of source. You're now kind of using p=<.2,.2,.2,.2,.2>.
Is that really "average"? Or maybe you can't define
a single average person. College-educated will probably have
a different vector than high-school dropouts.
So you ultimately have a five-dimensional (that is,
#sources-dimensional) people space, with each point in that
space having its own "likelihood distribution" for coming
across your words. ... Or something like that. The basic
point, again, being to weight the people.
--
John Forkosh ( mailto: j...@f.com where j=john and f=forkosh )
Peter Webb
Feb 24, 2012, 6:43:15 AM2/24/12
to
Like the others have said, you weight your data according to the relative
number of words the "average well educated person" hears/reads from each
source. Which you have no practical way of determining, because as far as I
know there has been no study of how "average well educated people" spend
their time absorbing information.
You could guess, but the answers you eventually get will get will be
meaningless, as the answers will reflect your weightings just as much as the
data, and you have no means of weighting the data. Or rather, each possible
weighting just reflects your arbitrary assumption of what an "average
educated person" sees or hears.
So for example (somebody else's example) the average educated person spends
zero time reading academic journals, so that's a zero. The readership of the
internet, of novels, watching TV etc varies enormously across "average
educated person". How do you "weight" words that appear in incidental
dialogue on some reality TV show with words read on a web page? These are
completely different media.
You have (I believe) zero chance on forming numbers which actually mean
something by weighting the data and adding it together. You are weighting
and adding apples and oranges.
The interesting thing is the variation between media of the vocabularies;
add these together and you are throwing away the most interesting data.
If you are doing this because somebody asked you to, you need to either get
a lot more information from them (as to why you are doing it) or tell them
its impossible.
"Jennifer Murphy" <JenM...@jm.invalid> wrote in message
news:so3dk713f73qgao2b...@4ax.com...
number of words the "average well educated person" hears/reads from each
source. Which you have no practical way of determining, because as far as I
know there has been no study of how "average well educated people" spend
their time absorbing information.
You could guess, but the answers you eventually get will get will be
meaningless, as the answers will reflect your weightings just as much as the
data, and you have no means of weighting the data. Or rather, each possible
weighting just reflects your arbitrary assumption of what an "average
educated person" sees or hears.
So for example (somebody else's example) the average educated person spends
zero time reading academic journals, so that's a zero. The readership of the
internet, of novels, watching TV etc varies enormously across "average
educated person". How do you "weight" words that appear in incidental
dialogue on some reality TV show with words read on a web page? These are
completely different media.
You have (I believe) zero chance on forming numbers which actually mean
something by weighting the data and adding it together. You are weighting
and adding apples and oranges.
The interesting thing is the variation between media of the vocabularies;
add these together and you are throwing away the most interesting data.
If you are doing this because somebody asked you to, you need to either get
a lot more information from them (as to why you are doing it) or tell them
its impossible.
"Jennifer Murphy" <JenM...@jm.invalid> wrote in message
news:so3dk713f73qgao2b...@4ax.com...
James Beck
May 24, 2012, 11:09:44 PM5/24/12
to
knows?
The core vocabulary in academic writing is actually very small
compared to the others, about 3,000 words. The broad sub-categories of
academic writing each have a technical core of about a 1,000 words
(also pretty common). However, each paper includes 3-5 specialized,
idiosyncratic words, usually familiar to the small group of people
interested in the paper, but not in wide use otherwise. Weighting the
people as you suggest preserves the worst aspect of the data and
perversely implies that the simple, core vocabulary is less likely to
be encountered than it is.
The OP is closer to the right track. As a practical matter, the
likelihood of encountering the idiosyncratic words at random is close
to zero. It would be more robust to extract the general and
subcategory cores and re-weight the rest.
aruzinsky
May 27, 2012, 11:57:24 AM5/27/12
to
On Feb 23, 5:19 pm, Ray Vickson <RGVick...@shaw.ca> wrote:
> On Feb 23, 11:20 am, Jennifer Murphy <JenMur...@jm.invalid> wrote:
>
>
>
>
>
> On Feb 23, 11:20 am, Jennifer Murphy <JenMur...@jm.invalid> wrote:
>
>
>
>
>
> > On Thu, 23 Feb 2012 13:56:58 -0500, Rich Ulrich
>
>
> > <rich.ulr...@comcast.net> wrote:
> > >You give no hint, that I notice, of what it is that you
> > >are trying to accomplish.
>
> > >For most purposes of inference that come to my mind,
> > >the extreme cases -- the ones that you seem to propose
> > >to drop -- are the most informative and most interesting.
> > >So I conclude that your interests are probably the opposite
> > >(in some fashion) from what my naive interests would be.
>
> > >I repeat-- What are you trying to do?
> > >You give no hint, that I notice, of what it is that you
> > >are trying to accomplish.
>
> > >For most purposes of inference that come to my mind,
> > >the extreme cases -- the ones that you seem to propose
> > >to drop -- are the most informative and most interesting.
> > >So I conclude that your interests are probably the opposite
> > >(in some fashion) from what my naive interests would be.
>
>
> > I am trying to calculate for each word the relative likeliness that it
> > would be encountered by an average well-educated person in their daily
> > activities: reading the paper, listening to the news, attending classes,
> > talking to other people, reading books, etc.
>
> > The raw scores that I have already do that, but I question the
> > weighting.I do not think that the average person encounters the types of
> > words typically found in academic journals at the same frequency as they
> > would those found in newspapers or magazines. Therefore, I want to
> > re-weight the five sources to reflect a more average experience.
>
> > I am trying to calculate for each word the relative likeliness that it
> > would be encountered by an average well-educated person in their daily
> > activities: reading the paper, listening to the news, attending classes,
> > talking to other people, reading books, etc.
>
> > The raw scores that I have already do that, but I question the
> > weighting.I do not think that the average person encounters the types of
> > words typically found in academic journals at the same frequency as they
> > would those found in newspapers or magazines. Therefore, I want to
> > re-weight the five sources to reflect a more average experience.
>
> The "average" well-educated person will never read an academic
> journal. Whether or not a well-educated person will read a novel (or
> fiction in general) will depend strongly on whether that person is
> male or female.
>
> RGV- Hide quoted text -
> journal. Whether or not a well-educated person will read a novel (or
> fiction in general) will depend strongly on whether that person is
> male or female.
>
>
> - Show quoted text -
I strongly disagree. Papers from scientific and engineering journals
are easily obtained on the internet, some for free and others for a
price. Abstracts are aways free. Almost eventually, a well educated
person will have a health problem which will prompt that person to
read papers from medical journals. For example, I have a Ph.D. in
electrical engineering and, at age 65, I have been diagnosed with T2
diabetes. I read many scientific papers about T2 diabetes.
Put it this way, if your life depended on it, you would be a God damn
fool not to read papers from medical journals.
0 new messages