Finding the range of collocational pairs: Counts of the number of texts in the .mut file in WS4
9 views
Skip to first unread message
Pat
Jan 25, 2008, 6:19:41 AM1/25/08
to WordSmith Tools
We are using a single large file of about 3 million words. The .mut
file for the collocational data often shows 2 texts in the column for
the number of texts in which a relationship between two words is
found.
In another project, we are working with 4 files as the sub-sets of the
corpus. Sometimes the .mut file reports 4 texts.
Is there something we can do to get consistent reports of the number
of texts in which a collocational pair is found?
Or does "text" in the .mut file mean something else that we are not
aware of? We've searched through the online user's guide and only see
"text" defined in terms of the number of different texts that have
been combined to make a corpus.
Thanks.
Pat
file for the collocational data often shows 2 texts in the column for
the number of texts in which a relationship between two words is
found.
In another project, we are working with 4 files as the sub-sets of the
corpus. Sometimes the .mut file reports 4 texts.
Is there something we can do to get consistent reports of the number
of texts in which a collocational pair is found?
Or does "text" in the .mut file mean something else that we are not
aware of? We've searched through the online user's guide and only see
"text" defined in terms of the number of different texts that have
been combined to make a corpus.
Thanks.
Pat
mi...@lexically.net
Jan 25, 2008, 7:33:36 AM1/25/08
to WordSmith Tools
The notion of "text" in WordSmith means what most people would call a
text. In a newspaper, for example, there might be 6 or 7 "texts" on
each page. Yes that is what I meant in the Help but I will go back and
try to clarify better, thanks for the tip!
One large 3m word file is quite a lot. If all your texts (in that
sense) are amalgamated in one file, well I'm afraid WordSmith would
normally have no way of knowing that. It will report the number of
instances of each word or cluster in terms of what it assumes is a
"text", and it assumes (falsely, often of course) that a text = a file
on disk. If it doesn't you're better off totally ignoring the "Texts"
column in WS1-WS5 output.
On the other hand I do know that if you have loads of tiny texts each
only say 400 words long, WordSmith slows up a lot when processing
them. Well, actually it's Windows that slows up, trying to open each
file and soon thereafter close it again. A lot of work if you had to
go over to a filing cabinet and often a drawer to get at a single text
of course. So sometimes it's best to split up extremely large files or
join up tiny little ones.
Hence "Splitter" and Joiner" in the File Utilities.
Cheers -- Mike
text. In a newspaper, for example, there might be 6 or 7 "texts" on
each page. Yes that is what I meant in the Help but I will go back and
try to clarify better, thanks for the tip!
One large 3m word file is quite a lot. If all your texts (in that
sense) are amalgamated in one file, well I'm afraid WordSmith would
normally have no way of knowing that. It will report the number of
instances of each word or cluster in terms of what it assumes is a
"text", and it assumes (falsely, often of course) that a text = a file
on disk. If it doesn't you're better off totally ignoring the "Texts"
column in WS1-WS5 output.
On the other hand I do know that if you have loads of tiny texts each
only say 400 words long, WordSmith slows up a lot when processing
them. Well, actually it's Windows that slows up, trying to open each
file and soon thereafter close it again. A lot of work if you had to
go over to a filing cabinet and often a drawer to get at a single text
of course. So sometimes it's best to split up extremely large files or
join up tiny little ones.
Hence "Splitter" and Joiner" in the File Utilities.
Cheers -- Mike
Pat
Jan 28, 2008, 10:26:18 AM1/28/08
to WordSmith Tools
That's what we assumed about the meaning of "text." So, if we load 1
text, shouldn't WS be saying that only 1 text is involved? While
working with a single text, we are often being told that 2 texts were
involved. We've looked at the text count data in both the latest WS4
and the latest WS5 and see the same inconsistency in reporting the
number of texts.
We're uneasy about using WS to locate the range of collocations in
more complicated setups where the corpus is divided into several sub-
corpora in separate .txt files. In a recent attempt to find the range
of collocates when we were working with 3 separate texts, we' were
told by WS that many collocates occur in 4 texts.
Are any changes underway in WS5 to get accurate text counts reported?
Thanks!
Pat
text, shouldn't WS be saying that only 1 text is involved? While
working with a single text, we are often being told that 2 texts were
involved. We've looked at the text count data in both the latest WS4
and the latest WS5 and see the same inconsistency in reporting the
number of texts.
We're uneasy about using WS to locate the range of collocations in
more complicated setups where the corpus is divided into several sub-
corpora in separate .txt files. In a recent attempt to find the range
of collocates when we were working with 3 separate texts, we' were
told by WS that many collocates occur in 4 texts.
Are any changes underway in WS5 to get accurate text counts reported?
Thanks!
Pat
mi...@lexically.net
Jan 28, 2008, 12:40:36 PM1/28/08
to WordSmith Tools
I will check this too, now I have actually UNDERSTOOD! What you mean
I think is that you get anomalous results where the file is enormous,
such that WS4 or WS5 report 2 files where there's only one. (It is not
really material whether the large file contains one large book or a
mass of small documents as for WS both those would be considered as
single "texts". Now I just need to decide where to get a single file
containing 3 million words...
I think is that you get anomalous results where the file is enormous,
such that WS4 or WS5 report 2 files where there's only one. (It is not
really material whether the large file contains one large book or a
mass of small documents as for WS both those would be considered as
single "texts". Now I just need to decide where to get a single file
containing 3 million words...
mi...@lexically.net
Jan 28, 2008, 1:21:53 PM1/28/08
to WordSmith Tools
Still need more clarification, Pat. Have just made an ordinary
wordlist using 4 large files, each containing about 9.2 million words.
The top words in the wordlist all report 4 texts. (When I say top,
actually I have to go 15,000 words down the frequency list before I
hit anything with less than 4. Not surprising if these large cdrom
files contain nearly 10 million words each. They're each 1/4 of the
Guiardian newspaper for 1995, since you ask.) Now I'll see what
happensif I make an index.
Cheers -- MIke
wordlist using 4 large files, each containing about 9.2 million words.
The top words in the wordlist all report 4 texts. (When I say top,
actually I have to go 15,000 words down the frequency list before I
hit anything with less than 4. Not surprising if these large cdrom
files contain nearly 10 million words each. They're each 1/4 of the
Guiardian newspaper for 1995, since you ask.) Now I'll see what
happensif I make an index.
Cheers -- MIke
mi...@lexically.net
Jan 28, 2008, 1:52:34 PM1/28/08
to WordSmith Tools
Well, in the resulting index all the words seem to have sensible Texts
scores, apart from the number (#) which reports 8, where there are
only 4 text files. The MUT list, though does report some regular words
with Texts scores of "5". I will see what I can do to fix this.
Cheers -- MIke
scores, apart from the number (#) which reports 8, where there are
only 4 text files. The MUT list, though does report some regular words
with Texts scores of "5". I will see what I can do to fix this.
Cheers -- MIke
mi...@lexically.net
Jan 28, 2008, 4:40:51 PM1/28/08
to WordSmith Tools
OK, have found the trouble. I will upload tonight. But unfortunately
the file-number information in existing indexes you may have made
earlier are not reliable. As far as I can tell this will affect the MI
computation Pat brought up and may affect the concgram or index
managing to concordance all the items correctly.
the file-number information in existing indexes you may have made
earlier are not reliable. As far as I can tell this will affect the MI
computation Pat brought up and may affect the concgram or index
managing to concordance all the items correctly.
Pat
Jan 30, 2008, 8:19:17 AM1/30/08
to WordSmith Tools
Thanks, Mike. I just tried the new version and the text count is
correct. Thank you!!!
Pat
correct. Thank you!!!
Pat
Pat
Jan 30, 2008, 8:19:53 AM1/30/08
to WordSmith Tools
Yes, I did a new index and new .mut file and all is well with the text
counts. Pat
counts. Pat
Reply all
Reply to author
Forward
0 new messages