Tokens in text and tokens used for word list

[Wai Ling]

Hello. I would like to know in what situation tokens in text and
tokens used for word list will be different. Could it be due to the
numbers in the texts are counted as 1 in the tokens used for word
list?

I have not started using stop list. My texts are tagged and I have
inverted the format into <tag>word.

Thanks in advance for your help.

Regards,
Wai Ling

[Mike]

Dear Wai Ling

Sorry for the delay in answering.

> Hello. I would like to know in what situation tokens in text and
> tokens used for word list will be different. Could it be due to the
> numbers in the texts are counted as 1 in the tokens used for word
> list?

Yes, some entries don't make it into the word-list because of your
user settings. If you don't want numbers to be included, you will get
all words which look like or contain numbers (e.g. 65 or $65 or Y587E)
omitted from the word list. When the stats are computed, those DO
count as "tokens within the text file", but don't count as "tokens
used for the word list".

Cheers -- Mike

[Wai Ling]

Thank you for the explanation.

[Thomas Wolsey]

Hi, just a follow up on this for v8.0.0.29

Tokens in text may be larger because WS excludes numbers and number-like items (y3s, for example).

Thanks! Tom

[Mike Scott]

Hi Tom

Your image shows an oddity in the headers, I see the ( ) brackets strangely reversed as )running words(. Weird! Do you always see that??? (I never have on my UK PC).

Cheers -- Mike

[Thomas Wolsey]

Hi Mike,

Yes, they are always reversed. I thought it odd, but now I think it’s even weirder.   I bought my Dell in the US about two years ago. 

Best, Tom

[Thomas Wolsey]