The formula for Entropy in N-gram

[Ahmet GÜNERİ]

Dear Prof.

I am analyzing phrase-frames via AntConc and I know the formula for TTR (variability). However, I am not aware of how the entropy value is calculated in AntConc. What is the formula for entropy?

One more question, I guess it is not exactly the same as predictability. What is the difference between entropy and predictability?

Kind regards,

Ahmet

[Laurence Anthony]

Hi Ahmet,

To answer your question, here is the definition of (normalized) entropy:

(normalized) entropy = -sum(prob_values * np.log(prob_values) / np.log(slot_types))

where "prob_values" are the probabilities of each word in a slot and "slot_types" are the number of types in a slot

Entropy is a measure of uncertainty. So, a low entropy of 0 means the system has only 1 possible state, whereas an entropy of 1 means all the possible states are equally likely. In some ways an entropy value of 0 means the p-frame is completely predictable as there is only one possible slot filler, but for a value of 1, all we can say is that each slot filler is equally likely.

I hope that helps!

Laurence.

###############################################################
Laurence ANTHONY, Ph.D.
Professor of Applied Linguistics
Faculty of Science and Engineering
Waseda University
3-4-1 Okubo, Shinjuku-ku, Tokyo 169-8555, Japan
E-mail: antho...@gmail.com
WWW: http://www.laurenceanthony.net/
###############################################################

--
You received this message because you are subscribed to the Google Groups "AntConc-Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to antconc+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/antconc/CANkj3nQPx8H1RP%2B-XpsTT-uESh52s2iaYh7zx2vBG3qgaw0RPA%40mail.gmail.com.

[Ahmet GÜNERİ]

Dear Prof.

Now I am totally sure of what entropy is. Thank you very much for your contribution to our career.

Kinds regards,

Ahmet

Laurence Anthony <antho...@gmail.com>, 26 Ağu 2024 Pzt, 13:12 tarihinde şunu yazdı:

To view this discussion on the web visit https://groups.google.com/d/msgid/antconc/CAL6Fgv1JJV6e7NH_v3hGwnU7QzKdeCtkd5tT4XdoPZRAkPiPJg%40mail.gmail.com.