Re: Questions....

[Adam Pauls]

1. Yes, the backoff and smoothing used by these methods are different. I recommend reading http://www.aclweb.org/anthology/D07-1090 for more detail.

2. We typically map unknown words to and <UNK> token, which is then treated like any other word. unknownWordLogProb is only used when no <UNK> token is provided, and when a completely unseen word is found.

3. The code is here: http://code.google.com/p/berkeleylm/source/browse/trunk/src/edu/berkeley/nlp/lm/io/KneserNeyLmReaderCallback.java

On Thu, Mar 12, 2015 at 8:14 PM, 한정수 <ky...@akaon.com> wrote:

Hi, I'm studying(?) BerkeleyLM(https://code.google.com/p/berkeleylm/) in korea. 

BerkeleyLM said that it provides two LMs(kneserNeyLM and stupid backoff LM). 

I wonder that 

      1. what is the difference between two LMs? that is backoff method?  

     

      2. In ConfigOptions(edu.berkeley.nlp.lm.ConfigOptions.java), there is "unknownWordLogProb =-100.0f;". This mean KneserNeyLM do not provide backoff method?? 

     

      3. If there is backoff method in KneserNeyLM, where is the code?

Sorry for my english writing skills.

Thank you!

--
You received this message because you are subscribed to the Google Groups "berkeleylm-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to berkeleylm-disc...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.