-Infinity log probability

[Roman Prokofyev]

Hello,

I'm trying to compute log probability on a test collection and notices that around 15 sentences out if 57k (Brown corpus) yield log probability of -Infinity.

So I'm wondering why's that? Isn't stupid backoff model is smoothed as well? Or only Knesser-Ney?

Thanks.

[Adam Pauls]

This happens with out of vocabulary words. You need to include an <UNK> token in your data to handle this properly.

--
You received this message because you are subscribed to the Google Groups "berkeleylm-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to berkeleylm-disc...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Roman Prokofyev]

I just did more debugging and figure out that there were no OOV words.

It turned out that the problem is in StupidBackoffLm.java, line 72:

pow(alpha, i - startPos)

alpha is set to 0.4 and if i>120 or so, then pow is 0, which results in logarithm being an infinity.

Probably it is better to expand the log calculation to ensure this situation never happens:

log(x/y*pow(a,b)) = log(x/y) + log(pow(a,b)) = log(x/y) + b*log(a),

this way it will be i*log(0.4), if i = 140, it will be just -128.

I could create a patch, but don't know how it's done on Google Code.

On Wednesday, July 16, 2014 3:51:12 AM UTC+2, Adam Pauls wrote:

This happens with out of vocabulary words. You need to include an <UNK> token in your data to handle this properly.

On Tue, Jul 15, 2014 at 8:33 AM, Roman Prokofyev <roman.p...@gmail.com> wrote:

Hello,

I'm trying to compute log probability on a test collection and notices that around 15 sentences out if 57k (Brown corpus) yield log probability of -Infinity.

So I'm wondering why's that? Isn't stupid backoff model is smoothed as well? Or only Knesser-Ney?

Thanks.

--
You received this message because you are subscribed to the Google Groups "berkeleylm-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to berkeleylm-discuss+unsub...@googlegroups.com.

[Adam Pauls]

Why would i ever be that large?

To unsubscribe from this group and stop receiving emails from it, send an email to berkeleylm-disc...@googlegroups.com.

[Roman Prokofyev]

In Brown corpus (http://en.wikipedia.org/wiki/Brown_Corpus), which is often used to test language models,

there are sentences that large.

But the question is not about why "i" is that large, the question is about the flaw in the library :)

On Wednesday, July 16, 2014 5:16:16 PM UTC+2, Adam Pauls wrote:

Why would i ever be that large?

On Wed, Jul 16, 2014 at 2:02 AM, Roman Prokofyev <roman.p...@gmail.com> wrote:

I just did more debugging and figure out that there were no OOV words.

It turned out that the problem is in StupidBackoffLm.java, line 72:

pow(alpha, i - startPos)

alpha is set to 0.4 and if i>120 or so, then pow is 0, which results in logarithm being an infinity.

Probably it is better to expand the log calculation to ensure this situation never happens:

log(x/y*pow(a,b)) = log(x/y) + log(pow(a,b)) = log(x/y) + b*log(a),

this way it will be i*log(0.4), if i = 140, it will be just -128.

I could create a patch, but don't know how it's done on Google Code.

On Wednesday, July 16, 2014 3:51:12 AM UTC+2, Adam Pauls wrote:

This happens with out of vocabulary words. You need to include an <UNK> token in your data to handle this properly.

On Tue, Jul 15, 2014 at 8:33 AM, Roman Prokofyev <roman.p...@gmail.com> wrote:

Hello,

I'm trying to compute log probability on a test collection and notices that around 15 sentences out if 57k (Brown corpus) yield log probability of -Infinity.

So I'm wondering why's that? Isn't stupid backoff model is smoothed as well? Or only Knesser-Ney?

Thanks.

--
You received this message because you are subscribed to the Google Groups "berkeleylm-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to berkeleylm-discuss+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Adam Pauls]

Right, but you should never have i-startPos > lm-order (so something like 4 or 5). 

Are you buy chance trying to call getLogProb on the whole sentence? Or do you really allow 100-grams in your model?

To unsubscribe from this group and stop receiving emails from it, send an email to berkeleylm-disc...@googlegroups.com.

[Roman Prokofyev]

Ok, there might be some other problem then.

It seems that the getLogProb is called on the whole sentence, but I'm just using ComputeLogProbabilityofTextStream class from .io package.

As far as I see, this class executes "computeProb" method in which it calls getLogProb on the list of all words in the sentence.

I don't know if this is right or wrong, but that's what I see in the code.

The same thing happens if I build the model from the test data provided with the code.

I'm attaching the test sentence that produces this result.

[iesus.c...@gmail.com]

I'm having the same problem. For some sentences the system tries to find ngrams that go beyond the maximum ngram order, I'm not exactly sure why or when this happens. For some sentences it works, but for some other it doesn't, using fairly common words. 

What I did in the same method, was adding the following, I'm not sure if at the end the computations are correct, but at least it doesn't crash (for me it crashes, null pointer exception):

if(probContextOrder>=map.getMaxNgramOrder()-1){//Jesus addition to avoid errors for trying to lookup 6grams

System.out.println("Max ngramorder:"+map.getMaxNgramOrder());

System.out.println("I broke!");

break;

This is also using the ComputeLogProbabilityOfTextStream class. So, what exactly does this class do? I'm interested in computing prefix probabilities in order to calculate surprisal scores, do you think I should use this class?

Thank you very much!

Jesús

 

[Adam Pauls]

Well that's embarrassing -- that's a glaring bug in ComputeLogProbabilityofTextStream. It should be calling this method: https://code.google.com/p/berkeleylm/source/browse/trunk/src/edu/berkeley/nlp/lm/ArrayEncodedNgramLanguageModel.java#46 on the List<string> of words.

I'm not near a computer where I can patch that just now. 

To unsubscribe from this group and stop receiving emails from it, send an email to berkeleylm-disc...@googlegroups.com.

[iesus.c...@gmail.com]

Thank you! Now the output looks much better, and no exceptions :D 

[Roman Prokofyev]

Thanks, looking forward to the fix!

[Adam Pauls]

Please update from the latest SVN and see if your problems are fixed.

On Sat, Jul 19, 2014 at 2:55 PM, Roman Prokofyev <roman.p...@gmail.com> wrote:

Thanks, looking forward to the fix!