Sample code for Google N-gram with stupid backoff?

[Joseph Turian]

Could you provide code that does that following?

Load stupid backoff model with English Google N-gram binary + vocab

For each line in stdin:

   Run stupid backoff on the string and output the probability

This would be very useful. I'm trying to figure out how to do this, but having difficulty.

I think a lot of people just want to get probabilities like this, and it lowers the friction for using the library a lot.

Thanks!

    Joseph

[Adam Pauls]

I've attached a file that loads either a google binary or regular Berkeley LM binary (which already has the vocab) and computes the log-prob of a set of files (or STDIN if none is given, in standard UNIX style). I'll release a new version of the code with this file available shortly. 

By the way, it's hard for me to tell why this is hard to figure out how to do (because I know the code so well). Can you tell me what documentation you read and what you blocked on, so I can figure out the weak points in the documentation?

Thanks,

Adam

[Joseph Turian]

On Monday, 20 August 2012 14:55:56 UTC-4, Adam Pauls wrote:

I've attached a file that loads either a google binary or regular Berkeley LM binary (which already has the vocab) and computes the log-prob of a set of files (or STDIN if none is given, in standard UNIX style). I'll release a new version of the code with this file available shortly. 

By the way, it's hard for me to tell why this is hard to figure out how to do (because I know the code so well). Can you tell me what documentation you read and what you blocked on, so I can figure out the weak points in the documentation?

Adam,

Thanks for this.

I believe I was confused about some of the parameters.

When I first dive into code, I highly prefer to work from example code. So my main barrier to hacking was the lack of examples.

[Joseph Turian]

Another barrier is that I don't work with the Java stack that often (I use C++ and Python more frequently), so there's friction for me simply in building and invoking Java code. But naturally this is to be expected.

[Harta Wijaya]

Hi Adam,

Thanks for the example, however I can get the probability right.
What I did was:
1. I convert SRILM ARPA format to berkeley binarized format.
I followed the instruction from Joshua decoder:

java -cp $JOSHUA/lib/berkeleylm.jar -server -mxMEM edu.berkeley.nlp.lm.io.MakeLmBinaryFromArpa lm.arpa lm.berkeleylm

2. Next, I tried to run your ComputePerplexity.java to get the probability of the text from output file.
However, I always get -100.0 for the probability.

Thanks a lot for your help.



[Adam Pauls]

Could you send me the file you and command that you are running in step 2 so I can replicate? Thanks,

Adam

[Harta Wijaya]

Hi Adam,

Thanks for your reply. You can download those files and replicate the steps at http://www.comp.nus.edu.sg/~harta01/

Thank you.

Harta Wijaya

[Adam Pauls]

Actually, it's probably just because I made some changes to the code in the meanwhile and ComputePerplexity needs them to work properly. I will release a new version shortly -- please rerun then and let me know it works. 

[Harta Wijaya]

Ok. Thanks

[Adam Pauls]

A new version has been uploaded to the Googlecode site which should fix this bug. 

[Harta Wijaya]

HI Adam,

Sorry for the late reply.
Yea, it works now.

Thanks a lot.

[Oren Melamud]

Hi Adam,

I'm using your web 1T binary with stupid back-off. Very helpful. Thanks!
I looked at the sample code that you provided here and it seems to me like it's not padding the sentences with start/end sentence symbols, which should normally be added.
Am I right?

Thanks,
Oren.

[Oren Melamud]

Actually, maybe I'm missing something, but it seems like the use of lm.getLogProb(words) for every text line returns the conditional probability of the last word in the line given the previous words.

I guess using scoreSentence() would be the appropriate choice there, right?

[Adam Pauls]

Yes, this is fixed at head in SVN. 

--
You received this message because you are subscribed to the Google Groups "berkeleylm-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to berkeleylm-disc...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.