berkeleylm-discuss

sentence boundary symbols are not included in the vocabulary, I am not sure why, and that causes NaNs

These NaNs are definitely because of OOV (out-of-vocabulary) words: beginning-of-sentence and end-of-

how can you solve this problem?? 在 2015年4月27日星期一 UTC-4上午3:07:05，李若冰写道： I've met a problem that, I

I want to make a Ngram model but get this error . Any suggestion ?

No problem, thank you for confirming my assumptions and saving me from further digging. Jack On

Dear Adam Could you ask your (old?) collegues about that please ? There is also a version of the

Hi I was trying to load a 3.8 billion tokens, 12GB LM in ARPA format using readArrayEncodedLmFromArpa

Yes, I will migrate to github at some point. On Fri, Apr 3, 2015 at 7:32 AM, Matthew Hatem <mhatem

Yes, you would need to preprocess, nothing special is implemented. On Wed, Mar 25, 2015 at 1:22 PM,

Can you show me the exact command you ran, along with the full exception? On Sun, Mar 22, 2015 at 11:

Thanks a lot!!! On Wednesday, March 18, 2015 at 5:20:18 PM UTC+9, 한정수 wrote: there is vocab_cs.gz in

1. Yes, the backoff and smoothing used by these methods are different. I recommend reading http://www

It should basically be 10GB. I'm sure the JVM has some extra overhead, but probably on he order

Hi, I made (and tested) a small change so that BerkeleyLM is smarter about gzipped files — instead of

Thanks again for your time and help. I've been trying to debug myself. And well, the issue

I am new to language model. I want to compute the probability of a sentence using language model.

lm.getWordIndexer().getStartSymbol(). On Tuesday, December 30, 2014 6:05:34 AM UTC-8, Kunal Singhal

I added a note on the front page of Google code. Basically, you have build the "export"

I have a class named WordListBuilder that reads input from a text file in any language. Output is to

Yes, this is fixed at head in SVN. On Wed, Sep 17, 2014 at 7:27 AM, Oren Melamud <oren.melamud@

Hi, So, just to let you know, I was able to get the language model after all! :D These were the

I want to use it for ngram frequency calculation. On Wed, Sep 10, 2014 at 9:08 AM, Adam Pauls <

Please update from the latest SVN and see if your problems are fixed. On Sat, Jul 19, 2014 at 2:55 PM

Sorry, getting back to debugging. I've lost context here. Can you give me a dataset and command

https://code.google.com/p/berkeleylm/source/browse/trunk/examples/make-kneserney-arpa-from-raw-text.

Ok. Thanks for the quick reply. On Monday, July 21, 2014 5:55:20 PM UTC+3, Adam Pauls wrote:

No, 40 was actually a count separated by tab, by I figured out this, there were again out of

Ok, I think I got it, the library seems to be hard-coded to build 3gram models, meaning that I need

Including quotes, semicolons, dashes, etc (not just sentence markers). On Monday, 30 June 2014 14:47:

Thanks, Adam! I appreciate your quick responses! On Thursday, 26 June 2014 10:06:26 UTC-5, Adam Pauls