about javalm and bloomfilterlm

1 view
Skip to first unread message

thuvh

unread,
Nov 16, 2009, 10:10:06 PM11/16/09
to JosHUa - JHU Open Source Architecture (Technical List)
I've 2 questions:
In folder example of joshua package, i see two configuration file:
example.config.bloomfilterlm and example.config.javalm. could you
please show me how to train data to get these file or can I use ZMERT
to train data to get these file?
The next question is about decode_example_javalm.sh
When I run script, I get exception and this is log of process:

Nov 17, 2009 9:53:06 AM joshua.decoder.JoshuaConfiguration
readConfigFile
INFO: you use a LM feature function, so make sure you have a LM
grammar
Nov 17, 2009 9:53:06 AM joshua.decoder.ff.lm.buildin_lm.LMGrammarJAVA
<init>
INFO: use java lm
Nov 17, 2009 9:53:06 AM joshua.decoder.ff.lm.buildin_lm.LMGrammarJAVA
read_lm_grammar_from_file
INFO: Reading grammar from file example/example.trigram.lm.gz
Nov 17, 2009 9:53:06 AM joshua.decoder.ff.lm.buildin_lm.LMGrammarJAVA
read_lm_grammar_from_file
INFO: begin to read ngrams with order 1
Nov 17, 2009 9:53:06 AM joshua.decoder.ff.lm.buildin_lm.LMGrammarJAVA
read_lm_grammar_from_file
INFO: begin to read ngrams with order 2
Nov 17, 2009 9:53:07 AM joshua.decoder.ff.lm.buildin_lm.LMGrammarJAVA
read_lm_grammar_from_file
INFO: begin to read ngrams with order 3
Nov 17, 2009 9:53:07 AM joshua.decoder.JoshuaDecoder
initializeGlueGrammar
INFO: Constructing glue grammar...
Nov 17, 2009 9:53:07 AM
joshua.decoder.ff.tm.hiero.MemoryBasedBatchGrammar print_grammar
INFO: ###########Grammar###########
Nov 17, 2009 9:53:07 AM
joshua.decoder.ff.tm.hiero.MemoryBasedBatchGrammar print_grammar
INFO: ####num_rules: 2; num_bins: 2; num_pruned: 0; sumest_cost:
0.00000
Nov 17, 2009 9:53:07 AM joshua.decoder.JoshuaDecoder
initializeTranslationGrammars
INFO: Using grammar read from file example/example.hiero.tm.gz
Nov 17, 2009 9:53:08 AM
joshua.decoder.ff.tm.hiero.MemoryBasedBatchGrammar print_grammar
INFO: ###########Grammar###########
Nov 17, 2009 9:53:08 AM
joshua.decoder.ff.tm.hiero.MemoryBasedBatchGrammar print_grammar
INFO: ####num_rules: 15939; num_bins: 2029; num_pruned: 0;
sumest_cost: 0.00000
Exception in thread "main" java.lang.ClassCastException:
joshua.decoder.ff.lm.buildin_lm.LMGrammarJAVA$LMHash cannot be cast to
java.lang.Double
at joshua.decoder.ff.lm.buildin_lm.LMGrammarJAVA.get_valid_prob
(LMGrammarJAVA.java:199)
at
joshua.decoder.ff.lm.buildin_lm.LMGrammarJAVA.ngramLogProbability_helper
(LMGrammarJAVA.java:165)
at joshua.decoder.ff.lm.AbstractLM.ngramLogProbability
(AbstractLM.java:68)
at
joshua.decoder.ff.lm.DefaultNGramLanguageModel.sentenceLogProbability
(DefaultNGramLanguageModel.java:82)
at joshua.decoder.ff.lm.AbstractLM.sentenceLogProbability
(AbstractLM.java:45)
at joshua.decoder.ff.lm.LanguageModelFF.score_chunk
(LanguageModelFF.java:279)
at joshua.decoder.ff.lm.LanguageModelFF.estimate_rule_prob
(LanguageModelFF.java:314)
at joshua.decoder.ff.lm.LanguageModelFF.estimate
(LanguageModelFF.java:125)
at joshua.decoder.ff.tm.MonolingualRule.estimateRuleCost
(MonolingualRule.java:194)
at joshua.decoder.ff.tm.BasicRuleCollection.sortRules
(BasicRuleCollection.java:104)
at joshua.decoder.ff.tm.BasicRuleCollection.sortRules
(BasicRuleCollection.java:119)
at joshua.decoder.ff.tm.AbstractGrammar.sort
(AbstractGrammar.java:118)
at joshua.decoder.ff.tm.AbstractGrammar.sort
(AbstractGrammar.java:153)
at joshua.decoder.ff.tm.AbstractGrammar.sortGrammar
(AbstractGrammar.java:70)
at joshua.decoder.JoshuaDecoder.initialize(JoshuaDecoder.java:
304)
at joshua.decoder.JoshuaDecoder.<init>(JoshuaDecoder.java:108)
at joshua.decoder.JoshuaDecoder.main(JoshuaDecoder.java:664)
Nov 17, 2009 9:53:08 AM joshua.decoder.ff.tm.GrammarReader finalize
SEVERE: Grammar file stream was not closed, this indicates a coding
error: example/example.hiero.tm.gz
Nov 17, 2009 9:53:08 AM joshua.decoder.ff.tm.GrammarReader finalize
SEVERE: Grammar file stream was not closed, this indicates a coding
error: grammars/hiero.glue
"javalm.log" 46L, 3488C

Plz, tell me the way to solve it.

Chris Callison-Burch

unread,
Nov 17, 2009, 2:29:37 PM11/17/09
to joshua_t...@googlegroups.com
Neither of these LMs is working at the moment. The only one that is
working properly is srilm. --Chris
> --
>
> You received this message because you are subscribed to the Google
> Groups "JosHUa - JHU Open Source Architecture (Technical List)" group.
> To post to this group, send email to
> joshua_t...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/joshua_technical?hl=
> .
>
Reply all
Reply to author
Forward
0 new messages