Tuner problem

[Reza Lesmana]

Hi devs,

Since ZMERT in my Joshua seems to be stalled (please refer to my other thread "ZMERT take long time") and not showing the error log,

I try PRO and MIRA tuner, but both shows error. 

PRO Tuner

After running PRO tuner, it gives me an error on the console like this :
-----------------------------------------------------------------------------------------------------------------------------

[tune-bundle] rebuilding...

  dep=/home/rezalesmana/joshua-v6.0.3/scripts/training/templates/tune/joshua.config

  dep=/media/thesis/working_directory/data/tune/grammar.filtered.gz

  dep=/media/thesis/working_directory/tune/model/joshua.config [NOT FOUND]

  cmd=/home/rezalesmana/joshua-v6.0.3/scripts/support/run_bundler.py --force --symlink --absolute --verbose /home/rezalesmana/joshua-v6.0.3/scripts/training/templates/tune/joshua.config /media/thesis/working_directory/tune/model --copy-config-options '-top-n 300 -output-format "%i ||| %s ||| %f ||| %c" -mark-oovs false -tm0/type hiero -tm0/owner pt -tm0/maxspan 20 -tm1/owner glue -search cky -weights "lm_0 1 tm_pt_0 1 tm_pt_1 1 tm_pt_4 1 tm_pt_2 1 tm_pt_5 1 tm_pt_3 1 tm_glue_0 1 " -feature-function "StateMinimizingLanguageModel -lm_order 5 -lm_file /media/thesis/working_directory/lm.kenlm"' --pack-tm /media/thesis/working_directory/data/tune/grammar.filtered.gz --tm /media/thesis/working_directory/data/tune/grammar.glue

  took 528 seconds (8m48s)

[pro] rebuilding...

  dep=/media/thesis/working_directory/data/tune/corpus.en

  dep=/media/thesis/working_directory/tune/model/joshua.config [CHANGED]

  dep=tune/model/grammar.filtered.gz.packed/slice_00000.source

  dep=/media/thesis/working_directory/tune/joshua.config.final [NOT FOUND]

  cmd=/home/rezalesmana/joshua-v6.0.3/scripts/training/run_zmert.py /media/thesis/working_directory/data/tune/corpus.en /media/thesis/working_directory/data/tune/corpus.id --tunedir /media/thesis/working_directory/tune --tuner pro --decoder-config /media/thesis/working_directory/tune/model/joshua.config

  took 1 seconds (1s)

[filter-test] cached, skipping...

[test-bundle] rebuilding...

  dep=/media/thesis/working_directory/tune/joshua.config.final [NOT FOUND]

  dep=/media/thesis/working_directory/data/test/grammar.filtered.gz [CHANGED]

  dep=/media/thesis/working_directory/test/joshua.config [NOT FOUND]

  cmd=/home/rezalesmana/joshua-v6.0.3/scripts/support/run_bundler.py --force --symlink --verbose /media/thesis/working_directory/tune/joshua.config.final test/model --copy-config-options '-top-n 300 -output-format "%i ||| %s ||| %f ||| %c" -mark-oovs false' --pack-tm /media/thesis/working_directory/data/test/grammar.filtered.gz --tm /media/thesis/working_directory/data/tune/grammar.glue

  JOB FAILED (return code 2)

ERROR:root:ERROR: argument config: can't open '/media/thesis/working_directory/tune/joshua.config.final': [Errno 2] No such file or directory: '/media/thesis/working_directory/tune/joshua.config.final'

--------------------------------------------------------------------------------------

This is the tail of pro.log

-----------------------------

Optimizing BLEU

Number of features: 10

Exception in thread "main" java.lang.IndexOutOfBoundsException: Index: 2, Size: 2

        at java.util.ArrayList.rangeCheck(ArrayList.java:653)

        at java.util.ArrayList.get(ArrayList.java:429)

        at joshua.corpus.Vocabulary.word(Vocabulary.java:184)

        at joshua.pro.PROCore.initialize(PROCore.java:456)

        at joshua.pro.PROCore.main(PROCore.java:3627)

Feature names: {"lm_0",

PRO exiting prematurely (PROCore returned 1)...

-----------------------------

It seems that the PRO tuner fail to run (see that the "[pro] rebuilding" only runs 1 second??) but the pipeline continues but 

doesn't seem to find the joshua.config.final file. If I'm correct, this file should be available if tuner succeeds, right?

MIRA Tuner

After running MIRA tuner, it gives me an error of ArrayIndexOutOfBoundsException

This is the tail of run1.log :
--------------------

Input 1214: 300-best extraction took 5.550 seconds

Input 1216: 300-best extraction took 2.610 seconds

Input 1210: 300-best extraction took 19.238 seconds

Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 1

        at joshua.decoder.JoshuaDecoder.main(JoshuaDecoder.java:91)

--------------

And this is the tail of mira.log :

--------------------------------

Executing: /media/thesis/working_directory/tune/model/run-joshua.sh -m 10g -threads 4 -moses  -output-format "%i ||| %s ||| %f ||| %c"  -config /media/thesis/working_directory/tune/model/joshua.config -weight-overwrite 'lm-0= 0.111111 WordPenalty= -0.111111 tm-pt-4= 0.111111 tm-pt-5= 0.111111 tm-pt-3= 0.111111 tm-glue-0= 0.111111 tm-pt-2= 0.111111 tm-pt-1= 0.111111 tm-pt-0= 0.111111'  -n-best-list /media/thesis/working_directory/tune/run1.best300.out 300 distinct  -input-file /media/thesis/working_directory/data/tune/corpus.en > /media/thesis/working_directory/tune/run1.out 2> /media/thesis/working_directory/tune/run1.log

Executing: /media/thesis/working_directory/tune/model/run-joshua.sh -m 10g -threads 4 -moses  -output-format "%i ||| %s ||| %f ||| %c"  -config /media/thesis/working_directory/tune/model/joshua.config -weight-overwrite 'lm-0= 0.111111 WordPenalty= -0.111111 tm-pt-4= 0.111111 tm-pt-5= 0.111111 tm-pt-3= 0.111111 tm-glue-0= 0.111111 tm-pt-2= 0.111111 tm-pt-1= 0.111111 tm-pt-0= 0.111111'  -n-best-list /media/thesis/working_directory/tune/run1.best300.out 300 distinct  -input-file /media/thesis/working_directory/data/tune/corpus.en > /media/thesis/working_directory/tune/run1.out 2> /media/thesis/working_directory/tune/run1.log

(1) run decoder to produce n-best lists

params = -m 10g -threads 4 -moses  -output-format "%i ||| %s ||| %f ||| %c"

decoder_config = -weight-overwrite 'lm-0= 0.111111 WordPenalty= -0.111111 tm-pt-4= 0.111111 tm-pt-5= 0.111111 tm-pt-3= 0.111111 tm-glue-0= 0.111111 tm-pt-2= 0.111111 tm-pt-1= 0.111111 tm-pt-0= 0.111111'

Exit code: 1

The decoder died. CONFIG WAS -weight-overwrite 'lm-0= 0.111111 WordPenalty= -0.111111 tm-pt-4= 0.111111 tm-pt-5= 0.111111 tm-pt-3= 0.111111 tm-glue-0= 0.111111 tm-pt-2= 0.111111 tm-pt-1= 0.111111 tm-pt-0= 0.111111'

-------------------------------------

Can anyone help me how to proceed with the tuning phase? I'm out of option right now :) 
Thank you very much

[Matt Post]

Hi — PRO functionality was restored in Joshua 6.0.4, just released. MIRA has also been updated to work better (and no longer depends on Moses). So please update and try again.

matt

--
You received this message because you are subscribed to the Google Groups "Joshua Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to joshua_develop...@googlegroups.com.
To post to this group, send email to joshua_d...@googlegroups.com.
Visit this group at http://groups.google.com/group/joshua_developers.
For more options, visit https://groups.google.com/d/optout.