Effects of killing a pipeline run

[Lewis John Mcgibbney]

Hi Folks,

What the the effect of killing a pipeline run?

Say I execute the following

lmcgibbn@LMC-032857 /usr/local/joshua/experiments(master) $ ../bin/pipeline.pl  --rundir . --type hiero --corpus input/commoncrawl.ru-en --tune input/commoncrawl.ru-en --test input/commoncrawl.ru-en --source en --target ru --rundir experiment1/run1 --readme "Experiment 1 Run 1 of ru --> en model training"

It sometimes becomes extremely difficult to know whether Joshua has died, or to understand what is really going on. Particularly within the Giza step.

Say I kill the task, and kick off another task using the same input but passing in a different --rundir, what is the effect? Is there some sort of merging which takes place or does this lead to major problems?

Thanks

Lewis

--

Lewis

[Matt Post]

It depends on how cleanly it dies, but you should just be able to rerun the same command and it will pick up at the last incomplete step.

If you pass in a different rundir, the runs are completely independent, nothing will be cached, etc.

Alignment logging goes to alignments/X/giza.log (when using GIZA), where X is the sub piece of the corpus that is being aligned (the corpus is split into lines of --aligner-chunk-size lines, default 1000000). alignments/giza.log isn't used (deceptive).

matt

--
You received this message because you are subscribed to the Google Groups "Joshua Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to joshua_develop...@googlegroups.com.
To post to this group, send email to joshua_d...@googlegroups.com.
Visit this group at https://groups.google.com/group/joshua_developers.
For more options, visit https://groups.google.com/d/optout.