Pipeline error and run-all-test with thrax in YOSEMITE

35 views
Skip to first unread message

diegomo...@gmail.com

unread,
Feb 25, 2016, 6:25:18 PM2/25/16
to Joshua Technical Support
I tried to execute quick start from JOSHUA's webpage using indian models but the following error appeared: 

getmerge: 

and I've executed run-all-tests, all of them were ok but again with the same message the thrax did not run. 

getmerge: `thrax/final': No such file or directory. 


I've seen in the thrax log and appeared as follows:

Running in mode: translation
[SCHED] planned job for class edu.jhu.thrax.hadoop.jobs.VocabularyJob
[SCHED] planned job for class edu.jhu.thrax.hadoop.jobs.ExtractionJob
[SCHED] planned job for class edu.jhu.thrax.hadoop.features.mapred.SourcePhraseGivenTargetFeature
[SCHED] planned job for class edu.jhu.thrax.hadoop.features.mapred.TargetPhraseGivenSourceFeature
[SCHED] planned job for class edu.jhu.thrax.hadoop.jobs.SourceWordGivenTargetWordProbabilityJob
[SCHED] planned job for class edu.jhu.thrax.hadoop.jobs.TargetWordGivenSourceWordProbabilityJob
[SCHED] planned job for class edu.jhu.thrax.hadoop.features.annotation.AnnotationFeatureJob
[SCHED] planned job for class edu.jhu.thrax.hadoop.jobs.OutputJob
[SCHED] class edu.jhu.thrax.hadoop.jobs.OutputJob in state WAITING
[SCHED] class edu.jhu.thrax.hadoop.features.mapred.TargetPhraseGivenSourceFeature in state WAITING
[SCHED] class edu.jhu.thrax.hadoop.jobs.ExtractionJob in state WAITING
[SCHED] class edu.jhu.thrax.hadoop.jobs.VocabularyJob in state WAITING
[SCHED] class edu.jhu.thrax.hadoop.jobs.VocabularyJob in state READY
[SCHED] class edu.jhu.thrax.hadoop.features.annotation.AnnotationFeatureJob in state WAITING
[SCHED] class edu.jhu.thrax.hadoop.jobs.SourceWordGivenTargetWordProbabilityJob in state WAITING
[SCHED] class edu.jhu.thrax.hadoop.jobs.VocabularyJob in state WAITING
[SCHED] class edu.jhu.thrax.hadoop.jobs.VocabularyJob in state READY
[SCHED] class edu.jhu.thrax.hadoop.jobs.TargetWordGivenSourceWordProbabilityJob in state WAITING
[SCHED] class edu.jhu.thrax.hadoop.jobs.VocabularyJob in state WAITING
[SCHED] class edu.jhu.thrax.hadoop.jobs.VocabularyJob in state READY
[SCHED] class edu.jhu.thrax.hadoop.jobs.ExtractionJob in state WAITING
[SCHED] class edu.jhu.thrax.hadoop.jobs.VocabularyJob in state WAITING
[SCHED] class edu.jhu.thrax.hadoop.jobs.VocabularyJob in state READY
[SCHED] class edu.jhu.thrax.hadoop.features.mapred.SourcePhraseGivenTargetFeature in state WAITING
[SCHED] class edu.jhu.thrax.hadoop.jobs.ExtractionJob in state WAITING
[SCHED] class edu.jhu.thrax.hadoop.jobs.VocabularyJob in state WAITING
[SCHED] class edu.jhu.thrax.hadoop.jobs.VocabularyJob in state READY
[SCHED] class edu.jhu.thrax.hadoop.jobs.VocabularyJob in state RUNNING
15/05/08 16:25:09 INFO input.FileInputFormat: Total input paths to process : 1
[SCHED] class edu.jhu.thrax.hadoop.jobs.VocabularyJob in state FAILED
[SCHED] class edu.jhu.thrax.hadoop.jobs.SourceWordGivenTargetWordProbabilityJob in state PREREQ_FAILED
[SCHED] class edu.jhu.thrax.hadoop.features.annotation.AnnotationFeatureJob in state PREREQ_FAILED
[SCHED] class edu.jhu.thrax.hadoop.jobs.OutputJob in state PREREQ_FAILED
[SCHED] class edu.jhu.thrax.hadoop.jobs.TargetWordGivenSourceWordProbabilityJob in state PREREQ_FAILED
[SCHED] class edu.jhu.thrax.hadoop.jobs.ExtractionJob in state PREREQ_FAILED
[SCHED] class edu.jhu.thrax.hadoop.features.mapred.TargetPhraseGivenSourceFeature in state PREREQ_FAILED
[SCHED] class edu.jhu.thrax.hadoop.features.mapred.SourcePhraseGivenTargetFeature in state PREREQ_FAILED
class edu.jhu.thrax.hadoop.features.mapred.TargetPhraseGivenSourceFeature PREREQ_FAILED
class edu.jhu.thrax.hadoop.jobs.OutputJob PREREQ_FAILED
class edu.jhu.thrax.hadoop.features.annotation.AnnotationFeatureJob PREREQ_FAILED
class edu.jhu.thrax.hadoop.jobs.SourceWordGivenTargetWordProbabilityJob PREREQ_FAILED
class edu.jhu.thrax.hadoop.jobs.TargetWordGivenSourceWordProbabilityJob PREREQ_FAILED
class edu.jhu.thrax.hadoop.features.mapred.SourcePhraseGivenTargetFeature PREREQ_FAILED
class edu.jhu.thrax.hadoop.jobs.VocabularyJob FAILED
class edu.jhu.thrax.hadoop.jobs.ExtractionJob PREREQ_FAILED


when I looked at Hadoop log. I've seen it: 

error code 127 

Do you have any idea?? 

YOSEMITE OS
JOSHUA 6.0.5
Hadoop 2.6.0

Thank you in advance

Matt Post

unread,
Feb 25, 2016, 7:36:05 PM2/25/16
to joshua_...@googlegroups.com
It looks like Hadoop failed. Do you have your own Hadoop installation, or did you use the standalone mode from the pipeline?


--
You received this message because you are subscribed to the Google Groups "Joshua Technical Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to joshua_suppor...@googlegroups.com.
To post to this group, send email to joshua_...@googlegroups.com.
Visit this group at https://groups.google.com/group/joshua_support.
For more options, visit https://groups.google.com/d/optout.

diegomo...@gmail.com

unread,
Feb 26, 2016, 7:23:46 AM2/26/16
to Joshua Technical Support
hmmm. I have my own hadoop installation. I switched the hadoop version and the problems changed.

 I will go through hadoop. thks

Matt Post

unread,
Feb 26, 2016, 9:19:14 AM2/26/16
to joshua_...@googlegroups.com
All right. Feel free to post problems (and solutions!) I've just noticed myself that a number of parameters changed from Hadoop 1 to 2. I've had to use the following in the "thrax-run" step in pipeline.pl for a very large job:

-D mapreduce.map.java.opts='-Xmx3g' -D mapreduce.reduce.java.opts='-Xmx3g' -D mapreduce.map.memory.mb=4096

(The old Hadoop used mapred.map.java.opts, and apparently keeping this as a backward-compatible option, or printing out an error, was too much to ask :))

matt
Reply all
Reply to author
Forward
0 new messages