Streaming command failed, Command returned non-zero exit status 256

[Wacha Bounliphone]

Hello,

I would like some newbe help with MRJob.

I'm running wrapper scripts from https://github.com/dgleich/simform/tree/sisc-rm locally on my Bitnami Hadoop 2.7.2-1 Stack, with Python 2.7.12 and MRJob v.0.5.4

I'm executing a command

python mr_exodus2seq_hadoop.py hdfs:///user/wacha/heat_datasets/output/input.txt -r hadoop -v --python-archive simform-deploy.tar.gz -t 10 -d hdfs:///user/wacha/heat_datasets/output/data.seq/ --variables TEMP --timestepfile timesteps.txt  --file timesteps.txt

the python script defining a wrapper subclassing MRJob. I have checked that the relevant input files exist on the HDFS system,

Below is the complete verbose output at the command line. I don't understand what could be happening, could someone point me to a reason of why the fails and how to debug this? I have looked for similar errors, e.g.
https://github.com/Yelp/mrjob/issues/1092
but to no avail as these correspond to older versions of MRJob.

Best regards,
Wacha Bounliphone





===

Looking for configs in /users/visics/wacha/.mrjob.conf
Looking for configs in /etc/mrjob.conf
No configs found; falling back on auto-configuration
Active configuration:
{'bootstrap_mrjob': None,
 'check_input_paths': True,
 'cleanup': ['ALL'],
 'cleanup_on_failure': ['NONE'],
 'cmdenv': {},
 'hadoop_bin': None,
 'hadoop_extra_args': [],
 'hadoop_home': None,
 'hadoop_log_dirs': [],
 'hadoop_streaming_jar': None,
 'hadoop_tmp_dir': 'tmp/mrjob',
 'hadoop_version': None,
 'interpreter': None,
 'jobconf': {},
 'label': None,
 'libjars': [],
 'local_tmp_dir': '/tmp',
 'owner': 'wacha',
 'python_archives': ['simform-deploy.tar.gz'],
 'python_bin': None,
 'setup': [],
 'setup_cmds': [],
 'setup_scripts': [],
 'sh_bin': ['sh', '-ex'],
 'steps_interpreter': None,
 'steps_python_bin': None,
 'strict_protocols': True,
 'upload_archives': [],
 'upload_files': ['timesteps.txt']}
Looking for hadoop binary in /users/visics/wacha/Develop/hadoop-2.7.2-1/hadoop/bin...
Found hadoop binary: /users/visics/wacha/Develop/hadoop-2.7.2-1/hadoop/bin/hadoop
> /users/visics/wacha/Develop/hadoop-2.7.2-1/hadoop/bin/hadoop fs -ls hdfs:///user/wacha/heat_datasets/output/input.txt
STDOUT: -rw-r--r--   1 wacha supergroup        171 2016-08-18 15:50 hdfs:///user/wacha/heat_datasets/output/input.txt
Creating temp directory /tmp/mr_exodus2seq_hadoop.wacha.20160818.150831.852145
archiving /users/visics/wacha/Develop/anaconda/lib/python2.7/site-packages/mrjob -> /tmp/mr_exodus2seq_hadoop.wacha.20160818.150831.852145/mrjob.tar.gz as mrjob/
Writing wrapper script to /tmp/mr_exodus2seq_hadoop.wacha.20160818.150831.852145/setup-wrapper.sh
WRAPPER: # store $PWD
WRAPPER: __mrjob_PWD=$PWD
WRAPPER:
WRAPPER: # obtain exclusive file lock
WRAPPER: exec 9>/tmp/wrapper.lock.mr_exodus2seq_hadoop.wacha.20160818.150831.852145
WRAPPER: python -c 'import fcntl; fcntl.flock(9, fcntl.LOCK_EX)'
WRAPPER:
WRAPPER: # setup commands
WRAPPER: {
WRAPPER:   export PYTHONPATH=$__mrjob_PWD/mrjob.tar.gz:$PYTHONPATH
WRAPPER:   export PYTHONPATH=$__mrjob_PWD/simform-deploy.tar.gz:$PYTHONPATH
WRAPPER: } 0</dev/null 1>&2
WRAPPER:
WRAPPER: # release exclusive file lock
WRAPPER: exec 9>&-
WRAPPER:
WRAPPER: # run task from the original working directory
WRAPPER: cd $__mrjob_PWD
WRAPPER: "$@"
> /users/visics/wacha/Develop/hadoop-2.7.2-1/hadoop/bin/hadoop version
Using Hadoop version 2.7.2
> /users/visics/wacha/Develop/hadoop-2.7.2-1/hadoop/bin/hadoop fs -mkdir -p hdfs:///user/wacha/tmp/mrjob/mr_exodus2seq_hadoop.wacha.20160818.150831.852145/files/
Copying local files to hdfs:///user/wacha/tmp/mrjob/mr_exodus2seq_hadoop.wacha.20160818.150831.852145/files/...
  /users/visics/wacha/myProjects/simform/src/mr_exodus2seq_hadoop.py -> hdfs:///user/wacha/tmp/mrjob/mr_exodus2seq_hadoop.wacha.20160818.150831.852145/files/mr_exodus2seq_hadoop.py
> /users/visics/wacha/Develop/hadoop-2.7.2-1/hadoop/bin/hadoop fs -put /users/visics/wacha/myProjects/simform/src/mr_exodus2seq_hadoop.py hdfs:///user/wacha/tmp/mrjob/mr_exodus2seq_hadoop.wacha.20160818.150831.852145/files/mr_exodus2seq_hadoop.py
  simform-deploy.tar.gz -> hdfs:///user/wacha/tmp/mrjob/mr_exodus2seq_hadoop.wacha.20160818.150831.852145/files/simform-deploy.tar.gz
> /users/visics/wacha/Develop/hadoop-2.7.2-1/hadoop/bin/hadoop fs -put simform-deploy.tar.gz hdfs:///user/wacha/tmp/mrjob/mr_exodus2seq_hadoop.wacha.20160818.150831.852145/files/simform-deploy.tar.gz
  timesteps.txt -> hdfs:///user/wacha/tmp/mrjob/mr_exodus2seq_hadoop.wacha.20160818.150831.852145/files/timesteps.txt
> /users/visics/wacha/Develop/hadoop-2.7.2-1/hadoop/bin/hadoop fs -put timesteps.txt hdfs:///user/wacha/tmp/mrjob/mr_exodus2seq_hadoop.wacha.20160818.150831.852145/files/timesteps.txt
  /tmp/mr_exodus2seq_hadoop.wacha.20160818.150831.852145/setup-wrapper.sh -> hdfs:///user/wacha/tmp/mrjob/mr_exodus2seq_hadoop.wacha.20160818.150831.852145/files/setup-wrapper.sh
> /users/visics/wacha/Develop/hadoop-2.7.2-1/hadoop/bin/hadoop fs -put /tmp/mr_exodus2seq_hadoop.wacha.20160818.150831.852145/setup-wrapper.sh hdfs:///user/wacha/tmp/mrjob/mr_exodus2seq_hadoop.wacha.20160818.150831.852145/files/setup-wrapper.sh
  /tmp/mr_exodus2seq_hadoop.wacha.20160818.150831.852145/mrjob.tar.gz -> hdfs:///user/wacha/tmp/mrjob/mr_exodus2seq_hadoop.wacha.20160818.150831.852145/files/mrjob.tar.gz
> /users/visics/wacha/Develop/hadoop-2.7.2-1/hadoop/bin/hadoop fs -put /tmp/mr_exodus2seq_hadoop.wacha.20160818.150831.852145/mrjob.tar.gz hdfs:///user/wacha/tmp/mrjob/mr_exodus2seq_hadoop.wacha.20160818.150831.852145/files/mrjob.tar.gz
> /users/visics/wacha/Develop/anaconda/bin/python /users/visics/wacha/myProjects/simform/src/mr_exodus2seq_hadoop.py --steps -d hdfs:///user/wacha/heat_datasets/output/data.seq/ --timestepfile timesteps.txt -t 10 --variables TEMP
Looking for Hadoop streaming jar in /users/visics/wacha/Develop/hadoop-2.7.2-1/hadoop...
Found Hadoop streaming jar: /users/visics/wacha/Develop/hadoop-2.7.2-1/hadoop/share/hadoop/tools/lib/hadoop-streaming-2.7.2.jar
Running step 1 of 1...
> /users/visics/wacha/Develop/hadoop-2.7.2-1/hadoop/bin/hadoop jar /users/visics/wacha/Develop/hadoop-2.7.2-1/hadoop/share/hadoop/tools/lib/hadoop-streaming-2.7.2.jar -files 'hdfs:///user/wacha/tmp/mrjob/mr_exodus2seq_hadoop.wacha.20160818.150831.852145/files/setup-wrapper.sh#setup-wrapper.sh,hdfs:///user/wacha/tmp/mrjob/mr_exodus2seq_hadoop.wacha.20160818.150831.852145/files/mr_exodus2seq_hadoop.py#mr_exodus2seq_hadoop.py,hdfs:///user/wacha/tmp/mrjob/mr_exodus2seq_hadoop.wacha.20160818.150831.852145/files/timesteps.txt#timesteps.txt' -archives 'hdfs:///user/wacha/tmp/mrjob/mr_exodus2seq_hadoop.wacha.20160818.150831.852145/files/mrjob.tar.gz#mrjob.tar.gz,hdfs:///user/wacha/tmp/mrjob/mr_exodus2seq_hadoop.wacha.20160818.150831.852145/files/simform-deploy.tar.gz#simform-deploy.tar.gz' -inputformat org.apache.hadoop.mapred.lib.NLineInputFormat -input hdfs:///user/wacha/heat_datasets/output/input.txt -output hdfs:///user/wacha/tmp/mrjob/mr_exodus2seq_hadoop.wacha.20160818.150831.852145/output -mapper 'sh -ex setup-wrapper.sh python mr_exodus2seq_hadoop.py --step-num=0 --mapper -d hdfs:///user/wacha/heat_datasets/output/data.seq/ --timestepfile timesteps.txt -t 10 --variables TEMP' -reducer 'sh -ex setup-wrapper.sh python mr_exodus2seq_hadoop.py --step-num=0 --reducer -d hdfs:///user/wacha/heat_datasets/output/data.seq/ --timestepfile timesteps.txt -t 10 --variables TEMP'
Invoking Hadoop via PTY
  packageJobJar: [/tmp/hadoop-unjar3460972900608461997/] [] /tmp/streamjob113767583308788074.jar tmpDir=null
  Connecting to ResourceManager at /0.0.0.0:8032
  Connecting to ResourceManager at /0.0.0.0:8032
  Total input paths to process : 1
  number of splits:3
  Submitting tokens for job: job_1471511636485_0011
  Submitted application application_1471511636485_0011
  The url to track the job: http://kochab.esat.kuleuven.be:8088/proxy/application_1471511636485_0011/
  Running job: job_1471511636485_0011
  Job job_1471511636485_0011 running in uber mode : false
   map 0% reduce 0%
  Task Id : attempt_1471511636485_0011_m_000000_0, Status : FAILED
Error: java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 2
    at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:322)
    at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:535)
    at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:130)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
    at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:34)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)

  Task Id : attempt_1471511636485_0011_m_000001_0, Status : FAILED
Error: java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 2
    at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:322)
    at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:535)
    at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:130)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
    at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:34)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)

  Task Id : attempt_1471511636485_0011_m_000002_0, Status : FAILED
Error: java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 2
    at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:322)
    at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:535)
    at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:130)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
    at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:34)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)

  Task Id : attempt_1471511636485_0011_m_000000_1, Status : FAILED
Error: java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 2
    at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:322)
    at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:535)
    at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:130)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
    at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:34)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)

  Task Id : attempt_1471511636485_0011_m_000001_1, Status : FAILED
Error: java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 2
    at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:322)
    at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:535)
    at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:130)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
    at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:34)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)

  Task Id : attempt_1471511636485_0011_m_000002_1, Status : FAILED
Error: java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 2
    at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:322)
    at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:535)
    at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:130)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
    at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:34)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)

  Task Id : attempt_1471511636485_0011_m_000000_2, Status : FAILED
Error: java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 2
    at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:322)
    at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:535)
    at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:130)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
    at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:34)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)

  Task Id : attempt_1471511636485_0011_m_000001_2, Status : FAILED
Error: java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 2
    at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:322)
    at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:535)
    at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:130)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
    at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:34)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)

  Task Id : attempt_1471511636485_0011_m_000002_2, Status : FAILED
Error: java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 2
    at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:322)
    at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:535)
    at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:130)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
    at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:34)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)

   map 100% reduce 100%
  Job job_1471511636485_0011 failed with state FAILED due to: Task failed task_1471511636485_0011_m_000000
Job failed as tasks failed. failedMaps:1 failedReduces:0

  Job not successful!
  Streaming Command Failed!
Counters: 16
    Job Counters
        Failed map tasks=10
        Killed map tasks=2
        Killed reduce tasks=1
        Launched map tasks=12
        Other local map tasks=12
        Total megabyte-milliseconds taken by all map tasks=31032320
        Total megabyte-milliseconds taken by all reduce tasks=0
        Total time spent by all map tasks (ms)=30305
        Total time spent by all maps in occupied slots (ms)=30305
        Total time spent by all reduce tasks (ms)=0
        Total time spent by all reduces in occupied slots (ms)=0
        Total vcore-milliseconds taken by all map tasks=30305
        Total vcore-milliseconds taken by all reduce tasks=0
    Map-Reduce Framework
        CPU time spent (ms)=0
        Physical memory (bytes) snapshot=0
        Virtual memory (bytes) snapshot=0
Scanning logs for probable cause of failure...
> /users/visics/wacha/Develop/hadoop-2.7.2-1/hadoop/bin/hadoop fs -ls hdfs:///tmp/hadoop-yarn/staging
STDOUT: Found 2 items
STDOUT: drwxr-xr-x   - wacha supergroup          0 2016-08-18 11:41 hdfs:///tmp/hadoop-yarn/staging/history
STDOUT: drwx------   - wacha supergroup          0 2016-08-18 11:41 hdfs:///tmp/hadoop-yarn/staging/wacha
Looking for history log in hdfs:///tmp/hadoop-yarn/staging...
> /users/visics/wacha/Develop/hadoop-2.7.2-1/hadoop/bin/hadoop fs -ls hdfs:///tmp/hadoop-yarn/staging
STDOUT: Found 2 items
STDOUT: drwxr-xr-x   - wacha supergroup          0 2016-08-18 11:41 hdfs:///tmp/hadoop-yarn/staging/history
STDOUT: drwx------   - wacha supergroup          0 2016-08-18 11:41 hdfs:///tmp/hadoop-yarn/staging/wacha
> /users/visics/wacha/Develop/hadoop-2.7.2-1/hadoop/bin/hadoop fs -ls -R hdfs:///tmp/hadoop-yarn/staging
> /users/visics/wacha/Develop/hadoop-2.7.2-1/hadoop/bin/hadoop fs -ls hdfs:///user/wacha/tmp/mrjob/mr_exodus2seq_hadoop.wacha.20160818.150831.852145/output/_logs
Looking for history log in /users/visics/wacha/Develop/hadoop-2.7.2-1/hadoop/logs...
> /users/visics/wacha/Develop/hadoop-2.7.2-1/hadoop/bin/hadoop fs -ls hdfs:///tmp/hadoop-yarn/staging/userlogs/application_1471511636485_0011
> /users/visics/wacha/Develop/hadoop-2.7.2-1/hadoop/bin/hadoop fs -ls hdfs:///user/wacha/tmp/mrjob/mr_exodus2seq_hadoop.wacha.20160818.150831.852145/output/_logs/userlogs/application_1471511636485_0011
Probable cause of failure:

Error: java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 2
    at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:322)
    at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:535)
    at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:130)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
    at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:34)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)

Step 1 of 1 failed: Command '['/users/visics/wacha/Develop/hadoop-2.7.2-1/hadoop/bin/hadoop', 'jar', '/users/visics/wacha/Develop/hadoop-2.7.2-1/hadoop/share/hadoop/tools/lib/hadoop-streaming-2.7.2.jar', '-files', 'hdfs:///user/wacha/tmp/mrjob/mr_exodus2seq_hadoop.wacha.20160818.150831.852145/files/setup-wrapper.sh#setup-wrapper.sh,hdfs:///user/wacha/tmp/mrjob/mr_exodus2seq_hadoop.wacha.20160818.150831.852145/files/mr_exodus2seq_hadoop.py#mr_exodus2seq_hadoop.py,hdfs:///user/wacha/tmp/mrjob/mr_exodus2seq_hadoop.wacha.20160818.150831.852145/files/timesteps.txt#timesteps.txt', '-archives', 'hdfs:///user/wacha/tmp/mrjob/mr_exodus2seq_hadoop.wacha.20160818.150831.852145/files/mrjob.tar.gz#mrjob.tar.gz,hdfs:///user/wacha/tmp/mrjob/mr_exodus2seq_hadoop.wacha.20160818.150831.852145/files/simform-deploy.tar.gz#simform-deploy.tar.gz', '-inputformat', 'org.apache.hadoop.mapred.lib.NLineInputFormat', '-input', 'hdfs:///user/wacha/heat_datasets/output/input.txt', '-output', 'hdfs:///user/wacha/tmp/mrjob/mr_exodus2seq_hadoop.wacha.20160818.150831.852145/output', '-mapper', 'sh -ex setup-wrapper.sh python mr_exodus2seq_hadoop.py --step-num=0 --mapper -d hdfs:///user/wacha/heat_datasets/output/data.seq/ --timestepfile timesteps.txt -t 10 --variables TEMP', '-reducer', 'sh -ex setup-wrapper.sh python mr_exodus2seq_hadoop.py --step-num=0 --reducer -d hdfs:///user/wacha/heat_datasets/output/data.seq/ --timestepfile timesteps.txt -t 10 --variables TEMP']' returned non-zero exit status 256

[David Marin]

Shoot, it looks like it’s having trouble finding your task logs, which would explain what’s actually going wrong in your script (the error it found just tells you that your MRJob raised an exception). Could be as simple as an import error.

You’ll want to look in <log dir>/userlogs/application_1471511636485_0011/container_*/stderr, but I can’t figure out where your log dir is from your email (possibly /users/visics/wacha/Develop/hadoop-2.7.2-1/hadoop/logs?).

Also, is the hadoop binary in $PATH from the point of view of your hadoop tasks? This particular script calls the hadoop binary to write to the filesystem (see https://github.com/dgleich/simform/blob/sisc-rm/src/mr_exodus2seq_hadoop.py#L268), so if your script can’t actually call “hadoop” without specifying the whole path, your job will fail.

-Dave

> --
> You received this message because you are subscribed to the Google Groups "mrjob" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to mrjob+un...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Wacha Bounliphone]

Thanks for your help.

- The whole path as been specified in the .bashrc with
export HADOOP_HOME="$HOME/Develop/hadoop-2.7.2-1/hadoop"
export PATH="$HADOOP_HOME/bin:$PATH"

-I have look at the Logs for container_1471511636485_0012_01_000001 , and the error is

====

Aug 19, 2016 11:52:08 AM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register
INFO: Registering org.apache.hadoop.mapreduce.v2.app.webapp.JAXBContextResolver as a provider class
Aug 19, 2016 11:52:08 AM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register
INFO: Registering org.apache.hadoop.yarn.webapp.GenericExceptionHandler as a provider class
Aug 19, 2016 11:52:08 AM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register
INFO: Registering org.apache.hadoop.mapreduce.v2.app.webapp.AMWebServices as a root resource class
Aug 19, 2016 11:52:08 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.9 09/02/2011 11:17 AM'
Aug 19, 2016 11:52:08 AM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider
INFO: Binding org.apache.hadoop.mapreduce.v2.app.webapp.JAXBContextResolver to GuiceManagedComponentProvider with the scope "Singleton"
Aug 19, 2016 11:52:09 AM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider
INFO: Binding org.apache.hadoop.yarn.webapp.GenericExceptionHandler to GuiceManagedComponentProvider with the scope "Singleton"
Aug 19, 2016 11:52:09 AM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider
INFO: Binding org.apache.hadoop.mapreduce.v2.app.webapp.AMWebServices to GuiceManagedComponentProvider with the scope "PerRequest"
log4j:WARN No appenders could be found for logger (org.apache.hadoop.ipc.Server).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.

I'm so sorry, I'm a newbe and don't understand what could be happening and how to debug this problem.
Thanks

[David Marin]

Try looking in a different container; container 1 probably isn’t actually going to run your script.

Also, that looks like syslog, and you want stderr.

-Dave