Failed to run the first job on dataproc

[Alexandra Faynburd]

Hi,

I'm trying to run the example (MRWordFrequencyCount job) on GCP DataProc using all default settings.
The cluster is created successfully but the job fails with the following output

Probable cause of failure:

Error: java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 1

at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:325)

at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:538)

at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:130)

at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)

at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:34)

at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:459)

at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)

at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:177)

at java.security.AccessController.doPrivileged(Native Method)

at javax.security.auth.Subject.doAs(Subject.java:422)

at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1893)

at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:171)

Step 1 of 1 failed

Any hint where to look for a problem?

Thanks

[Riaz Jahangir]

"Error: java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code XX" is the generic error you get from the hadoop framework. Unless you specify particular exit codes in your own code, this log is unlikely to be helpful.

There are other logs but they can be tough to find depending on your setup. If you have a YARN UI running, you can dig in through the "History" link in the "Tracking UI" column of the main dashboard. From there, you'll find links to the number of jobs that failed. Click through until you have a view of specific nodes/attempts that failed, and you'll see another "Logs" link that will actually show you the python stderr.

Reference that might help: https://cloud.google.com/dataproc/docs/concepts/accessing/cluster-web-interfaces

-Riaz