spark job failed by GCS api 503 error in dataproc

[stev...@liveramp.com]

21/12/07 04:10:33 INFO org.apache.hadoop.mapred.FileInputFormat: Total input files to process : 1

21/12/07 04:15:55 WARN org.apache.spark.scheduler.TaskSetManager: Lost task 728.0 in stage 272.0 (TID 362723, jcid-fox-1638847189324654111-w-64.c.select-eng-us-tv-prod.internal, executor 101): java.io.IOException: Error reading 'gs://xxxxxxxx/xxxxxxxx/xxx/xxx/xxx' at position 97828374442

at com.google.cloud.hadoop.repackaged.gcs.com.google.cloud.hadoop.gcsio.GoogleCloudStorageReadChannel.handleExecuteMediaException(GoogleCloudStorageReadChannel.java:1160)

at com.google.cloud.hadoop.repackaged.gcs.com.google.cloud.hadoop.gcsio.GoogleCloudStorageReadChannel.openStream(GoogleCloudStorageReadChannel.java:976)

at com.google.cloud.hadoop.repackaged.gcs.com.google.cloud.hadoop.gcsio.GoogleCloudStorageReadChannel.openContentChannel(GoogleCloudStorageReadChannel.java:769)

at com.google.cloud.hadoop.repackaged.gcs.com.google.cloud.hadoop.gcsio.GoogleCloudStorageReadChannel.performLazySeek(GoogleCloudStorageReadChannel.java:760)

at com.google.cloud.hadoop.repackaged.gcs.com.google.cloud.hadoop.gcsio.GoogleCloudStorageReadChannel.read(GoogleCloudStorageReadChannel.java:363)

at com.google.cloud.hadoop.fs.gcs.GoogleHadoopFSInputStream.read(GoogleHadoopFSInputStream.java:130)

at java.io.DataInputStream.read(DataInputStream.java:149)

at org.apache.hadoop.mapreduce.lib.input.UncompressedSplitLineReader.fillBuffer(UncompressedSplitLineReader.java:62)

at org.apache.hadoop.util.LineReader.readDefaultLine(LineReader.java:218)

at org.apache.hadoop.util.LineReader.readLine(LineReader.java:176)

at org.apache.hadoop.mapreduce.lib.input.UncompressedSplitLineReader.readLine(UncompressedSplitLineReader.java:94)

at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:255)

at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:48)

at org.apache.spark.rdd.HadoopRDD$$anon$1.getNext(HadoopRDD.scala:277)

at org.apache.spark.rdd.HadoopRDD$$anon$1.getNext(HadoopRDD.scala:214)

at org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:73)

at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)

at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)

at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)

at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:461)

at org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:191)

at org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:63)

at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96)

at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)

at org.apache.spark.scheduler.Task.run(Task.scala:109)

at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

at java.lang.Thread.run(Thread.java:748)

Caused by: com.google.cloud.hadoop.repackaged.gcs.com.google.api.client.googleapis.json.GoogleJsonResponseException: 503 Service Unavailable

Service Unavailable

[stev...@liveramp.com]

Our spark job was failed by "com.google.cloud.hadoop.repackaged.gcs.com.google.api.client.googleapis.json.GoogleJsonResponseException: 503 Service Unavailable" error and we have resubmitted it, it failed by other files again.