OutOfMemoryError: Java heap space problem

[Jingxin Ye]

Hi,

I am running a script on a singlenode hadoop system with 128GB ram. I can run any simple examples without any problems. I went to processing a 300mb file, the mapper function and simple reducer (like mean) works fine. But when I added the real calculation I want to do. It shows "java.lang.OutOfMemoryError: Java heap space". 

15/08/21 15:18:33 WARN streaming.PipeMapRed: java.lang.OutOfMemoryError: Java heap space

After some google, I find I can increase the heap size by configuring mapred-site.xml

<configuration>
 <property>
   <name>mapreduce.framework.name</name>
      <value>yarn</value>
   <name>mapreduce.map.memory.mb</name>
   <value>40960</value>
   <name>mapreduce.reduce.memory.mb</name>
   <value>102400</value>
   <name>mapreduce.map.java.opts</name>
   <value>-Xmx30720m</value>
   <name>mapreduce.reduce.java.opts</name>
   <value>-Xmx61440m</value>
   <name>mapred.child.java.opts</name>
       <value>-Xmx20480m</value>
       </property>
</configuration>

It means I have the heap of size 20GB, but it still gives me the error. But the log shows Total committed heap usage (bytes)=861929472 which is far less then 20GB.

Any ideas where the problem is and how to solve it.

Thanks,

Jingxin

15/08/21 15:18:03 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
15/08/21 15:18:04 INFO Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
15/08/21 15:18:04 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
15/08/21 15:18:04 INFO jvm.JvmMetrics: Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
15/08/21 15:18:04 INFO mapred.FileInputFormat: Total input paths to process : 1
15/08/21 15:18:04 INFO mapreduce.JobSubmitter: number of splits:1
15/08/21 15:18:04 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local1775502531_0001
15/08/21 15:18:04 INFO mapred.LocalDistributedCacheManager: Creating symlink: /tmp/hadoop-hadoop/mapred/local/1440195484827/rmr-local-env2cf018455396 <- /home/hadoop/hail_backup_720/src/rmr-local-env2cf018455396
15/08/21 15:18:04 INFO mapred.LocalDistributedCacheManager: Localized file:/tmp/Rtmpi70pfo/rmr-local-env2cf018455396 as file:/tmp/hadoop-hadoop/mapred/local/1440195484827/rmr-local-env2cf018455396
15/08/21 15:18:04 INFO mapred.LocalDistributedCacheManager: Creating symlink: /tmp/hadoop-hadoop/mapred/local/1440195484828/rmr-global-env2cf03a85a1ba <- /home/hadoop/hail_backup_720/src/rmr-global-env2cf03a85a1ba
15/08/21 15:18:04 INFO mapred.LocalDistributedCacheManager: Localized file:/tmp/Rtmpi70pfo/rmr-global-env2cf03a85a1ba as file:/tmp/hadoop-hadoop/mapred/local/1440195484828/rmr-global-env2cf03a85a1ba
15/08/21 15:18:04 INFO mapred.LocalDistributedCacheManager: Creating symlink: /tmp/hadoop-hadoop/mapred/local/1440195484829/rmr-streaming-map2cf068b80d1e <- /home/hadoop/hail_backup_720/src/rmr-streaming-map2cf068b80d1e
15/08/21 15:18:04 INFO mapred.LocalDistributedCacheManager: Localized file:/tmp/Rtmpi70pfo/rmr-streaming-map2cf068b80d1e as file:/tmp/hadoop-hadoop/mapred/local/1440195484829/rmr-streaming-map2cf068b80d1e
15/08/21 15:18:04 INFO mapred.LocalDistributedCacheManager: Creating symlink: /tmp/hadoop-hadoop/mapred/local/1440195484830/rmr-streaming-reduce2cf02716b8e5 <- /home/hadoop/hail_backup_720/src/rmr-streaming-reduce2cf02716b8e5
15/08/21 15:18:04 INFO mapred.LocalDistributedCacheManager: Localized file:/tmp/Rtmpi70pfo/rmr-streaming-reduce2cf02716b8e5 as file:/tmp/hadoop-hadoop/mapred/local/1440195484830/rmr-streaming-reduce2cf02716b8e5
15/08/21 15:18:04 INFO mapreduce.Job: The url to track the job: http://localhost:8080/
15/08/21 15:18:04 INFO mapred.LocalJobRunner: OutputCommitter set in config null
15/08/21 15:18:04 INFO mapreduce.Job: Running job: job_local1775502531_0001
15/08/21 15:18:04 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter
15/08/21 15:18:04 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1
15/08/21 15:18:05 INFO mapred.LocalJobRunner: Waiting for map tasks
15/08/21 15:18:05 INFO mapred.LocalJobRunner: Starting task: attempt_local1775502531_0001_m_000000_0
15/08/21 15:18:05 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1
15/08/21 15:18:05 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
15/08/21 15:18:05 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/user/jye/kriging/data/hail_input.dat:0+4501870
15/08/21 15:18:05 INFO mapred.MapTask: numReduceTasks: 1
15/08/21 15:18:05 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
15/08/21 15:18:05 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
15/08/21 15:18:05 INFO mapred.MapTask: soft limit at 83886080
15/08/21 15:18:05 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
15/08/21 15:18:05 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
15/08/21 15:18:05 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
15/08/21 15:18:05 INFO streaming.PipeMapRed: PipeMapRed exec [/usr/bin/Rscript, --vanilla, ./rmr-streaming-map2cf068b80d1e]
15/08/21 15:18:05 INFO Configuration.deprecation: mapred.work.output.dir is deprecated. Instead, use mapreduce.task.output.dir
15/08/21 15:18:05 INFO Configuration.deprecation: map.input.start is deprecated. Instead, use mapreduce.map.input.start
15/08/21 15:18:05 INFO Configuration.deprecation: mapred.task.is.map is deprecated. Instead, use mapreduce.task.ismap
15/08/21 15:18:05 INFO Configuration.deprecation: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id
15/08/21 15:18:05 INFO Configuration.deprecation: mapred.tip.id is deprecated. Instead, use mapreduce.task.id
15/08/21 15:18:05 INFO Configuration.deprecation: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir
15/08/21 15:18:05 INFO Configuration.deprecation: map.input.file is deprecated. Instead, use mapreduce.map.input.file
15/08/21 15:18:05 INFO Configuration.deprecation: mapred.skip.on is deprecated. Instead, use mapreduce.job.skiprecords
15/08/21 15:18:05 INFO Configuration.deprecation: map.input.length is deprecated. Instead, use mapreduce.map.input.length
15/08/21 15:18:05 INFO Configuration.deprecation: mapred.job.id is deprecated. Instead, use mapreduce.job.id
15/08/21 15:18:05 INFO Configuration.deprecation: user.name is deprecated. Instead, use mapreduce.job.user.name
15/08/21 15:18:05 INFO Configuration.deprecation: mapred.task.partition is deprecated. Instead, use mapreduce.task.partition
15/08/21 15:18:05 INFO streaming.PipeMapRed: R/W/S=1/0/0 in:NA [rec/s] out:NA [rec/s]
15/08/21 15:18:05 INFO streaming.PipeMapRed: R/W/S=10/0/0 in:NA [rec/s] out:NA [rec/s]
15/08/21 15:18:05 INFO streaming.PipeMapRed: R/W/S=100/0/0 in:NA [rec/s] out:NA [rec/s]
15/08/21 15:18:05 INFO streaming.PipeMapRed: R/W/S=1000/0/0 in:NA [rec/s] out:NA [rec/s]
Loading objects:
  .Random.seed
  input_dataframe
  input_file_format
Please review your hadoop settings. See help(hadoop.settings)
  input_subset
  mapper
  partition.label
  predict.grid
  reducer
Loading objects:
  backend.parameters
  combine
  combine.file
  combine.line
  debug
  default.input.format
  default.output.format
  in.folder
  in.memory.combine
  input.format
  libs
  map
  map.file
  map.line
  out.folder
  output.format
  pkg.opts
  postamble
  preamble
  profile.nodes
  reduce
  reduce.file
  reduce.line
  rmr.global.env
  rmr.local.env
  save.env
  tempfile
  vectorized.reduce
  verbose
  work.dir
Loading required package: methods
Loading required package: rmr2
Loading required package: rJava
15/08/21 15:18:05 INFO mapreduce.Job: Job job_local1775502531_0001 running in uber mode : false
15/08/21 15:18:05 INFO mapreduce.Job:  map 0% reduce 0%
Loading required package: rhdfs

HADOOP_CMD=/home/hadoop/hadoop/bin/hadoop

Be sure to run hdfs.init()
Loading required package: sp
Loading required package: automap
Loading required package: gstat
Loading required package: leaflet
Loading required package: htmltools
Loading objects:
  backend.parameters
  combine
  combine.file
  combine.line
  debug
  default.input.format
  default.output.format
  in.folder
  in.memory.combine
  input.format
  libs
  map
  map.file
  map.line
  out.folder
  output.format
  pkg.opts
  postamble
  preamble
  profile.nodes
  reduce
  reduce.file
  reduce.line
  rmr.global.env
  rmr.local.env
  save.env
  tempfile
  vectorized.reduce
  verbose
  work.dir
15/08/21 15:18:07 INFO streaming.PipeMapRed: R/W/S=10000/0/0 in:5000=10000/2 [rec/s] out:0=0/2 [rec/s]
15/08/21 15:18:11 INFO mapred.LocalJobRunner: hdfs://localhost:9000/user/jye/kriging/data/hail_input.dat:0+4501870 > map
15/08/21 15:18:11 INFO streaming.PipeMapRed: Records R/W=33942/1
15/08/21 15:18:11 INFO mapreduce.Job:  map 19% reduce 0%
15/08/21 15:18:14 INFO mapred.LocalJobRunner: Records R/W=33942/1 > map
15/08/21 15:18:14 INFO mapreduce.Job:  map 25% reduce 0%
15/08/21 15:18:17 INFO mapred.LocalJobRunner: Records R/W=33942/1 > map
15/08/21 15:18:17 INFO mapreduce.Job:  map 31% reduce 0%
15/08/21 15:18:20 INFO mapred.LocalJobRunner: Records R/W=33942/1 > map
15/08/21 15:18:22 INFO streaming.PipeMapRed: Records R/W=53993/9
15/08/21 15:18:26 INFO mapred.LocalJobRunner: Records R/W=53993/9 > map
15/08/21 15:18:26 INFO mapreduce.Job:  map 37% reduce 0%
15/08/21 15:18:28 INFO streaming.PipeMapRed: R/W/S=100000/23/0 in:4347=100000/23 [rec/s] out:1=23/23 [rec/s]
15/08/21 15:18:30 INFO streaming.PipeMapRed: MRErrorThread done
15/08/21 15:18:30 INFO streaming.PipeMapRed: mapRedFinished
15/08/21 15:18:30 INFO mapred.LocalJobRunner: Records R/W=53993/9 > map
15/08/21 15:18:30 INFO mapred.MapTask: Starting flush of map output
15/08/21 15:18:30 INFO mapred.MapTask: Spilling map output
15/08/21 15:18:30 INFO mapred.MapTask: bufstart = 0; bufend = 2191671; bufvoid = 104857600
15/08/21 15:18:30 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26214304(104857216); length = 93/6553600
15/08/21 15:18:30 INFO mapred.MapTask: Finished spill 0
15/08/21 15:18:30 INFO mapred.Task: Task:attempt_local1775502531_0001_m_000000_0 is done. And is in the process of committing
15/08/21 15:18:30 INFO mapred.LocalJobRunner: Records R/W=53993/9
15/08/21 15:18:30 INFO mapred.Task: Task 'attempt_local1775502531_0001_m_000000_0' done.
15/08/21 15:18:30 INFO mapred.LocalJobRunner: Finishing task: attempt_local1775502531_0001_m_000000_0
15/08/21 15:18:30 INFO mapred.LocalJobRunner: map task executor complete.
15/08/21 15:18:30 INFO mapred.LocalJobRunner: Waiting for reduce tasks
15/08/21 15:18:30 INFO mapred.LocalJobRunner: Starting task: attempt_local1775502531_0001_r_000000_0
15/08/21 15:18:30 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1
15/08/21 15:18:30 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
15/08/21 15:18:30 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@2b36d4f8
15/08/21 15:18:30 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=334338464, maxSingleShuffleLimit=83584616, mergeThreshold=220663392, ioSortFactor=10, memToMemMergeOutputsThreshold=10
15/08/21 15:18:30 INFO reduce.EventFetcher: attempt_local1775502531_0001_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
15/08/21 15:18:30 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local1775502531_0001_m_000000_0 decomp: 2191776 len: 2191780 to MEMORY
15/08/21 15:18:30 INFO reduce.InMemoryMapOutput: Read 2191776 bytes from map-output for attempt_local1775502531_0001_m_000000_0
15/08/21 15:18:30 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2191776, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->2191776
15/08/21 15:18:30 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
15/08/21 15:18:30 INFO mapred.LocalJobRunner: 1 / 1 copied.
15/08/21 15:18:30 INFO reduce.MergeManagerImpl: finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
15/08/21 15:18:30 INFO mapred.Merger: Merging 1 sorted segments
15/08/21 15:18:30 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 2191754 bytes
15/08/21 15:18:30 INFO reduce.MergeManagerImpl: Merged 1 segments, 2191776 bytes to disk to satisfy reduce memory limit
15/08/21 15:18:30 INFO reduce.MergeManagerImpl: Merging 1 files, 2191780 bytes from disk
15/08/21 15:18:30 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
15/08/21 15:18:30 INFO mapred.Merger: Merging 1 sorted segments
15/08/21 15:18:30 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 2191754 bytes
15/08/21 15:18:30 INFO mapred.LocalJobRunner: 1 / 1 copied.
15/08/21 15:18:30 INFO streaming.PipeMapRed: PipeMapRed exec [/usr/bin/Rscript, --vanilla, ./rmr-streaming-reduce2cf02716b8e5]
15/08/21 15:18:30 INFO Configuration.deprecation: mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
15/08/21 15:18:30 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
15/08/21 15:18:30 INFO streaming.PipeMapRed: R/W/S=1/0/0 in:NA [rec/s] out:NA [rec/s]
Loading objects:
  .Random.seed
  input_dataframe
  input_file_format
15/08/21 15:18:30 INFO mapreduce.Job:  map 100% reduce 0%
Please review your hadoop settings. See help(hadoop.settings)
  input_subset
  mapper
  partition.label
  predict.grid
  reducer
Loading objects:
  backend.parameters
  combine
  combine.file
  combine.line
  debug
  default.input.format
  default.output.format
  in.folder
  in.memory.combine
  input.format
  libs
  map
  map.file
  map.line
  out.folder
  output.format
  pkg.opts
  postamble
  preamble
  profile.nodes
  reduce
  reduce.file
  reduce.line
  rmr.global.env
  rmr.local.env
  save.env
  tempfile
  vectorized.reduce
  verbose
  work.dir
Loading required package: methods
Loading required package: rmr2
Loading required package: rJava
Loading required package: rhdfs

HADOOP_CMD=/home/hadoop/hadoop/bin/hadoop

Be sure to run hdfs.init()
Loading required package: sp
Loading required package: automap
Loading required package: gstat
Loading required package: leaflet
Loading required package: htmltools
Loading objects:
  backend.parameters
  combine
  combine.file
  combine.line
  debug
  default.input.format
  default.output.format
  in.folder
  in.memory.combine
  input.format
  libs
  map
  map.file
  map.line
  out.folder
  output.format
  pkg.opts
  postamble
  preamble
  profile.nodes
  reduce
  reduce.file
  reduce.line
  rmr.global.env
  rmr.local.env
  save.env
  tempfile
  vectorized.reduce
  verbose
  work.dir
15/08/21 15:18:32 INFO streaming.PipeMapRed: R/W/S=10/0/0 in:5=10/2 [rec/s] out:0=0/2 [rec/s]
15/08/21 15:18:33 WARN streaming.PipeMapRed: java.lang.OutOfMemoryError: Java heap space
15/08/21 15:18:36 INFO mapred.LocalJobRunner: reduce > reduce
15/08/21 15:18:36 INFO mapreduce.Job:  map 100% reduce 100%
15/08/21 15:20:55 INFO streaming.PipeMapRed: MRErrorThread done
15/08/21 15:20:55 INFO streaming.PipeMapRed: PipeMapRed failed!
java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 137
	at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:322)
	at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:535)
	at org.apache.hadoop.streaming.PipeReducer.close(PipeReducer.java:134)
	at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:453)
	at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392)
	at org.apache.hadoop.mapred.LocalJobRunner$Job$ReduceTaskRunnable.run(LocalJobRunner.java:319)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
15/08/21 15:20:55 INFO streaming.PipeMapRed: PipeMapRed failed!
java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 137
	at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:322)
	at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:535)
	at org.apache.hadoop.streaming.PipeReducer.close(PipeReducer.java:134)
	at org.apache.hadoop.io.IOUtils.cleanup(IOUtils.java:244)
	at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:459)
	at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392)
	at org.apache.hadoop.mapred.LocalJobRunner$Job$ReduceTaskRunnable.run(LocalJobRunner.java:319)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
15/08/21 15:20:55 INFO mapred.LocalJobRunner: reduce task executor complete.
15/08/21 15:20:55 WARN mapred.LocalJobRunner: job_local1775502531_0001
java.lang.Exception: java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 137
	at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462)
	at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:529)
Caused by: java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 137
	at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:322)
	at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:535)
	at org.apache.hadoop.streaming.PipeReducer.close(PipeReducer.java:134)
	at org.apache.hadoop.io.IOUtils.cleanup(IOUtils.java:244)
	at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:459)
	at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392)
	at org.apache.hadoop.mapred.LocalJobRunner$Job$ReduceTaskRunnable.run(LocalJobRunner.java:319)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
15/08/21 15:20:56 INFO mapreduce.Job: Job job_local1775502531_0001 failed with state FAILED due to: NA
15/08/21 15:20:56 INFO mapreduce.Job: Counters: 36
	File System Counters
		FILE: Number of bytes read=9427600
		FILE: Number of bytes written=12232504
		FILE: Number of read operations=0
		FILE: Number of large read operations=0
		FILE: Number of write operations=0
		HDFS: Number of bytes read=9003740
		HDFS: Number of bytes written=122
		HDFS: Number of read operations=10
		HDFS: Number of large read operations=0
		HDFS: Number of write operations=3
	Map-Reduce Framework
		Map input records=118188
		Map output records=24
		Map output bytes=2191671
		Map output materialized bytes=2191780
		Input split bytes=110
		Combine input records=0
		Combine output records=0
		Reduce input groups=4
		Reduce shuffle bytes=2191780
		Reduce input records=24
		Reduce output records=0
		Spilled Records=48
		Shuffled Maps =1
		Failed Shuffles=0
		Merged Map outputs=1
		GC time elapsed (ms)=75
		Total committed heap usage (bytes)=861929472
	Shuffle Errors
		BAD_ID=0
		CONNECTION=0
		IO_ERROR=0
		WRONG_LENGTH=0
		WRONG_MAP=0
		WRONG_REDUCE=0
	File Input Format Counters 
		Bytes Read=4501870
	File Output Format Counters 
		Bytes Written=122
	rmr
		reduce calls=3
15/08/21 15:20:56 ERROR streaming.StreamJob: Job not successful!
Streaming Command Failed!
15/08/21 15:20:56 WARN hdfs.DFSClient: DataStreamer Exception
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException): No lease on /user/jye/kriging/data/hail_output.dat/_temporary/0/_temporary/attempt_local1775502531_0001_r_000000_0/part-00000 (inode 17136): File does not exist. Holder DFSClient_NONMAPREDUCE_1923592826_1 does not have any open files.
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:3431)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.analyzeFileState(FSNamesystem.java:3236)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getNewBlockTargets(FSNamesystem.java:3074)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3034)
	at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:723)
	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:492)
	at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2043)

	at org.apache.hadoop.ipc.Client.call(Client.java:1476)
	at org.apache.hadoop.ipc.Client.call(Client.java:1407)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
	at com.sun.proxy.$Proxy9.addBlock(Unknown Source)
	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:418)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
	at com.sun.proxy.$Proxy10.addBlock(Unknown Source)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1430)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1226)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:449)
15/08/21 15:20:56 ERROR hdfs.DFSClient: Failed to close inode 17136
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException): No lease on /user/jye/kriging/data/hail_output.dat/_temporary/0/_temporary/attempt_local1775502531_0001_r_000000_0/part-00000 (inode 17136): File does not exist. Holder DFSClient_NONMAPREDUCE_1923592826_1 does not have any open files.
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:3431)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.analyzeFileState(FSNamesystem.java:3236)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getNewBlockTargets(FSNamesystem.java:3074)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3034)
	at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:723)
	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:492)
	at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2043)

	at org.apache.hadoop.ipc.Client.call(Client.java:1476)
	at org.apache.hadoop.ipc.Client.call(Client.java:1407)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
	at com.sun.proxy.$Proxy9.addBlock(Unknown Source)
	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:418)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
	at com.sun.proxy.$Proxy10.addBlock(Unknown Source)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1430)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1226)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:449)
Error in mr(map = map, reduce = reduce, combine = combine, vectorized.reduce,  : 
  hadoop streaming failed with error code 1

[Antonio Piccolboni]

Have you checked help(hadoop.settings)? rmr2 is also setting some of those properties on a per job basis and it may be actually confusing the issue for you.

--
post: rha...@googlegroups.com ||
unsubscribe: rhadoop+u...@googlegroups.com ||
web: https://groups.google.com/d/forum/rhadoop?hl=en-US
---
You received this message because you are subscribed to the Google Groups "RHadoop" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rhadoop+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.