Hive Dynamic Partitioning Memory Issues

[Ram Kedem]

Hi,

Performing Dynamic Partitioning on a huge record-set and keep failing on lack of memory (insufficient Java heap space), although I have enough.
In the Hadoop monitoring UI (http://cluster:8088/cluster/scheduler) I can see the maximum memory being allocated to one Hive job is about 200GB while I have free 3TB of memory.

These are my basic Hive configurations (to allow Hive dynamic partitioning ) -

set hive.exec.dynamic.partition.mode=nonstrict;
set hive.exec.max.dynamic.partitions=20480;
set hive.exec.max.dynamic.partitions.pernode=2560;
set hive.exec.max.created.files=200000;


In addition I've tried to adjust these memory parameters, but haven't seen any positive results, my memory limit is still 200GB

SET mapred.child.java.opts=-Xmx600g;
SET mapred.map.child.java.opts=-Xmx600g;
SET mapred.reduce.child.java.opts=-Xmx600g;
set mapreduce.map.memory.mb=1000000;
set mapreduce.reduce.memory.mb=1000000 ;

Please assist,

Many thanks !