hive.auto.convert.join.noconditionaltask.size and tez.runtime.pipelined.sorter.lazy-allocate.memory in MR3 1.7

[Sungwoo Park]

In MR3 1.7, we have changed the values for hive.auto.convert.join.noconditionaltask.size and tez.runtime.pipelined.sorter.lazy-allocate.memory in hive-site.xml and tez-site.xml as follows:

<property>
  <name>hive.auto.convert.join.noconditionaltask.size</name>
  <value>4000000000</value>
</property>

<property>
  <name>tez.runtime.pipelined.sorter.lazy-allocate.memory</name>
  <value>true</value>
</property>

Previously hive.auto.convert.join.noconditionaltask.size was conservatively set to 1145044992 and tez.runtime.pipelined.sorter.lazy-allocate.memory to false.

With the new settings for the above configuration parameters, map-side joins are more frequently triggered, and you might see a huge reduction in the execution time on join-heavy queries.

--- Sungwoo