I'm attempting to run a 'copy' command to copy from one large dataset to a new dataset with parquet format with different partition strategy. This keeps failing with this message:
Container [pid=1546,containerID=container_1452278047864_45579_01_107350] is running beyond physical memory limits. Current usage: 2.8 GB of 2 GB physical memory used; 3.4 GB of 4.2 GB virtual memory used. Killing container.
In order to set the java options via flags, I'm running into this error:
~~~~~~~~~~~~~~~~
$ flags="-Xmx4096m" kite-dataset copy src_table dest_table
Exception in thread "main" java.lang.ClassNotFoundException: -Xmx4096m
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:270)
at org.apache.hadoop.util.RunJar.run(RunJar.java:214)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
~~~~~~~~~~~~~~~~~~~
I'm on "1.0.0-cdh5.5.1" Kite version, please let me know how to fix these errors, thanks!