java.lang.OutOfMemoryError: Java heap space while running Database Batch Sink

[shhh]

Hi,

I'm getting java.lang.OutOfMemoryError: Java heap space errors while running Database Batch Sink inserting bulk records into Oracle. I tried increasing mapreduce.map memory and jva heap settings in Mapreduce2 in Hortonworks but did not help. Here are all the logs. Anyone has any suggestions? thank you

2020-01-25 15:50:36,835 - INFO [program.status:c.c.c.i.a.r.d.DistributedProgramRunner@472] - Starting Workflow Program 'DataPipelineWorkflow' with Arguments [logical.start.time=1579967427280]

2020-01-25 15:50:36,835 - INFO  [program.status:c.c.c.i.a.r.d.DistributedProgramRunner@472] - Starting Workflow Program 'DataPipelineWorkflow' with Arguments [logical.start.time=1579967427280]
2020-01-25 15:51:08,338 - INFO  [WorkflowDriver:c.c.c.d.SmartWorkflow@454] - Pipeline 'Oracle_Export_Pipeline_v11' is started by user 'yarn' with arguments {logical.start.time=1579967427280}
2020-01-25 15:51:08,394 - INFO  [WorkflowDriver:c.c.c.d.SmartWorkflow@488] - Pipeline 'Oracle_Export_Pipeline_v11' running
2020-01-25 15:51:08,492 - INFO  [WorkflowDriver:c.c.c.i.a.r.w.WorkflowDriver@611] - Starting workflow execution for 'DataPipelineWorkflow' with Run id '681b7c98-3f8a-11ea-a2c5-0242f6d55f01'
2020-01-25 15:51:24,416 - INFO  [action-phase-2-0:c.c.c.i.a.r.w.WorkflowDriver@342] - Starting MapReduce Program 'phase-2' in workflow
2020-01-25 15:52:15,656 - INFO  [MapReduceRunner-phase-2:c.c.c.i.a.r.b.MapReduceRuntimeService@379] - Submitted MapReduce Job: name=phase-2, jobId=job_1579965990413_0006, namespaceId=default, applicationId=Oracle_Export_Pipeline_v11, program=phase-2, runid=8a2775a1-3f8a-11ea-95e5-00001702b31d.
2020-01-25 15:52:25,663 - WARN  [main:o.a.h.h.s.DomainSocketFactory@117] - The short-circuit local reads feature cannot be used because libhadoop cannot be loaded.
2020-01-25 15:52:26,204 - WARN  [main:o.a.h.m.i.MetricsConfig@125] - Cannot locate configuration: tried hadoop-metrics2-mrappmaster.properties,hadoop-metrics2.properties
2020-01-25 15:52:26,622 - INFO  [main:o.m.log@67] - Logging to Logger[org.mortbay.log] via org.mortbay.log.Slf4jLog
2020-01-25 15:52:26,642 - WARN  [main:o.a.h.h.HttpRequestLog@100] - Jetty request log can only be enabled using Log4j
2020-01-25 15:52:27,187 - INFO  [main:c.s.j.g.s.c.GuiceComponentProviderFactory@104] - Registering org.apache.hadoop.mapreduce.v2.app.webapp.JAXBContextResolver as a provider class
2020-01-25 15:52:27,188 - INFO  [main:c.s.j.g.s.c.GuiceComponentProviderFactory@104] - Registering org.apache.hadoop.yarn.webapp.GenericExceptionHandler as a provider class
2020-01-25 15:52:27,189 - INFO  [main:c.s.j.g.s.c.GuiceComponentProviderFactory@107] - Registering org.apache.hadoop.mapreduce.v2.app.webapp.AMWebServices as a root resource class
2020-01-25 15:52:27,194 - INFO  [main:c.s.j.s.i.a.WebApplicationImpl@791] - Initiating Jersey application, version 'Jersey: 1.9 09/02/2011 11:17 AM'
2020-01-25 15:52:27,291 - INFO  [main:c.s.j.g.s.c.GuiceComponentProviderFactory@159] - Binding org.apache.hadoop.mapreduce.v2.app.webapp.JAXBContextResolver to GuiceManagedComponentProvider with the scope "Singleton"
2020-01-25 15:52:27,696 - INFO  [main:c.s.j.g.s.c.GuiceComponentProviderFactory@159] - Binding org.apache.hadoop.yarn.webapp.GenericExceptionHandler to GuiceManagedComponentProvider with the scope "Singleton"
2020-01-25 15:52:28,078 - INFO  [main:c.s.j.g.s.c.GuiceComponentProviderFactory@159] - Binding org.apache.hadoop.mapreduce.v2.app.webapp.AMWebServices to GuiceManagedComponentProvider with the scope "PerRequest"
2020-01-25 15:52:28,486 - INFO  [CommitterEvent Processor #0:c.c.c.i.a.r.b.MainOutputCommitter@94] - Setting up for MapReduce job: namespaceId=default, applicationId=Oracle_Export_Pipeline_v11, program=phase-2, runid=8a2775a1-3f8a-11ea-95e5-00001702b31d
2020-01-25 15:52:29,156 - WARN  [CommitterEvent Processor #0:o.a.h.m.l.o.FileOutputCommitter@347] - Output Path is null in setupJob()
2020-01-25 15:52:39,292 - WARN  [main:o.a.h.m.i.MetricsConfig@125] - Cannot locate configuration: tried hadoop-metrics2-maptask.properties,hadoop-metrics2.properties
2020-01-25 15:52:39,439 - WARN  [main:o.a.h.m.i.MetricsConfig@125] - Cannot locate configuration: tried hadoop-metrics2-maptask.properties,hadoop-metrics2.properties
2020-01-25 15:52:40,389 - INFO  [Socket Reader #1 for port 40503:S.o.a.h.i.Server@1573] - Auth successful for job_1579965990413_0006 (auth:SIMPLE)
2020-01-25 15:52:40,471 - INFO  [Socket Reader #1 for port 40503:S.o.a.h.i.Server@1573] - Auth successful for job_1579965990413_0006 (auth:SIMPLE)
2020-01-25 15:56:17,817 - WARN  [CommitterEvent Processor #1:o.a.h.m.l.o.FileOutputCommitter@619] - Output Path is null in abortTask()
2020-01-25 15:56:27,579 - WARN  [main:o.a.h.m.i.MetricsConfig@125] - Cannot locate configuration: tried hadoop-metrics2-maptask.properties,hadoop-metrics2.properties
2020-01-25 15:56:28,162 - INFO  [Socket Reader #1 for port 40503:S.o.a.h.i.Server@1573] - Auth successful for job_1579965990413_0006 (auth:SIMPLE)
2020-01-25 15:57:58,177 - WARN  [CommitterEvent Processor #2:o.a.h.m.l.o.FileOutputCommitter@619] - Output Path is null in abortTask()
2020-01-25 15:58:07,056 - WARN  [main:o.a.h.m.i.MetricsConfig@125] - Cannot locate configuration: tried hadoop-metrics2-maptask.properties,hadoop-metrics2.properties
2020-01-25 15:58:07,980 - INFO  [Socket Reader #1 for port 40503:S.o.a.h.i.Server@1573] - Auth successful for job_1579965990413_0006 (auth:SIMPLE)
2020-01-25 15:58:48,990 - ERROR [MessagingMetricsCollectionService:c.c.c.m.c.AggregatedMetricsCollectionService@137] - Failed in publishing metrics for timestamp 1579967915.
java.lang.OutOfMemoryError: Java heap space
	at java.util.Arrays.copyOfRange(Arrays.java:3664) ~[na:1.8.0_112]
	at java.lang.String.<init>(String.java:207) ~[na:1.8.0_112]
	at java.lang.StringBuilder.toString(StringBuilder.java:407) ~[na:1.8.0_112]
	at sun.net.www.MessageHeader.print(MessageHeader.java:297) ~[na:1.8.0_112]
	at sun.net.www.http.HttpClient.writeRequests(HttpClient.java:613) ~[na:1.8.0_112]
	at sun.net.www.http.HttpClient.writeRequests(HttpClient.java:624) ~[na:1.8.0_112]
	at sun.net.www.protocol.http.HttpURLConnection.writeRequests(HttpURLConnection.java:693) ~[na:1.8.0_112]
	at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1567) ~[na:1.8.0_112]
	at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1474) ~[na:1.8.0_112]
	at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480) ~[na:1.8.0_112]
	at co.cask.common.http.HttpRequests.execute(HttpRequests.java:114) ~[co.cask.common.common-http-0.11.0.jar:na]
	at co.cask.cdap.common.internal.remote.RemoteClient.execute(RemoteClient.java:93) ~[na:na]
	at co.cask.cdap.messaging.client.ClientMessagingService.performWriteRequest(ClientMessagingService.java:253) ~[na:na]
	at co.cask.cdap.messaging.client.ClientMessagingService.publish(ClientMessagingService.java:184) ~[na:na]
	at co.cask.cdap.metrics.collect.MessagingMetricsCollectionService$TopicPayload.publish(MessagingMetricsCollectionService.java:153) ~[na:na]
	at co.cask.cdap.metrics.collect.MessagingMetricsCollectionService.publishMetric(MessagingMetricsCollectionService.java:106) ~[na:na]
	at co.cask.cdap.metrics.collect.MessagingMetricsCollectionService.publish(MessagingMetricsCollectionService.java:101) ~[na:na]
	at co.cask.cdap.metrics.collect.AggregatedMetricsCollectionService.publishMetrics(AggregatedMetricsCollectionService.java:133) [na:na]
	at co.cask.cdap.metrics.collect.AggregatedMetricsCollectionService.run(AggregatedMetricsCollectionService.java:117) [na:na]
	at com.google.common.util.concurrent.AbstractExecutionThreadService$1$1.run(AbstractExecutionThreadService.java:52) [com.google.guava.guava-13.0.1.jar:na]
	at java.lang.Thread.run(Thread.java:745) [na:1.8.0_112]
2020-01-25 15:58:49,008 - ERROR [main:o.a.h.m.YarnChild@209] - Error running child : java.lang.OutOfMemoryError: Java heap space
	at oracle.jdbc.driver.OraclePreparedStatement.executeLargeBatch(OraclePreparedStatement.java:9454)
	at oracle.jdbc.driver.T4CPreparedStatement.executeLargeBatch(T4CPreparedStatement.java:1405)
	at oracle.jdbc.driver.OraclePreparedStatement.executeBatch(OraclePreparedStatement.java:9442)
	at oracle.jdbc.driver.OracleStatementWrapper.executeBatch(OracleStatementWrapper.java:234)
	at co.cask.hydrator.plugin.db.batch.sink.ETLDBOutputFormat$1.close(ETLDBOutputFormat.java:85)
	at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.close(MapTask.java:670)
	at org.apache.hadoop.mapred.MapTask.closeQuietly(MapTask.java:2019)
	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:797)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:170)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)
	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:164)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at co.cask.cdap.internal.app.runtime.batch.distributed.MapReduceContainerLauncher.launch(MapReduceContainerLauncher.java:114)
	at org.apache.hadoop.mapred.YarnChild.main(Unknown Source)

2020-01-25 15:58:49,012 - INFO  [Socket Reader #1 for port 40503:S.o.a.h.i.Server@1573] - Auth successful for job_1579965990413_0006 (auth:SIMPLE)
2020-01-25 15:58:49,015 - ERROR [IPC Server handler 16 on 40503:o.a.h.m.TaskAttemptListenerImpl@249] - Task: attempt_1579965990413_0006_m_000000_1 - exited : Java heap space
2020-01-25 15:58:49,029 - WARN  [CommitterEvent Processor #3:o.a.h.m.l.o.FileOutputCommitter@619] - Output Path is null in abortTask()
2020-01-25 15:58:57,093 - WARN  [main:o.a.h.m.i.MetricsConfig@125] - Cannot locate configuration: tried hadoop-metrics2-maptask.properties,hadoop-metrics2.properties
2020-01-25 15:58:57,991 - INFO  [Socket Reader #1 for port 40503:S.o.a.h.i.Server@1573] - Auth successful for job_1579965990413_0006 (auth:SIMPLE)
2020-01-25 16:01:15,249 - ERROR [MessagingMetricsCollectionService:c.c.c.m.c.AggregatedMetricsCollectionService@137] - Failed in publishing metrics for timestamp 1579968062.
java.lang.OutOfMemoryError: Java heap space
	at java.util.Arrays.copyOf(Arrays.java:3332) ~[na:1.8.0_112]
	at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124) ~[na:1.8.0_112]
	at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448) ~[na:1.8.0_112]
	at java.lang.StringBuilder.append(StringBuilder.java:136) ~[na:1.8.0_112]
	at sun.net.www.MessageHeader.print(MessageHeader.java:297) ~[na:1.8.0_112]
	at sun.net.www.http.HttpClient.writeRequests(HttpClient.java:613) ~[na:1.8.0_112]
	at sun.net.www.http.HttpClient.writeRequests(HttpClient.java:624) ~[na:1.8.0_112]
	at sun.net.www.protocol.http.HttpURLConnection.writeRequests(HttpURLConnection.java:693) ~[na:1.8.0_112]
	at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1567) ~[na:1.8.0_112]
	at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1474) ~[na:1.8.0_112]
	at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480) ~[na:1.8.0_112]
	at co.cask.common.http.HttpRequests.execute(HttpRequests.java:114) ~[co.cask.common.common-http-0.11.0.jar:na]
	at co.cask.cdap.common.internal.remote.RemoteClient.execute(RemoteClient.java:93) ~[na:na]
	at co.cask.cdap.messaging.client.ClientMessagingService.performWriteRequest(ClientMessagingService.java:253) ~[na:na]
	at co.cask.cdap.messaging.client.ClientMessagingService.publish(ClientMessagingService.java:184) ~[na:na]
	at co.cask.cdap.metrics.collect.MessagingMetricsCollectionService$TopicPayload.publish(MessagingMetricsCollectionService.java:153) ~[na:na]
	at co.cask.cdap.metrics.collect.MessagingMetricsCollectionService.publishMetric(MessagingMetricsCollectionService.java:106) ~[na:na]
	at co.cask.cdap.metrics.collect.MessagingMetricsCollectionService.publish(MessagingMetricsCollectionService.java:101) ~[na:na]
	at co.cask.cdap.metrics.collect.AggregatedMetricsCollectionService.publishMetrics(AggregatedMetricsCollectionService.java:133) [na:na]
	at co.cask.cdap.metrics.collect.AggregatedMetricsCollectionService.run(AggregatedMetricsCollectionService.java:117) [na:na]
	at com.google.common.util.concurrent.AbstractExecutionThreadService$1$1.run(AbstractExecutionThreadService.java:52) [com.google.guava.guava-13.0.1.jar:na]
	at java.lang.Thread.run(Thread.java:745) [na:1.8.0_112]
2020-01-25 16:01:15,264 - ERROR [main:o.a.h.m.YarnChild@209] - Error running child : java.lang.OutOfMemoryError: Java heap space
	at oracle.jdbc.driver.OraclePreparedStatement.executeLargeBatch(OraclePreparedStatement.java:9454)
	at oracle.jdbc.driver.T4CPreparedStatement.executeLargeBatch(T4CPreparedStatement.java:1405)
	at oracle.jdbc.driver.OraclePreparedStatement.executeBatch(OraclePreparedStatement.java:9442)
	at oracle.jdbc.driver.OracleStatementWrapper.executeBatch(OracleStatementWrapper.java:234)
	at co.cask.hydrator.plugin.db.batch.sink.ETLDBOutputFormat$1.close(ETLDBOutputFormat.java:85)
	at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.close(MapTask.java:670)
	at org.apache.hadoop.mapred.MapTask.closeQuietly(MapTask.java:2019)
	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:797)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:170)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)
	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:164)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at co.cask.cdap.internal.app.runtime.batch.distributed.MapReduceContainerLauncher.launch(MapReduceContainerLauncher.java:114)
	at org.apache.hadoop.mapred.YarnChild.main(Unknown Source)

2020-01-25 16:01:15,638 - INFO  [Socket Reader #1 for port 40503:S.o.a.h.i.Server@1573] - Auth successful for job_1579965990413_0006 (auth:SIMPLE)
2020-01-25 16:01:15,641 - ERROR [IPC Server handler 25 on 40503:o.a.h.m.TaskAttemptListenerImpl@249] - Task: attempt_1579965990413_0006_m_000000_2 - exited : Java heap space
2020-01-25 16:01:15,654 - WARN  [CommitterEvent Processor #4:o.a.h.m.l.o.FileOutputCommitter@619] - Output Path is null in abortTask()
2020-01-25 16:01:23,140 - WARN  [main:o.a.h.m.i.MetricsConfig@125] - Cannot locate configuration: tried hadoop-metrics2-maptask.properties,hadoop-metrics2.properties
2020-01-25 16:01:26,459 - INFO  [Socket Reader #1 for port 40503:S.o.a.h.i.Server@1573] - Auth successful for job_1579965990413_0006 (auth:SIMPLE)
2020-01-25 16:03:28,178 - WARN  [CommitterEvent Processor #0:o.a.h.m.l.o.FileOutputCommitter@619] - Output Path is null in abortTask()
2020-01-25 16:03:34,794 - WARN  [main:o.a.h.m.i.MetricsConfig@125] - Cannot locate configuration: tried hadoop-metrics2-maptask.properties,hadoop-metrics2.properties
2020-01-25 16:03:38,149 - INFO  [Socket Reader #1 for port 40503:S.o.a.h.i.Server@1573] - Auth successful for job_1579965990413_0006 (auth:SIMPLE)
2020-01-25 16:05:59,656 - WARN  [CommitterEvent Processor #1:o.a.h.m.l.o.FileOutputCommitter@619] - Output Path is null in abortTask()
2020-01-25 16:05:59,672 - WARN  [CommitterEvent Processor #2:o.a.h.m.l.o.FileOutputCommitter@619] - Output Path is null in abortTask()
2020-01-25 16:05:59,675 - WARN  [CommitterEvent Processor #3:o.a.h.m.l.o.FileOutputCommitter@517] - Output Path is null in cleanupJob()
2020-01-25 16:05:59,676 - INFO  [CommitterEvent Processor #3:c.c.c.i.a.r.b.MainOutputCommitter@181] - Invalidating transaction 1579967546027000000
2020-01-25 16:06:00,641 - INFO  [MapReduceRunner-phase-2:c.c.c.i.a.r.b.MapReduceRuntimeService@424] - MapReduce Job completed. Job details: [name=phase-2, jobId=job_1579965990413_0006, namespaceId=default, applicationId=Oracle_Export_Pipeline_v11, program=phase-2, runid=8a2775a1-3f8a-11ea-95e5-00001702b31d]
2020-01-25 16:06:02,680 - INFO  [MapReduceRunner-phase-2:c.c.c.e.b.m.ETLMapReduce@415] - Batch Run finished : status = ProgramState{status=FAILED, failureInfo='Task failed task_1579965990413_0006_m_000000
Job failed as tasks failed. failedMaps:1 failedReduces:0
'}
2020-01-25 16:06:02,998 - ERROR [MapReduceRunner-phase-2:c.c.c.i.a.r.ProgramControllerServiceAdapter@97] - MapReduce Program 'phase-2' failed.
java.lang.IllegalStateException: MapReduce JobId job_1579965990413_0006 failed
	at com.google.common.base.Preconditions.checkState(Preconditions.java:176) ~[com.google.guava.guava-13.0.1.jar:na]
	at co.cask.cdap.internal.app.runtime.batch.MapReduceRuntimeService.run(MapReduceRuntimeService.java:435) ~[na:na]
	at com.google.common.util.concurrent.AbstractExecutionThreadService$1$1.run(AbstractExecutionThreadService.java:52) ~[com.google.guava.guava-13.0.1.jar:na]
	at co.cask.cdap.internal.app.runtime.batch.MapReduceRuntimeService$2$1.run(MapReduceRuntimeService.java:469) [na:na]
	at java.lang.Thread.run(Thread.java:745) [na:1.8.0_112]
2020-01-25 16:06:03,002 - ERROR [MapReduceRunner-phase-2:c.c.c.i.a.r.ProgramControllerServiceAdapter@98] - MapReduce program 'phase-2' failed with error: MapReduce JobId job_1579965990413_0006 failed. Please check the system logs for more details.
java.lang.IllegalStateException: MapReduce JobId job_1579965990413_0006 failed
	at com.google.common.base.Preconditions.checkState(Preconditions.java:176) ~[com.google.guava.guava-13.0.1.jar:na]
	at co.cask.cdap.internal.app.runtime.batch.MapReduceRuntimeService.run(MapReduceRuntimeService.java:435) ~[na:na]
	at com.google.common.util.concurrent.AbstractExecutionThreadService$1$1.run(AbstractExecutionThreadService.java:52) ~[com.google.guava.guava-13.0.1.jar:na]
	at co.cask.cdap.internal.app.runtime.batch.MapReduceRuntimeService$2$1.run(MapReduceRuntimeService.java:469) [na:na]
	at java.lang.Thread.run(Thread.java:745) [na:1.8.0_112]
2020-01-25 16:06:03,017 - ERROR [WorkflowDriver:c.c.c.d.SmartWorkflow@544] - Pipeline 'Oracle_Export_Pipeline_v11' failed.
2020-01-25 16:06:03,306 - ERROR [WorkflowDriver:c.c.c.i.a.r.w.WorkflowProgramController@89] - Workflow service 'workflow.default.Oracle_Export_Pipeline_v11.DataPipelineWorkflow.681b7c98-3f8a-11ea-a2c5-0242f6d55f01' failed.
java.lang.IllegalStateException: MapReduce JobId job_1579965990413_0006 failed
	at com.google.common.base.Preconditions.checkState(Preconditions.java:176) ~[com.google.guava.guava-13.0.1.jar:na]
	at co.cask.cdap.internal.app.runtime.batch.MapReduceRuntimeService.run(MapReduceRuntimeService.java:435) ~[na:na]
	at com.google.common.util.concurrent.AbstractExecutionThreadService$1$1.run(AbstractExecutionThreadService.java:52) ~[com.google.guava.guava-13.0.1.jar:na]
	at co.cask.cdap.internal.app.runtime.batch.MapReduceRuntimeService$2$1.run(MapReduceRuntimeService.java:469) ~[na:na]
	at java.lang.Thread.run(Thread.java:745) [na:1.8.0_112]

[shhh]

Any guidance here?

[shhh]

Hi Team,

In CDAP, How can I assign more resources to MR job? In my case, Database batch sink is creating MR job and fails with out of memory errors on tasks. Here is the error on one of tasks -

Exception from container-launch. Container id: container_e15_1580092284179_0013_01_000003 Exit code: 1 Container exited with a non-zero exit code 1. Error file: prelaunch.err. Last 4096 bytes of prelaunch.err : Last 4096 bytes of stderr : SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/hdfs_data/hadoop/yarn/local/usercache/yarn/appcache/application_1580092284179_0013/filecache/10/job.jar/lib/ch.qos.logback.logback-classic-1.0.9.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/hdfs_data/hadoop/yarn/local/filecache/12/mapreduce.tar.gz/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [ch.qos.logback.classic.util.ContextSelectorStaticBinder] SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/hdfs_data/hadoop/yarn/local/usercache/yarn/appcache/application_1580092284179_0013/filecache/10/job.jar/lib/ch.qos.logback.logback-classic-1.0.9.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/hdfs_data/hadoop/yarn/local/filecache/12/mapreduce.tar.gz/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [ch.qos.logback.classic.util.ContextSelectorStaticBinder] Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "DistributedMapReduceTaskContextProvider STARTING-SendThread(host1.com:2181)" Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "client DomainSocketWatcher" Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "MessagingMetricsCollectionService" Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Thread for syncLogs" Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "ShortCircuitCache_Cleaner" Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "kafka-client-cleanup" Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "IPC Client (658107584) connection to /x.x.x.x:39269 from job_1580092284179_0013" Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "metrics-meter-tick-thread-1" Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Keep-Alive-Timer" Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "metrics-meter-tick-thread-2" Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "main" Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "communication thread"

Thanks in advance

[Sreevatsan Raman]

You can click on Configure in Pipeline details page and set resources for mappers and reducers. Please see attached screenshot

--
You received this message because you are subscribed to the Google Groups "CDAP User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cdap-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cdap-user/855a2e0e-f4c0-4db8-9747-bddd72fb6a85%40googlegroups.com.

[shhh]

Thanks Sreevatsan for the info. Now I added 5 CPUs and 2048M for my pipeline with database sink but I still see 2 mappers getting created for job. Also I see these errors in task:

Exception from container-launch. Container id: container_e16_1580148013572_0010_01_000002 Exit code: 255 Container exited with a non-zero exit code 255. Last 4096 bytes of stderr : SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/hdfs_data/hadoop/yarn/local/usercache/yarn/appcache/application_1580148013572_0010/filecache/10/job.jar/lib/ch.qos.logback.logback-classic-1.0.9.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/hdfs_data/hadoop/yarn/local/filecache/12/mapreduce.tar.gz/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [ch.qos.logback.classic.util.ContextSelectorStaticBinder] SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/hdfs_data/hadoop/yarn/local/usercache/yarn/appcache/application_1580148013572_0010/filecache/10/job.jar/lib/ch.qos.logback.logback-classic-1.0.9.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/hdfs_data/hadoop/yarn/local/filecache/12/mapreduce.tar.gz/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [ch.qos.logback.classic.util.ContextSelectorStaticBinder] Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "client DomainSocketWatcher" Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "DistributedMapReduceTaskContextProvider STARTING-SendThread(host.com:2181)" Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "IPC Client (658107584) connection to host/host:8020 from yarn" Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "communication thread" Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "kafka-client-cleanup" Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "IPC Client (658107584) connection to /ip from job_1580148013572_0010" Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "MessagingMetricsCollectionService" Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "ShortCircuitCache_Cleaner" Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Thread for syncLogs" Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "metrics-meter-tick-thread-2" Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "main"

I really don't understand these errors. Ports in the error log are from zookeeper and name node. Is CDAP running out of connections? 

Thanks a lot.

On Monday, January 27, 2020 at 7:50:12 AM UTC-8, Sreevatsan Raman wrote:

You can click on Configure in Pipeline details page and set resources for mappers and reducers. Please see attached screenshot

To unsubscribe from this group and stop receiving emails from it, send an email to cdap...@googlegroups.com.

[shhh]

It's stabilized and working good now..

[sudheer kumar]

May i know how you were able to solve 

[Vitalii Tymchyshyn]

To confirm: Do you have problems with Database Sink or Database Source?