kafka consumption through mapreduce or yarn failing with connection refused error

vinay k

unread,

Nov 17, 2016, 7:12:35 AM11/17/16

to gobblin-users

kafka consumption through mapreduce or yarn failing with connection refused error on hortonworks secure cluster. The destination host is getting displayed as 0.0.0.0:8032. No where i am providing this details. The destination work dir has metrics and working directory created as well. Failing when creating the output directory i guess. The standard gobblin application.conf and reference.conf files with all the necessary parameters is placed. Anyone aware of this kind of behaviour please let me know the solution or work arounds.

WARN [NativeCodeLoader] Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

WARN [BlockReaderLocal] The short-circuit local reads feature cannot be used because libhadoop cannot be loaded.

WARN [ZKHelixAdmin] Root directory exists.Cleaning the root directory:/GobblinYarnVin

WARN [RetryInvocationHandler] Exception while invoking class org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getApplications. Not retrying because failovers (30) exceeded maximum allowed (30)

Exception in thread "main" java.net.ConnectException: Call From xxxxxxx.xxxxx.com/xx.xx.xx.xxx to 0.0.0.0:8032 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused

at sun.reflect.GeneratedConstructorAccessor5.newInstance(Unknown Source)

at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

at java.lang.reflect.Constructor.newInstance(Constructor.java:422)

at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:783)

at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:730)

at org.apache.hadoop.ipc.Client.call(Client.java:1410)

at org.apache.hadoop.ipc.Client.call(Client.java:1359)

at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)

at com.sun.proxy.$Proxy19.getApplications(Unknown Source)

at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getApplications(ApplicationClientProtocolPBClientImpl.java:197)

at sun.reflect.GeneratedMethodAccessor1.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:497)

at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)

at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)

at com.sun.proxy.$Proxy20.getApplications(Unknown Source)

at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getApplications(YarnClientImpl.java:285)

at gobblin.yarn.GobblinYarnAppLauncher.getReconnectableApplicationId(GobblinYarnAppLauncher.java:434)

at gobblin.yarn.GobblinYarnAppLauncher.getApplicationId(GobblinYarnAppLauncher.java:421)

at gobblin.yarn.GobblinYarnAppLauncher.launch(GobblinYarnAppLauncher.java:254)

at gobblin.yarn.GobblinYarnAppLauncher.main(GobblinYarnAppLauncher.java:794)

Caused by: java.net.ConnectException: Connection refused

at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)

at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)

at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)

at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529)

at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493)

at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:601)

at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:696)

at org.apache.hadoop.ipc.Client$Connection.access$2700(Client.java:367)

at org.apache.hadoop.ipc.Client.getConnection(Client.java:1458)

at org.apache.hadoop.ipc.Client.call(Client.java:1377)

... 15 more

bash-4.1$

Issac Buenrostro

unread,

Nov 17, 2016, 12:11:51 PM11/17/16

to vinay k, gobblin-users

Hi Vinay,

Can you try running your application in MR mode? YARN mode is considered experimental and has very limited support.

Issac

--
You received this message because you are subscribed to the Google Groups "gobblin-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gobblin-user...@googlegroups.com.
To post to this group, send email to gobbli...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gobblin-users/54bbef2b-b068-4a8e-ae1c-fc872e5a2435%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

vinay k

unread,

Nov 18, 2016, 3:09:06 AM11/18/16

to gobblin-users, vinay.vi...@gmail.com

Hi Issac,

Through MR mode also getting similar error. Getting an addition output task state path does not exist error along with destination host error.

2016-11-17 06:12:17 CST WARN [TaskStateCollectorService RUNNING] gobblin.runtime.TaskStateCollectorService 119 - Output task state path hdfs://xxxx.xxxxx.com:8020/user/xxxxxx/gobblin-work-dir/working/GobblinKafkaQuickStartvin/job_GobblinKafkaQuickStartvin_1479384009037/output/job_GobblinKafkaQuickStartvin_1479384009037 does not exist

2016-11-17 06:12:34 CST INFO [main] org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider 100 - Failing over to rm2

2016-11-17 06:13:13 CST INFO [main] org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider 100 - Failing over to rm1

2016-11-17 06:13:13 CST WARN [main] org.apache.hadoop.io.retry.RetryInvocationHandler 118 - Exception while invoking class org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getNewApplication. Not retrying because failovers (30) exceeded maximum allowed (30)

java.net.ConnectException: Call From xxxxx.xxxx.com/xx.xx.xxxx.xxx to 0.0.0.0:8032 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused

at sun.reflect.GeneratedConstructorAccessor10.newInstance(Unknown Source)

at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

at java.lang.reflect.Constructor.newInstance(Constructor.java:422)

at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:783)

at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:730)

at org.apache.hadoop.ipc.Client.call(Client.java:1410)

at org.apache.hadoop.ipc.Client.call(Client.java:1359)

at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)

at com.sun.proxy.$Proxy12.getNewApplication(Unknown Source)

at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getNewApplication(ApplicationClientProtocolPBClientImpl.java:167)

at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:497)

at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)

at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)

at com.sun.proxy.$Proxy13.getNewApplication(Unknown Source)

at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getNewApplication(YarnClientImpl.java:133)

at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.createApplication(YarnClientImpl.java:141)

at org.apache.hadoop.mapred.ResourceMgrDelegate.getNewJobID(ResourceMgrDelegate.java:188)

at org.apache.hadoop.mapred.YARNRunner.getNewJobID(YARNRunner.java:231)

at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:357)

at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1285)

at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1282)

at java.security.AccessController.doPrivileged(Native Method)

at javax.security.auth.Subject.doAs(Subject.java:422)

at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)

at org.apache.hadoop.mapreduce.Job.submit(Job.java:1282)

at gobblin.runtime.mapreduce.MRJobLauncher.runWorkUnits(MRJobLauncher.java:203)

at gobblin.runtime.AbstractJobLauncher.launchJob(AbstractJobLauncher.java:296)

at gobblin.runtime.mapreduce.CliMRJobLauncher.launchJob(CliMRJobLauncher.java:84)

at gobblin.runtime.mapreduce.CliMRJobLauncher.run(CliMRJobLauncher.java:61)

at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)

at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)

at gobblin.runtime.mapreduce.CliMRJobLauncher.main(CliMRJobLauncher.java:106)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:497)

at org.apache.hadoop.util.RunJar.main(RunJar.java:212)