kafka-flink example

[geeka...@gmail.com]

Hi,
we are trying to execute a flink job example https://github.com/logicalclocks/hops-examples/blob/master/flink/src/main/java/io/hops/examples/flink/kafka/StreamingExample.java
on HopsWork 0.8 but we got a permission denied exception.
We used the karamel at
https://github.com/logicalclocks/karamel-chef/blob/master/cluster-defns/3.hopsworks-0.8.yml
merging flink information with the file at https://github.com/karamelchef/karamel/blob/master/karamel-core/src/test/resources/se/kth/karamel/client/model/test-definitions/flink_gce.yml
Could you help me to understand how to deal with the kafka topic?
thank you

Log Entry Detail

Timestamp

6-feb-2019 16:29:18.098

Log Level

WARNING

Logger

javax.enterprise.ejb.container

Name-Value Pairs

{_ThreadName=__ejb-thread-pool6}

Record Number

164

Message ID

Complete Message

javax.ejb.EJBException

at com.sun.ejb.containers.EJBContainerTransactionManager.processSystemException(EJBContainerTransactionManager.java:750)

at com.sun.ejb.containers.EJBContainerTransactionManager.checkExceptionNoTx(EJBContainerTransactionManager.java:640)

at com.sun.ejb.containers.EJBContainerTransactionManager.postInvokeTx(EJBContainerTransactionManager.java:482)

at com.sun.ejb.containers.BaseContainer.postInvokeTx(BaseContainer.java:4616)

at com.sun.ejb.containers.BaseContainer.postInvoke(BaseContainer.java:2089)

at com.sun.ejb.containers.EjbAsyncTask.call(EjbAsyncTask.java:114)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

at java.lang.Thread.run(Thread.java:748)

Caused by: java.lang.RuntimeException: Couldn't deploy Yarn cluster

at io.hops.hopsworks.common.jobs.flink.AbstractYarnClusterDescriptor.deploy(AbstractYarnClusterDescriptor.java:491)

at io.hops.hopsworks.common.jobs.yarn.YarnRunner.startAppMaster(YarnRunner.java:295)

at io.hops.hopsworks.common.jobs.yarn.YarnJob.startApplicationMaster(YarnJob.java:183)

at io.hops.hopsworks.common.jobs.yarn.YarnJob.runJob(YarnJob.java:288)

at io.hops.hopsworks.common.jobs.execution.HopsJob$1.run(HopsJob.java:180)

at io.hops.hopsworks.common.jobs.execution.HopsJob$1.run(HopsJob.java:159)

at java.security.AccessController.doPrivileged(Native Method)

at javax.security.auth.Subject.doAs(Subject.java:422)

at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1955)

at io.hops.hopsworks.common.jobs.execution.HopsJob.execute(HopsJob.java:159)

at io.hops.hopsworks.common.jobs.AsynchronousJobExecutor.startExecution(AsynchronousJobExecutor.java:97)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at org.glassfish.ejb.security.application.EJBSecurityManager.runMethod(EJBSecurityManager.java:1081)

at org.glassfish.ejb.security.application.EJBSecurityManager.invoke(EJBSecurityManager.java:1153)

at com.sun.ejb.containers.BaseContainer.invokeBeanMethod(BaseContainer.java:4836)

at com.sun.ejb.EjbInvocation.invokeBeanMethod(EjbInvocation.java:656)

at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:836)

at com.sun.ejb.EjbInvocation.proceed(EjbInvocation.java:608)

at org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:73)

at org.jboss.weld.ejb.SessionBeanInterceptor.aroundInvoke(SessionBeanInterceptor.java:52)

at sun.reflect.GeneratedMethodAccessor113.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at com.sun.ejb.containers.interceptors.AroundInvokeInterceptor.intercept(InterceptorManager.java:895)

at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:835)

at com.sun.ejb.EjbInvocation.proceed(EjbInvocation.java:608)

at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.doCall(SystemInterceptorProxy.java:163)

at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.aroundInvoke(SystemInterceptorProxy.java:140)

at sun.reflect.GeneratedMethodAccessor112.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at com.sun.ejb.containers.interceptors.AroundInvokeInterceptor.intercept(InterceptorManager.java:895)

at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:835)

at com.sun.ejb.containers.interceptors.InterceptorManager.intercept(InterceptorManager.java:374)

at com.sun.ejb.containers.BaseContainer.__intercept(BaseContainer.java:4808)

at com.sun.ejb.containers.BaseContainer.intercept(BaseContainer.java:4796)

at com.sun.ejb.containers.EjbAsyncTask.call(EjbAsyncTask.java:101)

... 4 more

Caused by: org.apache.hadoop.security.AccessControlException: Permission denied: user=TestJohnDoe__meb10000, access=WRITE, inode="user":hdfs:hadoop:drwxrwxr-t

at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:512)

at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:329)

at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:305)

at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:218)

at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6085)

at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6070)

at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkAncestorAccess(FSNamesystem.java:6046)

at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:2356)

at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2321)

at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.access$2700(FSNamesystem.java:302)

at org.apache.hadoop.hdfs.server.namenode.FSNamesystem$15.performTask(FSNamesystem.java:2281)

at io.hops.transaction.handler.TransactionalRequestHandler.execute(TransactionalRequestHandler.java:99)

at io.hops.transaction.handler.HopsTransactionalRequestHandler.execute(HopsTransactionalRequestHandler.java:50)

at io.hops.transaction.handler.RequestHandler.handle(RequestHandler.java:65)

at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2292)

at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:464)

at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:413)

at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)

at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:447)

at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:996)

at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:850)

at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:793)

at java.security.AccessController.doPrivileged(Native Method)

at javax.security.auth.Subject.doAs(Subject.java:422)

at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1955)

at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2775)

at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)

at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

at java.lang.reflect.Constructor.newInstance(Constructor.java:423)

at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121)

at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:88)

at org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1811)

at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1540)

at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1464)

at org.apache.hadoop.hdfs.DistributedFileSystem$6.doCall(DistributedFileSystem.java:435)

at org.apache.hadoop.hdfs.DistributedFileSystem$6.doCall(DistributedFileSystem.java:431)

at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)

at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:445)

at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:374)

at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:915)

at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:896)

at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:793)

at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:368)

at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:341)

at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1973)

at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1941)

at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1906)

at org.apache.flink.yarn.Utils.setupLocalResource(Utils.java:129)

at io.hops.hopsworks.common.jobs.flink.AbstractYarnClusterDescriptor.deployInternal(AbstractYarnClusterDescriptor.java:787)

at io.hops.hopsworks.common.jobs.flink.AbstractYarnClusterDescriptor.deploy(AbstractYarnClusterDescriptor.java:484)

... 43 more

Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): Permission denied: user=TestJohnDoe__meb10000, access=WRITE, inode="user":hdfs:hadoop:drwxrwxr-t

at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:512)

at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:329)

at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:305)

at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:218)

at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6085)

at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6070)

at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkAncestorAccess(FSNamesystem.java:6046)

at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:2356)

at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2321)

at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.access$2700(FSNamesystem.java:302)

at org.apache.hadoop.hdfs.server.namenode.FSNamesystem$15.performTask(FSNamesystem.java:2281)

at io.hops.transaction.handler.TransactionalRequestHandler.execute(TransactionalRequestHandler.java:99)

at io.hops.transaction.handler.HopsTransactionalRequestHandler.execute(HopsTransactionalRequestHandler.java:50)

at io.hops.transaction.handler.RequestHandler.handle(RequestHandler.java:65)

at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2292)

at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:464)

at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:413)

at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)

at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:447)

at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:996)

at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:850)

at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:793)

at java.security.AccessController.doPrivileged(Native Method)

at javax.security.auth.Subject.doAs(Subject.java:422)

at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1955)

at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2775)

at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1535)

at org.apache.hadoop.ipc.Client.call(Client.java:1481)

at org.apache.hadoop.ipc.Client.call(Client.java:1391)

at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:227)

at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)

at com.sun.proxy.$Proxy780.create(Unknown Source)

at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create(ClientNamenodeProtocolTranslatorPB.java:310)

at sun.reflect.GeneratedMethodAccessor628.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:409)

at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:163)

at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:155)

at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)

at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:346)

at com.sun.proxy.$Proxy781.create(Unknown Source)

at org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1807)

... 61 more

[Theofilos Kakantousis]

Hi,

Flink will be upgraded to its latest version in Hopsworks 0.10.x. In Hopsworks 0.8.x is in version 1.1.3 which means there are a few issues that won't be fixed till the upgrade.

To run it either you need to set the flink checkpoint dir to a directory under a project's dataset or add writable permissions to /user in hdfs. 

[geeka...@gmail.com]

Thank you Theofilos, with your suggestion I achieve to go further, I set permissions rwxrwxrwx on the hdfs folder /user, now the project user is able to write in the path but the job follow failing with UNDEFINED state.
this is the

da loro suggerito abbiamo impostato i permessi della directory hdfs /user a rwxrwxrwx ed ora l’utenza applicativa del project riesce a scrivere e creare gli oggetti sotto quel path, ma il job continua ad andare in errore con stato UNDEFINED,
this is the error stack trace
flink version: 1.1.3 and hopswork 0.10-snapshot
thank you so much for all the support


Log Entry Detail

Timestamp
8-feb-2019 10:39:15.461

Log Level
WARNING
Logger
javax.enterprise.ejb.container
Name-Value Pairs

{_ThreadName=__ejb-thread-pool12}
Record Number
1440

Caused by: io.hops.hopsworks.common.jobs.flink.AbstractYarnClusterDescriptor$YarnDeploymentException: The YARN application unexpectedly switched to state FAILED during deployment.
Diagnostics from YARN: Application application_1549618157918_0001 failed 1 times (global limit =2; local limit is =1) due to AM Container for appattempt_1549618157918_0001_000001 exited with exitCode: 31
Failing this attempt.Diagnostics: Exception from container-launch.
Container id: container_e02_1549618157918_0001_01_000001
Exit code: 31
Stack trace: org.apache.hadoop.yarn.server.nodemanager.containermanager.runtime.ContainerExecutionException: Launch container failed
at org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DefaultLinuxContainerRuntime.launchContainer(DefaultLinuxContainerRuntime.java:110)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DelegatingLinuxContainerRuntime.launchContainer(DelegatingLinuxContainerRuntime.java:90)
at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:415)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:306)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:84)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

Shell output: main : command provided 1
main : run as user is yarnapp
main : requested yarn user is rEiHWfZfUSZXqpp543WgsOuj3X7cCun54ZXiqZtigfU
Getting exit code file...
Creating script paths...
Writing pid file...
Writing to tmp file /opt/hops/hopsdata/tmp/nm-local-dir/nmPrivate/application_1549618157918_0001/container_e02_1549618157918_0001_01_000001/container_e02_1549618157918_0001_01_000001.pid.tmp
Writing to cgroup task files...
Creating local dirs...
Launching container...
Getting exit code file...
Creating script paths...


Container exited with a non-zero exit code 31
For more detailed output, check the application tracking page: http://hopmeazldev01:8088/cluster/app/application_1549618157918_0001 Then click on links to logs of each attempt.
. Failing the application.
If log aggregation is enabled on your cluster, use this command to further investigate the issue:
yarn logs -applicationId application_1549618157918_0001
at io.hops.hopsworks.common.jobs.flink.AbstractYarnClusterDescriptor.deployInternal(AbstractYarnClusterDescriptor.java:933)

at io.hops.hopsworks.common.jobs.flink.AbstractYarnClusterDescriptor.deploy(AbstractYarnClusterDescriptor.java:484)
... 43 more