Latest Weave on HDP 2.0 GA

[aperep...@hortonworks.com]

Hi guys,

Lots of progress made since last time I looked at weave, very happy to see it. I couldn't see any mention of HDP libs or repos, so I may contribute a pull request :)

However, before I do that I'm trying to follow the example and this code hangs consistently when trying to connect to a cluster:

WeaveController controller = runnerService.prepare(new EchoServer())

  .addLogHandler(new PrinterLogHandler(new PrintWriter(System.out)))

  .start();

I wonder if that's some wire incompatibility, but figured would ask here first. Thread dump below. I'm connecting to a ZK quorum at 'hdp20' address, which is configured in my /etc/hosts and works for any other access (e.g. external HBase client). Ideas?

Thanks!

Full thread dump Java HotSpot(TM) 64-Bit Server VM (20.65-b04-462 mixed mode):

"zk-client-EventThread" daemon prio=5 tid=7fb4030b0800 nid=0x113882000 waiting on condition [113881000]

   java.lang.Thread.State: WAITING (parking)

at sun.misc.Unsafe.park(Native Method)

- parking to wait for  <7f41374f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)

at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)

at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)

at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)

at java.lang.Thread.run(Thread.java:695)

"YarnWeaveRunnerService STARTING-EventThread" daemon prio=5 tid=7fb4037f7800 nid=0x11377f000 waiting on condition [11377e000]

   java.lang.Thread.State: WAITING (parking)

at sun.misc.Unsafe.park(Native Method)

- parking to wait for  <7f40a0080> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)

at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)

at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)

at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491)

"YarnWeaveRunnerService STARTING-SendThread(hdp20:2181)" daemon prio=5 tid=7fb4032b7000 nid=0x11349a000 runnable [113499000]

   java.lang.Thread.State: RUNNABLE

at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)

at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:136)

at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:69)

at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)

- locked <7f40ad6c8> (a sun.nio.ch.Util$2)

- locked <7f40ad6d8> (a java.util.Collections$UnmodifiableSet)

- locked <7f40ad680> (a sun.nio.ch.KQueueSelectorImpl)

at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)

at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:338)

at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)

"Monitor Ctrl-Break" daemon prio=5 tid=7fb401884800 nid=0x112d09000 runnable [112d08000]

   java.lang.Thread.State: RUNNABLE

at java.net.SocketInputStream.socketRead0(Native Method)

at java.net.SocketInputStream.read(SocketInputStream.java:129)

at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:264)

at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:306)

at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:158)

- locked <7f3cb3098> (a java.io.InputStreamReader)

at java.io.InputStreamReader.read(InputStreamReader.java:167)

at java.io.BufferedReader.fill(BufferedReader.java:136)

at java.io.BufferedReader.readLine(BufferedReader.java:299)

- locked <7f3cb3098> (a java.io.InputStreamReader)

at java.io.BufferedReader.readLine(BufferedReader.java:362)

at com.intellij.rt.execution.application.AppMain$1.run(AppMain.java:85)

at java.lang.Thread.run(Thread.java:695)

"Low Memory Detector" daemon prio=5 tid=7fb40180a000 nid=0x11295f000 runnable [00000000]

   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread1" daemon prio=9 tid=7fb3ff132800 nid=0x11285c000 waiting on condition [00000000]

   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread0" daemon prio=9 tid=7fb403800000 nid=0x112759000 waiting on condition [00000000]

   java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" daemon prio=9 tid=7fb403001000 nid=0x112656000 waiting on condition [00000000]

   java.lang.Thread.State: RUNNABLE

"Surrogate Locker Thread (Concurrent GC)" daemon prio=5 tid=7fb403000000 nid=0x112553000 waiting on condition [00000000]

   java.lang.Thread.State: RUNNABLE

"Finalizer" daemon prio=8 tid=7fb401809000 nid=0x112348000 in Object.wait() [112347000]

   java.lang.Thread.State: WAITING (on object monitor)

at java.lang.Object.wait(Native Method)

- waiting on <7f44fcf10> (a java.lang.ref.ReferenceQueue$Lock)

at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)

- locked <7f44fcf10> (a java.lang.ref.ReferenceQueue$Lock)

at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)

at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:171)

"Reference Handler" daemon prio=10 tid=7fb401808800 nid=0x112245000 in Object.wait() [112244000]

   java.lang.Thread.State: WAITING (on object monitor)

at java.lang.Object.wait(Native Method)

- waiting on <7f44f5ce0> (a java.lang.ref.Reference$Lock)

at java.lang.Object.wait(Object.java:485)

at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)

- locked <7f44f5ce0> (a java.lang.ref.Reference$Lock)

"main" prio=5 tid=7fb3fe800800 nid=0x10a3d6000 waiting on condition [10a3d3000]

   java.lang.Thread.State: TIMED_WAITING (sleeping)

at java.lang.Thread.sleep(Native Method)

at org.apache.hadoop.ipc.Client$Connection.handleConnectionFailure(Client.java:778)

at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:566)

- locked <7f3b193c8> (a org.apache.hadoop.ipc.Client$Connection)

at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:642)

- locked <7f3b193c8> (a org.apache.hadoop.ipc.Client$Connection)

at org.apache.hadoop.ipc.Client$Connection.access$2600(Client.java:314)

at org.apache.hadoop.ipc.Client.getConnection(Client.java:1399)

at org.apache.hadoop.ipc.Client.call(Client.java:1318)

at org.apache.hadoop.ipc.Client.call(Client.java:1300)

at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)

at com.sun.proxy.$Proxy8.getNewApplication(Unknown Source)

at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getNewApplication(ApplicationClientProtocolPBClientImpl.java:167)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

at java.lang.reflect.Method.invoke(Method.java:597)

at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)

at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)

at com.sun.proxy.$Proxy9.getNewApplication(Unknown Source)

at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getNewApplication(YarnClientImpl.java:127)

at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.createApplication(YarnClientImpl.java:135)

at com.continuuity.weave.internal.yarn.Hadoop21YarnAppClient.createLauncher(Hadoop21YarnAppClient.java:59)

at com.continuuity.weave.internal.yarn.Hadoop21YarnAppClient.createLauncher(Hadoop21YarnAppClient.java:129)

at com.continuuity.weave.yarn.YarnWeavePreparer.start(YarnWeavePreparer.java:222)

at com.continuuity.weave.yarn.EchoServerTestRun.testEchoServer(EchoServerTestRun.java:61)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

at java.lang.reflect.Method.invoke(Method.java:597)

at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)

at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)

at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)

at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)

at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)

at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)

at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)

at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)

at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)

at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)

at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)

at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)

at org.junit.runners.ParentRunner.run(ParentRunner.java:309)

at org.junit.runner.JUnitCore.run(JUnitCore.java:160)

at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:77)

at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:195)

at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:63)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

at java.lang.reflect.Method.invoke(Method.java:597)

at com.intellij.rt.execution.application.AppMain.main(AppMain.java:120)

"VM Thread" prio=9 tid=7fb401804000 nid=0x112142000 runnable 

"Gang worker#0 (Parallel GC Threads)" prio=9 tid=7fb3ff000000 nid=0x10d7d8000 runnable 

"Gang worker#1 (Parallel GC Threads)" prio=9 tid=7fb3ff001000 nid=0x10d8db000 runnable 

"Gang worker#2 (Parallel GC Threads)" prio=9 tid=7fb3ff001800 nid=0x10d9de000 runnable 

"Gang worker#3 (Parallel GC Threads)" prio=9 tid=7fb3ff002000 nid=0x10dae1000 runnable 

"Gang worker#4 (Parallel GC Threads)" prio=9 tid=7fb3ff002800 nid=0x10dbe4000 runnable 

"Gang worker#5 (Parallel GC Threads)" prio=9 tid=7fb3ff003800 nid=0x10dce7000 runnable 

"Gang worker#6 (Parallel GC Threads)" prio=9 tid=7fb3ff004000 nid=0x10ddea000 runnable 

"Gang worker#7 (Parallel GC Threads)" prio=9 tid=7fb3ff004800 nid=0x10deed000 runnable 

"Concurrent Mark-Sweep GC Thread" prio=9 tid=7fb3ff0e2000 nid=0x111db9000 runnable 

"Gang worker#0 (Parallel CMS Threads)" prio=9 tid=7fb3ff0e0800 nid=0x1113b3000 runnable 

"Gang worker#1 (Parallel CMS Threads)" prio=9 tid=7fb3ff0e1800 nid=0x1114b6000 runnable 

"VM Periodic Task Thread" prio=10 tid=7fb40181b800 nid=0x112a62000 waiting on condition 

"Exception Catcher Thread" prio=10 tid=7fb3fe801800 nid=0x10a601000 runnable 

JNI global references: 1126

Heap

 par new generation   total 19136K, used 13994K [7f3000000, 7f44c0000, 7f44c0000)

  eden space 17024K,  77% used [7f3000000, 7f3cde560, 7f40a0000)

  from space 2112K,  38% used [7f40a0000, 7f416c2d8, 7f42b0000)

  to   space 2112K,   0% used [7f42b0000, 7f42b0000, 7f44c0000)

 concurrent mark-sweep generation total 63872K, used 1051K [7f44c0000, 7f8320000, 7fae00000)

 concurrent-mark-sweep perm gen total 21248K, used 18948K [7fae00000, 7fc2c0000, 800000000)

CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.

[Andrew Perepelytsya]

Answering my own question :) I realized that Weave is connecting directly to RM before talking to ZK, after throwing in a minimal yarn-site.xml things moved forward:

<configuration>

    <property>

        <name>yarn.resourcemanager.address</name>

        <value>hdp20:8050</value>

    </property>

</configuration>

[Nitin Motgi]

Thank you.

A patch for support HDP GA would be great. 

Seems like u figured out the solution to the problem you were having :-)

--
You received this message because you are subscribed to the Google Groups "weave-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weave-user+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Terence Yim]

Great to know that you made it. The latest Weave should be compatible with Hadoop Yarn 2.2.0, which is the one used by HDP GA. If you find any further issues, please feel free to discuss or submit a patch.

Thanks,

Terence

[aperep...@hortonworks.com]

The last bit I had to put in was a core-site.xml like:

<configuration>

    <property>

        <name>fs.default.name</name>

        <value>hdfs://hdp20:8020</value>

    </property>

</configuration>

and create the 777 hdfs dir /weave

Weave deployed the app after that fine. I'll be figuring out permissions next, my workstation submits as a user which doesn't exist in hdfs :) 

[Terence Yim]

If you are not running with secure Hadoop, you could workaround with the HDFS user issue by supplying a LocationFactory to the YarnWeaveRunnerService like this:

String hdfsUser = "yarn";

Configuration conf = new YarnConfiguration();

LocationFactory lf = new HDFSLocationFactory(FileSystem.get(FileSystem.getDefaultUri(conf), conf, hdfsUser), "/weave");

WeaveRunnerService runner = new YarnWeaveRunnerService(conf, zkConnectStr, lf);

That will make Weave always access HDFS as "yarn" user. If you are running secure HDFS with Kerberos, then you don't need that as Weave will figure out the kerberos login and grab the right delegation token.

Terence