Run Weave with YARN cluster

[Murariu Fabian]

Hi,

I'm trying to get Weave to run on a YARN cluster and I'm stumbling with an error. I can't find any documentation on what configuration I need to YarnConfiguration to make it run. is /weave/EchoServer supposed to be in hdfs or on the local file system? If it were to be local then /tmp/weave/.. would be a better location

Exception in thread " STARTING" java.lang.RuntimeException: java.io.IOException: Mkdirs failed to create file:/weave/EchoServer/e01bf617-ad1d-4a55-b048-c1b52b89895f
    at com.google.common.base.Throwables.propagate(Throwables.java:160)
    at com.continuuity.weave.yarn.YarnWeaveController.doStartUp(YarnWeaveController.java:119)
    at com.continuuity.weave.internal.AbstractZKServiceController.startUp(AbstractZKServiceController.java:96)
    at com.continuuity.weave.internal.AbstractExecutionServiceController$ServiceDelegate.startUp(AbstractExecutionServiceController.java:106)
    at com.google.common.util.concurrent.AbstractIdleService$1$1.run(AbstractIdleService.java:43)
    at java.lang.Thread.run(Thread.java:744)
Caused by: java.io.IOException: Mkdirs failed to create file:/weave/EchoServer/e01bf617-ad1d-4a55-b048-c1b52b89895f
    at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:438)
    at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:424)
    at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:905)
    at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:886)
    at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:783)
    at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:772)
    at com.continuuity.weave.filesystem.HDFSLocation.getOutputStream(HDFSLocation.java:75)
    at com.continuuity.weave.internal.ApplicationBundler.createBundle(ApplicationBundler.java:135)
    at com.continuuity.weave.internal.ApplicationBundler.createBundle(ApplicationBundler.java:96)
    at com.continuuity.weave.yarn.YarnWeavePreparer.createAppMasterJar(YarnWeavePreparer.java:341)
    at com.continuuity.weave.yarn.YarnWeavePreparer.access$300(YarnWeavePreparer.java:103)
    at com.continuuity.weave.yarn.YarnWeavePreparer$1.call(YarnWeavePreparer.java:238)
    at com.continuuity.weave.yarn.YarnWeavePreparer$1.call(YarnWeavePreparer.java:226)
    at com.continuuity.weave.yarn.YarnWeaveController.doStartUp(YarnWeaveController.java:85)
    ... 4 more

Thanks

This is my code

        YarnConfiguration configuration = new YarnConfiguration(new Configuration());
        configuration.set(RM_ADDRESS, "hadoop2:8050");
        configuration.set(RM_SCHEDULER_ADDRESS, "hadoop2:8030");
        configuration.set(RM_WEBAPP_ADDRESS, "hadoop2:8088");
        YarnWeaveRunnerService runnerService = new YarnWeaveRunnerService(configuration, "hadoop1:2181");

        runnerService.startAndWait();

        WeaveController controller = runnerService.prepare(new EchoServer(),
                ResourceSpecification.Builder.with()
                        .setVirtualCores(1)
                        .setMemory(1, ResourceSpecification.SizeUnit.GIGA)
                        .setInstances(2)
                        .build())
                .addLogHandler(new PrinterLogHandler(new PrintWriter(System.out, true)))
                .withApplicationArguments("echo")
                .withArguments("EchoServer", "echo2")
                .start();

        final CountDownLatch running = new CountDownLatch(1);

        controller.addListener(new ServiceListenerAdapter() {
            @Override
            public void running() {
                running.countDown();
            }
        }, Threads.SAME_THREAD_EXECUTOR);

        Assert.assertTrue(running.await(30, TimeUnit.SECONDS));

[Murariu Fabian]

This was user error, obviously I needed YarnConfiguration to point to my cluster by adding all necessary xml files.