Configuring Mesos with Tachyon Framework

[Renan Capaverde]

Hello,

From http://tachyon-project.org/documentation/v0.8.2/Running-Tachyon-on-EC2-Mesos.html:

"Now you can launch the Mesos cluster and Tachyon Mesos Framework (...)"

I already have Mesos up and running with spark and I want to know how can I configure it to use Tachyon Mesos Framework to deploy Tachyon Master and Slaves.

I didnt find anything clear about it on google and on the discussions, sorry if it is a repeated question.

Cheers,

[Bin Fan]

Hi Renan,

I am not really familiar with mesos, but hope this may help:

since you are using Tachyon v0.8, check script

tachyon/integration/bin/tachyon-mesos.sh

of course you need to compile the tachyon-mesos integration first.

To do this, run "mvn clean install -Pmesos" which will compile this integration.

- Bin

[cc]

Hey,

These scripts contain all the steps to deploy Tachyon on Mesos described in http://tachyon-project.org/documentation/v0.8.2/Running-Tachyon-on-EC2-Mesos.html:

https://github.com/amplab/tachyon/blob/master/deploy/vagrant/provision/start_tachyon_on_mesos.yml

https://github.com/amplab/tachyon/tree/master/deploy/vagrant/provision/roles/mesos

The scripts are yaml files and bash scripts, there is NO NEED to understand ansible or vagrant first. 

Best,

Cheng

在 2016年1月27日星期三 UTC+8上午5:55:59，Renan Capaverde写道：

[Renan Capaverde]

Hi Cheng,

I am running into some issues running Tachyon on EC2 with Mesos, Spark and S3 as the Underfs.

1) File "deploy/vagrant/provision/roles/ufs_s3/files/config_spark.sh" was missing. I am using a file that contains only "#!/usr/bin/env bash" and it works.

2) I tried to set s3 as the underfs by setting deploy/vagrant/provision/roles/tachyon/tasks/start_mesos_framework.yml the following vars under "environment":

    TACHYON_UNDERFS_ADDRESS: "s3n://nu-spark/double-entry"

    S3_KEY: "my_aws_secret_key"

    S3_ID: "my_aws_access_key" 

And I got the following errors:
2.a) for TachyonMaster task from mesos:

2016-01-29 19:43:31,920 ERROR MASTER_LOGGER (MetricsConfig.java:loadConfigFile) - Error loading metrics configuration file.

2016-01-29 19:43:31,923 ERROR MASTER_LOGGER (TachyonMaster.java:main) - Uncaught exception terminating Master

java.lang.IllegalArgumentException: All eligible Under File Systems were unable to create an instance for the given path: s3n://nu-spark/double-entry

java.lang.RuntimeException: Invalid configuration key fs.s3n.awsAccessKeyId.

at tachyon.underfs.UnderFileSystemRegistry.create(UnderFileSystemRegistry.java:132)

at tachyon.underfs.UnderFileSystem.get(UnderFileSystem.java:100)

at tachyon.underfs.UnderFileSystem.get(UnderFileSystem.java:83)

at tachyon.master.TachyonMaster.connectToUFS(TachyonMaster.java:412)

at tachyon.master.TachyonMaster.startMasters(TachyonMaster.java:280)

at tachyon.master.TachyonMaster.start(TachyonMaster.java:261)

at tachyon.master.TachyonMaster.main(TachyonMaster.java:64)

at tachyon.mesos.TachyonMasterExecutor$1.run(TachyonMasterExecutor.java:71)

2.b) For TachyonWorker task on mesos:

2016-01-29 19:33:32,310 ERROR WORKER_LOGGER (ClientBase.java:connect) - Failed to connect (29) to BlockMaster master @ TachyonMaster/10.187.83.93:19998 : java.net.ConnectException: Connection refused

2016-01-29 19:33:32,311 ERROR WORKER_LOGGER (TachyonWorker.java:main) - Failed to initialize the block worker, exiting.

java.io.IOException: Failed to connect to BlockMaster master @ TachyonMaster/10.187.83.93:19998 after 29 attempts

at tachyon.ClientBase.connect(ClientBase.java:134)

at tachyon.client.WorkerBlockMasterClient.getId(WorkerBlockMasterClient.java:101)

at tachyon.worker.WorkerIdRegistry.registerWithBlockMaster(WorkerIdRegistry.java:59)

at tachyon.worker.block.BlockWorker.<init>(BlockWorker.java:200)

at tachyon.worker.TachyonWorker.main(TachyonWorker.java:42)

at tachyon.mesos.TachyonWorkerExecutor$1.run(TachyonWorkerExecutor.java:71)


Am I doing something wrong? Is this supposed to work? Why couldnt I set this config?

Cheers,

[Renan Capaverde]

It looks like mesos framework is downloading an older version of Tachyon. As default, It points to  "//s3.amazonaws.com/tachyon-mesos/tachyon.tar.gz" that contains Tachyon version 0.8.0. 

I think a nice idea is to have a config "tachyon.integration.mesos.tachyon.path" pointing to "tachyon-x.y.z.tar.gz" where x.y.z is the current version of the release. And also, having all released versions updated in S3 with that format.

[cc]

Could you help open a ticket at tachyon.atlassian.net to report the outdated downloading URL? Thanks.

For a temporary hack, you could modify that URL manually to the correct version of Tachyon at http://tachyon-project.org/downloads/files/0.8.2/

在 2016年2月2日星期二 UTC+8上午1:11:07，Renan Capaverde写道：

[Renan Capaverde]

For the record: https://tachyon.atlassian.net/projects/TACHYON/issues/TACHYON-1698

I compiled Tachyon myself, uploaded to s3 and set path to point to that location.

Still, I got the same problem with java.lang.RuntimeException: Invalid configuration key fs.s3n.awsAccessKeyId.

Tachyon Mesos Framework does not send the configured accesskey/secretkey on JAVA_OPTS to the running tasks of Tachyon. As a workaround, I put the accesskeys and secretkeys into my compiled tachyon.tar.gz (on tachyon-env.sh) and then it worked.

I think this also a bug and I filed another JIRA for that: https://tachyon.atlassian.net/projects/TACHYON/issues/TACHYON-1699

Cheers,