Deploying sparrow using ec2 scripts

[gkumar7]

To launch the cluster (before deploy), I use the following:

./ec2-exp.sh launch sparrowcluster -k sparrowkey -i ~/.ssh/aws/sparrowkey.pem -t m3.medium --spot-price=0.025 -z us-east-1a -f 1 -b 1

then, during the deploy step

./ec2-exp.sh deploy sparrowcluster -i ~/.ssh/aws/sparrowkey.pem -w 100 -l 10

there seems to be a few unresolved dependencies:

  - com.eed3si9n#sbt-assembly;0.8.5

  - org.spark-project#spark-core_2.9.3;0.8.0-SNAPSHOT

Any assistance would be greatly appreciated.

[Kay Ousterhout]

Sometimes that can happen due to transient connectivity issues; have you tried just running the command again?

-Kay

--
You received this message because you are subscribed to the Google Groups "Sparrow Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sparrow-scheduler...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[gkumar7]

Hi Kay,

Thanks for the reply! Yes, I have tried to run the command again, but the issue persists.

[gkumar7]

Actually, the issue was because of the out-of-date url used to retrieve the sbt plugins. I replaced http://scalasbt.artifactoryonline.com/scalasbt/sbt-plugin-releases with https://repo.scala-sbt.org/scalasbt/sbt-plugin-releases/ and now it works fine.

[Henrique Grando]

Hi guys, 

Even with this latest fix I wasn't able to get the ec2 script to work. During the deploy stage I get the compilation errors attached. I've tried looking into the Spark and Java documentations, but I couldn't find anything useful.

I would really appreciate any help on this matter.

[Kay Ousterhout]

It sounds like you're using the wrong version of Spark, so Shark isn't compiling correctly. What Spark version are you using? Are you compiling Shark based on the special Sparrow version of Spark?

[Lou]

Hi Henrique,

Should Sparrow only work (as a scheduling plugin) with a forked version of Spark, i.e. 0,7? AFAIR, the package URL of the Hadoop core libs in use should be accessible, too. And Shark has been subsumed by Spark SQL since the second half of 2014 (?).

If you are (highly) interested in giving such de-centralized bit a try with a later version of Spark, other than 0,7, there are two good references (among others) one can have a look, i.e. 

1. A good tutorial on the task scheduler of Spark, via https://jaceklaskowski.gitbooks.io/mastering-apache-spark/content/spark-taskscheduler.html 

2. A tutorial on how Spark works using a few illustrative examples, via http://nbviewer.jupyter.org/github/amplab/datascience-sp14/blob/master/lab8/lab8-withsolutions.ipynb

Last but not least, Memcached (memcached.org) seems to be another system of interest to play with (the latency-sensitive workloads), and the HCloud paper from the Stanford Platform Lab is worth reading.

Mvh,

Lou

[张晓达]

Hello, gkumar7,

I meet the same problem when deploying Sparrow. So I want to know where should I replace the URL with the other one.

Thanks,

Xiaoda

在 2016年3月7日星期一 UTC+1下午8:24:40，gkumar7写道：

[Henrique Grando]

Hi, Kay.

Thanks for your answer. I was just running the script for deploy (sparrow-master/deploy/ec2/ec2-exp.sh). It already installs the correct version of Spark, doesn't it?

To unsubscribe from this group and stop receiving emails from it, send an email to sparrow-scheduler-users+unsub...@googlegroups.com.

[Henrique Grando]

Hi Lou, 

Thanks for your help. I'm trying to setup Sparrow with Spark just as a scheduling plugin for performance tests purpose. I intended to explore the possibility to develop a distributed scheduler on task level for Spark (since Sparrow can only send jobs to different schedulers). For this purpose I think the older version of Spark might be enough. 

Thanks,

Henrique

[Henrique Grando]

Hi Xiaoda, 

gkumar7 has a forked repository of the Sparrow with the necessary fix already made: https://github.com/spark-hadoop-profiling/sparrow

Hope this helps,

Henrique

[Lou]

Hi Henrique,

>I intended to explore the possibility to develop a distributed scheduler on task level for Spark (since Sparrow can only send jobs to different schedulers). 

Sounds reasonable, and stress that when Sparrow works with Spark, it is a task-level scheduler, since it has replaced the original task scheduler of Spark in dispatching short-lived jobs to Spark worker nodes. More interestingly, I think when it comes to performance analysis of Spark, first and foremost, not only the root cause of scheduling bottleneck of Spark is necessarily to be identified (by answering an/the important question raised by one's intuition), but also the underlying classical or novel (domain-specific) performance analysis model should be defined (as in different models give us different answers, and it is just a belief). Next, any further improvement from "not-working to working" will make (some) sense. And I agree, by taking an easy path, Spark 0,7 works at its utmost.

Last but not least, the performance of Spark, comparing Hadoop 1.x, is quite impressive as measured and cited in an own reference, and the eco-system centering around Spark looks great (among others). Maybe when workloads are not solely data-intensive anymore, and the super-fast flash is getting more and more popular, the de-centralized/distributed bit would/should fly higher and higher.  

Bien à toi

Lou

[gkumar7]

Hi Xiaoda,

The url to replace can be seen here: https://github.com/spark-hadoop-profiling/spark/commit/05fe6fadfa11ff10f640f2d468f98e46b3ffe6ea (must be updated in the forked version of spark).

On Sunday, July 3, 2016 at 4:18:13 AM UTC-5, 张晓达 wrote:

[Bishwajit Saha]

To launch the cluster, I have used the following command:
./ec2-exp.sh launch sparrowcluster -k thesis_1 -i thesis_1.pem -t m3.medium  -z us-east-1b -f 1 -b 1

It works well for me.

 during deploy i am using the following command:
./ec2-exp.sh deploy sparrowcluster -i sparrow.pem -w 100 -l 10
But this cpmmand is not working for me. It gives following error:

ERROR: Could not find full cluster named sparrowcluster: fe=[] be=[]

Any assistance would be greatly appreciated. Thanks in advance.

[Kay Ousterhout]

I'd suggest checking the AWS console to make sure that you have instances running in the security group sparrowcluster.  In your first two messages, it looks like you used a different cluster name to launch and deploy, which won't work.

--

[Bishwajit Saha]

When i am going to build your sparrow integrated spark in my computer then the errors like below are found.'

 

compiler-interface' not yet compiled for Scala 2.9.3. Compiling...

error: error while loading CharSequence, class file '/usr/lib/jvm/java-8-oracle/jre/lib/rt.jar(java/lang/CharSequence.class)' is broken

(bad constant pool tag 18 at byte 10) 

error: error while loading AnnotatedElement, class file '/usr/lib/jvm/java-8-oracle/jre/lib/rt.jar(java/lang/reflect/AnnotatedElement.class)' is broken

(bad constant pool tag 18 at byte 76)

error: error while loading Arrays, class file '/usr/lib/jvm/java-8-oracle/jre/lib/rt.jar(java/util/Arrays.class)' is broken

(bad constant pool tag 18 at byte 765)

error: error while loading Comparator, class file '/usr/lib/jvm/java-8-oracle/jre/lib/rt.jar(java/util/Comparator.class)' is broken

(bad constant pool tag 18 at byte 20)

/tmp/sbt_cc378550/API.scala:384:

 

error: java.util.Comparator does not take type parameters

private[this] val sortClasses = new Comparator[Symbol] {

                                            ^

5 errors found

Any assistance would be greatly appreciated.

Thank You in advance. 

[Kay Ousterhout]

This looks like an issue with your version of Java -- have you been able to compile other things (not Sparrow, e.g., a simple Java application) successfully against it?

--

[Bishwajit Saha]

Yes i have compiled simple java application in spark. And there seems no problem to build spark in my computer. Then why this errors are being showed at the time of building sparrow integrated spark?

Thanks in advance.

[Bishwajit Saha]

I am using java version like below.

 

 java version "1.8.0_131"

Java(TM) SE Runtime Environment (build 1.8.0_131-b11)

Java HotSpot(TM) 64-Bit Server VM (build 25.131-b11, mixed mode)

 and scala version like below.

Scala code runner version 2.11.6 -- Copyright 2002-2013, LAMP/EPFL

I have also tried to build this with scala version 2.9.3.

But in all cases i am getting errors like below.

[error] error while loading CharSequence, class file '/usr/lib/jvm/java-8-oracle/jre/lib/rt.jar(java/lang/CharSequence.class)' is broken

[error] (bad constant pool tag 18 at byte 10)

[error] error while loading Comparator, class file '/usr/lib/jvm/java-8-oracle/jre/lib/rt.jar(java/util/Comparator.class)' is broken

[error] (bad constant pool tag 18 at byte 20)

[error] two errors found

[error] (compile:compile) Compilation failed

I am trying to build this sparrow integrated spark with the command:

sbt/sbt package assembly

Is the version of java and scala i am using is right version to build?
If not then please tell me the right version. 

Thanks in advance.

[Kay Ousterhout]

I haven't seen this problem before and it doesn't seem to be pointing to code in the Sparrow code base -- so I'm not sure what the issue is here! Sorry!