EC2 -- why is cluster setup time 5-10 minutes on Amazon's tutorial, but 30-40 minutes on AmpCamp's tutorial?

[gustavs...@gmail.com]

I am just learning spark, and I am trying both of these tutorials:

• Amazon
• link: http://aws.amazon.com/articles/4926593393724923
  
• cluster setup time: "The cluster should be ready in 5-10 minutes."
• AmpCamp
• link: http://ampcamp.berkeley.edu/big-data-mini-course/launching-a-spark-shark-cluster-on-ec2.html
  
• cluster setup time: "This command may take a 30-40 minutes or longer"
  

I am just wondering, why is the amazon tutorial so much faster? What factors influence the setup time of a cluster?

Thanks!

Eric

[Patrick Wendell]

The AMPCamp scripts copy a large dataset from s3 that the tutorial uses. This is just used for the tutorial, it is not included in the default spark ec2 launcher that is packaged with Spark. The coying time is the vast majority of the 30-40 minutes.

In general, launching a cluster on either EMR or using the Spark ec2 scripts takes about 5 minutes.

- Patrick

--
You received this message because you are subscribed to the Google Groups "Spark Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to spark-users...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[gustavs...@gmail.com]

Thanks!