Copy Spark 0.7 AMI to US West (Northern California), us-west-1

[gustavs...@gmail.com]

Hi,

Would it be possible to copy the 0.7 AMI to us-west-1, as was possible here:

https://groups.google.com/forum/?fromgroups=#!searchin/spark-developers/tried$20to$20copy$20ami/spark-developers/Vhc0KtWZGDE/253lC8pnr-MJ

I did not see any Spark AMI's when searching us-west-1: https://console.aws.amazon.com/ec2/v2/home?region=us-west-1#Images:filter=all-images;platform=all-platforms;visibility=public-images;search=spark

Let me know if there are costs that we may be able to cover for doing this.

Thanks,

Eric

[Patrick Wendell]

Hey Eric,

FYI - I've finished a re-writing of the scripts to support other availability zones. Hopefully this will end up in 0.7.2.

In the mean time, this will require Matei to do the copy since he owns the image. However, here are my notes for "manually" cloning the existing Image - it's not too bad. If Matei can't get around to it you can follow these and create your own image in US-west:

===
- Start with base Amazon linux image
 - Allow root login in sshd config file
- Rsync /root/ from old AMI:rsync --archive -e "ssh -i /root/.ssh/patkey.pem" * -r ec2-54-242-52-94.compute-1.amazonaws.com:/root/ --delete
- Copy /etc/cloud/cloud.cfg from existing image
- yum install -y git
- yum install -y dstat
- yum remove -y java-1.6.0-openjdk[-devel] (remove/reinstall to fix symlink)
  - yum install -y java-1.6.0-openjdk-devel
- yum install -y java-1.7.0-openjdk-devel
- manually select 1.7 alternative for javac and java (others inherit)
  - do this using sudo update-alternatives --config java[c] and select 1.7

--
You received this message because you are subscribed to the Google Groups "Spark Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to spark-develope...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Alessandro Gagliardi]

Hi Patrick,

I just installed the latest version of Spark (0.7.3) and I'm trying to start it up in us-west-2, but it's not working. I'm wondering if your change didn't make it in or if I'm doing something wrong.

Thanks in advance,

-Alessandro

[Patrick Wendell]

Hey There,

The new changes to the ec2 scripts are only in 0.8 and newer (i.e. un-released). So if you want to test them you'll have to git clone master. They will create a cluster with the latest released version of Spark (0.7.3) though.

Another option is to launch the existing 0.7.3 AMI, save a copy of it on EC2 then copy that to the Oregon region. Then you can launch with your own AMI copy.

- Patrick

[Alessandro Gagliardi]

I think I installed it from the master branch on github yesterday. How would I check the version number? 

[Patrick Wendell]

Ah sorry - I mistyped. You need to clone the ec2-updates branch from my own repo. We didn't merge this yet. Right now we are testing this branch internally:

git clone https://github.com/pwendell/spark.git -b ec2-updates.

[Alessandro Gagliardi]

I don't know if this helps, but here's what I try and here's the response:

11:01:15 spark $ec2/spark-ec2 -k <KEY_PAIR> -i <IDENTITY_FILE> -r us-west-2 -s 1 launch test_spark

Setting up security groups...

Creating security group test_spark-master

Creating security group test_spark-slaves

Creating security group test_spark-zoo

Searching for existing cluster test_spark...

Latest Spark AMI: ami-530f7a3a

Launching instances...

ERROR:boto:400 Bad Request

ERROR:boto:<?xml version="1.0" encoding="UTF-8"?>

<Response><Errors><Error><Code>InvalidAMIID.NotFound</Code><Message>The image id '[ami-530f7a3a]' does not exist</Message></Error></Errors><RequestID>e44e50aa-c9d7-4b51-86ee-d0d918a0a799</RequestID></Response>

Could not find AMI ami-530f7a3a

It works if I do -r us-east-1.

[Patrick Wendell]

Also just a heads up - my branch is somewhat experimental, so please report any issues. 

--

[Alessandro Gagliardi]

Will do. So far it's working great. Thanks.

[Alessandro Gagliardi]

I've been stopping and starting this cluster at the end and beginning of the day and until now that worked fine but this afternoon I tried starting it again and have been getting some errors:

error: Failed connect to github.com:443; Connection timed out while accessing https://github.com/mesos/spark-ec2.git/info/refs?service=git-upload-pack

fatal: HTTP request failed

Connection to ec2-...-...-...-....us-west-2.compute.amazonaws.com closed.

Error connecting to host Command 'ssh -t -o StrictHostKeyChecking=no -i /Users/alessandro/.ssh/....pem root@ec2-...-...-...-....us-west-2.compute.amazonaws.com 'rm -rf spark-ec2 && git clone https://github.com/mesos/spark-ec2.git -b v2'' returned non-zero exit status 128, sleeping 30

It's still going, and it looks like it might recover, but I don't remember this happening before. 

[Patrick Wendell]

I think github might be down right now...