Copy Spark 0.7 AMI to AWS Europe region

[Selwyn McCracken]

Hi,

I tried to copy the latest spark AMI (ami-a60193cf ) to Amazon's Europe region, but this fails as I am not the owner of the image.

Is it possible to get a copy of an image that I can publish for Europe? We would be happy to meet any associated costs. 

 

Running in the US-East region is not a realistic option, as we have a large volume of data already in EU S3 buckets that we do not want to incur traffic costs for. 

Many thanks,

Selwyn

p.s. Spark and it's python integration in particular is amazing stuff.  Great work.

[Matei Zaharia]

I just did this for you; the new AMI is ami-5ca8a328, for the EU Ireland region. It may take a while to actually copy, but when it does, you should be able to pass it to the script with the -a flag.

Matei

> --
> You received this message because you are subscribed to the Google Groups "Spark Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to spark-develope...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[Selwyn McCracken]

Thanks Matei. 

I've just checked and it hasn't appeared in the publicly available images yet, but will check again over the weekend and let you know.

Best regards,

Selwyn

[Selwyn McCracken]

Hi Matei,

Has that AMI image finished copying across? If so, have you set it to be publicly visible? I've just checked and that ID does not appear in the Irish region.

Best regards,

Selwyn  

[Grega Kešpret]

Hello,

where is it possible to see different Spark AWS amis?

Thanks!

LP,

Grega

--

[Selwyn McCracken]

If you are logged into the AWS management console, within the EC2 section you can click on AMIs link on the left. You can then choose to view All Images using the 'Viewing' drop-down items at the top.

For the US East region if you type Spark into the search box you should see several AMIs appear.

However, you probably want to use ami-a60193cf according to https://s3.amazonaws.com/mesos-images/ids/latest-spark-0.7, which is what the spark EC2 launch script points at.

[Matei Zaharia]

Hi Selwyn,

The AWS console says that it's copied, and publicly visible. The ID is ami-5ca8a328. Make sure that you pass both -a ami-5ca8a328 and -z eu-west-1 to the spark-ec2 script.

Matei

[Selwyn McCracken]

Still no joy I'm afraid - see the output below:

selwyn@thekraken:~/spark-0.7.0/ec2$ ./spark-ec2 -i ~/aws/mykey.pem -k mykey launch spark -a ami-5ca8a328 -z eu-west-1 -s 2

Setting up security groups...

Searching for existing cluster spark...

Launching instances...

ERROR:boto:400 Bad Request

ERROR:boto:<?xml version="1.0" encoding="UTF-8"?>

<Response><Errors><Error><Code>InvalidAMIID.NotFound</Code><Message>The image id '[ami-5ca8a328]' does not exist</Message></Error></Errors><RequestID>10161e7d-00d1-4de1-8aec-f4a1407ac89a</RequestID></Response>

Could not find AMI ami-5ca8a328

[Selwyn McCracken]

Hi Matei,

Just a quick update to say that the EU 0.7 AMI is actually up and running correctly.

The *region* flag needs to be set as eu-west-1, not the zone (I was following your instructions a bit too literally... ;-)

Thanks again for copying this across.

Best regards,

Selwyn 

[Alessandro Gagliardi]

Could you do the same for us-west-2 please?

Thank you!

Alessandro Gagliardi

Data Scientist

Path, Inc.