Re: [gs-discussion] Copying directly from S3 to GS

[Google Cloud Storage Team]

Hi Garve,

The simplest and easiest way to transfer them would be to use the gsutil tool.  It would look something like this:

gsutil mb mybucket1
gsutil -m cp -r s3://mybucket1/** gs://mybucket1/

The only hiccup is that you might find the bucket names you have reserved on S3 are already used in Google Cloud Storage.  To mitigate that, you can either choose new names for them individually, or use valid subdomain names for a domain you own.  I'd go with the latter and name all the buckets on Google Cloud Storage as domains, as in "mybucket.mydomain.com".  

If you're on a Linux or OSX machine, you could script this pretty easily.  I'd create a file with all the bucket names in it, one per line, called "buckets.txt", and then type the following into my terminal:

for bucket in `cat buckets.txt`
do
    echo "Copying $bucket"

    gsutil mb gs://$bucket.mydomain.com

    gsutil -m cp -r s3://$bucket/** gs:/$bucket.mydomain.com

done

That's just a barebones script, but it should do what you need. 

Best Regards,

Benson

Google Cloud Storage Team

On Sat, Mar 30, 2013 at 8:47 AM, <ga...@plexusmedia.co.uk> wrote:

I have around 5GB of images stored in around 40 different buckets on S3. What is the easiest way to copy them to Google Cloud Storage?

cheers

Garve

--
You received this message because you are subscribed to the Google Groups "Google Cloud Storage" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gs-discussio...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Google Cloud Storage Team]

Hi Garve,

There were a few errors in my suggestion; I'm sorry about that. 

First, the initial command I gave you to create the bucket needs a gs:// prefix in the bucket name, and should say 

gsutil mb gs://mybucket1

gsutil -m cp -r s3://mybucket1/* gs://mybucket1

Second, the use of ** would flatten your directory structure, so the copy script should instead look like this:

for bucket in `cat buckets.txt`

do
    echo "Copying $bucket"

    gsutil mb gs://$bucket.mydomain.com

    gsutil -m cp -r s3://$bucket/* gs://$bucket.mydomain.com

done

Best Regards,

Benson

Google Cloud Storage Team