Re: [gs-discussion] Copying directly from S3 to GS

1,119 views
Skip to first unread message

Google Cloud Storage Team

unread,
Apr 1, 2013, 2:23:57 PM4/1/13
to gs-discussion
Hi Garve,

The simplest and easiest way to transfer them would be to use the gsutil tool.  It would look something like this:

gsutil mb mybucket1
gsutil -m cp -r s3://mybucket1/** gs://mybucket1/

The only hiccup is that you might find the bucket names you have reserved on S3 are already used in Google Cloud Storage.  To mitigate that, you can either choose new names for them individually, or use valid subdomain names for a domain you own.  I'd go with the latter and name all the buckets on Google Cloud Storage as domains, as in "mybucket.mydomain.com".  

If you're on a Linux or OSX machine, you could script this pretty easily.  I'd create a file with all the bucket names in it, one per line, called "buckets.txt", and then type the following into my terminal:

for bucket in `cat buckets.txt`
do
    echo "Copying $bucket"
    gsutil mb gs://$bucket.mydomain.com
    gsutil -m cp -r s3://$bucket/** gs:/$bucket.mydomain.com
done

That's just a barebones script, but it should do what you need. 

Best Regards,
Benson
Google Cloud Storage Team


On Sat, Mar 30, 2013 at 8:47 AM, <ga...@plexusmedia.co.uk> wrote:
I have around 5GB of images stored in around 40 different buckets on S3. What is the easiest way to copy them to Google Cloud Storage?

cheers

Garve


--
You received this message because you are subscribed to the Google Groups "Google Cloud Storage" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gs-discussio...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

Google Cloud Storage Team

unread,
Apr 1, 2013, 2:43:39 PM4/1/13
to gs-discussion
Hi Garve,

There were a few errors in my suggestion; I'm sorry about that. 

First, the initial command I gave you to create the bucket needs a gs:// prefix in the bucket name, and should say 

gsutil mb gs://mybucket1
gsutil -m cp -r s3://mybucket1/* gs://mybucket1

Second, the use of ** would flatten your directory structure, so the copy script should instead look like this:

for bucket in `cat buckets.txt`
do
    echo "Copying $bucket"
    gsutil mb gs://$bucket.mydomain.com
    gsutil -m cp -r s3://$bucket/* gs://$bucket.mydomain.com
done

Best Regards,
Benson
Google Cloud Storage Team
Reply all
Reply to author
Forward
0 new messages