Can I use BOTO to syncronize two S3 buckets in different providers?

67 views
Skip to first unread message

aay...@qdqmedia.com

unread,
Nov 16, 2015, 10:27:10 AM11/16/15
to boto-users
Hello!

I have my own object storage cluster with Riak-CS and I would like to synchronize it with Amazon S3 with backups purpose.

I have made a testing environment with two object storage servers (also with RIAK-CS), and I'm trying to synchronize two buckets with the same name, but I'm not able to make the synchronization process works, it returns a 404 error.

I'm using boto  2.38.

here is the code I'm using:

from boto.s3.key import Key
from boto.s3.connection import S3Connection
from boto.s3.connection import OrdinaryCallingFormat

apikey04='GFR3O0HFPXQ-BWSXEMAG'
secretkey04='eIiigR4Rov2O2kxuSHNW7WPoJE2KmrtMpzzqlg=='

apikey02='J0TT_C9MJPWPGHW-KEWY'
secretkey02='xcLOt3ANqyNJ0kAjP8Mxx68qr7kgyXG3eqJuMA=='
cf=OrdinaryCallingFormat()

conn04=S3Connection(aws_access_key_id=apikey04,aws_secret_access_key=secretkey04,
                      is_secure=False,host='rk04.ejemplo.com',port=8080,calling_format=cf)

conn02=S3Connection(aws_access_key_id=apikey02,aws_secret_access_key=secretkey02,
                      is_secure=False,host='rk02.ejemplo.com',port=8080,calling_format=cf)

bucket04=conn04.get_bucket('testbucket')
bucket02=conn02.get_bucket('testbucket')

rs04 = bucket04.list()

for k in rs04:
    print k.name
    bucket02.copy_key(k.key, bucket04, k.key)


When this script is executed it returns:

Traceback (most recent call last):
  File "s3_connect_2.py", line 38, in <module>
    bucket02.copy_key(k.key, bucket04, k.key)
  File "/home/alberto/.virtualenvs/boto/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 888, in copy_key
    response.reason, body)
boto.exception.S3ResponseError: S3ResponseError: 404 Not Found
<?xml version="1.0" encoding="UTF-8"?><Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Resource>/&lt;Bucket: testbucket&gt;$

Has anyone does something similar before? 

Tanks a lot.

Victor Trac

unread,
Nov 16, 2015, 10:45:25 AM11/16/15
to boto-users
copy_key copies keys from one key to another, not across connections, so the source key is not going to work. You'll need to download the file and re-upload using get_* and send_file or set_*.

--
Victor Trac  |  victortrac.com  |  twitter.com/victortrac 

--
You received this message because you are subscribed to the Google Groups "boto-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to boto-users+...@googlegroups.com.
To post to this group, send email to boto-...@googlegroups.com.
Visit this group at http://groups.google.com/group/boto-users.
For more options, visit https://groups.google.com/d/optout.

Reply all
Reply to author
Forward
0 new messages