Account Options

  1. Sign in
The old Google Groups will be going away soon, but your browser is incompatible with the new version.
Google Groups Home
« Groups Home
Clean up bucket s3
There are currently too many topics in this group that display first. To make this topic appear first, remove this option from another topic.
There was an error processing your request. Please try again.
flag
  9 messages - Collapse all  -  Translate all to Translated (View all originals)
The group you are posting to is a Usenet group. Messages posted to this group will make your email address visible to anyone on the Internet.
Your reply message has not been sent.
Your post was successful
 
From:
To:
Cc:
Followup To:
Add Cc | Add Followup-to | Edit Subject
Subject:
Validation:
For verification purposes please type the characters you see in the picture below or the numbers you hear by clicking the accessibility icon. Listen and type the numbers you hear
 
alexgr007  
View profile  
 More options Mar 2 2011, 9:46 am
From: alexgr007 <alexgr...@gmail.com>
Date: Wed, 2 Mar 2011 06:46:31 -0800 (PST)
Local: Wed, Mar 2 2011 9:46 am
Subject: Clean up bucket s3
Hi everyone.

I wish clean up one directory into s3. This directory have a lot of
images, but i don't want to delete image by image. So, when I try to
do something like:

bucket = "sk-dev"

directory = storage.directories.get(bucket)

directory.destroy

I get, this message: The bucket you tried to delete is not empty

My aim is delete everything in the bucket, and then uploads new
images.

Thanks.


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
geemus  
View profile  
 More options Mar 2 2011, 1:18 pm
From: geemus <wbe...@engineyard.com>
Date: Wed, 2 Mar 2011 10:18:58 -0800 (PST)
Local: Wed, Mar 2 2011 1:18 pm
Subject: Re: Clean up bucket s3
Unfortunately, due to the way that storage providers work there
generally isn't a way to do this without deleting thing image by
image.

Sorry I can't be of more help than that, but it is really the only
viable option.

wes

On Mar 2, 6:46 am, alexgr007 <alexgr...@gmail.com> wrote:


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
John Vincent  
View profile  
 More options Mar 2 2011, 1:24 pm
From: John Vincent <lusis....@gmail.com>
Date: Wed, 2 Mar 2011 13:24:10 -0500
Local: Wed, Mar 2 2011 1:24 pm
Subject: Re: Clean up bucket s3
As Wes said, there's not a uniform way to do that. You'll have to
iterate the contents.

However, depending on the size of the bucket, you'll run into the 1k
key limit. I'll save you some searching now ;)

https://gist.github.com/763977

--
John E. Vincent
http://about.me/lusis

 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Nic  
View profile  
 More options Apr 13 2011, 11:17 am
From: Nic <nicolas.fedd...@gmail.com>
Date: Wed, 13 Apr 2011 08:17:10 -0700 (PDT)
Local: Wed, Apr 13 2011 11:17 am
Subject: Re: Clean up bucket s3
Is there a way to access a nested directory so that you can iterate
through it?  So if I have a directory 6 levels deep with 10 images in
it how would I get that directory that I want to iterate through?

On Mar 2, 1:24 pm, John Vincent <lusis....@gmail.com> wrote:


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
geemus  
View profile  
 More options Apr 13 2011, 1:00 pm
From: geemus <wbe...@engineyard.com>
Date: Wed, 13 Apr 2011 10:00:01 -0700 (PDT)
Local: Wed, Apr 13 2011 1:00 pm
Subject: Re: Clean up bucket s3
The cloud providers don't actually provide for nested directories.  In
the end they just have long keys that contain many '/' characters.  As
such you can probably limit the list down to stuff with the prefix
option and then iterate over those.

On Apr 13, 8:17 am, Nic <nicolas.fedd...@gmail.com> wrote:


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
John Vincent  
View profile   Translate to Translated (View Original)
 More options Apr 13 2011, 1:14 pm
From: John Vincent <lusis....@gmail.com>
Date: Wed, 13 Apr 2011 13:14:27 -0400
Local: Wed, Apr 13 2011 1:14 pm
Subject: Re: Clean up bucket s3
Guess I missed this when it came in, sorry. Again Wes beat me to it.
In the case of S3, the "directory" structure you're seeing is just
bucket names or keys with "/" in the name. S3 originally didn't even
have support for that. It was bucket + contents.

One option you might want to try down the road is, depending on how
you're putting stuff in the bucket, is to maintain an "index" bucket
somewhere. It's not as pretty but you could have a bucket called, say,
"my-bucket-indexes" and put a single file per bucket in there. The
file would contain a list of items in a given bucket. The you just
iterate over the contents of the file to clean out the bucket when it
gets to a certain size.

--
John E. Vincent
http://about.me/lusis

 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Nic  
View profile  
 More options Apr 13 2011, 1:16 pm
From: Nic <nicolas.fedd...@gmail.com>
Date: Wed, 13 Apr 2011 10:16:54 -0700 (PDT)
Local: Wed, Apr 13 2011 1:16 pm
Subject: Re: Clean up bucket s3
Okay thanks Wes, that's what I'll do!

On Apr 13, 12:00 pm, geemus <wbe...@engineyard.com> wrote:


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Nic  
View profile  
 More options Apr 18 2011, 1:45 pm
From: Nic <nicolas.fedd...@gmail.com>
Date: Mon, 18 Apr 2011 10:45:05 -0700 (PDT)
Local: Mon, Apr 18 2011 1:45 pm
Subject: Re: Clean up bucket s3
Does fog have a built-in iterator that I can use for this?

On Apr 13, 12:00 pm, geemus <wbe...@engineyard.com> wrote:


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
geemus (Wesley Beary)  
View profile  
 More options Apr 18 2011, 1:55 pm
From: "geemus (Wesley Beary)" <gee...@engineyard.com>
Date: Mon, 18 Apr 2011 10:55:20 -0700
Local: Mon, Apr 18 2011 1:55 pm
Subject: Re: Clean up bucket s3

fog has support for the regular s3 iterators, but not really anything built
in beyond that (at least at present).  You can see some discussion of using
the s3 stuff here:
https://groups.google.com/group/ruby-fog/browse_thread/thread/b34ba69...

Let me know if that doesn't help and I can try to further clarify or provide
an example.


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
End of messages
« Back to Discussions « Newer topic     Older topic »