S3 get_bucket limited to 1000 items

[Darrin Eden]

Hi, What to do when a bucket has more than 1K objects?

https://github.com/geemus/fog/blob/v0.3.34/lib/fog/aws/requests/storage/get_bucket.rb#L84

Thanks.

[geemus]

You will need to do pagination, which s3 does through a notion of
'marker'. So you'll want to get the key of object 1000 and say that
this is the marker and make the request again to get the 2nd 1000.
You can repeat as necessary, and I think the IsTruncated value should
be false when you get to the last page. Let me know if that isn't
clear enough or you have further questions or issues.
Thanks!
wes

On Jan 2, 6:16 pm, Darrin Eden <darrin.e...@gmail.com> wrote:
> Hi, What to do when a bucket has more than 1K objects?
>

> https://github.com/geemus/fog/blob/v0.3.34/lib/fog/aws/requests/stora...
>
> Thanks.

[Darrin Eden]

Thanks. I ended up writing this inelegant snippet (lame excuse: I was
getting tired): https://gist.github.com/763977

[geemus (Wesley Beary)]

It used to be the default to run through all the pages and concatenate for you, but as you might imagine that doesn't scale very well.  So as users started to run into issues I removed it as the default.

I've intended for a while to try to add something to make it a bit easier at least, something like files.next or something, but never settled on how I wanted the api to be and then promptly forgot about it.

I created an issue so I won't forget again, but not sure I'll get to it any time soon: https://github.com/geemus/fog/issues/issue/129

Thanks!

wes

[Darrin Eden]

I think the way you've written now is quite good. It encouraged me to
think more about design and as a result my code is much improved. It
just took a bit of digging to understand the API. Don't know if
there's an easy way to leave hints for the next developer. In any case
thanks again and take care.

On Jan 3, 1:55 pm, "geemus (Wesley Beary)" <gee...@engineyard.com>
wrote: