Update a batch of S3 objects' metadata?

4,176 views
Skip to first unread message

John Bachir

unread,
Feb 14, 2012, 2:10:39 PM2/14/12
to ruby...@googlegroups.com
I need to change some metadata (Content-Type) on hundreds or thousands of objects on S3. As far as I can tell there is no way to save only metadata with fog, the entire object must be re-saved including the body. Is this true?

Thanks!
John

geemus (Wesley Beary)

unread,
Feb 15, 2012, 11:31:26 AM2/15/12
to ruby...@googlegroups.com
Unfortunately I don't think this is a fog specific issue. There isn't
a way to do what you would like on S3 at all (that I'm aware of).
Feel free to peruse the docs to see if I missed something though:
http://docs.amazonwebservices.com/AmazonS3/latest/API/RESTObjectPOST.html

You might be able to save yourself some trouble doing copies (so you
don't have to reupload) but that would mean changing the keys in
addition to other things I think. You might try copying something to
the same bucket/key with
the different stats, I have no idea if it will work, but probably
worth a shot on an object you aren't worried about.

That might get you what you want, otherwise you may be in for a lot of
waiting while stuff does its thing. Let us know how it goes or if you
have any other questions.

Thanks!
wes

John Bachir

unread,
Feb 17, 2012, 9:20:39 PM2/17/12
to ruby...@googlegroups.com
Copy-- great idea -- it works! Not only that, But I can do an in-place copy, so I can update metadata with one command:

myfile.copy(directory.key, myfile.key, 'x-amz-metadata-directive' => 'REPLACE', 'Content-Type' => 'foo/bar')

Problem: any metadata not specified in options at the end ends up being nil. So, one must construct a hash of the original metadata and merge in the new stuff:

({'Content-Disposition' => myfile.content_disposition, 'Content-Type' => 'foo/bar', … })

The risk here is forgetting to include a metadata value or starting to overwrite things with nil if a new type of metadata value is introduced in the future. Is there a nicer way to to this? Does this hash already existing on Fog::Storage::AWS::File somewhere? Then we could just specify the stuff that changes:

myfile.s3_metadata.merge( 'Content-Type' => 'foo/bar' )


geemus (Wesley Beary)

unread,
Feb 17, 2012, 11:52:32 PM2/17/12
to ruby...@googlegroups.com
Awesome, good to hear it worked and thanks for the update!

John Joseph Bachir

unread,
Feb 18, 2012, 12:02:21 AM2/18/12
to ruby...@googlegroups.com
Is there an easy way to get the hash of metadata out of a Fog::Storage::AWS::File?

geemus (Wesley Beary)

unread,
Feb 18, 2012, 11:03:08 AM2/18/12
to ruby...@googlegroups.com
I don't know about getting JUST the metadata, but you should be able
to call #attributes on a fog model to get a hash of all its
attributes. That should probably contain what you want, but might
have extra stuff too. Let me know if that doesn't cut it though and
we can try again. Thanks!
wes

Rupak Ganguly

unread,
Feb 18, 2012, 11:17:19 AM2/18/12
to ruby...@googlegroups.com
You can get the metadata, merge in your new stuff, and replace with a copy. The get will be an overhead though.

Thanks,
Rupak Ganguly

Kevin Menard

unread,
Feb 20, 2012, 8:13:38 PM2/20/12
to ruby...@googlegroups.com
You can fetch just the metadata with a HEAD, which should be a good deal faster.

--
Kevin

Rob Yurkowski

unread,
Jun 14, 2013, 3:22:47 PM6/14/13
to ruby...@googlegroups.com
I know that this is a year late, but this topic was a pretty good help. Still, I had some difficulties operating on a selection of those files, so here's what I did to make it work:


Hopefully this will be helpful to someone in the future, too!

geemus (Wesley Beary)

unread,
Jun 14, 2013, 3:39:49 PM6/14/13
to ruby...@googlegroups.com
Thanks!


--
You received this message because you are subscribed to the Google Groups "ruby-fog" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ruby-fog+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

Rupak Ganguly

unread,
Jun 14, 2013, 6:55:41 PM6/14/13
to ruby...@googlegroups.com
OpenStack have a post_object which is an update call and does not need a body. Also preserves the existing metadata.

Does'nt S3 have a post_object API?

Thanks,
Rupak Ganguly

geemus (Wesley Beary)

unread,
Jun 17, 2013, 12:59:49 PM6/17/13
to ruby...@googlegroups.com
There isn't an explicit thing quite like that, I gather you can use copy to write over the top of an existing thing in order to get this effect though. See: http://stackoverflow.com/questions/4754383/how-to-change-metadata-on-an-object-in-amazon-s3
Reply all
Reply to author
Forward
0 new messages