Update a batch of S3 objects' metadata?

Showing 1-12 of 12 messages
Update a batch of S3 objects' metadata? John Bachir 2/14/12 11:10 AM
I need to change some metadata (Content-Type) on hundreds or thousands of objects on S3. As far as I can tell there is no way to save only metadata with fog, the entire object must be re-saved including the body. Is this true?

Thanks!
John

Re: Update a batch of S3 objects' metadata? geemus 2/15/12 8:31 AM
Unfortunately I don't think this is a fog specific issue.  There isn't
a way to do what you would like on S3 at all (that I'm aware of).
Feel free to peruse the docs to see if I missed something though:
http://docs.amazonwebservices.com/AmazonS3/latest/API/RESTObjectPOST.html

You might be able to save yourself some trouble doing copies (so you
don't have to reupload) but that would mean changing the keys in
addition to other things I think.  You might try copying something to
the same bucket/key with
the different stats, I have no idea if it will work, but probably
worth a shot on an object you aren't worried about.

That might get you what you want, otherwise you may be in for a lot of
waiting while stuff does its thing.  Let us know how it goes or if you
have any other questions.

Thanks!
wes

Re: Update a batch of S3 objects' metadata? John Joseph Bachir 2/17/12 6:20 PM
Copy-- great idea -- it works! Not only that, But I can do an in-place copy, so I can update metadata with one command:

myfile.copy(directory.key, myfile.key, 'x-amz-metadata-directive' => 'REPLACE', 'Content-Type' => 'foo/bar')

Problem: any metadata not specified in options at the end ends up being nil. So, one must construct a hash of the original metadata and merge in the new stuff:

({'Content-Disposition' => myfile.content_disposition, 'Content-Type' => 'foo/bar', … })

The risk here is forgetting to include a metadata value or starting to overwrite things with nil if a new type of metadata value is introduced in the future. Is there a nicer way to to this? Does this hash already existing on Fog::Storage::AWS::File somewhere? Then we could just specify the stuff that changes:

myfile.s3_metadata.merge( 'Content-Type' => 'foo/bar' )


Re: Update a batch of S3 objects' metadata? geemus 2/17/12 8:52 PM
Awesome, good to hear it worked and thanks for the update!
Re: Update a batch of S3 objects' metadata? John Joseph Bachir 2/17/12 9:02 PM
Is there an easy way to get the hash of metadata out of a Fog::Storage::AWS::File?
Re: Update a batch of S3 objects' metadata? geemus 2/18/12 8:03 AM
I don't know about getting JUST the metadata, but you should be able
to call #attributes on a fog model to get a hash of all its
attributes.  That should probably contain what you want, but might
have extra stuff too.  Let me know if that doesn't cut it though and
we can try again.  Thanks!
wes
Re: Update a batch of S3 objects' metadata? rupakg 2/18/12 8:17 AM
You can get the metadata, merge in your new stuff, and replace with a copy. The get will be an overhead though.

Thanks,
Rupak Ganguly
678-648-7434
Re: Update a batch of S3 objects' metadata? Kevin Menard 2/20/12 5:13 PM
You can fetch just the metadata with a HEAD, which should be a good deal faster.

--
Kevin

Re: Update a batch of S3 objects' metadata? Rob Yurkowski 6/14/13 12:22 PM
I know that this is a year late, but this topic was a pretty good help. Still, I had some difficulties operating on a selection of those files, so here's what I did to make it work:


Hopefully this will be helpful to someone in the future, too!
Re: Update a batch of S3 objects' metadata? geemus 6/14/13 12:39 PM
Thanks!


--
You received this message because you are subscribed to the Google Groups "ruby-fog" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ruby-fog+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

Re: Update a batch of S3 objects' metadata? rupakg 6/14/13 3:55 PM
OpenStack have a post_object which is an update call and does not need a body. Also preserves the existing metadata.

Does'nt S3 have a post_object API?

Thanks,
Rupak Ganguly
678-648-7434
--
You received this message because you are subscribed to the Google Groups "ruby-fog" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ruby-fog+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Re: Update a batch of S3 objects' metadata? geemus 6/17/13 9:59 AM
There isn't an explicit thing quite like that, I gather you can use copy to write over the top of an existing thing in order to get this effect though. See: http://stackoverflow.com/questions/4754383/how-to-change-metadata-on-an-object-in-amazon-s3