Change object metadata without re-uploading

[Eric James Soltys]

Is there any way to change an object's metadata (Cache-Control specifically) without having to re-upload it and use the -h option?

I was hoping I'd be able to set a bucket's default headers using a process similar to how setdefacl works.

[Mike Schwartz (Google Storage Team)]

Hi Eric,

Currently there's no way to do it via gsutil; there's an open issue to add such support.

In the mean time, you could run curl commands to do it. Essentially you want to do a PUT to the bucket/object, specifying x-goog-copy-source:<that bucket/object>, and also specifying headers containing the replacement metadata. If you decide to try this please test it out first before you attempt it on your real data.

Mike

[Eric James Soltys]

Thanks Mike.

I've starred the issue and and I'll see what I can do using your workaround.

Eric

On Monday, July 9, 2012 2:32:17 PM UTC-7, Mike Schwartz (Google Storage Team) wrote:

Hi Eric,

Currently there's no way to do it via gsutil; there's an open issue to add such support.

In the mean time, you could run curl commands to do it. Essentially you want to do a PUT to the bucket/object, specifying x-goog-copy-source:<that bucket/object>, and also specifying headers containing the replacement metadata. If you decide to try this please test it out first before you attempt it on your real data.

Mike

[Sebastián Marconi]

On Monday, July 9, 2012 6:32:17 PM UTC-3, Mike Schwartz (Google Storage Team) wrote:

Hi Eric,

Currently there's no way to do it via gsutil; there's an open issue to add such support.

In the mean time, you could run curl commands to do it. Essentially you want to do a PUT to the bucket/object, specifying x-goog-copy-source:<that bucket/object>, and also specifying headers containing the replacement metadata. If you decide to try this please test it out first before you attempt it on your real data.

Didn't worked for me, the object seems to keep the old headers intact. Am I making it wrong?

A) gsutil ls -L gs://bucket/object

...

Cache control: public, max-age=3600

...

B) curl -X PUT --verbose -H "x-goog-api-version: 2" -H "Content-Length:0" -H "x-goog-project-id: PROJECT" -H "Authorization: AUTH" -H "Cache-Control: no-cache" -H "x-goog-copy-source: bucket/object" "http://commondatastorage.googleapis.com/bucket/object"

...

<?xml version='1.0' encoding='UTF-8'?><CopyObjectResult><LastModified>2012-08-01T12:55:51.089Z</LastModified><ETag>"744dc1fc02b5f14bbefc30be39b89e7c"</ETag></CopyObjectResult>

....

C) gsutil ls -L gs://bucket/object

...

Cache control: public, max-age=3600

...

I'm trying to update +170.000 ~1MB files, I don't want to copy them all again to update the Cache-Control option. Thanks!! 

[Google Storage Team]

Original Message Follows:
------------------------
From: Sebastián Marconi<sebastia...@gmail.com>
Subject: Re: Change object metadata without re-uploading
Date: Wed, 1 Aug 2012 06:10:16 -0700 (PDT)

>
>
> On Monday, July 9, 2012 6:32:17 PM UTC-3, Mike Schwartz (Google Storage
> Team) wrote:
> >
> > Hi Eric,
> >
> > Currently there's no way to do it via gsutil; there's an open

issue<http://code.google.com/p/gsutil/issues/detail?id=89>to add such

You may find it easier to use the new Cloud Storage JSON API to perform
this update, since you can use the PATCH method
(https://developers.google.com/storage/docs/json_api/v1/objects/patch) to
update it. e.g.,

curl --request PATCH -H 'Content-Type: application/json' --data
'{cacheControl:"public, max-age=3600"}' -H "Authorization: Bearer
OAUTH_ACCESS_TOKEN_HERE"
https://www.googleapis.com/storage/v1beta1/b/bucket/o/object\?projection=full

NOTE: At some point, the projection argument should be unnecessary; we're
working on it.
NOTE: the object needs to be %-encoded, so slashes in the object name need
to be turned into %2f. e.g., gs://mybucket/path/to/object would be
https://www.googleapis.com/storage/v1beta1/b/mybucket/o/path%2Fto%2Fobject.

You can also use the batch mechanism to send multiple PATCH requests up at
one time per
https://developers.google.com/storage/docs/json_api/v1/how-tos/batch, so
you can batch them in batch requests of size less than 10MB. But if this
is a one-time operation, you're find doing it one-at-a-time.

If you need access to the Cloud Storage JSON API, currently in Limited
Availability, click Request Access... link in the APIs Console
(https://developers.google.com/console) next to the API, or you can
respond to this e-mail with your numeric project ID, and I can provision
your project.

-nh

[Mike Schwartz (Google Storage Team)]

Hi Sebastián,

This is a pretty frequently requested feature, so I implemented a new setmeta command for gsutil that does it. That code is currently out for review. I will probably put out a new version of gsutil with this code in next couple of weeks.

Mike

[Sebastián Marconi]

Hi Sebastián,

This is a pretty frequently requested feature, so I implemented a new setmeta command for gsutil that does it. That code is currently out for review. I will probably put out a new version of gsutil with this code in next couple of weeks.

Mike

Great, thanks! 

[Mike Schwartz (Google Storage Team)]

Hi,

gsutil v3.15 is now released, and it includes a new setmeta command that allows you to set headers on objects without re-uploading the data (see "gsutil help setmeta"). It also includes support for Windows path names containing drive letters, and support for treating URIs as directories if other tools created folder objects (like dir_$folder$, or URIs ending with "/"), as well as a number of bug fixes. You can get this version of gsutil by running:

gsutil update

Thanks,

Mike

On Fri, Aug 3, 2012 at 9:30 AM, Sebastián Marconi sebastianmarconi-at-gmail.com <trax-proxy-disa...@google.com> wrote:

Hi Sebastián,

This is a pretty frequently requested feature, so I implemented a new setmeta command for gsutil that does it. That code is currently out for review. I will probably put out a new version of gsutil with this code in next couple of weeks.

Mike

Great, thanks!