Google Cloud Storage: Introducing composite objects

[Google Storage Team]

Hello,

You can now quickly combine multiple objects into a single new object. A total of 32 objects can be combined into any target object. Composite objects enable a few significant use cases:

• Parallel upload: divide your data into as many chunks as required to fully utilize your available bandwidth, upload each chunk to a distinct object, compose your final object, and delete any temporary objects.

• Limited append: upload data to a temporary new object, compose it with the object you wish to append to, and delete the temporary object.

We’ve updated gsutil with support for composite objects and several other notable features:

• Support for service accounts. This allows running gsutil without depending on an individual’s credentials. It’s useful whenever you want to run gsutil on behalf of a service or application. For example: when running gsutil as a scheduled job to upload data.

• gsutil now checks for available updates periodically.

• New chdefacl command, enabling easy modification of default object ACLs. It works similarly to chacl, but modifies the default object ACL on buckets instead of object ACLs.

• gsutil built-in help is also available online: https://developers.google.com/storage/docs/gsutil (expand the "gsutil Commands" section on the side bar)

• And many other enhancements and bug fixes.

We think you’ll find object composition to be a powerful addition to your toolkit. Please take a few minutes to give it a try today.

Brian Dorsey

Google Cloud Storage team

----  Documentation links and quick start ----

Release notes for gsutil this release (3.26):

http://pub.storage.googleapis.com/gsutil_3.26.ReleaseNotes.txt

Configure a service account (see "gsutil help creds" for details):

  $ gsutil config -e

Composite Object documentation:

https://developers.google.com/storage/docs/composite-objects

Composite object examples (from "gsutil help compose"):

To upload in parallel, split your file into smaller pieces, upload them using "gsutil -m cp", compose the results, and delete the pieces.

  $ split -n 10 big-file big-file-part-

  $ gsutil -m cp big-file-part-* gs://bucket/dir/

  $ rm big-file-part-*

  $ gsutil compose gs://bucket/dir/big-file-part-* gs://bucket/dir/big-file

  $ gsutil -m rm gs://bucket/dir/big-file-part-*

Appending simply entails uploading your new data to a temporary object, composing it with the growing append-target, and deleting the temporary object.

  $ echo 'new data' | gsutil cp - gs://bucket/data-to-append

  $ gsutil compose gs://bucket/append-target gs://bucket/data-to-append gs://bucket/append-target

  $ gsutil rm gs://bucket/data-to-append

If you have specific questions about Google Cloud Storage, please post them to the google-cloud-storage tag on StackOverflow. http://stackoverflow.com/tags/google-cloud-storage  We monitor questions to the tag, and this is the primary venue for all Google Cloud Storage support. You will likely get a quick answer to any questions, either from the active community or a member of the Cloud Storage team.