Announcing bdutil-0.33.0 and version 1.2.3 of the GCS Connector for Hadoop

[Hadoop on Google Cloud Platform Team]

Greetings Hadoop on Google Cloud Platform users!

We have just released an update to our scripts, support for configuration templates,  and some bug fixes to the Google Cloud Storage connector. First the names of following scripts been changed:

• ghadoop is now called bdutil

• ghconfig is now called bdconfig

• ghadoop_env.sh  is now called bdutil_env.sh

We now provide configuration template files that are merged into the “*-site.xml” files on deployment. Property values set in “bdutil-0.33.0/conf/*-site.xml.template” will be applied to the master and worker nodes in your deployed cluster. To set site specific values you can use the tag '<envVar name="FOO" />' which will be resolved to environment variable '$FOO' in the deployment scripts. You can also edit the scripts to manually set specific keys as shown in bdutil-0.33.0/libexec/set_default_fs.sh.

You can download the new bdutil-0.33.0.tar.gz or bdutil-0.33.0.zip directly, or visit the developer documentation where the download links now point to the latest version.

As always, please send any questions or comments to gcp-hadoo...@google.com

All the best,

Your Google Team

gcs-connector-1.2.3: CHANGES.txt

1.2.3 - 2014-03-21

 1. Fixed a bug where renaming a directory could cause the file contents to get

    shuffled between files when the fully-qualified file paths have different

    lengths. Does not apply to renames on files directly, such as when using

    a glob expression inside a flat directory.

 2. Changed the behavior of batch request API calls such that they are retried

    on failure in the same manner as non-batch requests.

 3. Eliminated an unnecessary dependency on com/google/protobuf which could

    cause version-incompatibility issues with other software packages.

bdutil-0.33.0: CHANGES.txt

0.33.0 - 2014-03-21

 1. Renamed 'ghadoop' to 'bdutil' and ghadoop_env.sh to bdutil_env.sh.

 2. Bundled a collection of *-site.xml.template files in conf/ subdirectory

    which are integrated into the hadoop conf/ files in the remote scripts.

 3. Switched core-site template to new 'fs.gs.auth.*' syntax for

    enabling service-account auth.

bdconfig-0.28.0: CHANGES.txt

0.28.0 - 2014-03-21

 1. Renamed ghconfig to bdconfig.

 2. Allowed for direct editing and querying of configuration *site.xml files.

    Run 'bdconfig help' to see the new commands.

 3. Allowed for in line replacement of '<envVar name="FOO" />' tags with the

    environment variable '$FOO'.

 4. Deprecated configure_hadoop, configure_ghfs, and set_default_fs commands.

 5. Switched to new 'fs.gs.auth.*' syntax for the various auth-related keys

    like client_id, client_secret, and enabling service-account auth.