2.1.0 Release Candidate Available

[Jordan Kiang]

A release candidate of Synapse Python Client version 2.1.0 is available for preview and testing.

It is available from test.pypi.org and can be installed as follows:

pip3 install --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple synapseclient==2.1.0.24

2.1.0 (2020-05-29)

Highlights:

• A max_threads property of the Synapse object has been added to customize the number of concurrent threads that will be used during file transfers.

  import synapseclient
  syn = synapseclient.login()
  syn.max_threads = 20
  

  If not customized the default value is (CPU count + 4). Adjusting this value higher may speed up file transfers if the local system resources can take advantage of the higher setting. Currently this value applies only to files whose underlying storage is AWS S3.

  Alternately, a value can be stored in the synapseConfig configuration file that will automatically apply as the default if a value is not explicitly set.

  [transfer]
  max_threads=16
  

• This release includes support for directly accessing S3 storage locations using AWS Security Token Service credentials. This allows use of external AWS clients and libraries with Synapse storage, and can be used to accelerate file transfers under certain conditions. To create an STS enabled folder and set-up direct access to S3 storage, see STS Storage Locations below.

• The getAnnotations and setAnnotations methods of the Synapse object have been deprecated in favor of newer get_annotations and set_annotations methods, respectively. The newer versions are parameterized with a typed Annotations dictionary rather than a plain Python dictionary to prevent existing annotations from being accidentally overwritten. The expected usage for setting annotations is to first retrieve the existing Annotations for an entity before saving changes by passing back a modified value.

  annos = syn.get_annotations('syn123')
  
  # set key 'foo' to have value of 'bar' and 'baz'
  annos['foo'] = ['bar', 'baz']
  # single values will automatically be wrapped in a list once stored
  annos['qwerty'] = 'asdf'
  
  annos = syn.set_annotations(annos)
  

  The deprecated annotations methods may be removed in a future release.

A full list of issues addressed in this release are below.

Bug

• [SYNPY-1036] - different users storing same file to same folder results in 403

• [SYNPY-913] - Travis Build badge for develop branch is pointing to pull request

• [SYNPY-960] - AppVeyor build badge appears to be failed while the builds are passed

Improvement

• [SYNPY-1036] - Make upload speeds comparable to those of the AWS S3 CLI

• [SYNPY-1049] - Expose STS-related APIs

STS Storage Locations

Create an STS enabled folder to use AWS Security Token Service credentials with S3 storage locations. These credentials can be used with external S3 tools such as the awscli and the boto3 library separately from Synapse to read and write files to and from Synapse storage.

Creating an STS enabled folder

Creating an STS enabled folder is similar to creating an external storage folder as described above, but this time passing an additional sts_enabled=True keyword parameter.

# create a new folder to use with STS and external S3 storage
folder = syn.store(Folder(name=folder_name, parent=parent))
syn.create_s3_storage_location(
    folder=folder,
    bucket_name='my-external-synapse-bucket',
    base_key='path/within/bucket',
    sts_enabled=True,
 )

Using credentials with the awscli

This example illustrates obtaining STS credentials and using them with the awscli command line tool. The first command outputs the credentials as shell commands to execute which will then be picked up by subsequent aws cli commands.

$ synapseclient get-sts-token -o shell syn123 read_write

export SYNAPSE_STS_S3_LOCATION="s3://my-external-synapse-bucket/path/within/bucket"
export AWS_ACCESS_KEY_ID="<access_key_id>"
export AWS_SECRET_ACCESS_KEY="<secret_access_key>"
export AWS_SESSION_TOKEN="<session_token>

# if the above are executed in the shell, the awscli will automatically apply them

# e.g. copy a file directly to the bucket using the exported credentials
$ aws s3 cp /path/to/local/file $SYNAPSE_STS_S3_LOCATION

Using credentials with boto3 in python

This example illustrates retrieving STS credentials and using them with boto3 within python code, in this case to upload a file.

# the boto output_format is compatible with the boto3 session api.
credentials = syn.get_sts_storage_token('syn123', 'read_write', output_format='boto')

s3_client = boto3.client('s3', **credentials)
s3_client.upload_file(
    Filename='/path/to/local/file,
    Bucket='my-external-synapse-bucket',
    Key='path/within/bucket/file',
)

Automatic transfers to/from STS storage locations using boto3 with synapseclient

The Python Synapse client can be configured to automatically use STS tokens to perform uploads and downloads to enabled storage locations using an installed boto3 library rather than through the traditional Synapse client APIs. This can improve performance in certain situations, particularly uploads of large files, as the data transfer itself can be conducted purely against the AWS S3 APIs, only invoking the Synapse APIs to retrieve the necessary token and to update Synapse metadata in the case of an upload. Once configured to do so, retrieval of STS tokens for supported operations occurs automatically without any change in synapseclient usage.

To enable STS/boto3 transfers on all get and store operations, do the following:

1. Ensure that boto3 is installed in the same Python installation as synapseclient.

pip install boto3

2. To enable automatic transfers on all uploads and downloads, update your Synapse client configuration file (typically “.synapseConfig” in your $HOME directory, unless otherwise configured) with the [transfer] section, if it is not already present. To leverage STS/boto3 transfers on a per Synapse client object basis, set the use_boto_sts_transfers property.

# add to .synapseConfig to automatically apply as default for all synapse client instances
[transfer]
use_boto_sts=true

# alternatively set on a per instance basis within python code
syn.use_boto_sts_transfers = True

Note that if boto3 is not installed, then these settings will have no effect.