Alternatives to IAM key pairs for s3 connectors

[Michael Gutteridge]

 Hi

My org wants to end the use of key pairs for AWS access.  Our AWS accounts are integrated with SSO: that works for many applications but it's not clear how that would work for credentials for the S3 connector.

One proposed method is to use IAM instance roles (if we run DTNs in AWS) and IAM roles anywhere (for on-campus systems).  S3 also has Access Grants which look like an interesting way of mapping identities to S3 datasets.

It's not completely keyless, but I'm thinking that we can accomplish some of the goals by using the --admin-managed-credential option and having an external process rotate keys (at least there won't be stale keys lying around).  There's a fair bit of engineering in there and I'm not sure how Globus will cope with that key being rotated mid-transfer.

I'm not even sure that the S3 connector can deal with a "keyless" situation- adding the key looks to be pretty integral to creating the connection.  I suppose if we used the "--s3-unauthenticated" option we could rely on the role/privilege of the underlying DTN instance (using instance roles/IAM anywhere) for access, but then mapping to individuals gets tricky.

It feels like madness lies in every direction we'd try.  But I'm doing some due-diligence and thought I'd see if anyone else in the Globus community has explored other ways of authorizing the connection to S3.

Thanks

 - Michael

[John Bresnahan]

Michael,

Globus currently only supports key pairs for access. That said we are exploring other solutions and thus feedback like this is very helpful.

Your idea of rotating admin managed credentials should work. There is potential for transfers which are in flight to fail as a result, however in that event we will retry and the next attempt would be successful. To mitigate the inefficiencies introduced by a retry you could rotate keys in the following way:

1: Add a new key pair in S3.

2: Set that new key as the admin managed credential in the Globus connector.

3: wait a window of time

4: de-register the old key with S3

The window where a transfer in flight would fail due to a key change is pretty small but it will depend on the details of the specific transfer. This approach would help minimize retries. But again, even if an in-flight transfer does fail it will retry and succeed.

John