S3 server-side encryption using AWS KMS

[Carl Bolduc]

I'm trying to upload a file to S3 using boto while specifying a KMS key that I created in IAM.

I want to do what is described here in Java (second sample):
http://java.awsblog.com/post/Tx19OLB7M3D6DS8/S3-Encryption-with-AWS-Key-Management-Service

Looking at the documentation, this does not seem to be supported. set_contents_from_file allows to specify encrypt_key to true but there does not seem to be a way to provide my own key ID. I was able to provide my key by setting custom headers but I get an error from in the response because my request is not signed.

As you can see in the Java sample, all the heavy lifting is done by the AWS SDK for Java, you simply need to provide the key ID.

Can this be achieved with boto?

[Thomas O'Dowd]

Hi Carl,

This works with Boto but you have to explicitly set the headers yourself
as you suggested. You also must use HTTPS for SSE-C requests. If you
still have trouble post some code.

That said, Boto could add an API for this but right now there isn't one.

Tom.

--
Cloudian KK - http://cloudian.jp/
S3 REST API Compliant Cloud Storage with Cloudian®

[Carl Bolduc]

Hi Tom,

Based on the Amazon python sample, this is what I'm trying:

from boto.s3.connection import S3Connection
from boto.s3.key import Key
import hmac
import hashlib
import binascii

AWS_ACCESS_KEY = "***"
AWS_SECRET_KEY = "***"
BUCKET_NAME = "mybucket"
S3_FILE_PATH = "encrypted.txt"

result="la bibi da bibi dum"
def sign(key, msg):
    return hmac.new(key, msg.encode("utf-8"), hashlib.sha256).digest()

def getSignatureKey(key, dateStamp, regionName, serviceName):
    kDate = sign(("AWS4" + key).encode("utf-8"), dateStamp)
    kRegion = sign(kDate, regionName)
    kService = sign(kRegion, serviceName)
    kSigning = sign(kService, "aws4_request")
    return kSigning
    
signature = sign(getSignatureKey('arn:aws:kms:...', "20150126", "us-east-1", "s3"), result)

def uploadToS3(fileName, fileContent):
    connection = S3Connection(AWS_ACCESS_KEY, AWS_SECRET_KEY,debug=2)
    bucket = connection.get_bucket(BUCKET_NAME)
    key = Key(bucket)
    key.key = fileName
    key.set_contents_from_string(fileContent, headers={'Authorization' : signature,'x-amz-server-side-encryption' : 'aws:kms','x-amz-server-side-encryption-aws-kms-key-id' : 'arn:aws:kms:...'})

uploadToS3(fileName=S3_FILE_PATH, fileContent=result)

I still get the following error in the response:

<Code>InvalidArgument</Code>
<Message>Requests specifying Server Side Encryption with AWS KMS managed keys require AWS Signature Version 4.</Message>
<ArgumentName>Authorization</ArgumentName>
<ArgumentValue>null</ArgumentValue>

Three things:

1. My "Authorization" header is ignored;
2. I doubt that I am calculating the signature correctly;
3. What do I need to do to use HTTPS? I thought that boto was already making requests in HTTPS.

Thanks,
Carl

[Thomas O'Dowd]

Hi Carl,

Ah KMS... Sorry I missed that part. I thought you were referring to
SSE-C which I have used and I know works. I haven't tried KMS yet.

> Three things:
> 1. My "Authorization" header is ignored;

Boto normally does the Authorization calculation for you so you
shouldn't have to set that header I believe nor do any signature
calculation yourself. Leave that to Boto.

> 2. I doubt that I am calculating the signature correctly;

See 1.

> 3. What do I need to do to use HTTPS? I thought that boto was already
> making requests in HTTPS.

yes. Unless you are explicitly setting the insecure on your connection
you should be using HTTPS. The error back from AWS would also indicate
this if this were the case.

The AWS error says that you need to use Signature V4 which looking at
the documentation for KMS is true always. I haven't used that myself but
a quick look at the code suggests that you might try setting the
environment variable "S3_USE_SIGV4=True".

Hope that helps. Maybe someone else has a better solution also.

Tom.

> 1. My "Authorization" header is ignored;
> 2. I doubt that I am calculating the signature correctly;
> 3. What do I need to do to use HTTPS? I thought that boto was

> --
> You received this message because you are subscribed to the Google
> Groups "boto-users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to boto-users+...@googlegroups.com.
> To post to this group, send email to boto-...@googlegroups.com.
> Visit this group at http://groups.google.com/group/boto-users.
> For more options, visit https://groups.google.com/d/optout.

[Carl Bolduc]

Thanks! Still having trouble though.

New code, letting boto do the signature:
--------------------------------------------------

from boto.s3.connection import S3Connection
from boto.s3.key import Key
import hmac
import hashlib
import binascii

import os

os.environ['S3_USE_SIGV4'] = 'True'

AWS_ACCESS_KEY = "***"
AWS_SECRET_KEY = "***"
BUCKET_NAME = "mybucket"
S3_FILE_PATH = "encrypted.txt"

result="la bibi da bibi dum"

def uploadToS3(fileName, fileContent):
connection = S3Connection(AWS_ACCESS_KEY, AWS_SECRET_KEY, host='s3.amazonaws.com', debug=2)

bucket = connection.get_bucket(BUCKET_NAME)
key = Key(bucket)
key.key = fileName

key.set_contents_from_string(fileContent, headers={'x-amz-server-side-encryption' : 'aws:kms','x-amz-server-side-encryption-aws-kms-key-id' : 'arn:aws:kms:...'})

uploadToS3(fileName=S3_FILE_PATH, fileContent=result)
--------------------------------------------------

What I changed:
1. Added os.environ['S3_USE_SIGV4'] = 'True'. After doing this, I got an error saying that I needed to provide a host parameter.
2. Added host='s3.amazonaws.com' in the connection.
3. Removed the signature stuff.

Now, I am getting a 400 Bad Request, the script fails on this line: bucket = connection.get_bucket(BUCKET_NAME)
--------------------------------------------------
Traceback (most recent call last):
File "TestServerSideEncryptionS3.py", line 24, in <module>
uploadToS3(fileName=S3_FILE_PATH, fileContent=result)
File "TestServerSideEncryptionS3.py", line 19, in uploadToS3
bucket = connection.get_bucket(BUCKET_NAME)
File "c:\Python27\lib\site-packages\boto\s3\connection.py", line 502, in get_bucket
return self.head_bucket(bucket_name, headers=headers)
File "c:\Python27\lib\site-packages\boto\s3\connection.py", line 549, in head_bucket
response.status, response.reason, body)
boto.exception.S3ResponseError: S3ResponseError: 400 Bad Request

The exact same code works if I remove the encryption related code.

Here is the header of the request:
--------------------------------------------------
HEAD / HTTP/1.1\r\nAccept-Encoding: identity\r\nx-amz-content-sha256: ***\r\nContent-Length: 0\r\nUser-Agent: Boto/2.34.0 Python/2.7.8 Windows/8\r\nHost: mybucket.s3.amazonaws.com\r\nX-Amz-Date: 20150127T145915Z\r\nAuthorization: AWS4-HMAC-SHA256 Credential=***/20150127/us-east-1/s3/aws4_request,SignedHeaders=host;user-agent;x-amz-content-sha256;x-amz-date,Signature=***\r\n\r\n
--------------------------------------------------

What's wrong with my request?

Carl

[Thomas O'Dowd]

Hi Carl,

What does the error response from AWS say? The XML should have some
reason for the 400 bad request at least which may clarify the next
step.

Tom.

[Carl Bolduc]

Hi Tom,

Unfortunately, there is nothing more. I even proxy the request to fiddler and the response is simply "HTTP/1.1 400 Bad Request" with an empty body.

Carl

[Carl Bolduc]

After I realised that get_bucket was failing, I lookup my bucket properties and realized that it's in Oregon. My working Java code uses a bucket located in US Standard. I switched to the US Standard bucket and got further. I am now getting a new error:

SignatureDoesNotMatch.

The request signature we calculated does not match the signature you provided. Check your key and signing method.

Since I have java code that can do what I am trying to achieve here with boto, I enabled logging to see the put request and compare it with the one done by boto.

Here they are:

Python
========

PUT /encrypted.txt HTTP/1.1\r\n
Accept-Encoding: identity\r\n
x-amz-content-sha256: ***\r\n
Content-Length: 20\r\n
Host: testjavaencryption.s3.amazonaws.com\r\n
Content-MD5: ***\r\n
x-amz-server-side-encryption-aws-kms-key-id: arn:aws:kms:us-east-1:***\r\n
Expect: 100-Continue\r\n
X-Amz-Date: 20150128T195418Z\r\n
Authorization: AWS4-HMAC-SHA256 Credential=***/20150128/us-east-1/s3/aws4_request,SignedHeaders=content-length;content-md5;content-type;expect;host;user-agent;x-amz-content-sha256;x-amz-date;x-amz-server-side-encryption;x-amz-server-side-encryption-aws-kms-key-id,Signature=***\r\n
x-amz-server-side-encryption: aws:kms\r\n
Content-Type: application/octet-stream\r\n
User-Agent: Boto/2.34.0 Python/2.7.8 Windows/8\r\n\r\n'

Java
======

PUT /hello_s3_sse_kms.txt HTTP/1.1[\r][\n]
x-amz-content-sha256: STREAMING-AWS4-HMAC-SHA256-PAYLOAD[\r][\n]
Content-Length: 194[\r][\n]
Host: testjavaencryption.s3-external-1.amazonaws.com[\r][\n]
x-amz-server-side-encryption-aws-kms-key-id: arn:aws:kms:us-east-1:**[\r][\n]
Expect: 100-continue[\r][\n]
[\r][\n]
HTTP/1.1 100 Continue[\r][\n]
[\r][\n]
15; chunk-signature=***[\r][\n]
la bibi da bibi dum[\r][\n]
[\r][\n]
0;chunk-signature=***[\r][\n]
X-Amz-Date: 20150128T195535Z[\r][\n]
Authorization: AWS4-HMAC-SHA256 Credential=***/20150128/us-east-1/s3/aws4_request, SignedHeaders=content-length;content-type;host;user-agent;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length;x-amz-server-side-encryption;x-amz-server-side-encryption-aws-kms-key-id, Signature=***[\r][\n]
x-amz-server-side-encryption: aws:kms[\r][\n]
Content-Type: application/octet-stream[\r][\n]
User-Agent: aws-sdk-java/1.9.6 Windows_8.1/6.3 Java_HotSpot(TM)_64-Bit_Server_VM/25.25-b02/1.8.0_25[\r][\n]
x-amz-decoded-content-length: 21[\r][\n]
Connection: Keep-Alive[\r][\n]

As you can see, there are a few differences.
1. x-amz-content-sha256 is different, Python request contains a hash while the Java request seems to contain a type of hashing.
2. Content-Length is much smaller with boto. I noticed that my Java code adds a metadata to the document with the length of the document, this is absent from the Python code.
3. Host is different, although I doubt that it matters.
4. Java is missing Content-MD5, but it works.
5. With Java, Expect contains the file data. It might be the case with Python if I wasn't getting an error with the signature.
6. The signature is different in the Authorization header (expected since Python signature is bad and the one from Java is good).

Why is the boto code not generating the correct singnature?

[Thomas O'Dowd]

Hi Carl,

Hopefully someone else can jump into this thread and give you more help.
I haven't yet looked at using KMS or using V4 signatures using Boto so I
don't know myself by looking at the request what to expect. It does
smell like Boto is not working properly though for your usecase. I'm
just not sure if the problem is how Boto is calculating the V4 signature
in general or a problem when it comes to KMS in particular.

One thing you could try is to upload your object using a V4 signature
without the KMS encryption. If that works then it would seem that the
problem is when KMS headers are used and if it doesn't work the problem
is more general and it would see that there is an issue with V4
signatures somewhere.

Tom.

[Carl Bolduc]

Hi Tom,

Confirmed, if I remove the kms headers but keep the env var S3_USE_SIGV4 set to True, it works.

I will report a bug.

Thanks for all your help.

Carl