Error when attempting to download a file from S3 with server side encryption

[Peter Clark]

Thanks you Janko for this gem and your continued work on it.

I have the following code in an Uploader:

  plugin :upload_options, store: -> (io, context) do

    {

      sse_customer_algorithm: 'AES256',

      sse_customer_key: context[:record].sse_customer_key,

      sse_customer_key_md5: context[:record].sse_customer_key_md5

    }

  end

 

so that files uploaded to S3 are encrypted via a user specific key.

However, when I attempt to retrieve the file using either #download or #url, I get an error message that says: 

Aws::S3::Errors::InvalidRequest: The object was stored using a form of Server Side Encryption. The correct parameters must be provided to retrieve the object.

What parameters do I need provide and where/how do I provide them?

Thanks you in advance for your help.

[Janko Marohnić]

I think that when you use SSE encryption per S3 object, then you also have to pass those same parameters when downloading the object or generating a presigned URL. Any additional options passed to Shrine::UploadedFile#download and Shrine::UploadedFile#url are forwarded to the underlying storage, so you should be able to make it work by passing the SSE options:

  uploaded_file.download(

    sse_customer_algorithm: 'AES256',

    sse_customer_key: record.sse_customer_key,

    sse_customer_key_md5: record.sse_customer_key_md5

  )

or

  uploaded_file.url(

    sse_customer_algorithm: 'AES256',

    sse_customer_key: record.sse_customer_key,

    sse_customer_key_md5: record.sse_customer_key_md5

  )

where `uploaded_file` is the Shrine::UploadedFile object returned by the attachment attribute on the model.

Kind regards,

Janko

--
You received this message because you are subscribed to the Google Groups "Shrine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ruby-shrine...@googlegroups.com.
To post to this group, send email to ruby-...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ruby-shrine/2a627b19-e249-405e-996d-ca9f5e0254e7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Peter Clark]

Thank you for your quick reply Janko - I had the same thought, and previously tried this approach, however it returned a 

Aws::S3::Errors::BadRequest

Further investigation led me to this AWS documentation which, if I am reading correctly, states that I cannot generate a url to fetch an encrypted object but must instead provide these sse parameters in http request headers programmatically. I've looked through the shrine source to see how I might go about this - does shrine provide a mechanism to pass the necessary request headers?

Presigned URL and SSE-C

You can generate a presigned URL that can be used for operations such as upload a new object, retrieve an existing object, or object metadata. Presigned URLs support the SSE-C as follows:

• When creating a presigned URL, you must specify the algorithm using the x-amz-server-side​-encryption​-customer-algorithm in the signature calculation.

• When using the presigned URL to upload a new object, retrieve an existing object, or retrieve only object metadata, you must provide all the encryption headers in your client application.

  Note

  For non-SSE-C objects, you can generate a presigned URL and directly paste that into a browser, for example to access the data.

  However, this is not true for SSE-C objects because in addition to the presigned URL, you also need to include HTTP headers that are specific to SSE-C objects. Therefore, you can use the presigned URL for SSE-C objects only programmatically.

[Peter Clark]

Update: I was able to successfully download the encrypted file from S3 using the following, but I suspect there is a better way:

  s3_object = @document.file.storage.object(@document.file.id)

  file      = s3_object.get(@document.sse_options)

  body      = file.body.read

Passing the options through Shrine did not work, but passing them in via the underlying s3 object did.

[Janko Marohnić]

Hey Peter,

It took me a long time to figure out why forwarding :sse_* parameters to S3#download isn't working, and I finally managed to find the root case and push a fix to master. It should be working now.

The issue was that, in addition to the #get_object request that was being made (to which the :sse_* parameters were correctly being forwarded), there was also an implicit #head_object request being made to which the :sse_* parameters weren't being forwarded. And that's what was raising the exception.

I don't know yet when the next release is going to be, so for now you can pull from master.

Kind regards,

Janko

--

You received this message because you are subscribed to the Google Groups "Shrine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ruby-shrine...@googlegroups.com.
To post to this group, send email to ruby-...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/ruby-shrine/72d1192a-a3bb-48b8-84f8-1ea182d7a6b8%40googlegroups.com.

[Peter Clark]

Thanks for this fix Janko, glad you were able to figure it out.

Peter