S3 Direct Multi-part Upload

[Rodrigo Rosenfeld Rosas]

Hi Janko, I'm about to start implementing our documents management module so I started to read more seriously about Shrine, Upper and S3 direct uploads.

You've covered a great example in your article regarding S3 direct uploads with Uppy:

https://twin.github.io/better-file-uploads-with-shrine-direct-uploads/

It even covers resumable uploads but if I understood it correctly, that means giving up on direct uploads to S3 since all parts would be uploaded to the tus server, that would then upload to S3.

I looked at the Uppy plugins and noticed that they seem to support multipart uploading to S3:

https://uppy.io/docs/aws-s3-multipart/

However it seems to require their bundled Companion server in order to support that (see the serverUrl parameter):

https://uppy.io/docs/companion/

At this point I'm not sure how it works. I wonder if their Companion server would send many pre-signed URLs so that each part would be uploaded directly to S3 or whether the Companion server would use a similar approach as you described in your article.

Do you know whether it's possible to send the parts directly to S3 by using a presigned URL for each part with Shrine and Uppy alone? If this is what happens when using the S3 Multipart plugin already, would it make sense to provide an end-point in Shrine to basically do the same as the Companion server does for that scenario?

I've also found this related discussions:

https://github.com/aws/aws-sdk-js/issues/1603

https://forums.aws.amazon.com/thread.jspa?threadID=254534

Any suggestions on how to implement multi-part direct upload to S3 using pre-signed urls would be greatly appreciated :)

Thanks in advance,

Rodrigo.

[Janko Marohnić]

Hey Rodrigo,

Yes, everything you said is correct. As Uppy's AwsS3Multipart plugin has been added fairly recently, I haven't yet properly investigated how to use it with Shrine.

It seems that one would need to re-write the same endpoints that Uppy Companion provides in Ruby. If you look at the implementations of the endpoints, it seems trivial, as each one is just calling a single method on the aws-sdk library. These endpoints are actually documented in the AwsS3Multipart plugin documentation, they match the createMultipartUpload(...), listParts(...), prepareUploadPart(...) etc. options that you can override (which by default call the Uppy Companion endpoints).

I should be able to come up with an implementation in a day or two, unless you would like to take a stab at it. I thought it would make sense to create a separate gem, e.g. shrine-uppy-s3_multipart (as I would prefer not to couple the Shrine gem itself to Uppy), that would provide a Rack application. Actually, it could also probably be a generic gem, e.g. uppy-s3_multipart, which could additionally ship with a Shrine plugin. How does that sound?

Kind regards,

Janko

--
You received this message because you are subscribed to the Google Groups "Shrine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ruby-shrine...@googlegroups.com.
To post to this group, send email to ruby-...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ruby-shrine/d97dc5e7-c0a9-47f3-b4be-93de053c7932%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.