Shrine + Uppy + direct upload to S3: can upload to cache but not permanent store

[jma...@gmail.com]

First of all, thank you so much for creating such a useful and well-documented software for developers.

I have followed documentations on Github and a few other places to try to get this basic function to work but cannot - I must be missing something obvious; I'd appreciate any pointers!

I'm trying to use Shrine with Uppy on the frontend to do a direct upload to S3.

I'm using Rails 6.0.0.rc1, Shrine 2.18.1, ruby 2.6.3.

Shrine initializer:

require 'shrine/storage/s3'

s3_options = {
  bucket:            'bucket-name',
  access_key_id:     ENV['AWS_ACCESS_KEY_ID'],
  secret_access_key: ENV['AWS_SECRET_ACCESS_KEY'],
  region:            'us-west-1',
}

Shrine.storages = {
  cache: Shrine::Storage::S3.new(public: true, prefix: 'cache', **s3_options),
  store: Shrine::Storage::S3.new(public: true, prefix: 'store', **s3_options)
}

Shrine.plugin :activerecord
Shrine.plugin :restore_cached_data # refresh metadata when attaching the cached file
Shrine.plugin :determine_mime_type

Shrine.plugin :presign_endpoint, presign_options: -> (request) do
  # Uppy will send these two query parameters
  filename = request.params['filename']
  type     = request.params['type']

  {
    content_disposition:  ContentDisposition.inline(filename),
    content_type:         type,
    content_length_range: 0..(10*1024*1024)
  }
end

models:

class ImageUploader < Shrine
  plugin :refresh_metadata
end

class Image < ApplicationRecord
  include ImageUploader::Attachment.new(:image)

  belongs_to :imageable, polymorphic: true
 ...
end

js:

    const uppy = Uppy({
      debug: true,
      restrictions: {
        maxFileSize: 3000000,
        maxNumberOfFiles: 5,
        allowedFileTypes: ['image/*']
      },
    })
      .use(Dashboard, {
        inline: false,
        trigger: '#open_uploader',
        showProgressDetails: true,
        note: 'Images only, up to 5 files, up to 3 MB',
        metaFields: [
          { id: 'name', name: 'Name', placeholder: 'file name' },
          { id: 'caption', name: 'Caption', placeholder: 'describe what the image is about' }
        ],
      })
      .use(AwsS3, {
        companionUrl: '/', // will call `GET /s3/params` on our app
      })
      .on('upload-success', (file, response) => {
        const uploadedFileData = JSON.stringify({
          id: file.meta.key.match(/cache\/(.+)/)[1], // remove the Shrine storage prefix
          storage: 'cache',
          metadata: {
            size:      file.size,
            filename:  file.name,
            mime_type: file.type,
          }
        })

        console.log(uploadedFileData)
      })

For development, I'm currently taking the output of `uploadedFileData` from the js console and copy pasting that to my Rails console.

Rails console:

image = Image.new(imageable: Portfolio.find(1), filename: "file.jpg", mimetype: "image/jpeg", size: 78653)
image.save

json = {"id":"a_long_string.jpg","storage":"cache","metadata":{"size":78653,"filename":"file.jpg","mime_type":"image/jpeg"}}.to_json # this is output from the js console

image.image = json
# This command returns error: "Shrine::Error ({} isn't valid uploaded file data)"

How should I be using Uppy's upload-success response here?

Note, if I do:

image.update(image_data: json)

My object will get saved, but I am not able to promote the image to permanent storage.

Thanks so much.

[Maxence M]

Hello,

I have made a medium post for direct to S3 uploads with Shrine and Uppy. 

At then end it explains how you can attach the newly uploaded file to your Rails model (pretty straightforward actually)

Maybe it can help

https://medium.com/@maxencemalbois/rails-multiple-direct-uploads-to-s3-with-shrine-and-uppy-with-live-versionning-thanks-to-an-ajax-6482b85dad2b

--
You received this message because you are subscribed to the Google Groups "Shrine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ruby-shrine...@googlegroups.com.
To post to this group, send email to ruby-...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ruby-shrine/0abc057a-0809-4ee1-9a1f-c733a297aa79%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Janko Marohnić]

That's a very strange error, I don't know what could have caused it. The JSON string you've shown looks completely fine. I've tried reproducing the error, but assigning that JSON string locally works fine for me (and that needs to work for direct upload flow to be possible):

require "shrine"

require "shrine/storage/file_system"

require "active_record"

ActiveRecord::Base.establish_connection(

  adapter: "sqlite3",

  database: ":memory:",

)

Shrine.storages = {

  cache: Shrine::Storage::FileSystem.new(Dir.tmpdir),

  store: Shrine::Storage::FileSystem.new(Dir.tmpdir),

}

Shrine.plugin :activerecord

Shrine.plugin :restore_cached_data # refresh metadata when attaching the cached file

Shrine.plugin :determine_mime_type

ActiveRecord::Base.connection.create_table :images do |t|

  t.text :image_data

end

class ImageUploader < Shrine

  plugin :refresh_metadata

end

class Image < ActiveRecord::Base

  include ImageUploader::Attachment.new(:image)

end

uploaded_file = Shrine.upload(StringIO.new, :cache)

image = Image.new

image.save

json = {"id":uploaded_file.id,"storage":"cache","metadata":{"size":78653,"filename":"file.jpg","mime_type":"image/jpeg"}}.to_json

image.image = json

image.image # => #<ImageUploader::UploadedFile:0x00007fe24bb1a9b0 @data={"id"=>"14bdb70ea6f12a51ff28b3c0d4262fb2", "storage"=>"cache", "metadata"=>{"size"=>0, "filename"=>"file.jpg", "mime_type"=>nil}}>

It would be great if you could create a self-contained script like above which reproduces the issue, because I don't know what could be wrong.

Kind regards,

Janko

--

[jma...@gmail.com]

Thank you Maxence and Janko for replying! Thanks Janko for taking the time to try to reproduce it. I think I will fork shrine and try debugging it that way.

To unsubscribe from this group and stop receiving emails from it, send an email to ruby-...@googlegroups.com.

[jma...@gmail.com]

I found the issue! I am using postgres and have the `image_data` column as jsonb with a default value '{}':

t.jsonb "image_data", default: "{}", null: false

I'm actually not too familiar with the nitty gritty of using jsonb, like if there should even be a default value for the column, and if that default should be '{}' or {}. Regardless, the jsonb column has served me fine until now.

That said, I was able to get Shrine's uploader to work with the json by making a small change in `def read` in attacher.rb:

      # Reads from the `<attachment>_data` attribute on the model instance.
      # It returns nil if the value is blank.
      def read
        value = record.send(data_attribute)
        convert_after_read(value) unless value.nil? || value.empty? || value == "{}" # <----
      end

Not pretty. Perhaps a better option for me is to not use jsonb and just use regular text instead.

On Tuesday, July 9, 2019 at 3:02:41 PM UTC-7, Janko Marohnić wrote:

To unsubscribe from this group and stop receiving emails from it, send an email to ruby-...@googlegroups.com.

[jma...@gmail.com]

I didn't expand too much on the actual problem in my previous reply --

The problem was that when Shrine was trying to read the value of image_data, it came across '{}', which is the default set in my db schema.

In this method:

      def read
        value = record.send(data_attribute)
        convert_after_read(value) unless value.nil? || value.empty? || value == "{}"

      end

`value` was returning '{}' (as a string), and the rest of the code was trying parse that. Eventually it gets to uploaded_file.rb's `def initialize(data)`, with `data` being {}:

      def initialize(data)
        raise Error, "#{data.inspect} isn't valid uploaded file data" unless data["id"] && data["storage"]

        @data = data
        @data["metadata"] ||= {}
        storage # ensure storage is registered
      end

It then errors out because data['id'] and data['storage'] are absent.

The correct behavior should be to ignore this initial value of '{}' altogether.

[jma...@gmail.com]

Sorry, I thought I had this figured out but I don't. I'm now having a problem where I can't get Shrine to save my image in permanent storage. In the Rails console, sometimes the following code execution results in an image being in permanent storage:

image = Image.find 1
json = {"id":"yyy.jpg","storage":"cache","metadata":{"size":230886,"filename":"photo.jpg","mime_type":"image/jpeg"}}.to_json

image.image = json
image.save

But, sometimes this results in the image being in cache:

> image.image_data = {"id":"510221eacb2cefe9b5f662abb434xxx.jpg","storage":"cache","metadata":{"size":529790,"filename":"Icon.jpg","mime_type":"image/jpeg"}}.to_json
> image.save
 => true
> image.image
 => #<ImageUploader::UploadedFile:0x00007f9bc4b09390 @data={"id"=>"510221eacb2cefe9b5f662abb434xxx.jpg", "storage"=>"cache", "metadata"=>{"size"=>529790, "filename"=>"Icon.jpg", "mime_type"=>"image/jpeg"}}>

As you can see, in the last output, the storage remains in "cache".

I haven't been able to pin down when the image gets saved to storage and when it remains in cache. Is there somewhere in the code I should dig further?

Thank you.

[jma...@gmail.com]

I think typing out my question helped me solve my own issue. I'm supposed to be doing `image.image = json`, and not `image.image_data = json`. I'm all good to go now with regards to saving in permanent storage.

[Janko Marohnić]

Glad you've figured it out. Shrine normally supports JSON(B) columns, but it doesn't support a default value of `{}`. So I would suggest that you remove that default value on the database level, rather than patching Shrine.

And correct that Shrine won't activate when you assign directly to the column attribute, it's designed to be ORM-agnostic, so it doesn't check ActiveRecord's dirty tracking or anything. Assigning directly to the column attribute is also known as a way to bypass Shrine if you don't like what it's doing.

Kind regards,

Janko

--
You received this message because you are subscribed to the Google Groups "Shrine" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ruby-shrine...@googlegroups.com.

To post to this group, send email to ruby-...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/ruby-shrine/69e09fd7-adb5-42b3-84a3-eaa4040ff35f%40googlegroups.com.

[jma...@gmail.com]

Thank you for the clarifications & all the work you do!