Media Library thumbnails backed by S3, and running on GAE

[Marcos Scriven]

Hi

I appreciate this might be a relatively rare scenario, but I'm running Mezzanine/Django on Google App Engine, backed with a MySQL instance on Google Cloud, and using S3 for uploads (not static)

Just a little thing really, the thumbnail generation on line 327 (of latest version in GitHub at time of writing) of mezzanine_tags.py assumes access to the local file system:

 ImageFile.MAXBLOCK = image.size[0] * image.size[1]

    try:

        image = ImageOps.fit(image, (width, height), Image.ANTIALIAS)

        image = image.save(thumb_path, filetype, quality=quality, **image_info)

        # Push a remote copy of the thumbnail if MEDIA_URL is

        # absolute.

        if "://" in settings.MEDIA_URL:

            with open(thumb_path, "r") as f:

                default_storage.save(thumb_url, File(f))

    except Exception:

        # If an error occurred, a corrupted image may have been saved,

        # so remove it, otherwise the check for it existing will just

        # return the corrupted image next time it's requested.

        try:

            os.remove(thumb_path)

        except Exception:

            pass

        return image_url

    return thumb_url

This of course fails on GAE, and the subsequent call to save to default storage then never happens. This doesn't cause much of an issue, except it's going to be slower in the long run as thumbnails never get cached.

I do have my own thumbnail generation code of sorts, and it uses a little snippet I got somewhere on StackOverflow:

    thumb_io = StringIO.StringIO()

    img.save(thumb_io, format='JPEG')

    # Create a new Django file-like object to be used in models as ImageField using

    # InMemoryUploadedFile.  If you look at the source in Django, a

    # SimpleUploadedFile is essentially instantiated similarly to what is shown here

    thumb_file = InMemoryUploadedFile(thumb_io, None, 'foo.jpg', 'image/jpg',

                                      thumb_io.len, None)

 This allows 'saving' the result of PIL operations to an in memory file, ready to be saved straight to one's default storage, without the need to store it locally first. Is that something that could be put into Mezzanine?

Thanks

Marcos

[Stephen McDonald]

Assuming it still achieves the same result for local storage backends, this sounds like a very worthwhile change - could you set up a pull request?

Marcos

--
You received this message because you are subscribed to the Google Groups "Mezzanine Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mezzanine-use...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
Stephen McDonald
http://jupo.org

[Nathan Keller]

+1 this would be useful to me, and I didn't quite follow the solution mentioned above. 

[Brad Bode]

Marcos, 

Your proposal would be very useful for me as well. What is the status of this? I am using S3 as well, but on Amazon Elastic Beanstalk. 

Stephen, Mezzanine rocks. There are definitely a few sticking points when deploying on Amazon EC2 / Elastic Beanstalk + S3, but most of them I have overcome. The biggest issue is the thumbnail template tag (Not for reasons that are your fault). For the sake of info, here is the issue:

Amazon EB and EC2 uses an interesting app deployment mechanism. Each time you update your app it increments a number in a directory, deploys your app into that directory, and starts the server. 

For instance, if you go with out of the box Mezzanine settings your static and media content goes here:

/opt/python/bundle/1/app/static/media

On subsequent deploys the new deploy will go here:

/opt/python/bundle/2/app/static/media

Note the incremental number change. 

So, the first issue is that the uploaded media disappears with each deploy because it is IN the app directory. The solution was to use django storages with S3 (and help from s3_folder_storage). That fixes the issue for uploading media and not have it be erased across deploys. 

If you didn't use S3, you could write your own script that runs on each deploy and copies over previous uploaded images. No fun and quite a patch. 

So that leads me to the final problem. Thumbnails does not work with S3 (you already know this). I understand the issue, but is there any other workaround? I suppose I could monkey patch the tag to do something completely different and upload the thumbnail to S3. 

Any thoughts on the issue?