Unable to create thumbnails with django-storages, boto and S3

[EricSI]

I've ran into this bug that I have not been able to resolve and am hoping you may have some insight.

I have read through the issues on github and related questions here, and I can say that it is not an issue with PIL or S3 permissions as I can access easy-thumbnails directly to generate thumbnails and they get generated correctly.

I've created a very barebones basic django project with just enough setup needed to test django-filer.  Here are the requirements I'm running on Python 2.7.4:

Django==1.5.1
PIL==1.1.7
South==0.8.1
boto==2.9.6
dj-database-url==0.2.1
django-filer==0.9.4
django-mptt==0.5.5
django-polymorphic==0.5
django-storages==1.1.8
easy-thumbnails==1.3
psycopg2==2.5.1

The bug:
When selecting a new image to upload from the django-filer admin, the original image is correctly uploaded and successfully saved to my amazon s3 bucket.  When django-filer immediately tries to generate the icons using easy_thumbnails, it fails and this is the error message received:

Internal Server Error: /admin/filer/clipboard/operations/upload/
Traceback (most recent call last):
  File "/Users/EricRessler/virtual_envs/filers3/lib/python2.7/site-packages/django/core/handlers/base.py", line 115, in get_response
    response = callback(request, *callback_args, **callback_kwargs)
  File "/Users/EricRessler/virtual_envs/filers3/lib/python2.7/site-packages/django/views/decorators/csrf.py", line 77, in wrapped_view
    return view_func(*args, **kwargs)
  File "/Users/EricRessler/virtual_envs/filers3/lib/python2.7/site-packages/filer/admin/clipboardadmin.py", line 88, in ajax_upload
    'thumbnail': file_obj.icons['32'],
  File "/Users/EricRessler/virtual_envs/filers3/lib/python2.7/site-packages/filer/models/imagemodels.py", line 170, in icons
    raise e
AttributeError: 'NoneType' object has no attribute 'modified'

Note: This error does not occur when using default local file system storage, it only happens when using S3.


Here are the settings I'm currently using, although I've been testing by changing various values the result is always the same.

DEBUG = True
TEMPLATE_DEBUG = True
THUMBNAIL_DEBUG = True
FILER_DEBUG = True
FILER_ENABLE_LOGGING = True

DATABASES = {
    'default': dj_database_url.config(default='postgres://localhost:5432/filers3')
}

AWS_ACCESS_KEY_ID = env('AWS_ACCESS_KEY_ID', 'removed for security')
AWS_SECRET_ACCESS_KEY = env('AWS_SECRET_ACCESS_KEY', 'removed for security')
AWS_STORAGE_BUCKET_NAME = env('AWS_STORAGE_BUCKET_NAME', 'filer-test-bucket')
AWS_S3_SECURE_URLS = True
AWS_QUERYSTRING_AUTH = False
AWS_QUERYSTRING_EXPIRE = False

DEFAULT_FILE_STORAGE = 'storages.backends.s3boto.S3BotoStorage'
STATICFILES_STORAGE = 'storages.backends.s3boto.S3BotoStorage'

THUMBNAIL_DEFAULT_STORAGE = 'storages.backends.s3boto.S3BotoStorage'

FILER_STORAGES = {
    'public': {
        'main': {
            'ENGINE': 'storages.backends.s3boto.S3BotoStorage',
            'OPTIONS': {

            },
            'UPLOAD_TO': 'filer.utils.generate_filename.by_date',
        },
        'thumbnails': {
            'ENGINE': 'storages.backends.s3boto.S3BotoStorage',
            'OPTIONS': {
               
            },
        },
    },
}

I am at a complete loss as to why the thumbnails won't generate.  I've used pdb to follow the path of the code down to the point of failure:

This line triggers the exception handler and outputs the error message above

site-packages/filer/models/imagemodels.py: line 161

thumb = self.file.get_thumbnail(thumbnail_options)

That function calls thumbnail_exists()

site-packages/easy_thumbnails/files.py: line 389

if self.thumbnail_exists(filename):
     ...

Inside that function, these two lines are where the error is triggered:

site-packages/easy_thumbnails/files.py lines 445-446

thumbnail = self.get_thumbnail_cache(thumbnail_name)
thumbnail_modtime = thumbnail.modified

In the first line, thumbnail is getting back None from self.get_thumbnail_cache, which causes the next line to throw the error seen above.

If anyone has any ideas I could really use some help.  I have completely run out of ideas.

[EricSI]

I also installed 0.9.4a1 to see if that had any effect, but the issue still occurs.

The thing I don't get is that the settings are so simple that I can't see anything wrong with it, however if it's an error in filer, easy thumbnails, or storages, then I shouldn't be the only person experiencing this problem, yet I haven't seen anyone with the same problem.

[EricSI]

Some more information, which may be helpful.

So the exception is definitely happening in the easy_thumbnails code, but the reason why I still believe this is an issue in filer is because if I bypass filer altogether and just use easy thumbnails, everything works perfectly.  So I'm trying to figure out if filer is doing something that is modifying a value along the way that could be causing easy_thumbnails to fail.  So far I have not had any luck finding anything.

Here's the management command I created to test easy_thumbnails and it succeeded:

from django.core.management import BaseCommand
from easy_thumbnails.files import get_thumbnailer

class Command(BaseCommand):

    def handle(self, *args, **options):
        """
        Example of easy_thumbnails successfully creating and uploading
        a thumbnail image to S3 using the S3boto storage.
        """
        source = open('./bootstrap/media/test.jpg')
        thumbnailer = get_thumbnailer(source, relative_name='test_thumb.png')
        self.stdout.write('Thumbnailer instance: %s\n' % thumbnailer)

        thumb_options = {
            'size': (32, 32),
            'crop': True,
            'upscale': True,
        }

        thumbnail = thumbnailer.get_thumbnail(thumb_options)
        self.stdout.write('Thumbnail instance: %s\n' % thumbnail)

        # url only works for remote storage
        try:
            self.stdout.write('Thumbnail URL: %s\n' % thumbnail.url)
        except Exception, e:
            self.stderr.out('The URL property is only accessible when using remote storage\n')

        # path only works for local storage
        try:
            self.stdout.write('Thumbnail Path: %s\n' % thumbnail.path)
        except Exception, e:
            self.stderr.write('The path property is only accessible when using local storage\n')

The output:
Thumbnailer instance: test_thumb.png
Thumbnail instance: test_thumb.png.32x32_q85_crop_upscale.jpg
Thumbnail URL: https://filer-test-bucket.s3.amazonaws.com/test_thumb.png.32x32_q85_crop_upscale.jpg
The path property is only accessible when using local storage


I do have a temporary workaround in place, which I don't believe is viable solution because it doesn't actually solve the issue, it just catches the exception.

The exception occurs when calling thumbnail_exists() from get_thumbnail(). By putting a try/except around that call, the generate_thumbnail() call is allowed to proceed.  However i don't know what kind of repercussions that may cause.

[Stefan Foulis]

easy-thumbnails has some database tables it uses to cache information about remote images and thumbnails. Maybe those tables are missing. Try running syncdb/migrate just to be sure.

[Ryan Bagwell]

upgrading to django-filer==0.9.4 from 0.9.3 fixed this for me.

[Jorge Rodríguez-Flores Esparza]

Hello,
Sorry by reopening the thread.
I'm having the same issue.
The response of the POST is:

Traceback:
File "/home/jorro/.virtualenvs/filer/lib/python2.7/site-packages/django/core/handlers/base.py" in get_response
  115.                         response = callback(request, *callback_args, **callback_kwargs)
File "/home/jorro/.virtualenvs/filer/lib/python2.7/site-packages/django/views/decorators/csrf.py" in wrapped_view
  77.         return view_func(*args, **kwargs)
File "/home/jorro/.virtualenvs/filer/lib/python2.7/site-packages/filer/admin/clipboardadmin.py" in ajax_upload
  88.                     'thumbnail': file_obj.icons['32'],

Exception Type: KeyError at /admin/filer/clipboard/operations/upload/
Exception Value: '32'
Request information:
GET:
qqfile = u'fondo.jpg'

POST: No POST data

FILES: No FILE

Traceback:
File "/home/jorro/.virtualenvs/filer/lib/python2.7/site-packages/django/core/handlers/base.py" in get_response
  115.                         response = callback(request, *callback_args, **callback_kwargs)
File "/home/jorro/.virtualenvs/filer/lib/python2.7/site-packages/django/views/decorators/csrf.py" in wrapped_view
  77.         return view_func(*args, **kwargs)
File "/home/jorro/.virtualenvs/filer/lib/python2.7/site-packages/filer/admin/clipboardadmin.py" in ajax_upload
  88.                     'thumbnail': file_obj.icons['32'],

Exception Type: KeyError at /admin/filer/clipboard/operations/upload/
Exception Value: '32'
Request information:
GET:
qqfile = u'fondo.jpg'

POST: No POST data

FILES: No FILES data

COOKIES:
csrftoken = 's7Yqs18I8xt5dEs5e8rUmta28l4K33z4'
sessionid = 'end1mn3irj7r37qvyam9gqjok05d3iz1'

With this settings:

########## storages CONFIGURATION
if True:
    AWS_QUERYSTRING_AUTH = False
    AWS_STORAGE_BUCKET_NAME = 'xxx'
    AWS_ACCESS_KEY_ID = 'xxx'
    AWS_SECRET_ACCESS_KEY = 'xxx'
    AWS_S3_FILE_OVERWRITE = False
    STATICFILES_STORAGE = 'storages.backends.s3boto.S3BotoStorage'
    S3_URL = 'http://%s.s3.amazonaws.com/' % AWS_STORAGE_BUCKET_NAME
    STATIC_URL = S3_URL
    DEFAULT_FILE_STORAGE = 'storages.backends.s3boto.S3BotoStorage'
########## END storages CONFIGURATION

########## filer (common) CONFIGURATION
THUMBNAIL_PROCESSORS = (
    'easy_thumbnails.processors.colorspace',
    'easy_thumbnails.processors.autocrop',
    'filer.thumbnail_processors.scale_and_crop_with_subject_location',
    'easy_thumbnails.processors.filters',
)
########## END filer (common) CONFIGURATION

And this env:

Django==1.5.2
Pillow==2.1.0
boto==2.11.0
django-filer==0.9.5
django-mptt==0.5.5
django-polymorphic==0.5.1
django-s3-folder-storage==0.1
django-storages==1.1.8
easy-thumbnails==1.3

I've tried PIL and Pillow (both apparently had JPEG and ZLIB) support.

I've tried on sqlite and pgsql.

When it's configured to write in local works perfect, but on S3 only creates 'filer_public'.

Also I've read that maybe making reset to 'easy_thumbnail' fix some problems with cache tables, but doesn't works.

The only clue I have is this:

• try to upload an image
• shows progressbar
• on the clipboard: undefined undefined
• in this moment, reset easy_thumbnails
• F5 to reload
• show the thumbnail on the clipboard and the thumbnail of 32px is created on the bucket
• move from the clipboard or reload and nothing works again

Thanks.

Traceback:
File "/home/jorro/.virtualenvs/filer/lib/python2.7/site-packages/django/core/handlers/base.py" in get_response
  115.                         response = callback(request, *callback_args, **callback_kwargs)
File "/home/jorro/.virtualenvs/filer/lib/python2.7/site-packages/django/views/decorators/csrf.py" in wrapped_view
  77.         return view_func(*args, **kwargs)
File "/home/jorro/.virtualenvs/filer/lib/python2.7/site-packages/filer/admin/clipboardadmin.py" in ajax_upload
  88.                     'thumbnail': file_obj.icons['32'],

Exception Type: KeyError at /admin/filer/clipboard/operations/upload/
Exception Value: '32'
Request information:
GET:
qqfile = u'fondo.jpg'

POST: No POST data

FILES: No FILES data

COOKIES:
csrftoken = 's7Yqs18I8xt5dEs5e8rUmta28l4K33z4'
sessionid = 'end1mn3irj7r37qvyam9gqjok05d3iz1'

Traceback:
File "/home/jorro/.virtualenvs/filer/lib/python2.7/site-packages/django/core/handlers/base.py" in get_response
  115.                         response = callback(request, *callback_args, **callback_kwargs)
File "/home/jorro/.virtualenvs/filer/lib/python2.7/site-packages/django/views/decorators/csrf.py" in wrapped_view
  77.         return view_func(*args, **kwargs)
File "/home/jorro/.virtualenvs/filer/lib/python2.7/site-packages/filer/admin/clipboardadmin.py" in ajax_upload
  88.                     'thumbnail': file_obj.icons['32'],

Exception Type: KeyError at /admin/filer/clipboard/operations/upload/
Exception Value: '32'
Request information:
GET:
qqfile = u'fondo.jpg'

POST: No POST data

FILES: No FILES data

COOKIES:
csrftoken = 's7Yqs18I8xt5dEs5e8rUmta28l4K33z4'
sessionid = 'end1mn3irj7r37qvyam9gqjok05d3iz1'

Traceback:
File "/home/jorro/.virtualenvs/filer/lib/python2.7/site-packages/django/core/handlers/base.py" in get_response
  115.                         response = callback(request, *callback_args, **callback_kwargs)
File "/home/jorro/.virtualenvs/filer/lib/python2.7/site-packages/django/views/decorators/csrf.py" in wrapped_view
  77.         return view_func(*args, **kwargs)
File "/home/jorro/.virtualenvs/filer/lib/python2.7/site-packages/filer/admin/clipboardadmin.py" in ajax_upload
  88.                     'thumbnail': file_obj.icons['32'],

Exception Type: KeyError at /admin/filer/clipboard/operations/upload/
Exception Value: '32'
Request information:
GET:
qqfile = u'fondo.jpg'

POST: No POST data

FILES: No FILE

Traceback:
File "/home/jorro/.virtualenvs/filer/lib/python2.7/site-packages/django/core/handlers/base.py" in get_response
  115.                         response = callback(request, *callback_args, **callback_kwargs)
File "/home/jorro/.virtualenvs/filer/lib/python2.7/site-packages/django/views/decorators/csrf.py" in wrapped_view
  77.         return view_func(*args, **kwargs)
File "/home/jorro/.virtualenvs/filer/lib/python2.7/site-packages/filer/admin/clipboardadmin.py" in ajax_upload
  88.                     'thumbnail': file_obj.icons['32'],

Exception Type: KeyError at /admin/filer/clipboard/operations/upload/
Exception Value: '32'
Request information:
GET:
qqfile = u'fondo.jpg'

POST: No POST data

FILES: No FILE

Traceback:
File "/home/jorro/.virtualenvs/filer/lib/python2.7/site-packages/django/core/handlers/base.py" in get_response
  115.                         response = callback(request, *callback_args, **callback_kwargs)
File "/home/jorro/.virtualenvs/filer/lib/python2.7/site-packages/django/views/decorators/csrf.py" in wrapped_view
  77.         return view_func(*args, **kwargs)
File "/home/jorro/.virtualenvs/filer/lib/python2.7/site-packages/filer/admin/clipboardadmin.py" in ajax_upload
  88.                     'thumbnail': file_obj.icons['32'],

Exception Type: KeyError at /admin/filer/clipboard/operations/upload/
Exception Value: '32'
Request information:
GET:
qqfile = u'fondo.jpg'

POST: No POST data

FILES: No FILE

[Allen Tom]

I'm having exactly the same problem too -- the thumbnail is generated and served when I'm using local storage, but the thumbnail fails to be generated when I switch to S3. 

Has anyone figured this out?

Thanks,

Allen

[Zach Frank]

I am also having this problem. I am trying to deploy a Django app using Django-CMS and Django Filer on Heroku, using S3 storage. Here is what I keep getting in the heroku logs. When I try to upload an image it shows the loader then the thumb shows up undefined.

2013-09-05T21:07:39.253509+00:00 app[web.1]: /app/.heroku/python/lib/python2.7/site-packages/django/db/models/fields/__init__.py:827: RuntimeWarning: DateTimeField received a naive datetime (2013-09-05 16:07:39.002420) while time zone support is active.
2013-09-05T21:07:39.253509+00:00 app[web.1]: RuntimeWarning)
2013-09-05T21:07:39.253509+00:00 app[web.1]:
2013-09-05T21:07:39.348801+00:00 app[web.1]: Internal Server Error: /en/admin/filer/clipboard/operations/upload/
2013-09-05T21:07:39.348801+00:00 app[web.1]: Traceback (most recent call last):
2013-09-05T21:07:39.348801+00:00 app[web.1]: File "/app/.heroku/python/lib/python2.7/site-packages/django/core/handlers/base.py", line 115, in get_response
2013-09-05T21:07:39.348801+00:00 app[web.1]: File "/app/.heroku/python/lib/python2.7/site-packages/filer/admin/clipboardadmin.py", line 88, in ajax_upload
2013-09-05T21:07:39.348801+00:00 app[web.1]: response = callback(request, *callback_args, **callback_kwargs)
2013-09-05T21:07:39.348801+00:00 app[web.1]: File "/app/.heroku/python/lib/python2.7/site-packages/django/views/decorators/csrf.py", line 77, in wrapped_view
2013-09-05T21:07:39.348801+00:00 app[web.1]: return view_func(*args, **kwargs)
2013-09-05T21:07:39.348801+00:00 app[web.1]: 'thumbnail': file_obj.icons['32'],
2013-09-05T21:07:39.348801+00:00 app[web.1]: File "/app/.heroku/python/lib/python2.7/site-packages/filer/models/imagemodels.py", line 176, in icons
2013-09-05T21:07:39.348801+00:00 app[web.1]: return self._generate_thumbnails(required_thumbnails)
2013-09-05T21:07:39.349098+00:00 app[web.1]: thumb = self.file.get_thumbnail(opts)
2013-09-05T21:07:39.349098+00:00 app[web.1]: File "/app/.heroku/python/lib/python2.7/site-packages/filer/models/imagemodels.py", line 156, in _generate_thumbnails
2013-09-05T21:07:39.349098+00:00 app[web.1]: File "/app/.heroku/python/lib/python2.7/site-packages/easy_thumbnails/files.py", line 389, in get_thumbnail
2013-09-05T21:07:39.349098+00:00 app[web.1]: if self.thumbnail_exists(filename):
2013-09-05T21:07:39.349098+00:00 app[web.1]: File "/app/.heroku/python/lib/python2.7/site-packages/easy_thumbnails/files.py", line 446, in thumbnail_exists
2013-09-05T21:07:39.349098+00:00 app[web.1]: thumbnail_modtime = thumbnail.modified
2013-09-05T21:07:39.349098+00:00 app[web.1]: AttributeError: 'NoneType' object has no attribute 'modified'

Any help appreciated.

Z

[EricSI]

Sorry for not getting back to this thread sooner.  I am still unsure what is causing this, however I was able to get it working temporarily by forking the easy_thumbnails repo and adding a try/except around the thumbnail exists check.  If you'd like you can test using my fork, found here:

git+git://github.com/eressler/easy-thumbnails.git

However, later on in the project I tried pulling the latest easy_thumbnails from the main git repo instead of pip and it would appear it is fixed there:

git+git://github.com/SmileyChris/easy-thumbnails.git@bbba52bb11fdd63f80416b85780b4f9dc3e61bc8

Using that repo with that commit resolves the issue for my project.  Hopefully it works for everyone else as well.

[Zach Frank]

I'm a novice python-er. In my requirements file for my Heroku project can I do something like below then? Is there a way I can tell pip to use that code? Or do I have to do something special?

i.e. something like this?..

easy-thumbnails==git+git://github.com/SmileyChris/easy-thumbnails.git@bbba52bb11fdd63f80416b85780b4f9dc3e61bc8

I'll keep researching how in the mean time.

Thanks,

[Allen Tom]

Hi there novice python-er -- you can try running "pip install easy-thumbnails==dev" which seems to have pulled the latest dev version, with the S3 fix.

After installing the dev version, I now have django-filer running with the photos and thumbnails on S3. I've only been playing around with this for a few minutes, but it seems to be working now.

Also, thanks to EricSI for tracking this down for the group.

Allen

[Eric Ressler]

Yeah you can just add this to your requirements file:

git+git://github.com/SmileyChris/easy-thumbnails.git@bbba52bb11fdd63f80416b85780b4f9dc3e61bc8

without the easy_thumbnails== part

On Thu, Sep 5, 2013 at 3:16 PM, Zach Frank <zfrank...@gmail.com> wrote:

--
You received this message because you are subscribed to a topic in the Google Groups "django-filer" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/django-filer/w-u3NMVYZOk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to django-filer...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[EricSI]

I didn't know you could specify dev as the version number, that's good to know.  If that works, then awesome!  Glad that this is fixed in a newer version.

[Zach Frank]

Crazy! I had to redo my Heroku setup but the easy_thumbnails==dev in my requirements worked!!! Thanks for your help guys. Glad it was something simple. Love this app.

Z

[Jorge Rodríguez-Flores Esparza]

Working!!
Updating to dev branch fixed the problem.
Thanks for the help.