Product thumbnail cache URL goes 'bad' after one pageview

[tsemi...@gmail.com]

For some reason yet unknown to me, a product's thumbnails disappear from view after just a single load of a page (both in Dashboard and Product Detail view) because the template seems to be truncating the URLs. This problem doesn't happen to statics at all.

To replicate:

1. Upload an image for a product using the dashboard. The image gets stored in the correct S3 bucket, media path,is accessible by all, and so on. (confirmed with django admin, AWS Console and sorl debug logging)

2. System returns the correct working URL https://s3.amazonaws.com/mybucket/media/cache/XXX/YYY/ZZZ.jpg that works on first page view request.

3. Reload the page. System returns broken URL https://s3.amazonaws.com/mybucket/cache/XXX/YYY/ZZZ.jpg (missing 'media' path)

4. Running python manage.py thumbnail cleanup, python manage.py thumbnail clear, restart gunicorn takes up back to step 2


Am I missing something? Should I be looking at sorl.thumbnail or oscar templates?




Some relevant settings and data:

Snippet from pip requirements
sorl-thumbnail==12.3
boto==2.38.0
django-storages==1.1.8
awscli==1.7.39

Snippet from settings.py
THUMBNAIL_DEBUG = True
THUMBNAIL_PRESERVE_FORMAT=True
OSCAR_IMAGE_FOLDER = 'images/products/'

AWS_STORAGE_BUCKET_NAME = 'mybucket'    
AWS_ACCESS_KEY_ID = 'mykey'
AWS_SECRET_ACCESS_KEY = 'mysecret'
AWS_S3_CUSTOM_DOMAIN = 's3.amazonaws.com/%s' % AWS_STORAGE_BUCKET_NAME

STATICFILES_LOCATION = 'static'
MEDIAFILES_LOCATION = 'media'
DEFAULT_FILE_STORAGE = 'utils.MediaRootS3BotoStorage'
STATICFILES_STORAGE = 'utils.StaticRootS3BotoStorage'
STATIC_ROOT = location('static')
MEDIA_ROOT = location('media')
STATIC_URL = "https://%s/%s/" % (AWS_S3_CUSTOM_DOMAIN, STATICFILES_LOCATION)
MEDIA_URL = "https://%s/%s/" % (AWS_S3_CUSTOM_DOMAIN, MEDIAFILES_LOCATION)

Entire utils.py
from storages.backends.s3boto import S3BotoStorage

StaticRootS3BotoStorage = lambda: S3BotoStorage(location='static')
MediaRootS3BotoStorage  = lambda: S3BotoStorage(location='media')

Entire thumbnail_kvstore db
                             key                              |                                                                 value
--------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------
 sorl-thumbnail||image||e14a50914a152da773157a34d30d4ab4      | {"storage": "storages.backends.s3boto.S3BotoStorage", "name": "images/products/ZZZ.jpg", "size": [2000, 3000]}
 sorl-thumbnail||image||5cab03b2ed5b48fccd2309abdddd8d7e      | {"storage": "storages.backends.s3boto.S3BotoStorage", "name": "cache/ee/25/ee25c9c57c544b0eadaa892c6bd1ce84.jpg", "size": [267, 400]}
 sorl-thumbnail||thumbnails||e14a50914a152da773157a34d30d4ab4 | ["5cab03b2ed5b48fccd2309abdddd8d7e"]
 sorl-thumbnail||image||8a767e549b0b784968c1e5aca445532a      | {"storage": "storages.backends.s3boto.S3BotoStorage", "name": "cache/60/84/608471a1fac28fb8b5c885e95d3d8797.jpg", "size": [103, 155]}

[Gaurav Wadhwani]

I had the exact same issue for both local and AWS deployment. I fixed it by using the following code:

DEFAULT_FILE_STORAGE = 'utils.MediaS3BotoStorage'
STATICFILES_STORAGE = 'utils.StaticS3BotoStorage'

AWS_ACCESS_KEY_ID = 'YOUR_KEY_GOES_HERE'
AWS_SECRET_ACCESS_KEY = 'SECRET_SSHHH'
AWS_STORAGE_BUCKET_NAME = 'mybucket'

S3_URL = 'http://%s.s3.amazonaws.com' % AWS_STORAGE_BUCKET_NAME
MEDIA_DIRECTORY = '/media/'
STATIC_DIRECTORY = '/static/'

STATIC_URL = "/files/"

MEDIA_URL = "/content/"

STATIC_ROOT = S3_URL + STATIC_DIRECTORY
MEDIA_ROOT = S3_URL + MEDIA_DIRECTORY
OSCAR_MISSING_IMAGE_URL = MEDIA_ROOT + 'image_not_found.jpg'

[tsemi...@gmail.com]

Indeed Gaurav!

I also reworked utils.py to be so (and serve from Amazon's CDN)

class StaticStorage(S3BotoStorage):
    """uploads to 'mybucket/static/', serves from 'cloudfront.net/static/'"""
    location = settings.STATICFILES_LOCATION

    def __init__(self, *args, **kwargs):
        kwargs['custom_domain'] = settings.AWS_CLOUDFRONT_DOMAIN
        super(StaticStorage, self).__init__(*args, **kwargs)

class MediaStorage(S3BotoStorage):
    """uploads to 'mybucket/media/', serves from 'cloudfront.net/media/'"""
    location = settings.MEDIAFILES_LOCATION

    def __init__(self, *args, **kwargs):
        kwargs['custom_domain'] = settings.AWS_CLOUDFRONT_DOMAIN
        super(MediaStorage, self).__init__(*args, **kwargs)


However, I noticed that the following:

OSCAR_MISSING_IMAGE_URL = MEDIA_ROOT + 'image_not_found.jpg'

Causes a S3 403 error when Oscar/Sorl tries to get a media cache URL that doesn't exist.

[Gaurav Wadhwani]

For the missing image, add the image_not_found.jpg to your media location in S3 manually and save it as a public image (everyone can see the image) (permissions: set everyone to open/view file).

It should be good to go then (that's how everything started to work for me).

—
Sent from Mailbox

[Gaurav Wadhwani]

Did you fix this? I started having the same issue now (though its just slightly different).

My project isn't appending the /media/ tag before the /cache/1a....... which results in a 404. Any word?

[tsemi...@gmail.com]

Nope I've not solved it.

I've actually left this lower down the backlog since listings are required to have a minimum of 1 photo each.

As for the missing /media/ URL, this occurred on my dev server once earlier on but a complete wipe and reinstall of the machine fixed it (I didn't have the time to dig in further).

[tsemi...@gmail.com]

Ok so I looked into my problem and here is what I found

>>> x = Product.objects.get(id=<ID of a product with images>)
>>> pprint.pprint(vars(Product.primary_image(x)))
{'_product_cache': <Product: PRODUCT NAME>,
 '_state': <django.db.models.base.ModelState object at 0x7ff3caf74590>,
 'caption': u'',
 'date_created': datetime.datetime(2015, 8, 4, 13, 57, 17, 257409, tzinfo=<UTC>),
 'display_order': 0,
 'id': 11,
 'original': u'<VALID FILE PATH OF IMAGE>',
 'product_id': 11}

>>> y = Product.objects.get(id=<ID of a product with no images>)
>>> pprint.pprint(vars(Product.primary_image(y)))
Error: argument must have __dict__ attribute
>>> pprint.pprint(Product.primary_image(y))
{'caption': '',
 'is_missing': True,
 'original': <oscar.apps.catalogue.abstract_models.MissingProductImage object at 0x7ff3caf74f50>}
 
--- Sorl debug log for X
DEBUG:sorl.thumbnail.base:Getting thumbnail for file [<VALID PRODUCT IMAGE PATH FOR X>] at [100x100]

--- Sorl debug log for Y
DEBUG:sorl.thumbnail.base:Getting thumbnail for file [<oscar.apps.catalogue.abstract_models.MissingProductImage object at 0x7f87e9c1e790>] at [100x100]

It seems that product.get_missing_image() ends up with the MissingProductImage object for 'original' but Sorl expects a filepath and fails to update thumbnail_kvstore. So any subsequent cache requests to S3/Cloudfront fails.

AFAIK, get_missing_image method is only used by Product so I overrode that to simply point to the right path instead of calling MissingProductImage()

On Saturday, 1 August 2015 18:28:47 UTC+8, Gaurav Wadhwani wrote:

[King Olami]

Were you able to fix it please.

[IL]

This was ages ago but yes, as I mentioned, I simply overwrote the abstract model's get_missing_image

    def get_missing_image(self):
        """
        Returns a missing image object.
        """
        # This class should have a 'name' property so it mimics the Django file
        # field.
        return MissingProductImage()

by forking the catalogue app with a hard coded default path to a placeholder image.

    def get_missing_image(self):
        """Return a missing image object.

        :return: path of the image
        :rtype: str
        """

        return 'media/image_not_found.jpg'