Multiple Renditions error using S3

[Michael Godshall]

I have Wagtail running on Heroku using gunicorn, gevent, Postgres and S3 as the storage backend for media files.  I'm running into an issue every couple days where multiple Renditions are being created for an image.  I believe it's an issue with new images that are added and then accessed for the first time via the "image" templatetag to be resized.  Instead of failing silently it causes an internal server error when a page with that image is loaded:

MultipleObjectsReturned: get() returned more than one Rendition -- it returned 2!

In order to fix the error, I have to delete the image and upload it again.

Any direction on how to solve this would be appreciated.

Here's the full traceback:

Traceback (most recent call last):

 File "/app/.heroku/python/lib/python2.7/site-packages/django/core/handlers/base.py", line 137, in get_response
   response = response.render()

 File "/app/.heroku/python/lib/python2.7/site-packages/django/template/response.py", line 105, in render
   self.content = self.rendered_content

 File "/app/.heroku/python/lib/python2.7/site-packages/django/template/response.py", line 82, in rendered_content
   content = template.render(context)

 File "/app/.heroku/python/lib/python2.7/site-packages/django/template/base.py", line 140, in render
   return self._render(context)

 File "/app/.heroku/python/lib/python2.7/site-packages/django/template/base.py", line 134, in _render
   return self.nodelist.render(context)

 File "/app/.heroku/python/lib/python2.7/site-packages/django/template/base.py", line 840, in render
   bit = self.render_node(node, context)

 File "/app/.heroku/python/lib/python2.7/site-packages/django/template/base.py", line 854, in render_node
   return node.render(context)

 File "/app/.heroku/python/lib/python2.7/site-packages/django/template/loader_tags.py", line 123, in render
   return compiled_parent._render(context)

 File "/app/.heroku/python/lib/python2.7/site-packages/django/template/base.py", line 134, in _render
   return self.nodelist.render(context)

 File "/app/.heroku/python/lib/python2.7/site-packages/django/template/base.py", line 840, in render
   bit = self.render_node(node, context)

 File "/app/.heroku/python/lib/python2.7/site-packages/django/template/base.py", line 854, in render_node
   return node.render(context)

 File "/app/.heroku/python/lib/python2.7/site-packages/django/template/loader_tags.py", line 62, in render
   result = block.nodelist.render(context)

 File "/app/.heroku/python/lib/python2.7/site-packages/django/template/base.py", line 840, in render
   bit = self.render_node(node, context)

 File "/app/.heroku/python/lib/python2.7/site-packages/django/template/base.py", line 854, in render_node
   return node.render(context)

 File "/app/.heroku/python/lib/python2.7/site-packages/django/template/defaulttags.py", line 305, in render
   return nodelist.render(context)

 File "/app/.heroku/python/lib/python2.7/site-packages/django/template/base.py", line 840, in render
   bit = self.render_node(node, context)

 File "/app/.heroku/python/lib/python2.7/site-packages/django/template/base.py", line 854, in render_node
   return node.render(context)

 File "/app/.heroku/python/lib/python2.7/site-packages/django/template/defaulttags.py", line 305, in render
   return nodelist.render(context)

 File "/app/.heroku/python/lib/python2.7/site-packages/django/template/base.py", line 840, in render
   bit = self.render_node(node, context)

 File "/app/.heroku/python/lib/python2.7/site-packages/django/template/base.py", line 854, in render_node
   return node.render(context)

 File "/app/.heroku/python/lib/python2.7/site-packages/wagtail/wagtailimages/templatetags/wagtailimages_tags.py", line 55, in render
   rendition = image.get_rendition(self.filter)

 File "/app/.heroku/python/lib/python2.7/site-packages/wagtail/wagtailimages/models.py", line 146, in get_rendition
   focal_point_key=None,

 File "/app/.heroku/python/lib/python2.7/site-packages/django/db/models/manager.py", line 151, in get
   return self.get_queryset().get(*args, **kwargs)

 File "/app/.heroku/python/lib/python2.7/site-packages/django/db/models/query.py", line 313, in get
   (self.model._meta.object_name, num))

MultipleObjectsReturned: get() returned more than one Rendition -- it returned 2!

[Michael Godshall]

I also added this to a related issue on github: https://github.com/torchbox/wagtail/issues/312

[Karl Hobley]

Are you using a custom image model?

If so, make sure you have put "focal_point_key" into the unique_together setting on your Rendition model

[Karl Hobley]

Looking a bit closer at the stack trace, it appears that Django is requesting a rendition with "focal_point_key = None" which means that the unique_together constraint on Rendition is not working or not present.

What database backend are you using?

If you're using a custom Image model, does you rendition model have a unique_together constraint like this:
https://github.com/torchbox/wagtail/blob/master/wagtail/wagtailimages/models.py#L375-L378

Thanks,

Karl

[Michael Godshall]

I am using the default image model for wagtail.

    image = models.ForeignKey(

        'wagtailimages.Image',

        null=True,

        blank=True,

        on_delete=models.SET_NULL,

        related_name='+'

    )

I looked at the south migrations for wagtailimages and noticed 0003_focal_point_fields.py which adds the unique constraint.  I confirmed that the migration has been applied successfully to the Rendition table.  I am using Postgres.

[sachit adhikari]

I am facing exactly the same issue. My development and deployment is stack is same as yours. Is there any solution for this?

[Michael Godshall]

I have not seen this error in over a week, but there also hasn’t been any new content added to the site in that time so unfortunately I don’t have anything new to report.

Are you calling the {% image %} tag multiple times for the same image and size in the same template?  It seems like it would be possible for wagtail to create multiple renditions in this scenario.  We updated our code to only call an image at a particular size one time in the template, so that may be helping as well.  I have not been able to confirm that this fixed the problem, but I’m curious if your code is doing the same thing.

-- 
You received this message because you are subscribed to a topic in the Google Groups "Wagtail" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/wagtail/Ts_ytEJbQmw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to wagtail+u...@googlegroups.com.
To post to this group, send email to wag...@googlegroups.com.
Visit this group at http://groups.google.com/group/wagtail.
To view this discussion on the web, visit https://groups.google.com/d/msgid/wagtail/1cc7686b-fc79-44ae-a167-09de1cdebad3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Michael Godshall]

I posted a comment earlier but it seems it wasn't saved.

I am still running into this issue, and it seems to only be an problem when a rendition is requested for an image that was recently added.  

After further investigation, I believe I have identified the source of the problem.  I added my comments to the github issue, but I will paste them below for convenience.

https://github.com/torchbox/wagtail/issues/312#issuecomment-56438146

I believe the issue is related to the focal_point_key field. The current definition looks like this:

focal_point_key = models.CharField(max_length=255, null=True, editable=False)

I suspect that the unique_together constraint is not working correctly with null=True for this field. Instead, it should be set to blank=True so that the default value will be stored as an empty string rather than null.

focal_point_key = models.CharField(max_length=255, blank=True, editable=False)

In addition, the Django docs suggest that null=True should not be used for CharFields, so this probably needs to be changed either way (https://docs.djangoproject.com/en/1.7/ref/models/fields/#null). 

Here is some related reading:

http://stackoverflow.com/a/17510291
http://stackoverflow.com/a/20330228

On Thursday, August 21, 2014 9:54:44 AM UTC-7, Michael Godshall wrote:

[Joss Ingram]

We've just upgraded to 0.8.3, and now we're getting this on our live site!!!

MultipleObjectsReturned at /images/bGMgEgXsRDhVT-Xeq1ytEVuQxoI=/974/fill-450x341/

get() returned more than one Filter -- it returned 3!

Request Method:	GET
Request URL:	http://foxtailprod.southwales.ac.uk/images/bGMgEgXsRDhVT-Xeq1ytEVuQxoI=/974/fill-450x341/
Django Version:	1.6.5
Exception Type:	MultipleObjectsReturned
Exception Value:	get() returned more than one Filter -- it returned 3!
Exception Location:	/usr/local/lib/python2.7/dist-packages/django/db/models/query.py in get, line 313
Python Executable:	/usr/bin/python
Python Version:	2.7.6
Python Path:	['/var/www/django/foxtail', '/usr/local/bin', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-x86_64-linux-gnu', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/usr/local/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages', '/var/www/django/foxtail/foxtail']
Server time:	Wed, 26 Nov 2014 16:22:03 +0000

On Thursday, 21 August 2014 17:54:44 UTC+1, Michael Godshall wrote:

[Joss Ingram]

We've rolled back to 0.7 again now! Any ideas on this? We're stuck on 0.7 now!

[Matthew Westcott]

Hi Joss,
It looks like this error isn't specific to 0.8.3 - it was previously reported back in June, although we haven't been able to replicate it ourselves:
https://github.com/torchbox/wagtail/issues/312 (NB everything after the second message refers to a similar-looking but unrelated bug that was fixed in 0.7)

Taking a fresh look, though, I think I've spotted the culprit - we're missing a unique constraint on the filter's 'spec' field. We'll add a fix for this in the next release, but in the meantime, running the following command on the postgres command line (before or after upgrading to 0.8.3) should hopefully prevent this happening again:

ALTER TABLE wagtailimages_filter ADD CONSTRAINT filter_spec_uniq UNIQUE (spec);

- Matt

[Joss Ingram]

Ok, Thanks Matt, we'll try this temp fix tomorrow on our staging server.

[Joss Ingram]

Hi Matt,

if we run this command  ALTER TABLE wagtailimages_filter ADD CONSTRAINT filter_spec_uniq UNIQUE (spec); 

is this only going to fix new records that are added, what happens to the existing duplicate values? will this command fail without a novalidate option?

I'm not that familiar with postgres, just want to check first.

Joss

On Wednesday, 26 November 2014 17:21:56 UTC, Matthew Westcott wrote:

[Matthew Westcott]

Hi Joss,
I'm presuming you restored a backup of the database when you rolled back to 0.7 - if that's the case, and the site appears to be working correctly now, you can be reasonably confident that there are no duplicate values - duplicate values are (from the available evidence...) a rare occurrence caused by an unusual race condition, and will tend to fail loudly when they do happen.

Running the ALTER TABLE command is safe either way - if there are any duplicates, it will just fail with an error, without making any changes.

If it does fail, you can discover the duplicates by running

SELECT * FROM wagtailimages_filter WHERE spec IN (SELECT spec FROM wagtailimages_filter GROUP BY spec HAVING COUNT(id) > 1);

which will then return something like:

id | spec
----+-----------
10 | width-100
13 | width-100
(2 rows)

You can then remove the duplicates manually with a command like:
UPDATE wagtailimages_rendition SET filter_id = 10 WHERE filter_id = 13;
DELETE FROM wagtailimages_filter WHERE id = 13

Cheers,
- Matt

> --
> You received this message because you are subscribed to the Google Groups "Wagtail" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to wagtail+u...@googlegroups.com.

> To post to this group, send email to wag...@googlegroups.com.
> Visit this group at http://groups.google.com/group/wagtail.

> To view this discussion on the web, visit https://groups.google.com/d/msgid/wagtail/e52a9db7-36a3-4327-8f81-c2c4516c51ad%40googlegroups.com.

[Joss Ingram]

Matt,

Just to let you know, that SQL did the trick, and we have 0.8.3 in production now

Thanks

Joss