Error when Editing a Document: 's3.ObjectSummary' object has no attribute 'content_length'

[Tim Allen]

I'm getting an error when I attempt to edit an existing document through Wagtail; we're using Wagtail 1.6.7, and django-storages and boto3 to store Django media files on S3:

's3.ObjectSummary' object has no attribute 'content_length'

The document downloads just fine when I click on the document title, I've verified it exists in S3, and checked the wagtaildocs_document table in the database, which all look correct.

Here's the pertinent parts of the error:

Internal Server Error: /documents/13/investments-mutual-fund-turnover-spreadsheet.xlsx
AttributeError at /documents/13/investments-mutual-fund-turnover-spreadsheet.xlsx
's3.ObjectSummary' object has no attribute 'content_length'

Request Method: GET
Request URL: https://wrds-classroom.wharton.upenn.edu/documents/13/investments-mutual-fund-turnover-spreadsheet.xlsx
Django Version: 1.10.3
Python Executable: /usr/local/bin/python3 Python Version: 3.5.1

This is happening for all of our documents. Has anyone seen an error like this before, before I start going down the rabbit hole?

Here's the row from the database for the document:

 id |              title               |                            file                             |          created_at           | uploaded_by_user_id | collection_id



----+----------------------------------+-------------------------------------------------------------+-------------------------------+---------------------+---------------

 13 | Mutual Fund Turnover Spreadsheet | documents/investments-mutual-fund-turnover-spreadsheet.xlsx | 2016-10-24 11:27:46.656506-04 |                   5 |             1

Thanks in advance for any information!

[Tim Allen]

Just to follow up, I did try upgrading to v1.7, and am still getting the same error. Thanks.

[Matthew Westcott]

Hi Tim,

Can you share the full stack trace of the error, please?

I see that boto3 support in django-storages is pretty new (1.5.0, released on August 2nd) and probably hasn't had much testing with Wagtail yet. It's possible that the boto3 backend doesn't support retrieving the file size - Django storage backends aren't required to implement this, and the fact that Wagtail assumes they do is a known issue:
https://github.com/torchbox/wagtail/blob/51b4f0f53d93a69f3beaba67e3211cd001ff4087/wagtail/wagtaildocs/views/serve.py#L61
It may be that S3Boto3Storage is the first real-world instance we've encountered of a storage backend not supporting this.

Cheers,
- Matt

[Tim Allen]

Thanks Matt, the issue is definitely in the boto3 / django-storages layer of the stack, as you can see. Thanks for having a look, I'll continue to poke around when I get a chance, but figured I'd bring it up.

Traceback:  

File "/var/django/virtualenvs/wrds-classroom-prod/lib/python3.5/site-packages/django/core/handlers/exception.py" in inner

  39.             response = get_response(request)

File "/var/django/virtualenvs/wrds-classroom-prod/lib/python3.5/site-packages/django/core/handlers/base.py" in _legacy_get_response

  249.             response = self._get_response(request)

File "/var/django/virtualenvs/wrds-classroom-prod/lib/python3.5/site-packages/django/core/handlers/base.py" in _get_response

  187.                 response = self.process_exception_by_middleware(e, request)

File "/var/django/virtualenvs/wrds-classroom-prod/lib/python3.5/site-packages/django/core/handlers/base.py" in _get_response

  185.                 response = wrapped_callback(request, *callback_args, **callback_kwargs)

File "/var/django/virtualenvs/wrds-classroom-prod/lib/python3.5/site-packages/django/views/decorators/cache.py" in _cache_controlled

  43.             response = viewfunc(request, *args, **kw)

File "/var/django/virtualenvs/wrds-classroom-prod/lib/python3.5/site-packages/wagtail/wagtailadmin/decorators.py" in decorated_view

  24.             return view_func(request, *args, **kwargs)

File "/var/django/virtualenvs/wrds-classroom-prod/lib/python3.5/site-packages/wagtail/wagtailadmin/utils.py" in wrapped_view_func

  103.                 return view_func(request, *args, **kwargs)

File "/var/django/virtualenvs/wrds-classroom-prod/lib/python3.5/site-packages/wagtail/wagtaildocs/views/documents.py" in edit

  157.             filesize = doc.file.size

File "/var/django/virtualenvs/wrds-classroom-prod/lib/python3.5/site-packages/django/db/models/fields/files.py" in _get_size

  76.         return self.storage.size(self.name)

File "/var/django/virtualenvs/wrds-classroom-prod/lib/python3.5/site-packages/storages/backends/s3boto3.py" in size

  514.                 return entry.content_length

Exception Type: AttributeError at /cms/documents/edit/13/ Exception Value: 's3.ObjectSummary' object has no attribute 'content_length'

[Matt Davies]

Did you ever get a resolution to this Tim?

Maybe the conversation moved to stackoverflow, but I can't seem to find it there :-(

It's probably an issue with AWS or boto3, or a bit of both I suspect.

Matt

[Moritz Roth]

Hi Matt

I also stumbled upon this and I think it's a fairly straight-forward problem to fix.

The django-storages s3boto3 backend has an 'entries' dictionary with locally cached files for the S3 bucket. The type for these entries is ObjectSummary, which doesn't have a content_length attribute like the Object class, but instead a 'size'. content_length and size appear to be equivalent (although the documentation doesn't explicitly state that).

So I fixed the line in s3boto3.py that uses that attribute to use .size instead. No more crash in Wagtail documents and the file size appears to be correct also. 

I'll try finding some documentation about these to make sure that the fix is actually correct, and submit a pull request to django-storages. In the mean time you can try building a version of django-storages with the fix applied.

https://github.com/mroth23/django-storages/commit/f681cc970ce10ae4a604db3c6f7749087d6dec3d

Cheers

Moritz

[Matt Davies]

Moritz, I congratulate you on your detective skills, outstanding work fella 

Tell your boss to give you a pay rise, now, not next week

I'll definitely have a go at your fix.

It works most of the time though, intermittent, the best kind of mittent.

thanks for answering

On Thursday, 17 November 2016 21:29:58 UTC, Tim Allen wrote:

[Tim Allen]

Howdy! I actually found the same problem with Django Storages as Moritz, and issued a PR which has been sitting since December: https://github.com/jschneier/django-storages/pull/233

I'm in the process of switching over to Django S3 Storage. I've used other packages from the developer, Dave Hall, and he's very good at making packages that do one thing and do them well: https://github.com/etianen/django-s3-storage

If you don't need the other storage backends, and are only using S3, I'd recommend having a look. I checked to ensure the package is using "size" rather than "content_length", and it is.

--
You received this message because you are subscribed to a topic in the Google Groups "Wagtail support" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/wagtail/LGeaMo0AZKA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to wagtail+unsubscribe@googlegroups.com.
To post to this group, send email to wag...@googlegroups.com.
Visit this group at https://groups.google.com/group/wagtail.
To view this discussion on the web, visit https://groups.google.com/d/msgid/wagtail/7d79cc17-3db2-44b6-9331-4eb2799d1a18%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.