SuspiciousOperation at filebrowser and s3

[Darren Ma]

I'm getting this SuspiciousOperation error when I swtich to s3boto storage and try add a new Blogpost or open the Media Library.

Exception Value:	Attempted access to '/app/backlash/media/uploads' denied.

I'm not sure why it's reading the local file path to begin with. All the other static seems to be loading fine.

I have created an 'uploads' in my bucket and set it to public.

I have also tried setting FILEBROWSER_DIRECTORY = MEDIA_URL + 'uploads/' 

which then gives me

Exception Value:	Attempted access to '/app/backlash/media/https:/myproject.s3.amazonaws.com/media/media/uploads' denied.

Any suggestions?

Sorry if this has been asked before. Didn't find a solution in the resources I looked at.

http://stackoverflow.com/questions/12535123/django-storages-and-amazon-s3-suspiciousoperation

http://community.webfaction.com/questions/6485/filebrowser-raises-a-suspiciousoperation-while-trying-to-access-my-static-folder

https://groups.google.com/forum/?fromgroups=#!searchin/mezzanine-users/SuspiciousOperation/mezzanine-users/HyZ83R0GVd8/GKmtDwE5ANUJ

[Sachin Shende]

Are you sure its not a permission problem?

[Darren Ma]

I've made all the folders in my bucket public. Still got the same issue.

Sometimes I wonder if all the pain of setting up S3 is worth it...

[Stephen McDonald]

This is an ongoing issue which is being tracked here:

https://github.com/stephenmcd/mezzanine/issues/530

Personally I don't use S3, so I don't have the available space or motivation to dive into this - desperately hoping someone who actually needs it can put in the work to resolve it.

--
You received this message because you are subscribed to the Google Groups "Mezzanine Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mezzanine-use...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
Stephen McDonald
http://jupo.org

[Darren Ma]

Thanks for the reply Stephen.

I knew s3 was going to be a dark road!

We are just starting a small news blog site and decided to host on Heroku because it's free until we get bigger.

Does anyone have any suggestions for alternative static/media storage on Heroku or do I have to change my hosting platform?

[Marcos Scriven]

Can you post your S3 boto settings in full please (obviously minus your secret key!)

I've not seen this error, but I'll see if I can replicate it.

[Darren Ma]

Hi Marcos,

I'm on 

• Django 1.5.1
  
• Mezzanine 1.4.5
  
• boto 2.8.0
  
• django-storages 1.1.8
  

My s3 settings are:

########## S3 STATIC CONFIGURATION
DEFAULT_FILE_STORAGE = 'backlash.storage.S3ProtectedStorage'
STATICFILES_STORAGE = 'storages.backends.s3boto.S3BotoStorage'
AWS_ACCESS_KEY_ID = get_env_setting('AWS_ACCESS_KEY_ID')
AWS_SECRET_ACCESS_KEY = get_env_setting('AWS_SECRET_ACCESS_KEY')
AWS_STORAGE_BUCKET_NAME = get_env_setting('AWS_STORAGE_BUCKET_NAME')
AWS_MEDIA_BUCKET_NAME = get_env_setting('AWS_MEDIA_BUCKET_NAME')
STATIC_URL = 'https://%s.s3.amazonaws.com/static/' % AWS_STORAGE_BUCKET_NAME
MEDIA_URL = 'https://%s.s3.amazonaws.com/media/' % AWS_MEDIA_BUCKET_NAME
# Used to make sure that only changed files are uploaded with collectstatic
AWS_PRELOAD_METADATA = True
AWS_LOCATION = 'static'
AWS_QUERYSTRING_EXPIRE = 7200
#turns off https for static files (necessary)
AWS_QUERYSTRING_AUTH = False
########## S3 STATIC CONFIGURATION

Here is the link to the storage class I'm using. http://pastebin.com/2XW0DPaQ (not sure perhaps one can use default s3botostorage but I had issues with that on previous deployments so that's why I use this one.)

Works with local filestorage but not with s3. That is why I'm curious to see your setup.

I should also mention that with my current setup I have an older version of mezzanine and django working(kinda, buggy media library).

Thanks for the effort!

[Marcos Scriven]

Hi Darren

I don't use that S3boto class (I use the default one), but I do think I found the reason for your issue:

http://stackoverflow.com/questions/12535123/django-storages-and-amazon-s3-suspiciousoperation

Let me know if that helps, if not I'll look into it further.

Marcos

[Darren Ma]

That post on SO doesn't offer a solution. It is where the error is bubbling up from but I don't seem to understand the s3boto well enough to make changes to fix this.

I also reverted to DEFAULT_FILE_STORAGE = 'storages.backends.s3boto.S3BotoStorage'

And set my FILEBROWSER_DIRECTORY = MEDIA_URL + 'uploads/'

Unfortunately no luck. Still get this error:

Attempted access to '/home/darren/Projects/heroku/backlash_project/backlash/media/https:/authentictrendmedia.s3.amazonaws.com/media/uploads' denied.

I tried playing around with _clean_name() like removing the replace('\\', '/')

and also messing with _normalize_name() by commenting out and just using pass.

No luck though.

[Marcos Scriven]

Oh....

Then I see the problem right away - it's literally trying to find a directory called:

'/home/darren/Projects/heroku/backlash_project/backlash/media/https:/authentictrendmedia.s3.amazonaws.com/media/uploads' denied.

On your machine. 

But I'm still not clear when exactly this error occurs for you? When trying to add a file in the media library? Or just when loading the media library at all?

Personally, I set my MEDIA_ROOT to empty string : ""

As MEDIA_ROOT doesn't make any sense when using S3 - it's an absolute file path.

[Darren Ma]

Hi Marco,

Yes you are right the MEDIA_ROOT was being set in my base.py setting. 

By editing these setting I can finally load and kind of use my Media Library!

########## S3 STATIC CONFIGURATION
MEDIA_ROOT = ''
# FILEBROWSER_DIRECTORY = MEDIA_URL + 'uploads/'

I still wouldn't say it's usable though. I tried to upload a file and it completes by everytime I am greeted with this error:

Request Method: GET
Request URL: http://127.0.0.1:8000/admin/media-library/browse/?ot=desc&o=date
Traceback:
File "/home/darren/.virtualenvs/bl/local/lib/python2.7/site-packages/django/core/handlers/base.py" in get_response
  115.                         response = callback(request, *callback_args, **callback_kwargs)
File "/home/darren/.virtualenvs/bl/local/lib/python2.7/site-packages/django/contrib/admin/views/decorators.py" in _checklogin
  17.             return view_func(request, *args, **kwargs)
File "/home/darren/.virtualenvs/bl/local/lib/python2.7/site-packages/django/views/decorators/cache.py" in _wrapped_view_func
  89.         response = view_func(request, *args, **kwargs)
File "/home/darren/.virtualenvs/bl/local/lib/python2.7/site-packages/filebrowser_safe/views.py" in browse
  108.         if fileobject.filetype == request.GET.get('filter_type', fileobject.filetype) and get_filterdate(request.GET.get('filter_date', ''), fileobject.date):
File "/home/darren/.virtualenvs/bl/local/lib/python2.7/site-packages/filebrowser_safe/base.py" in _date
  85.             self._date_stored = time.mktime(default_storage.modified_time(self.path).timetuple())
File "/home/darren/Projects/heroku/backlash_project/backlash/storages/backends/s3boto.py" in modified_time
  455.         return parse_ts_extended(entry.last_modified)
File "/home/darren/Projects/heroku/backlash_project/backlash/storages/backends/s3boto.py" in parse_ts_extended
  39.         rv = parse_ts(ts)
File "/home/darren/.virtualenvs/bl/lib/python2.7/site-packages/boto/utils.py" in parse_ts
  390.     ts = ts.strip()
Exception Type: AttributeError at /admin/media-library/browse/
Exception Value: 'NoneType' object has no attribute 'strip'

The REALLY strange part is that this error persists and I can't load my Media Library until a do a restart on the server. Stop the runserver and start it again for example. Then all of a sudden the Media Library loads and I can see the files uploaded there.

[Marcos Scriven]

Hi Darren

From the bottom of that stack trace looks to me like S3 isn't providing a timestamp (hence 'NoneType'), and by restarting I suspect you are forcing S3 to reload the files (and hence pickup a timestamp)

I would definitely recommend putting a debug point in here:

File "/home/darren/.virtualenvs/bl/local/lib/python2.7/site-packages/filebrowser_safe/base.py" in _date
  85.             self._date_stored = time.mktime(default_storage.modified_time(self.path).timetuple())

And follow into the call to modified_time to see why that's coming back as None for new files.

Marcos

[Darren Ma]

Marco you are a good man.

In the s3boto.py class there is this 

    def modified_time(self, name):
        name = self._normalize_name(self._clean_name(name))
        entry = self.entries.get(name)
        # only call self.bucket.get_key() if the key is not found
        # in the preloaded metadata.
        if entry is None:
            entry = self.bucket.get_key(self._encode_name(name))
        # Parse the last_modified string to a local datetime object.
        return parse_ts_extended(entry.last_modified)

The issue is that entry.last_modified was None which was breaking everything.

I've edited the def to the following:

    def modified_time(self, name):
        name = self._normalize_name(self._clean_name(name))
        entry = self.entries.get(name)
        # only call self.bucket.get_key() if the key is not found
        # in the preloaded metadata.
        if entry is None:
            entry = self.bucket.get_key(self._encode_name(name))
        # Parse the last_modified string to a local datetime object.
        if not entry.last_modified:
            try:
                entry.last_modified = datetime.datetime.now().strftime(ISO8601)
            except ValueError:
                entry.last_modified = datetime.datetime.now().strftime(ISO8601_MS)
        return parse_ts_extended(entry.last_modified)

It is working. Not sure if this will have negative consequences further down the line but I'm just happy it's working.

Thank you again for steering me in the right direction.

[Stephen McDonald]

Really enjoying this thread! I'd love to get to a point where we can say do X, Y and Z and S3 will just work.

Are the issues:

- Set MEDIA_URL to an empty string

- s3boto needs a patch for modified time

Is that it? Anything else?

Is there anything we could do in Mezzanine to monkey-patch our way into getting the modified time issue working?

[Marcos Scriven]

The MEDIA_ROOT needs to be set to empty string, rather than MEDIA_URL. You definitely need MEDIA_URL to be set to wherever you're serving your media from.

While the patch to S3 boto might work, I would worry about a couple of things:

• If the issue doesn't occur after restart (without the patch), it would be preferable to set the last modified time via the normal code path. If that means reloading the file from S3 to get the timestamp, so be it. Much preferable to get the timestamp according to S3, than just putting one in
• Timezones/daylight savings - I'd use datetime.datetime.utcnow().replace(tzinfo=utc) (or whatever timezone you want). Even if you're not working across timezones, you can get burnt twice a year when the clocks change over. If you happen to have a cron job running (for example), that depends on the last modified timestamp, you could be caught out.

I didn't actually experience this timestamp issue - so perhaps it's a new bug (or an old one). I'll check what versions I have.

The only issues I have with it at the moment are:

• Media Library is very slow indeed - I rather suspect that's due to it trying to pickup meta data like file sizes and timestamps, when really all you want is a filename and a thumbnail. I'd have to look and see if that's Mezzanine or FIlebrowser (I suspect the latter)
• With the thumbnail thing in mind, I found that Mezzanine tries to write the thumbnail directly to the local filesystem, rather than using default_storage (hence this topic https://groups.google.com/forum/?fromgroups=#!topic/mezzanine-users/WGVaNhD5vRs). Initially I thought we could use an in memory file to save the PIL output, but looking again I don't really see the need to save it anywhere other than default storage at all, since it's only ever served from MEDIA_URL?

Thanks

Marcos

[Stephen McDonald]

Yes I was thinking the same thing with the change you proposed - if we can have the thumbnail written directly via the storage api, then everything should just work.

Thanks for all your help in troubleshooting this.

 

Thanks

Marcos

--
You received this message because you are subscribed to the Google Groups "Mezzanine Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mezzanine-use...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Darren Ma]

So far using storages for S3 I'd recommend

MEDIA_ROOT = ''

FILEBROWSER_DIRECTORY does not need to be set.

You should make your media/uploads folder in your bucket and make them public.

As for the little timestamp issue I've been having I'll try see how to do it better and get back to you all if I ever get there.

[Darren Ma]

Sorry to harp on.

Was going to submit an issue to django-storages regarding this timestamp issue but I believe it's already been placed.

https://bitbucket.org/david/django-storages/issue/163/s3boto-modified_time-fails-for-new-file

Cheers

[Stephen McDonald]

Awesome - I just added a link back to this thread in that issue.

[Marcos Scriven]

Ah... well spotted!

Looks like it is indeed a new issue. Was introduced here a couple of weeks ago:

 

https://bitbucket.org/david/django-storages/commits/a3b2540a10a99e7332fa4ba2a9cd9a262a9a63f9

Whereas I've not updated since Jan (version 1.1.6)

Marcos

[Flavio Barros]

I'm still having this problem. Any solutions?

[Flavio Barros]

I Tryed the solution here but is not woking yet.

Flavio Barros Doutorando, Unicamp w:www.flaviobarros.net

Meu Blog Handwritten digit recognition – Part1

"In every CHOICES that we choose, There's always a RISK; But always remember that there's also a chance" - Kent Solatorio Lopez

Get a signature like this: Click here!

--
You received this message because you are subscribed to the Google Groups "Mezzanine Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mezzanine-use...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Tom Longson]

I'm having the ts.strip error still, tried using django-storages-redux, but ran into another error:

ImproperlyConfigured at /admin/media-library/browse/

Error finding Upload-Folder. Maybe it does not exist?

Looking for a solution still.

Tom

[Daniel Blasco]

Hi,

A different way to deal with this problem by overriding modified_time function, which actually fetches the object again from storage if modified_time is None:

from storages.backends.s3boto import S3BotoStorage, parse_ts_extended

class MyS3BotoStorage(S3BotoStorage):

    def modified_time(self, name):
        name = self._normalize_name(self._clean_name(name))
        entry = self.entries.get(name)
        # only call self.bucket.get_key() if the key is not found

        # in the preloaded metadata or if modified time stamp is empty.
        if entry is None or not entry.last_modified:
            entry = self.bucket.get_key(self._encode_name(name))
            self._entries[name] = entry

        # Parse the last_modified string to a local datetime object.
        return parse_ts_extended(entry.last_modified)

And then reference this new class from STATICFILES_STORAGE and DEFAULT_FILE_STORAGE of your settings.

By the way, it looks like the original project in bitbucket https://bitbucket.org/david/django-storages is not being maintained. Is there any fork or alternatives?

[Derek Adair]

Django Storages Redux is now on github.  I think they even took over the pypi namespace.