static expire headers

[Niphlod]

hello everybody...
For w2p_tvseries I'd like to set an expire header for banners etc.
Usually my apps where behind a real webserver, so with rewrite rules one can always intercept /static/ requests and don't pass them along to web2py.

For w2p_tvseries, though, I think that for the 70% of the time the app will be fired when requested and shut down, as a desktop app, so users will benefit from rocket only.
Web2py is a little conservative on that matter: I know that between requests a file could change so it is better always to check it for modifications but hey, the 99% of the time the static folder remains the same for a loooong time.

I see 2 options with current web2py:
1) move static files to uploads, tune up the standard download function and add an expire header there
2) create another supersmall app without models, only one controller serving images with response.stream and adding the expire header

1 is bad because modules will be reloaded each time for the relatively stupid image serving, 2 is bad because there will be 2 app to install to web2py to make 1 worth using :D

Is anyone came up with a solution ? it would be wonderful if with routes for a particular folder one could add a header to tune up a little bit the standards ...

BTW: I'm thinking about setting an expire header of 1 hour to the static folder, it could mean for subsequent requests something like 50 less requests for loading a page.
I know that web2py's stream_file_or_304_or_206 is smart enough to return a 304, but it's an occupied thread nonetheless os.stating() a file.

BTW2 : no flame wars on "it's bad to set expire headers in the future". We can use cache() to fine-tune many things planning carefully the "timeout",  but not static files, think this as another use of cache ;-)

[Anthony]

1) move static files to uploads, tune up the standard download function and add an expire header there
2) create another supersmall app without models, only one controller serving images with response.stream and adding the expire header

1 is bad because modules will be reloaded each time for the relatively stupid image serving

I assume you mean models will be re-executed for each static file request in case #1. To avoid that, perhaps you could add some logic to serve static files at the beginning of the first model file (or create a model file for this purpose that executes first) -- that way, you can serve the static files before any of the models are executed.

Anyway, I agree it would be nice to have more control over how static files are served, though as you noted, in production this typically isn't handled by web2py anyway.

Anthony

[Niphlod]

yes, I meant models

do you mean something like

models/00.py

with in that

def servstatic():
     .......
     return response.stream(file)

?

so, having a model "return" will skip all models and controller/function execution ?

[Massimo Di Pierro]

I model can stop the control flow of execution with

raise HTTP(....)

not with return

[Anthony]

No, you can't return from a model, but you can raise an HTTP exception to abort the models and return immediately -- something like:

import os
if request.controller == 'w2pstatic':
    filepath = os.path.join(request.env.applications_parent, 'applications',
        request.env.path_info[1:]).replace('w2pstatic', 'static')
    try:
        file = open(filepath, 'rb')
    except:
        raise HTTP(404)
    else:
        raise HTTP(200, response.stream(file), Expires='[expiration time]')

The above assumes your static URLs specify 'w2pstatic' as the controller, but the files are really in the 'static' folder.

Anthony

[Niphlod]

well, pinpointed in the right direction, but sadly it doesn't work.....after creating a dummy controller just to let the request be accepted, I think I found the culprit:

raise HTTP(200, response.stream(file), headers)

will stop execution, but raising actually from response.stream, that calls stream_file_or_304_or_206, so headers are set in that call, and not the one raised from the model itself.

next step, check stream_file_or_304_or_206, Cache-Control : private is set as fixed, so overriding it in response.stream call or even calling directly stream_file_or_304_or_206 won't work.
Adding an Expires header won't work nonetheless, because for HTTP specs Cache-Control "wins" over Expires header ......

The only way is to use streamer directly, but all other headers, content-type detection, etc would need to be rewritten

[Anthony]

If you pass the open file object to response.stream, it should not call stream_file_or_304_or_206 -- it only calls that if you pass the file path as a string to response.stream. When I tried it (with the code I posted), it worked fine (i.e., the header was set as specified in the HTTP() call).

Anthony

[Niphlod]

I know, but it's quite an undocumented feature ....

I guess I spoke from my bad english, but actually I was proposing to modify stream_file_or_304_or_206 to check if headers set in the method are yet defined before overwriting them.

Sorry again and thanks again ^_^

[Anthony]

I know, but it's quite an undocumented feature ....

I agree this could be documented (and handled) better -- I was just trying to offer a current workaround.

 

I guess I spoke from my bad english, but actually I was proposing to modify stream_file_or_304_or_206 to check if headers set in the method are yet defined before overwriting them.

Sounds reasonable. Though that still wouldn't help with standard static file serving (i.e., when specifying "static" as the controller) because web2py serves static files before hitting the app code at all (so no opportunity for the app to set any headers).

Anthony

[Niphlod]

Sounds reasonable. Though that still wouldn't help with standard static file serving (i.e., when specifying "static" as the controller) because web2py serves static files before hitting the app code at all (so no opportunity for the app to set any headers).

Yes, that's true, but at least for app-managed files the modification to streamer would allow "internal" images to be served in a flexible way. A global setting for app, as for auth ones, for managing static file serving would allow even greater flexibility. That could afford rocket usage also for medium intranets, actually I think that static file serving through rocket is the "option" that make people decide to go for a "real webserver" when the number of users increase. For usual page serving with 5-10 files for css and images, also for very small sites, could mean the difference from serving, e.g., 4 users * 5 pages/minute * 7 files + 1*actual webpage = 160 requests/minute to 20.

Static file serving is handled by web2py in every case, it's just in main.py that the static file is returned with response.stream without executing models/controllers/etc.

We have options to minify,concat,inline css and js files, but that's an overoptimization (still doesn't properly work when you concat or minify files in different folders under static) if you can set cache headers for static files.

An actual dream of mine, just to further "support" the whole "ideas we stole, ideas we had", is web2py "stealing" from rails, because apparently

Rails automatically adds timestamps to all references to static files if you use the according helper methods (like stylesheet_link_tag, javascript_include_tag and image_tag). This means, that the URL of a static file is changed if the file’s timestamp is changed. This way, browsers automatically request the new file.

So, maybe, we could implement a settings for serving application's static assets in this way:
- no caching at all (just like now)
- versioning with settings.static_assets_version = '1.2.3', rewriting the subfolder of static files dinamically in urls (e.g. /static/thisfile.css becomes /static/1.2.3/thisfile.css for the browser, no need to change css's references to images if set relative), setting a max-age of 31536000 (one year)

When you develop, no cache at all, when you deploy, if you know you changed some javascript/css/image/whatever, set settings_static_assets_version to something else and voilà