Pylons on nginx, SSI, and memcached

[Tycon]

Does anyone have such a configuration working ?

Basically it means that on every request nginx will use the URL as the
key to perform a lookup on memcached and serve the page from there if
it exists. If it's not cached then it will forward the request to
pylons which will produce the page and store it in mecached, so
subsequent request for the same page will be served by nginx using
mecached without forwarding the request to pylons.

Also, the parts of the page that should not be cached (such as the
current user name, shopping basket, breaking news, etc) are handled
using SSI (server side include) which means that the page rendered by
pylons include a special SSI tag for each dynamic part of the page
(instead of the actual dynamic content).

When nginx gets the page (either from pylons or memcached) it will
look for those special SSI tags and for each one it will issue a
request to pylons to render the dynamic part using the URL that is
specified in the tag.

So the bottom line is that for each page only the dynamic parts of the
page will be rendered on each user request (dynamic parts are the ones
that can be different even for request for the same URL). As I
understand it from reading rails blogs, this is the holy grail of web
app frameworks, as it can give tremendous performance even on low end
hardware. At least it does on my development virtual machine.

[Tycon]

Of course for optimal performance and scalability, I'm talking about a
reverse proxy configuration with load balancing between multiple
paster app servers. Nginx rocks it serves static pages twice as fast
as apache, and proxies requests 50% faster, all while using much less
memory and CPU time.

[mk]

Tycon wrote:
> So the bottom line is that for each page only the dynamic parts of the
> page will be rendered on each user request (dynamic parts are the ones
> that can be different even for request for the same URL).

I would only like to say that if you get such configuration working,
please be sure to post [ gory details | ideally, howto ] here or on some
blog and let us know here. At least I would be very interested to see
how such solution is built.

Regards,
mk

[TJ Ninneman]

Yep, we've used it here to cache both Pylons and ASP and it absolutely
screams.

Including the SSI tags works exactly as you said. What I do is use a
replacement tag in my templates (ie. {* user_status_line *}) and add
code in my lib/base.py file to conditionally handle the replacement:

http://pylonshq.com/pasties/1043

Obviously, I also have an ssi controller that renders and outputs the
user_status_line fragments (to be called directly from nginx).

TJ

[Josh]

I'm currently playing with Varnish as a caching layer, which has
support for Edge Side Includes. Have you considered a setup like this?
If so, I'd be interested to hear what made memcached + SSI more
appealing. If not, I'd be happy to provide some links I've come across
if you're interested in exploring this further.

[TJ Ninneman]

On Jan 9, 2009, at 1:52 PM, Josh wrote:
>
> I'm currently playing with Varnish as a caching layer, which has
> support for Edge Side Includes. Have you considered a setup like this?
> If so, I'd be interested to hear what made memcached + SSI more
> appealing. If not, I'd be happy to provide some links I've come across
> if you're interested in exploring this further.

We looked heavily into Varnish as we run everything on FreeBSD and it
looked just darn cool. I just never really had the time to get it
implemented as a reverse proxy.

One of our requirements was the ability for the backside application
to invalidate cached items. Is this possible?

Another requirement was the ability to redirect backend requests by
URL (ie. *.asp to one server and everything else to another). As I
recall, Varnish can do this, right?

TJ

[Joshua Bronson]

Both of these should be possible. Have a look at http://bart.motd.be/experimenting-with-varnish and http://bart.motd.be/using-varnish-http-accelerator-experiences-so-far. See if the configurations posted there contain enough clues...

[Joshua Bronson]

Actually, I'm not sure if his configuration covers dispatching to different backends by url. Something like this ought to work:

backend asp_server {

  .host = "127.0.0.1";

  .port = "XYZ";

}

backend pylons_server {

  .host = "127.0.0.1";

  .port = "ZYX";

}

sub vcl_recv {

  if (req.url ~ "\.asp$") {

    set req.backend = asp_server;

  } else {

    set req.backend = pylons_server;

  }

  // ...

}

I also found http://varnish.projects.linpro.no/wiki/VCLExampleLongerCaching to be an informative read.

[Tycon]

Here's my nginx configuration for SSI + Memcahced (nginx.conf):

location / {
ssi on;

proxy_set_header X_FORWARDED_PROTO $scheme;
proxy_set_header X_FORWARDED_HOST $server_name;
proxy_set_header X_FORWARDED_FOR $remote_addr;

if ($request_method = POST) {
proxy_pass http://pylons;
}

set $memcached_key $request_uri;
memcached_pass localhost:11211;
default_type text/html;
error_page 404 = /dynamic;
}

location /dynamic {
ssi on;

proxy_set_header X_FORWARDED_PROTO $scheme;
proxy_set_header X_FORWARDED_HOST $server_name;
proxy_set_header X_FORWARDED_FOR $remote_addr;

proxy_pass http://pylons;
}

This is the cache class for storing in memcached:

class Cache(object):
def __init__(self, url, timeout=None, prefix=None):
self.mc = memcache.Client([url])
self.timeout = timeout or 300
self.prefix = prefix
def get(self, key):
return self.mc.get(key)
def set(self, key, value, timeout=None):
if self.prefix:
key = self.prefix + key
self.mc.set(key, value, timeout or self.timeout)

You can initialize it in lib/app_globals.py:

class Globals(object):
def __init__(self):
self.cache = Cache(url='127.0.0.1:11211')

Here's the dynamic controller used by SSI (controllers/dynamic.py):

class DynamicController(BaseController):
def display(self):
return render('/dynamic/' + str(request.params['template']))

Here's the SSI helper and cache decorator (lib/helpers.py):

def ssi(action=None, template=None):
if template:
qry_str = '?template=' + template + '&$args'
else:
qry_str = '?$args'
if not action:
action = request.urlvars['action']
uri = '/dynamic/' + (action or 'display') + qry_str
return '<!--# include virtual="' + uri + '" -->'

def cache(timeout=None):
def wrapper(func, *args, **kwargs):
request = args[0]._py_object.request
content = func(*args, **kwargs)
key = request.path_qs
g.cache.set(key, str(content), timeout)
return content
return decorator(wrapper)

Here's the action that renders and caches a page (controllers/
page.py):

@h.cache(600)
def view(self):
c.name = db.lookup(params['info'])
return render('view.html')

Here's the template for the above action (view.html) - it includes a
dynamic part using the ssi helper:

${h.ssi(template='user.html')|n}
Page name: ${c.name}

Here's a dynamic template (user.html) used by the cached page
view.html:

% if session.has_key('user'):
${session.get('user')}
% else:
${h.link_to(u'signin', h.url_for('/account/signin'))}
% endif

[Tycon]

The nginx/ssi/memcached config for pylons is shown here:

http://www.resheteretz.com/app/blog/index.php?option=com_content&view=article&id=46:pylons-nginx-ssi-and-memcached&catid=31:general&Itemid=46

On Jan 9, 12:51 pm, "Joshua Bronson" <jabron...@gmail.com> wrote:

[Tycon]

correct link

http://www.reshetseret.com/app/blog/?p=3

On Jan 10, 4:59 pm, Tycon <adie...@gmail.com> wrote:
> The nginx/ssi/memcached config for pylons is shown here:
>

> http://www.resheteretz.com/app/blog/index.php?option=com_content&view...

[Ben Bangert]

On Jan 11, 2009, at 2:27 PM, Tycon wrote:

> correct link
>
> http://www.reshetseret.com/app/blog/?p=3

Looks good. Maybe we can get some of these functions included either
in Pylons, or possibly WebHelpers for the performance obsessed? I'd
love to see these packaged for ease of use by all.

Cheers,
Ben

[Ben Bangert]

Alternatively, this might be handy to have in the new Pylons snippets
section on the new website:
http://beta.pylonshq.com/snippets

(it uses restructed text, you can get highlighting for Mako with:
.. code-block: html+mako

template code

I still need to add a preview button.... heh

- Ben