Issue with "Multi" cache

[luke]

My problem:
When I first fired up my server it was using the default settings,
i.e. just one layer and memcache as the cache. This worked fine but
was a bit slow, and I wanted to have a secondary disk cache and seed
it with tilestache-seed.py. So I modified my config to use "Multi"
cache type (see tilestache.cfg below), thinking it will work just like
before only better, particularly once I've seeded it. I can see all
the PNGs it makes on disk so all is well there.

I then ran the seed tool and found that for the levels/areas that I
seeded it works really nicely. The bits that weren't seeded and need
to be rendered take FOREVER to render. Point the preview page to a new
part of the map and wait patiently and you'll see the server's CPU go
nuts for a while, then *maybe* one or two tiles appear and the server
activity stops. Sometimes there's a timeout from nginx, in which case
it appears to be that PostGres was taking too long and gunicorn killed
the connection after 30 seconds. Thinking it's just slow, I increase
the timeout and find that tilestache seems to be doing the queries and
rendering the tiles (I'm judging this solely from watching htop) and
then stops, but nothing is delivered to the client in the browser.
Presumably nothing is written to disk because the next time the
browser goes there it renders the tiles again.

It looks to me as though it's failing to complete rendering the tiles,
maybe timing out and giving up, or something. No idea and not sure how
to find out why. Tilestache doesn't seem to log anything informative
(all I see in tiledrawer-gunicorn.log is the start/stop messages of
the workers- before increasing the timeout I would see [CRITICAL]
WORKER TIMEOUT but I don't see that anymore).

I switched the cache back to memcache only, restarted it, and it all
works nicely (albeit slowly), as before.

Any tips for how to debug this? Know what might be wrong?

My setup:
Based on the Jan '12 version of Tiledrawer on Ubuntu.
Using nginx, gunicorn and memcache.
I can confirm that memcache is running, I can see the TCP connections
to it. In htop I can see CPU activity for PGSQL and the gunicorn
worker threads. All seems to be set up and running correctly as far as
I can tell.

Here's my tilestache.cfg:
{
"layers": {
"tiles": {
"preview": {
"lat": 46.7967,
"lon": -2.1350,
"zoom": 15
},
"png options": {
"palette": "palette.act",
"optimize": true
},
"bounds": {
"high": 18,
"north": 59.142,
"low": 0,
"west": -12.151,
"east": 10.349,
"south": 45.536
},
"metatile": {
"buffer": 256,
"rows": 4,
"columns": 4
},
"provider": {
"name": "mapnik",
"mapfile": "style.xml"
}
}
},
"cache": {
"name": "Multi",
"tiers": [
{
"name": "Memcache",
"servers": [
"127.0.0.1:11211"
]
},
{
"name": "Disk",
"path": "/tmp/stache"
}
]
}
}

An extract from my tilestache-gunicorn.sh:

ADDRESS=localhost:8080
GUNICORN=/usr/bin/gunicorn
DIRNAME=/usr/local/tiledrawer/gunicorn
CONFIG=tilestache.cfg

PIDFILE="/var/run/tiledrawer-gunicorn.pid"
LOGFILE="/var/log/tiledrawer-gunicorn.log"
COMMAND="$GUNICORN --daemon --user www-data --workers 12 --worker-
class egg:gunicorn#gevent_wsgi --bind $ADDRESS --log-file $LOGFILE"

[luke]

problem solved. it was a permissions issue whereby mapnik couldn't
save the rendered tiles to disk because i screwed up the permissions
on the tmp dir!

whilst the permissions looked fine to me at first glance, turns out
that not every directory between / and /tmp/stache/tiles had full read
and execute permissions for the www-data user.