Fast Caching with Django and Nginx

I've been toying with optimizing the caching on my blog recently – for my own interest (this humble blog doesn't get all that much traffic). All the same, any speed improvements will only mean snappier page-loads and greater capacity to handle a slashdotting, or similar.

I discovered that Nginx has a memcached module that can serve pages directly from memcached without touching the file-system, or a downstream web-app. Which makes for very fast response times. To get it working, you need to set a cache key named with the url of the page, and the value to the HTML. Alas, this means that it would not work with Django's caching mechanism due to the way Django caches unique pages based on the contents of the request header.

Still, the promise of serving up cached pages without even touching Django was most tempting. So I took the cache middleware from Django and butchered it so that it created a simple enough cache key for Nginx to handle.

Here's the code that generates a cache key, given a request:

def get_cache_key ( request , key_prefix = None ): if key_prefix is None : key_prefix = settings . CACHE_MIDDLEWARE_KEY_PREFIX cache_key = ' %s . %s ' % ( key_prefix , request . path ) return cache_key

The following snippet, taken from my Nginx conf file, creates a variable called $memcached_key that matches the cache_key generated in the Python code. If that key exists in memcache it is served directly, otherwise it proxies through to the Django app.

location / {

set $memcached_key .$uri;

memcached_pass 127.0.0.1:11211;

default_type text/html;

error_page 404 = /dynamic$uri;

}

location /dynamic {

proxy_pass http://127.0.0.1:80/;

include /etc/nginx/proxy.conf;

}

Unfortunately my butchering of Django's cache code meant that it would no longer handle dynamic pages. I compensated for this by re-writing any urls with dynamic content so that they went directly to the app.

rewrite /xhr/ /dynamic$uri;

rewrite /search/ /dynamic$uri;

Yet another casualty was Django auth – I couldn't access any pages as a logged in user. The work-around for this was to create a sub-domain that pointed at the same IP, but didn't do the caching via Nginx. That way, I can use the sub-domain for any site admin work.

So far, this Frankenstein cache mechanism seems to be working nicely – pages are served as rapidly as memcached can pluck them out of memory.

If you would like to see the code, Django Techblog is open source. It was meant to satisfy my own needs in a blogging engine, but hopefully it will be of use to others. And it would be very cool if there were other Techblogs on the interwebs!