As my career evolves, I'm dealing more and more with restful APIs. While endpoint caching made a lot of sense when rendering rich Jinja templates output, (and still does for API) it is not as efficient when your client queries your objects database one by one and through listings.

Object layer caching

Endpoint caching would be caching the response of a controller. We do that a lot at SodaHead when it comes to logged out pages. The trick is to use a middleware to store dynamic data into the session to keep it dynamic.

Object layer caching, on the other hand, would be the caching of your ORM objects. SQLAlchemy recommend dogpile.cache for that.

There are only two hard things in Computer Science: cache invalidation, naming things, and off-by-one error. -- Some Genius + Phil Karlton

Caching is tricky and very simple at the same time. Caching one object is easy: serializing an object and storing it in memcached is pretty simple. Invalidation of this object and its dependancies is not that easy. The way I define caching is like a large safe road but allowing you to wander out of it. I'll provide simple safe solutions that will help you gain in performance while being low maintenance. For more complex cases you'll be able to still cache query results but will have to manually invalidate those caches.

Use cases

I see two major cases for caching:

pulling one object from the database through its primary key

pulling a list of (related) objects.

Single object

SQLAchemy provides query.get to simply retrieve an object using its primary key, so we will replicate that behavior.

# pulling one User object user = User.query.get(1) # pulling one User object from cache user = User.cache.get(1)

For convenience, you can even provide a get_or_404 method: do it yourself, it's easy.

Listing of objects

For this specific case, we'll go with email addresses, a 1..n relationship (meaning one user can have multiple email addresses, but an address object can only have one user)

# user is the object we pulled earlier (either from cache or not) email_addresses = EmailAddress.cache.filter(user_id=1)

You can even define a property on the User object itself to handle it:

class User(db.Model...): ... @property def cached_email_addresses(self): return EmailAddress.cache.filter(user_id=self.id)

And voilà, you can now call it from the object too (cached or not, btw)

email_addresses = user.cached_email_addresses

Note: I cannot stress it enough, caching is tricky. The filter method is purposely simple and will only accept one column which should belong to the object.

For listings, I picked a way of caching that you might not be familiar with. Listing caches are just a list of primary keys and rely on single object cache pulling. The logic is approximately the following (but optimized to pull batches of objects from cache):

Retrieve list of IDs from cache if no IDs cached: pull list of IDs from database store list of IDs in cache for every ID in list of IDs: yield object with pk == ID retrieved from cache

Storing object IDs is incredibly much smaller. The benefit is that a single object invalidation will only impact a small part of the listing it is related to. Listing calls will warm up single object caches and this technique allows some basic query features like offset, limit and order_by (only ASC / DESC , not a real column order_by ) without the need to invalidate the cache. This will allow better performance as the less you request, the less will be retrieved and instanciated from your cache backend. Overall, that should cover at least 50% of the cases (absolutely no study to back-up this number though).

Flask Models

Flask-SQLAlchemy inverts the logic a little, associating the query on the model itself which can be unsettling at first. The Model class provided by the Flask-SQLAlchemy library allows you to pass your own custom query object as query_class property, we'll use our callable here:

from caching import CacheableMixin, query_callable, regions class User(db.Model, CacheableMixin): cache_label = "default" # region's label to use cache_regions = regions # regions to store cache # Query handeling dogpile caching query_class = query_callable(regions) id = db.Column(db.Integer, primary_key=True) username = db.Column(db.String(80), unique=True) email = db.Column(db.String(120), unique=True) views = db.Column(db.Integer, default=0) def __init__(self, username, email): self.username = username self.email = email def __repr__(self): return '<User %r>' % self.username

The Caching Library

Here is where the magic lives. Every model with caching capability will inherit the CacheableMixin class. The mixin will also add event listener to smart flush caches on update.

# caching.py import functools import hashlib from flask.ext.sqlalchemy import BaseQuery from sqlalchemy import event, select from sqlalchemy.orm.interfaces import MapperOption from sqlalchemy.orm.attributes import get_history from sqlalchemy.ext.declarative import declared_attr from dogpile.cache.region import make_region from dogpile.cache.api import NO_VALUE def md5_key_mangler(key): """ Encodes SELECT queries (key) into md5 hashes """ if key.startswith('SELECT '): key = hashlib.md5(key.encode('ascii')).hexdigest() return key def memoize(obj): """ Local cache of the function's return value """ cache = obj.cache = {} @functools.wraps(obj) def memoizer(*args, **kwargs): key = str(args) + str(kwargs) if key not in cache: cache[key] = obj(*args, **kwargs) return cache[key] return memoizer # if your app runs on multiple machine behind a # load balancer, I'd recommend the memcache backend. cache_config = { 'backend': 'dogpile.cache.memory', 'expiration_time': 3600, # 1 hour } regions = dict( default=make_region(key_mangler=md5_key_mangler).configure(**cache_config) ) class CachingQuery(BaseQuery): """ A Query subclass which optionally loads full results from a dogpile cache region. """ def __init__(self, regions, entities, *args, **kw): self.cache_regions = regions BaseQuery.__init__(self, entities=entities, *args, **kw) def __iter__(self): """ override __iter__ to pull results from dogpile if particular attributes have been configured. """ if hasattr(self, '_cache_region'): return self.get_value(createfunc=lambda: list(BaseQuery.__iter__(self))) else: return BaseQuery.__iter__(self) def _get_cache_plus_key(self): """ Return a cache region plus key. """ dogpile_region = self.cache_regions[self._cache_region.region] if self._cache_region.cache_key: key = self._cache_region.cache_key else: key = _key_from_query(self) return dogpile_region, key def invalidate(self): """ Invalidate the cache value represented by this Query. """ dogpile_region, cache_key = self._get_cache_plus_key() dogpile_region.delete(cache_key) def get_value(self, merge=True, createfunc=None, expiration_time=None, ignore_expiration=False): """ Return the value from the cache for this query. Raise KeyError if no value present and no createfunc specified. """ dogpile_region, cache_key = self._get_cache_plus_key() assert not ignore_expiration or not createfunc, \ "Can't ignore expiration and also provide createfunc" if ignore_expiration or not createfunc: cached_value = dogpile_region.get(cache_key, expiration_time=expiration_time, ignore_expiration=ignore_expiration) else: cached_value = dogpile_region.get_or_create( cache_key, createfunc, expiration_time=expiration_time) if cached_value is NO_VALUE: raise KeyError(cache_key) if merge: cached_value = self.merge_result(cached_value, load=False) return cached_value def set_value(self, value): """ Set the value in the cache for this query. """ dogpile_region, cache_key = self._get_cache_plus_key() dogpile_region.set(cache_key, value) def query_callable(regions, query_cls=CachingQuery): return functools.partial(query_cls, regions) def _key_from_query(query, qualifier=None): """ Given a Query, create a cache key. """ stmt = query.with_labels().statement compiled = stmt.compile() params = compiled.params return " ".join( [str(compiled)] + [str(params[k]) for k in sorted(params)]) class FromCache(MapperOption): """Specifies that a Query should load results from a cache.""" propagate_to_loaders = False def __init__(self, region="default", cache_key=None): """Construct a new FromCache. :param region: the cache region. Should be a region configured in the dictionary of dogpile regions. :param cache_key: optional. A string cache key that will serve as the key to the query. Use this if your query has a huge amount of parameters (such as when using in_()) which correspond more simply to some other identifier. """ self.region = region self.cache_key = cache_key def process_query(self, query): """Process a Query during normal loading operation.""" query._cache_region = self class Cache(object): def __init__(self, model, regions, label): self.model = model self.regions = regions self.label = label # allow custom pk or default to 'id' self.pk = getattr(model, 'cache_pk', 'id') def get(self, pk): """ Equivalent to the Model.query.get(pk) but using cache """ return self.model.query.options(self.from_cache(pk=pk)).get(pk) def filter(self, order_by='asc', offset=None, limit=None, **kwargs): """ Retrieve all the objects ids then pull them independently from cache. kwargs accepts one attribute filter, mainly for relationship pulling. offset and limit allow pagination, order by for sorting (asc/desc). """ query_kwargs = {} if kwargs: if len(kwargs) > 1: raise TypeError('filter accept only one attribute for filtering') key, value = kwargs.items()[0] if key not in self._columns(): raise TypeError('%s does not have an attribute %s' % self, key) query_kwargs[key] = value cache_key = self._cache_key(**kwargs) pks = self.regions[self.label].get(cache_key) if pks is NO_VALUE: pks = [o.id for o in self.model.query.filter_by(**kwargs)\ .with_entities(getattr(self.model, self.pk))] self.regions[self.label].set(cache_key, pks) if order_by == 'desc': pks.reverse() if offset is not None: pks = pks[pks:] if limit is not None: pks = pks[:limit] keys = [self._cache_key(id) for id in pks] for pos, obj in enumerate(self.regions[self.label].get_multi(keys)): if obj is NO_VALUE: yield self.get(pks[pos]) else: yield obj[0] def flush(self, key): """ flush the given key from dogpile.cache """ self.regions[self.label].delete(key) @memoize def _columns(self): return [c.name for c in self.model.__table__.columns if c.name != self.pk] @memoize def from_cache(self, cache_key=None, pk=None): """ build the from cache option object the the given object """ if pk: cache_key = self._cache_key(pk) # if cache_key is none, the mangler will generate a MD5 from the query return FromCache(self.label, cache_key) @memoize def _cache_key(self, pk="all", **kwargs): """ Generate a key as query format: '<tablename>.<column>[<value>]' 'user.id[all]': all users 'address.user_id=4[all]': all address linked to user id 4 'user.id[4]': user with id=4 """ q_filter = "".join("%s=%s" % (k, v) for k, v in kwargs.items()) or self.pk return "%s.%s[%s]" % (self.model.__tablename__, q_filter, pk) def _flush_all(self, obj): for column in self._columns(): added, unchanged, deleted = get_history(obj, column) for value in list(deleted) + list(added): self.flush(self._cache_key(**{column: value})) # flush "all" listing self.flush(self._cache_key()) # flush the object self.flush(self._cache_key(getattr(obj, self.pk))) class CacheableMixin(object): @declared_attr def cache(cls): """ Add the cache features to the model """ return Cache(cls, cls.cache_regions, cls.cache_label) @staticmethod def _flush_event(mapper, connection, target): """ Called on object modification to flush cache of dependencies """ target.cache._flush_all(target) @classmethod def __declare_last__(cls): """ Auto clean the caches, including listings possibly associated with this instance, on delete, update and insert. """ event.listen(cls, 'before_delete', cls._flush_event) event.listen(cls, 'before_update', cls._flush_event) event.listen(cls, 'before_insert', cls._flush_event)

Caching Specific Filters

The default filtering is limited to one attribute, this is to guarantee cache flush on property update. Each time you update a property of an object, the corresponding listing cache will be flushed. If you need more specific filtering but still would like to cache the result, you can resort to basic dogpile caching:

users = User.query.\ options(User.cache.from_cache("user_examples")).\ filter(User.email.endswith('@example.com')).\ all()

Results would then be stored in dogpile cache backend as "user_examples". To flush that record:

User.cache.flush("user_examples")

Performance

So, is it faster ? Overall the gain is at least 2x faster then SQLite, either retrieval of single objects or list of objects.

Single Object

SQLite

Brices-MacBook-Air:app briceleroy$ ab -n 5000 -c 50 http://127.0.0.1:5000/1/ This is ApacheBench, Version 2.3 <$Revision: 655654 $> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Licensed to The Apache Software Foundation, http://www.apache.org/ Benchmarking 127.0.0.1 (be patient) Completed 500 requests Completed 1000 requests Completed 1500 requests Completed 2000 requests Completed 2500 requests Completed 3000 requests Completed 3500 requests Completed 4000 requests Completed 4500 requests Completed 5000 requests Finished 5000 requests Server Software: Werkzeug/0.9.6 Server Hostname: 127.0.0.1 Server Port: 5000 Document Path: /1/ Document Length: 89 bytes Concurrency Level: 50 Time taken for tests: 15.413 seconds Complete requests: 5000 Failed requests: 0 Write errors: 0 Total transferred: 1210000 bytes HTML transferred: 445000 bytes Requests per second: 324.40 [#/sec] (mean) Time per request: 154.129 [ms] (mean) Time per request: 3.083 [ms] (mean, across all concurrent requests) Transfer rate: 76.67 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect: 0 0 0.2 0 2 Processing: 14 153 10.2 151 196 Waiting: 13 153 10.2 151 193 Total: 16 153 10.2 151 196 Percentage of the requests served within a certain time (ms) 50% 151 66% 155 75% 158 80% 160 90% 163 95% 168 98% 175 99% 181 100% 196 (longest request)

Dogpile memory

Brices-MacBook-Air:app briceleroy$ ab -n 5000 -c 50 http://127.0.0.1:5000/1/ This is ApacheBench, Version 2.3 <$Revision: 655654 $> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Licensed to The Apache Software Foundation, http://www.apache.org/ Benchmarking 127.0.0.1 (be patient) Completed 500 requests Completed 1000 requests Completed 1500 requests Completed 2000 requests Completed 2500 requests Completed 3000 requests Completed 3500 requests Completed 4000 requests Completed 4500 requests Completed 5000 requests Finished 5000 requests Server Software: Werkzeug/0.9.6 Server Hostname: 127.0.0.1 Server Port: 5000 Document Path: /1/ Document Length: 89 bytes Concurrency Level: 50 Time taken for tests: 7.988 seconds Complete requests: 5000 Failed requests: 0 Write errors: 0 Total transferred: 1210000 bytes HTML transferred: 445000 bytes Requests per second: 625.93 [#/sec] (mean) Time per request: 79.881 [ms] (mean) Time per request: 1.598 [ms] (mean, across all concurrent requests) Transfer rate: 147.93 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect: 0 0 0.1 0 1 Processing: 4 79 8.2 76 111 Waiting: 3 79 8.2 76 111 Total: 5 79 8.2 76 111 Percentage of the requests served within a certain time (ms) 50% 76 66% 79 75% 82 80% 84 90% 90 95% 97 98% 103 99% 106 100% 111 (longest request)

Listing of Objects

SQLite

Brices-MacBook-Air:app briceleroy$ ab -n 5000 -c 50 http://127.0.0.1:5000/ This is ApacheBench, Version 2.3 <$Revision: 655654 $> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Licensed to The Apache Software Foundation, http://www.apache.org/ Benchmarking 127.0.0.1 (be patient) Completed 500 requests Completed 1000 requests Completed 1500 requests Completed 2000 requests Completed 2500 requests Completed 3000 requests Completed 3500 requests Completed 4000 requests Completed 4500 requests Completed 5000 requests Finished 5000 requests Server Software: Werkzeug/0.9.6 Server Hostname: 127.0.0.1 Server Port: 5000 Document Path: / Document Length: 5247 bytes Concurrency Level: 50 Time taken for tests: 24.206 seconds Complete requests: 5000 Failed requests: 0 Write errors: 0 Total transferred: 27010000 bytes HTML transferred: 26235000 bytes Requests per second: 206.56 [#/sec] (mean) Time per request: 242.059 [ms] (mean) Time per request: 4.841 [ms] (mean, across all concurrent requests) Transfer rate: 1089.69 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect: 0 0 0.6 0 10 Processing: 17 240 27.1 237 519 Waiting: 17 240 27.0 236 519 Total: 27 241 26.8 237 520 Percentage of the requests served within a certain time (ms) 50% 237 66% 244 75% 247 80% 250 90% 257 95% 267 98% 310 99% 342 100% 520 (longest request)

dogpile memory

Brices-MacBook-Air:app briceleroy$ ab -n 5000 -c 50 http://127.0.0.1:5000/ This is ApacheBench, Version 2.3 <$Revision: 655654 $> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Licensed to The Apache Software Foundation, http://www.apache.org/ Benchmarking 127.0.0.1 (be patient) Completed 500 requests Completed 1000 requests Completed 1500 requests Completed 2000 requests Completed 2500 requests Completed 3000 requests Completed 3500 requests Completed 4000 requests Completed 4500 requests Completed 5000 requests Finished 5000 requests Server Software: Werkzeug/0.9.6 Server Hostname: 127.0.0.1 Server Port: 5000 Document Path: / Document Length: 5247 bytes Concurrency Level: 50 Time taken for tests: 9.678 seconds Complete requests: 5000 Failed requests: 0 Write errors: 0 Total transferred: 27010000 bytes HTML transferred: 26235000 bytes Requests per second: 516.62 [#/sec] (mean) Time per request: 96.784 [ms] (mean) Time per request: 1.936 [ms] (mean, across all concurrent requests) Transfer rate: 2725.35 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect: 0 0 0.1 0 1 Processing: 3 96 14.3 91 167 Waiting: 3 96 14.3 91 167 Total: 4 96 14.3 91 168 Percentage of the requests served within a certain time (ms) 50% 91 66% 93 75% 97 80% 102 90% 113 95% 125 98% 138 99% 159 100% 168 (longest request)

Gist

If you want, there is a gist with the app and the library on github.

Comments on Hacker News or Reddit