I have multiple backend servers continuously building and refresing the public parts of an api in order to cache it. The backend servers are builing depending on what has to be done in the job queue.

At a time, backend server 1 will build :

/article/1.json /article/5.json

backend server 2 will build :

/article/3.json /article/9.json /article/6.json

I need to serve these files from the front-end servers. The cache is stored as file in order to be directly served by nginx without going through the rails stack.

The issue is to manage to have the cache up to date on the front-end servers in a scalable way (adding new servers should be seamless).

I've considered :

NFS / S3 (but too slow)

Memcached (but can't serve directly from nginx - might be wrong ?)

CouchDB direcly serving JSON (I feel this is too big for the job)

Backend to write json in redis, job in fronted to re-write files at the good place (currently my favorite option)

Any experience or great idea on a better way to achieve this ?