November 05, 2015 Raphaël Dubigny 2 min read

Caching data with varnish allows to deal with heavy data traffic at a limited cost.



The question is, how can I serve fresh data even if no one have requested them recently? The solution is to request your cache regularly.



This is how I managed to auto warm up a list of URLs in my varnish cache every hour.



First, I add the urls I want refreshed in a urls.txt file like this:

/dashboard/1 /dashboard/2 /dashboard/3

Note that this url list is static but you can easily feed it automatically with a background job of yours.



Then, I need to add specific code to my config.vcl file:

acl warmuper_ip { "10.20.30.40"; } sub vcl_recv { # the script varnish-cache-warmup.sh must always refresh the cache if (client.ip ~ warmuper_ip && req.http.Cache-Control ~ "no-cache") { set req.hash_always_miss = true; } }

Then, I create a varnish-cache-warmup.sh script to actually warmup the varnish cache:

#!/bin/bash wget --output-document=/dev/null --header='Cache-Control: no-cache' --tries=1 --quiet --base=http://domain.com --input-file=/path/to/urls.txt

In order to test your script you'll have to look at varnish logs. Here is a command that may help varnishlog -c | grep ReqURL .

Eventually, I add this cron task with crontab -e :

0 * * * * cd /path/to/varnish-cache-warmup.sh

And that is it!



Here are some helpful resources: