I’ve written a small tool to warm-up HTTP caches, e.g. services like nginx.

Source code as well as compiled releases are available at Github.

While most tools in this area are designed to apply a certain load to a web server (often for a specified time), cache_warmer is designed to explicitly GET a set of URLs from a file to warm-up a cache.

Usage

Warm-up URLs from a file using a mobile User-Agent

$ cache_warmer --mobile urls.txt Spawning 4 threads to warm cache with 10000 URIs 10000 / 10000 [===================================================================] 100.00 % 133.99/s Processed 10000 URLs X-Cache-Status header statistics: Miss: 8991 Hit: 1009 HTTP Status Code statistics: Ok: 9850 NotFound: 150 Total time taken: 450.002s

Features

Multi-Threaded

cache_warmer is multi-threaded. Threads can be specified using the --threads option.

HTTP keep-alive

It uses keep-alive by default, which can be disabled with --no-keep-alive .

Captcha detection

It supports a --captcha-string option, which scans the response body for certain strings to detect (and abort) when running into e.g. captchas.

Base URI

If your URL file only contains the base URL (like /products/spoons ), you can add a --base-uri to prepend the host and scheme, e.g. --base-uri https://example.com

Request delay

Outgoing requests can be toned down with the --delay flag.

X-Cache-Status header support

If your backend sets the X-Cache-Status header, you’ll get nice statistics about your cache hit rates at the end of the run.

When using nginx, such a header can be added with this directive:

add_header X-Cache-Status $upstream_cache_status;

Custom User-Agent

cache_warmer defaults to a Googlebot-like User-Agent. You can use the corresponding mobile User-Agent when specifying the --mobile flag.

In case you need a custom User-Agent, you can set it with --user-agent 'Your-User-Agent' .

Cookie support

Attach arbitrary cookies with the --cookie key=value option.

Compile