\$\begingroup\$

The first, purely aesthetic impression that strikes me is the level of nesting in your main crawl function. Consider extracting the core of that function. This might only be to a local function within crawl , but trying to make functions which have less than 7 (*or some fairly small number) generally makes code easier to follow.

Consider wrapping primitives in single-case discriminated unions to make passing variables safer. E.g. you could create a type to wrap the string url which is passed around such as:

type Url = Url of string

Another option to reduce nesting is to pass a function into crawl which will be executed having completed the request, so would be of signature DownloadResult -> Url list (or maybe DownloadResult -> Async<Url list> ) allowing you to return the set of next URLs to fetch. By doing this you've decoupled the ability to fetch a URL from what you ultimately want to do with the result. This pattern is the functional equivalent of the dependency inversion principle.