Crawlbot and Bulk Processing

Crawlbot lets you apply any Diffbot API to an entire site, structuring hundreds or thousands of pages into a single, searchable index. With Bulk Processing, extract structured data from hundreds to millions of URLs in a single job.