DAWNBench is a benchmark suite for end-to-end deep learning training and inference. Computation time and cost are critical resources in building deep models, yet many existing benchmarks focus solely on model accuracy. DAWNBench provides a reference set of common deep learning workloads for quantifying training time, training cost, inference latency, and inference cost across different optimization strategies, model architectures, software frameworks, clouds, and hardware.

Building on our experience with DAWNBench, we helped create MLPerf as an industry-standard for measuring machine learning system performance. Now that both the MLPerf Training and Inference benchmark suites have successfully launched, we ended rolling submissions to DAWNBench on 3/27/2020 to consolidate benchmarking efforts.

The original results before the April 20, 2018 deadline are archived for reference. To learn more about key takeaways from DAWNBench, check out our analysis of DAWNBench.